IPAK ML Studio
Model bake-off
Machine Learning Studio compares the generalizability of the prediction performance classifiers with test validation sets.
Research instrument
ipakedu.online/ml-studio
Run your own data
The demo is free to try. To create a project (your title, description & dataset) and run every classifier on it, an active subscription or a project purchase is required. One project includes unlimited runs.
Working with clinical data? De-identify it first — no PHI may be uploaded or stored.
checking access…
Instantly shows the full bake-off — 12 models precomputed on a 100 vs 100 × 1000 simulation. Not billed.
01
Inputs
Data · GEDA matrix
Group 1
Group 2
optional — auto-filled from the […] header line
Shared test-validation set
One TVS, 30 percent held out, seed recorded — or upload an independent validation set as the shared TVS. Every model scores these identical rows.
Run
Feature pre-filter Selected on the learning set only; k-of-m and all classifiers then train on that feature space. PPST/ABA find subset & both-tails markers a t-test misses.
PACE ranking Lyons-Weiler 2005 — ACE vs permutation null across feature-space size. Available after a bake-off completes.
Runs off the main thread in two Web Workers; k-of-m plus 11 classifiers score the shared held-out rows.
02
Results
03
Methods
What each model and metric means. Click to expand.
The 12 models
k-of-m (weighted DPML)the platform’s own interpretable classifier
A patient is scored by how many of m markers are aberrant — beyond a control-derived percentile cutpoint, in each marker’s disease-associated direction. The rule calls a patient a case when at least k of m markers are aberrant. Per-marker cutpoints, directions and weights are learned by a Type-2 genetic algorithm that maximizes balanced accuracy across 40 resampled splits, with an informed initialization seeded from control-percentile aberration; the reported model is the mode across those 40 splits, scored on the shared held-out set. Its distinctive value is interpretability — it returns a ranked, directional panel of markers (see Marker recovery), not just a black-box score.
Diagnostic Process Modeling Language / k-of-m framework (Shi & Lyons-Weiler).
Nearest-centroidclosest class mean
Computes each class’s mean vector (centroid) on the learning set and assigns a held-out sample to the nearest centroid by Euclidean distance. Simple, fast, and robust when classes are compact; sensitive to feature scaling and to features with very different variances.
Gaussian naive Bayesindependent per-feature Gaussians
Models each feature as class-conditionally Gaussian and independent, then classifies by Bayes’ rule (product of per-feature likelihoods times the class prior). “Naive” because it assumes conditional independence. Fast and often very strong when signal is spread across many roughly-Gaussian features (as in this simulation); it degrades when features are strongly correlated.
k-nearest neighbors (k = 5)majority vote of 5 neighbors
Classifies a held-out sample by majority vote of its 5 nearest learning-set samples (Euclidean). Non-parametric with no distributional assumption; captures local structure but is sensitive to scaling, to class imbalance, and to irrelevant features — the curse of dimensionality, acute here with 1000 markers.
Logistic regressionlinear log-odds boundary
Fits a linear decision boundary by maximizing the Bernoulli likelihood (here L2-regularized gradient descent). Yields calibrated probabilities and interpretable coefficients; strong when classes are close to linearly separable, with regularization guarding against overfitting in high dimensions.
Decision tree (CART)recursive Gini splits
Recursively splits on the single threshold that most reduces Gini impurity, forming an axis-aligned partition. Interpretable and captures interactions, but one deep tree is high-variance and overfits — which is exactly why tree ensembles usually beat it.
Bagging (15 trees)variance reduction by resampling
Bootstrap aggregating: trains 15 trees on bootstrap resamples and averages their votes, cutting the variance of a single tree without adding bias. A stepping stone to the random forest.
Random forest (40 trees)decorrelated bagged trees
Bagged trees with an added decorrelation step: each split considers only a random √m subset of features. Averaging many decorrelated trees gives a low-variance, overfitting-resistant classifier and useful feature-importance signal — a strong general-purpose baseline.
Boosting — AdaBoost (30 stumps)sequential error re-weighting
Fits 30 shallow trees (stumps) in sequence, each re-weighting the samples its predecessors misclassified, then combines them by weighted vote. Reduces bias and can reach very low training error, but is more sensitive to label noise and outliers than bagging or forests.
Linear SVM (Pegasos)maximum-margin hyperplane
A large-margin linear classifier trained by the Pegasos stochastic sub-gradient solver on the hinge loss. Here the decision threshold is calibrated on training balanced error rather than fixed at zero, so sensitivity and specificity are balanced honestly rather than by luck of the intercept.
Gaussian process (Laplace)Bayesian non-parametric, RBF kernel
Places a Gaussian-process prior over the latent decision function with an RBF kernel (lengthscale by marginal likelihood) and approximates the non-Gaussian posterior by the Laplace method. Gives probabilistic predictions with uncertainty; GP classification has no closed-form posterior, so Laplace is the canonical approximation.
Rasmussen & Williams, Gaussian Processes for Machine Learning (Alg. 3.1–3.2).
BARTBayesian additive regression trees (probit)
A sum-of-trees Bayesian model with a probit link. Latent utilities are drawn by Albert–Chib data augmentation; trees are updated by grow/prune Metropolis–Hastings with conjugate leaf draws; predictions average over the posterior. Powerful and self-regularizing through its priors, but MCMC is compute-heavy — on 1000 features it is the slowest method here and benefits from a longer run or a feature pre-filter.
Chipman, George & McCulloch, Ann. Appl. Stat. 2010.
Metrics & methods
Shared test-validation set (TVS)one held-out set, every model
One held-out set (30% of samples, split seed recorded) that all 12 models score. Because the rows, split and seed are identical across methods, leaderboard differences reflect the models, not a lucky split. You can also upload an independent validation set to serve as the shared TVS.
Balanced accuracy error (ACEw)the headline metric — lower is better
The average of the false-positive and false-negative rates (equivalently 1 − balanced accuracy). Lower is better; 0.5 is chance. Weighted/balanced so unequal group sizes never flatter a model that merely predicts the majority class.
ROC and directed AUCthreshold sweep, honestly signed
Each model’s true-positive vs false-positive rate as its threshold sweeps; AUC is the area under that curve. Reported directed: a worse-than-chance, inverted model shows AUC < 0.5 and is flagged rather than folded up to look competent.
Confusion cardsan optimistic ceiling
The 2×2 table at each model’s optimal ROC point. Labeled an optimistic ceiling because that threshold is chosen on the held-out set itself; the leaderboard’s k* is the honest, deployable threshold.
k-of-m marker recoverythe interpretable payoff
The top markers by learned weight, each with its aberration direction. This is what k-of-m offers that a pure score does not: a ranked, directional panel you can take back to the bench and test.
Over-training checklearning-set vs held-out gap
k-of-m’s average ROC across its 40 learning-set splits versus its ROC on the held-out TVS. A large gap (“optimism”) signals the model memorized the learning set; the panel flags gaps over 0.10.
Permutation importancemodel-agnostic feature importance
Shuffle one feature and measure the drop in held-out AUC — the Fisher–Rudin–Dominici approach. Computed for the affordable methods; k-NN, GP and BART are excluded because per-feature refitting is too slow in the browser. Correlated features can share or mask each other’s importance.
PACE significanceis the accuracy better than chance?
Permutation-Achieved Classification Error: permute the class labels, re-select features and re-standardize inside each permutation (leakage-safe), and compare the real balanced accuracy to the null band across feature-space size. A real curve outside the 95th/99th null band is significant. Opt-in; runs live.
Lyons-Weiler, Pelikan, Zeh et al., Cancer Informatics 2005;1:53–77.
Feature pre-filterleakage-safe top-k selection
Optional: rank markers on the learning set only and keep the top k, then train k-of-m and every classifier on that reduced space — faster heavy models, often sharper signal. Because selection never sees the held-out set, the estimate stays honest. Criteria: t-statistic, Fisher-like, AUC, J5, PPST, ABA.
J5 (delta score)variance-free ranking
Ranks a marker by the magnitude of its between-group mean difference relative to the average difference across all markers — a variance-free complement to the t-statistic.
caGEDA: Patel & Lyons-Weiler, Applied Bioinformatics 2004;3:49–62.
PPST / ABAsubset & both-tails markers a t-test misses
The Percentile Separability Test counts how many samples of one group fall beyond the other group’s percentile tails — surfacing markers that separate a subset of patients. The ABA test uniquely finds markers over- and under-expressed across different patients (both tails at once), a pattern a population test like the t-test structurally cannot detect.
Lyons-Weiler, Patel, Becich & Godfrey, BMC Bioinformatics 2004;5:110.
New project
Need to de-identify first? Use the De-Identification tool.