Prof. Nada’s recommendation (see
decisions/2026-08-04-professor-feedback.md)
was to look for structure the data can actually support instead of forcing a supervised
outcome it cannot. Clustering fits that: it needs no labels, so it is not blocked by the
absence of incident outcomes.
It is exploratory for three reasons that will not change with better code:
So the honest framing is: clustering generates hypotheses about patient/metabolic phenotypes for a future longitudinal study. A cluster is never a cancer diagnosis, a cancer subtype or a cancer site, and the output schema has no field for one.
api/clustering.py consumes the frozen self-supervised artifact:
preprocessor.joblib), so it stays fit on the
training partition only;splits.npz when the row count
matches, and are otherwise recomputed with the same seeded, participant-grouped policy —
the report states which happened;--space latent (default) clusters the 16-dimensional representation;
--space features clusters the shared preprocessed feature matrix.Models are fit on the training partition only; k-means and Gaussian mixtures are then assigned out of sample to the holdout, and the density arm reports train-partition diagnostics because it has no out-of-sample assignment.
| Arm | Implementation | Notes |
|---|---|---|
| Centroid | sklearn.cluster.KMeans |
k = 2…8 |
| Model-based | sklearn.mixture.GaussianMixture |
full covariance, gives membership posteriors |
| Density | sklearn.cluster.HDBSCAN (in-tree since scikit-learn 1.3) |
degrades to DBSCAN and then to “unavailable” rather than adding a dependency (scikit-learn; HDBSCAN, Campello et al.) |
method_availability in every report states which density arm was actually used.
Consensus clustering in the sense of Monti et al.
is the natural next step and is deliberately deferred; the resample machinery needed for it
is already in place.
A candidate is reported only if it passes every gate:
| Gate | Threshold | Why |
|---|---|---|
| Cluster count | ≥ 2 | A single cluster is not a finding. |
| Silhouette | ≥ 0.15 | Basic separation (Rousseeuw 1987). |
| Smallest cluster | ≥ 5% of rows | Tiny clusters are noise or outlier pockets. |
| Bootstrap ARI | ≥ 0.60 mean | Resample-refit agreement. |
| Cluster-wise Jaccard | no cluster ≤ 0.50 | A cluster that dissolves under resampling is not real, even when global ARI is high (Hennig). |
| Seed ARI | ≥ 0.60 | Not an artefact of initialisation. |
| Permuted-null gain | silhouette − null silhouette ≥ 0.05 | Column-permutation destroys joint structure while keeping marginals; the solution must beat that. |
| Outlier sensitivity | ARI ≥ 0.60 after dropping the top 1% deviation rows | A high silhouette can be produced by outliers rather than separation. |
| Negative controls | every gating control ≤ 0.30 | See below. |
Davies-Bouldin and Calinski-Harabasz are reported for every candidate as additional internal-validity context.
Selection among passing candidates is by silhouette + bootstrap ARI. No disease label takes part in fitting or selection.
Stability is necessary, not sufficient: a stable clustering can be a stable artefact, which is why the permuted null and the negative controls are gates too.
| Control | Statistic | Gating |
|---|---|---|
| Survey cycle | bias-corrected Cramér’s V | yes |
| Missingness burden | correlation ratio (η) | yes |
| Assay-availability burden (quartile bins of observed features) | bias-corrected Cramér’s V | yes |
| Assay-availability pattern (exact per-row string) | bias-corrected Cramér’s V | no — high-cardinality diagnostic only |
| Age | correlation ratio | yes |
| Sex | bias-corrected Cramér’s V | yes |
Cramér’s V is bias-corrected because the raw statistic inflates with the number of categories; without the correction any high-cardinality control appears dominant regardless of the clustering. The exact availability pattern is reported for transparency but never gates a decision.
A solution above threshold on a gating control is marked is_data_artefact and cannot be
selected.
When nothing passes, the report status is no_stable_clusters with an
abstain_reason and a per-candidate gate_failure_summary. This is a result, not a
failure: the API returns it, the dashboard displays it, and no phenotype is invented.
nhanes_multicycle_v2.csv)Status: no_stable_clusters, in both the all-adults and the complete-case sensitivity
analysis.
solution_driven_by_top_outliers (k-means k=4,5 and
GMM k=2,3 lose their labelling once the top 1% deviation rows are dropped) and
silhouette_not_better_than_permuted_null (GMM k=2, whose silhouette of 0.243 does not
clear the permuted-null baseline by the required margin). All ten candidates failed at
least one gate.usable_now features (23,228 of 63,041 adult
rows) removes the missingness/assay drivers but not the cycle effect.Interpretation: on the pooled cross-section, the dominant recoverable structure is when and how a participant was measured, not how their metabolism differs. That is a data finding worth reporting, and it is exactly what the negative controls exist to catch.
cd api
# full research pass: reliability + evidence + both clustering variants + charts
../.venv/bin/python run_research_pass.py
# clustering alone, complete-case sensitivity analysis
../.venv/bin/python clustering.py --complete-cases-only \
--methods kmeans,gaussian_mixture,hdbscan --k-values 2,3,4,5,6 \
--output-dir ../model_artifacts/clustering/sensitivity
Outputs per run: clustering_report.json, candidate_metrics.csv,
negative_controls.csv, and when a solution is selected projection_points.csv and
cluster_feature_panels.csv. The research pass adds accessible SVG charts plus the CSV
behind each one.