Recursive Embedding and Clustering

Clustering Needs More Than K-Means

Finding meaningful groups in user behavior data is rarely as simple as picking an algorithm and running it. High-dimensional data, unclear definitions, and the need to explain results to stakeholders all get in the way. The standard toolbox — principal component analysis (PCA) for reduction, k-means for clustering — tends to fall short on exactly the questions that matter most.

A more effective route combines four stages: reduce dimensionality, cluster with a density-aware algorithm, recurse into clusters that show internal structure, and finish with a supervised model to interpret and predict the groupings. The last step doubles as a production-ready classifier, so the expensive embedding and clustering steps don't have to run again for every new data point.

Dimensionality Reduction Without the Blob

Visual exploration is the usual first move with unfamiliar data. The problem: when a dataset has hundreds or thousands of features, the result of a linear reduction like PCA is often a round, featureless blob. This is the curse of dimensionality in practice — points live at the edges of the space, and a projection that preserves global variance does little to expose local structure.

Uniform manifold approximation and projection (UMAP) handles this far better. UMAP tries to preserve both local and global structure during reduction, which makes it suitable for revealing shape that PCA misses entirely.

For high-dimensional, loosely defined data, this is the practical foundation: reduce with UMAP first.

Why K-Means Fails on the Result

With a UMAP projection in hand that visually separates into groups, k-means seems like the obvious next step. But k-means upsets three properties worth demanding from a clustering method:

• Points should be assigned to clusters only where a cluster actually exists.

• Any required parameters should be intuitive to set.

• Results should be stable across data order and initialization.

Applied to UMAP output with k=6, where six groups are clearly visible, k-means produces assignments that cut straight through visible structure:

HDBSCAN (hierarchical density-based spatial clustering of applications with noise) matches the criteria instead. It doesn't force every point into a cluster, doesn't demand a cluster count up front, and produces stable partitions.

Recursive Clustering: Zooming In on Structure

Even with HDBSCAN, some clusters clearly contain internal variation. The fix is to treat that as a feature, not a bug. The "zoom in" approach works like this: take a cluster, keep only the original data points that belong to it, and run UMAP and HDBSCAN again on that subset.

Restricting the projection to a single cluster changes what "local" and "global" mean, revealing subdivisions that were invisible at the top level. A middle cluster that looked like a single group of handwritten digits splits into three distinct subclusters under recursion — average images show they separate into stroke patterns common to 8s, 3s, and 5s.

Even clusters that appear to lack any internal structure at first often resolve under recursion. A population of zeros, for instance, separates by roundness and slant of the handwriting.

The tradeoffs: recursion is computationally expensive, and you must decide in advance how deep to go. Zooming into a cluster that represents only 1% of the data is rarely worthwhile unless there's a specific question about it.

Explainability With XGBoost and SHAP

Stacked nonlinear algorithms don't explain themselves. A human can't trace why UMAP plus HDBSCAN put a given point in a given cluster. This is where supervised learning turns the pipeline into something interpretable.

Because every data point now has a cluster label from the process above, classification is straightforward. Training XGBoost as a one-versus-all model per cluster yields fast, accurate predictions. Its integrated SHAP (Shapley additive explanations) values then attribute each prediction to the original features — before any embedding or clustering was applied.

SHAP summary plots reveal which raw features drive each cluster assignment. Data scientists can validate that a cluster is meaningful, refine it by inspecting important variables, and iterate on the underlying question. The same trained model serves in production: original pipeline data feeds the classifier directly, and UMAP and HDBSCAN never run again.

From Clusters to Communication

Once clusters are validated, they can be cross-referenced with other data sources — demographics, platform activity, survey responses. This deepens the qualitative picture and supports additional analysis on questions outside the original scope, such as feature adoption by segment.

The explainability layer makes the case to stakeholders. A presentation deck built on this process shows the clusters, the SHAP evidence for why users land in each one, targeted research on those groups, and the relative size of each segment.

The full workflow — reduce, cluster, recurse, explain:

Recursion and the interpretability layer are where this approach departs most from standard practice. Getting finer-grained, human-interpretable groups from the start supports better-targeted research and, ultimately, better product decisions. Work remains to make the process more stable and robust, but the direction is clear: embedding, clustering, and explanation belong in the same loop.