Refining Data Before the Model: Lessons from Support and Security Pipelines
Data science work in customer support and security often hinges on preprocessing. Before feeding text or traffic into a machine learning model, refining the input can produce dramatically better results than swapping out the algorithm itself. Three examples from Cloudflare's own pipelines illustrate this: language classification, fuzzy string matching for ticket classification, and threat fingerprinting for attack traffic.
Language Classification: Know the Limits of Your Signal
Short chat messages are notoriously difficult to classify by language. An off-the-shelf classifier frequently mislabels brief user messages, and accuracy degrades with message length. When Cloudflare analyzed support chats, comparing classified language against the browser's Accept-Language header and the languages associated with the user's country, they found useful agreement across signals.
In 67% of sampled tickets, all three signals agreed. In 15% of cases, the classified language agreed only with the Accept-Language header, and in 5% it agreed only with country-associated languages.
The solution was not to replace the classifier but to build a model that weighs all available signals—classified language, Accept-Language, country header, and the classifier's own confidence score. A simpler heuristic—requiring two of three properties to agree—can also work in many cases. The key is acknowledging the classifier's weaknesses and compensating with context.
Token Matching: Why Levenshtein Isn't Always the Answer
Fuzzy string matching is a staple of natural language processing, often used to extract error messages from support tickets. Many engineers default to Levenshtein distance, the algorithm behind Python's fuzzywuzzy library. But Levenshtein is computationally expensive: for two strings of length k and l, it runs in O(k * l) time.
To find a better fit for ticket classification, Cloudflare compared several algorithms—Cosine, Dice, Damerau, LCS, and Levenshtein—measuring true positive and false positive rates. The Cosine algorithm won out not only for accuracy but for efficiency, reducing runtime to O(k + l).
Cosine similarity works by representing words as vectors in a space where each unique letter is a dimension; the smaller the angle between two vectors, the closer the match. The full mathematical comparison is documented in a peer-reviewed paper: M. Pikies and J. Ali, "String similarity algorithms for a ticket classification system," 2019 6th International Conference on Control, Decision and Information Technologies (CoDIT), Paris, France, 2019, pp. 36-41, available at https://doi.org/10.1109/CoDIT.2019.8820497.
Additional refinements improved results further. Similarity thresholds are tuned by evaluating true and false positive rates on sample data. A custom tokenization approach handles phrases and numeric strings, while the FastText natural language processing library helps determine candidate values for matching, boosting overall accuracy.
Threat Fingerprinting: Finding Clusters in the Noise
Attack alerting requires distinguishing real attacks from ordinary origin issues. DDoS attacks show up in granular metrics: changes in request or error rates over baseline, the relationship between errors and request volume, and other attack indicators. One reliable differentiator is the ratio of HTTP 499 codes to 5xx codes. Cloudflare's edge returns a 499 when a client disconnects before the origin responds; 5xx codes signal origin-side errors handling the request.
DDoS attacks show a linear relationship between these metrics, while origin problems typically spike one or the other. The same principles extend to more complex scenarios, such as spotting credential stuffing in aggregate. By looking at anonymized HTTP header data from prolific attackers of WordPress login portals—820 unique IPs targeting 16,248 distinct zones—patterns emerge that are invisible on a per-request basis. Since WordPress returns HTTP 200 on a failed login and HTTP 302 on success, status codes alone reveal success rates.
Plotting the data in three dimensions—reach across zones, login success rate, and a "variety ratio" measuring abnormal 4xx/5xx responses like firewall blocks and rate limiting—clear clusters emerge. An unsupervised agglomerative hierarchical clustering algorithm groups them.
Cluster analysis exposes the tactics: one cluster had 99.45% of requests from the same country and 99.45% from the same User-Agent. Another cluster had 89% of requests spread across just three User-Agents (75%, 12.3%, and 1.7%). These correlations allow attacks to be fingerprinted even when individual requests come from different IPs and headers, making them undetectable request-by-request.
For product-level decisions, per-request signals matter—should this request be challenged or allowed? But for alerting, aggregation is the key. Viewing data across multiple dimensions reduces false positives dramatically and surfaces anomalous clusters that single-signal analysis misses.
Preprocessing Counts
Newer machine learning approaches process data more efficiently, but preprocessing remains indispensable. In each case above, the breakthrough came not from changing the underlying strategy but from refining how data was selected, filtered, and structured before being handed to algorithms. When optimizing a data pipeline, it pays to examine not just the tools, but the shape of the input itself.
