Automated data discovery at scale
Knowing what data an organization stores—and where—is essential for enforcing privacy and security controls. Traditional data loss prevention (DLP) tools fingerprint specific data and watch endpoints for matches. That approach breaks down when data assets are numerous, heterogeneous, and constantly changing. A more robust method uses metadata, machine learning, and a scalable scanning pipeline to classify the semantic type of data across both persistent and live data stores.
The system described here detects semantic types automatically across dozens of data sources, with no constraints on data format or storage type. The resulting classifications feed downstream policy engines, such as access control, and can be retrieved either from aggregated scan results or through a real-time catalog API.
Two data paths, two detection strategies
Data flows through an organization in different ways: some is persisted in offline data stores, while other data is transient and moves in real time. Each path requires a distinct classification approach.
Persisted data stores
For offline storage, the system first builds a catalog of data assets by collecting basic metadata from each data store. This catalog lets the system estimate the resources needed to scan an asset, then dispatch a job to perform the scan. The scanning job is a compiled binary that performs a Bernoulli sample on the latest data available. Each sample is broken into columns, and features are extracted per column. These features are sent to a prediction service that combines rule-based and ML classifiers to output data labels for each column.
Downstream consumers can read the predictions from aggregated result data sets or query the real-time data catalog API.
Online traffic
Nonpersisted data also needs protection. The system provides an online API for real-time classification of transient traffic. This live prediction path is used to classify egress traffic, inbound data for ML models, and other streaming data.
Rules and models together
Manual heuristics alone deliver poor accuracy on unstructured data such as user-generated content. To handle that, the system incorporates an ML component built on a modular PyTorch architecture. The modular design allows individual parts of the model to be changed independently without regressing overall performance. Rule-based detection remains in the pipeline, complementing the ML predictions for a hybrid classification engine.
Enforcement-ready visibility
Privacy policies define how data should be handled, but enforcement depends on knowing which piece of metadata belongs to which data item. The classification system provides that visibility across all stored assets, letting organizations act quickly and with fewer errors. Because the classification design is not tied to fingerprints of specific values, adding detection for new data types is straightforward, while classification results still arrive with low latency and within a tight memory budget.
Work is ongoing to extend the system to unschematized files, to classify sensitivity levels in addition to data types, and to use self-supervised learning to improve accuracy for data classes with few manual labels.
Read the full paper: Secure and scalable data classification
