Detecting Threats At Tens of Terabytes a Day

Dropbox's Detection and Response Team (DART) is responsible for identifying and mitigating threats to employees, infrastructure, and customer data. The team ingests security-relevant logs for detection, hunting, and incident response, handling an average of tens of terabytes of data daily.

DART’s alerting and response pipeline uses Python and Jupyter notebooks to create new tools.

Beyond building detections and triaging incidents, DART spends considerable time filtering false positives and adding context to individual alerts. This manual work detracts from active threat hunting. The team's log volume also creates storage constraints: logs are distributed across multiple data stores based on volume, source, and the queries needed. During investigations, analysts must search multiple stores to assemble a complete picture, and the variety of query languages across these systems complicates onboarding and makes it difficult to add context to triggered rules.

DART's goal was a unified language and interface for querying across all data stores, plus a shared set of tools for the full incident response lifecycle: building detections, contextualizing alerts, hunting, and responding.

Alertbox: Automating Triage Context

To reduce triage time, DART first built Alertbox. The goal was to encode alert response runbooks as code, executing them before triage begins. Since runbooks often start by gathering context on involved users, hosts, and processes, and this data often lives outside the SIEM, DART needed a way to run code in response to alerts and integrate with various internal data sources.

Alertbox is built around the concept of a Workflow, a Python class mapped to a specific alert. Python was chosen due to its prevalence at Dropbox and in the security and data science communities. A default workflow serves as a catch-all for alerts without a specific one.

class DefaultSIEMEventWorkflow(Workflow[SIEMEvent]):
    def __init__(self, siem_client, jira, forerunner):
        # type: (SIEMClient, JiraAPI, ForeRunnerAPI) -> None
        self.jira = jira
        self.siem_client = siem_client
        self.forerunner = forerunner

    @classmethod
    def create_instance(cls):
        # type: () -> DefaultSIEMEventWorkflow
        jira = JiraAPI('dart-bot')
        siem_client = create_siem_client()
        forerunner = ForeRunnerAPI()
        return DefaultSIEMEventWorkflow(siem_client, jira, forerunner)

    def process_event(self, workflow_context):
        # type: (WorkflowContext[SIEMEvent]) -> None
        alert_name = workflow_context.event.name

        event = workflow_context.event
        url = event.url

        runbook = extract_runbook_url(event.logs[0])

        jupyter_notebook_url = self.forerunner.generate_jupyter_notebook_url(
            alert_name=alert_name, alert_content=event.logs[0]
        )

        tags = extract_tags(event.logs)
        table = Table.convert_to_table('Raw Logs', event.logs)

        try_create_jira_ticket(self.jira, alert_name, url, jupyter_notebook_url, tags, runbook, table)

Datasources: Abstracting Storage Details

With Alertbox providing orchestration, the next step was building libraries to pull context more easily and avoid implementation details like where data lives. DART created a 'datasources' Python library with modules that abstract away underlying data store specifics.

class AuditExecLog(Log):
    def __init__(
            self,
            auid, # type: int
            node,  # type: str
            uid, # type: int
            pid, # type: int
            ppid, # type: int
            comm, # type: str
            euid,  # type: str
            proctitle,  # type: str
            timestamp,  # type: int
            username,  # type: str
    ):
        pass

    def get_children(self):
        # type: () -> List[AuditExecLog]
        """
            Find execution of processes where the parent process is `self`
        :return: List of AuditExecLog, representing child processes
        """@cached
def get_machine_profile(client, hostname):
    # type: (MPClient, str) -> Optional[MachineProfileEntry]
    query = build_mp_query(hostname)
    response = MachineProfileEntry.query(client, query)
    return response or None

The library offers low-level abstractions that map directly to data stores. For example, AuditExecLog is a Python wrapper for the Linux audit subsystem's log format. When investigating a suspicious execution, analysts often need related executions like child processes. Raw queries for this are complex, involving issues like PID collisions. The datasources library encapsulates this complexity in a simple get_children method call.

DART also tracks information on every asset across Dropbox's environments — tooling status, owners, IPs, hostnames. This aggregate data is stored in an entity called Machine Profile. A helper function can look up a MachineProfileEntry by hostname, handling result caching, raw query construction, and parsing.

@cached
def get_machine_profile(client, hostname):
    # type: (MPClient, str) -> Optional[MachineProfileEntry]
    query = build_mp_query(hostname)
    response = MachineProfileEntry.query(client, query)
    return response or None

Covenant: Investigations and Hunting on Jupyter

While Alertbox and the datasources library automated alert responses, investigations still required knowing exact data locations, abstractions, and query languages. To bridge this gap, DART built Covenant, an investigation tool based on Jupyter Notebooks. Jupyter provides a powerful Python REPL with cells for code and Markdown, allowing analysts to modify and re-execute code. It is popular in data science for data slicing, model building, and reporting — work similar to DART's.

Here you can see these cells and how our notebook provides us with helpful autocompletion to navigate our various abstractions.

A key design decision was that Covenant should share the same fundamental data abstractions and tools as the automation platform. Covenant uses Bazel, an open-source build system, to create a custom Jupyter kernel with a dependency on the datasources library. This allows analysts to interact with data and hunt using the same primitives used for automated alert response.

An example investigation.

Securing a Remote Python Shell

Covenant is effectively a remote Python shell, and Jupyter itself has known vulnerabilities. Security measures include:

  • No direct internet access; no packages are pulled from the internet during the build process.
  • Access is behind a proxy enforcing strong 2FA and authorization, with application-level authentication for defense in depth.
  • A strict Content Security Policy and CSRF protection for both GET and POST requests using SameSite cookies.

Forerunner: Connecting Automation to Analysis

Since Jupyter notebooks intermingle code and output, they self-document the analysis process. DART wanted to leverage this to record investigations and tie them to individual alerts. Forerunner acts as the glue between Alertbox and Covenant. When an alert fires, Alertbox calls Forerunner via RPC. Forerunner returns a Jupyter notebook for that alert, and Alertbox embeds the notebook's URL into the alert ticket. Forerunner also runs the notebook asynchronously in the background.

These notebooks contain heavier queries pulling additional context. The on-call analyst can investigate within the notebook using datasources primitives, and the investigation is automatically recorded.

A sample alert investigation using Forerunner.

Common Abstractions Across the Response Cycle

The traditional approach to building detection and response tools often decouples automation from investigation. DART found this creates significant friction. Instead, Dropbox invested in a common underlying abstraction for logs, available throughout the entire incident response cycle via Alertbox, Covenant, and Forerunner. This integration of open source tools lets the team explore data quickly while automating away routine alerts, enabling focus on more sophisticated threats.