Pulling Datadog Metrics Into Jupyter for Deeper Analysis

Datadog is a monitoring and security platform for cloud applications, commonly used to track service status and infrastructure health. Its dashboards are useful for spotting issues, but they have two notable constraints when you need to do serious analysis.

  • Limited analysis tooling: Datadog offers a fixed set of visualizations and doesn't support complex work like building statistical models.
  • Granularity trade-offs: Dashboards have a fixed width. Zooming out to a 30-day view aggregates metrics to a 2-hour interval, smoothing over short-lived events. A 15-minute window, by contrast, shows 1-second intervals. That aggregation can hide exactly the kind of interesting activity you might want to investigate.

For those reasons, it can be worth extracting the underlying metric data and exploring it locally. Datadog's REST API provides the raw data; a Jupyter notebook is a convenient environment for working with it in Python.

What You Need to Get Started

The extraction process requires only two things:

  1. API credentials: An API key and an APP key to authenticate against the Datadog API.
  2. A metric query: For example, a request to track CPU utilization over time.

Once those are in hand, the extraction follows a simple sequence.

Step 1: Set Up Libraries and Credentials

Start by importing the required Python libraries and configuring your API keys for the requests.

Step 2: Define the Time Window

Next, set the parameters for the time-series query. The following example specifies a window from November 22, 2022 at 16:11:49 GMT to November 25, 2022 at 16:11:49 GMT.

A practical consideration: Datadog enforces rate limits on API requests. If you hit those limits, increasing the time_delta value will reduce the number of calls your script makes to the API.

Step 3: Run the Extraction Logic

The core of the process is to take the start and stop timestamps and split them into buckets, each of width time_delta. Then loop through those buckets, making a Datadog API call for each window and appending the results to a list.

After the loop completes, convert the accumulated lists into a pandas dataframe and return it.

Step 4: Analyze the Resulting Data

With the data in a dataframe, you'll have much finer granularity than Datadog's dashboard provides. This opens the door to a wider range of visualizations and statistical techniques.

For instance, you could use seaborn to build KDE plots of the system's CPU utilization distribution. This kind of analysis can reveal patterns, such as the shape of the distribution or the presence of modes, that the aggregated dashboard view misses.