pgwatch v6 Beta: Pulling Prometheus Metrics Into the Same Store as Your SQL Data

The v6.0.0-beta release of pgwatch is the most significant update since v5, and its headline feature changes how the tool can be deployed. For the first time, pgwatch can treat a Prometheus exporter as a first-class data source, operating alongside regular PostgreSQL sources rather than requiring a separate collection pipeline.

Why Scrape Prometheus at All?

pgwatch has always been built around translating SQL queries against pg_stat_* views into stored measurements. That approach works well for anything Postgres can report about itself, but it breaks down when the information lives outside the database. Consider the questions a Postgres view can't answer: whether the current node is the primary or a replica per the cluster manager, how replication lag looks from Patroni's perspective rather than from pg_stat_replication, or what node_exporter observes about disk I/O.

Previously, filling that gap meant running a separate collector alongside pgwatch. In v6.0.0-beta, pgwatch handles it directly.

Configuring a Prometheus Source

A prometheus source mirrors the structure of a Postgres source, but takes a URL instead of a DSN:

- name: patroni-prod-node1
  kind: prometheus
  conn_str: "http://patroni-node1:8008/metrics"
  preset_metrics: patroni
  is_enabled: true

Set conn_str to any endpoint that serves the Prometheus text exposition format, select a preset or define custom_metrics families with their own intervals, and pgwatch manages the rest. The built-in patroni preset alone spans twelve metric families, including patroni_primary, patroni_replica, patroni_xlog_location, patroni_dcs_last_seen, and patroni_pending_restart.

Authentication and transport security follow standard URL conventions:

conn_str: "https://user:secret@patroni-node1:8008/metrics?tlsskipverify=true"

The tlsrootcert=<path> and tlsskipverify=true parameters are removed before the request is sent, so they never reach the exporter. Passwords are redacted from every log line, keeping secrets out of your journal even if someone greps for a source name.

Monitoring an HA Cluster in One Block

The practical payoff comes with high-availability setups. A three-node Patroni cluster becomes three source entries:

- name: patroni-prod-node1
  kind: prometheus
  conn_str: "http://patroni-node1:8008/metrics"
  preset_metrics: patroni
  custom_tags:
    cluster: prod
    node: dc1-alpha
  is_enabled: true

- name: patroni-prod-node2
  kind: prometheus
  conn_str: "http://patroni-node2:8008/metrics"
  preset_metrics: patroni
  custom_tags:
    cluster: prod
    node: dc1-beta
  is_enabled: true

- name: patroni-prod-node3
  kind: prometheus
  conn_str: "http://patroni-node3:8008/metrics"
  preset_metrics: patroni
  custom_tags:
    cluster: prod
    node: dc2-single
  is_enabled: true

Any custom_tags attached to a source are written to every stored data point, enabling dashboards to filter or group by cluster and node values without manually querying Patroni. When you add normal kind: postgres sources for the same databases and direct everything to one sink, SQL-derived and Prometheus-derived metrics live together in the same measurement store.

How Metric Mapping Works

A prometheus source bypasses pgwatch's built-in metric catalogue entirely. The family name you list in custom_metrics — whether patroni_primary, pg_stat_activity_count, or node_cpu_seconds_total — is used verbatim as both the collection key and the value column of the resulting measurement. Prometheus labels become tag_<label> columns; the sample's numeric value goes into the family-named column; epoch_ns is taken from the sample timestamp if the exporter supplied one, otherwise from the current time. Values of +Inf, -Inf, and NaN pass through untouched, so pgwatch doesn't override the exporter's output — though a strict numeric column in your sink might object.

If you specify no metrics at all, pgwatch scrapes every exposed family every 60 seconds. That's useful for exploring an unfamiliar exporter, but the tool logs a warning when doing so because an unbounded set of families can lead to an unbounded number of measurement tables further downstream.

Prometheus Sinks Skip the Namespace Prefix

When the sink target is itself Prometheus, pgwatch applies a shortcut: measurements originating from a prometheus source are re-exposed under their original family names rather than being wrapped in the usual pgwatch_ namespace. A pg_stat_activity_count family scraped from postgres_exporter reappears as pg_stat_activity_count, not pgwatch_pg_stat_activity_count. This preserves existing exporter dashboards unchanged, even with pgwatch inserted into the collection path.

Beta Caveats and What's Next

The Prometheus source is new code in territory pgwatch previously hadn't explored. Point it at a staging exporter first, review what it collects, and report any anomalies via an issue before putting it in front of production dashboards. This release also includes dashboards migrated to Grafana's v13 schema, new Prometheus-datasource dashboards, and a reaper core hardened against the network flakiness typical of HA environments.