Monitoring

XTDB can expose metrics about a node via Dropwizard to Prometheus, AWS’s CloudWatch, and Java’s JMX.

We expose metrics about transaction ingestion and running queries. If you use RocksDB you can also expose internal RocksDB metrics.

Project Dependency

In order to use any of the xtdb-metrics reporters, you will need to include the following dependency on xtdb-metrics:

  • deps.edn

  • pom.xml

com.xtdb/xtdb-metrics {:mvn/version "1.23.2"}
<dependency>
    <groupId>com.xtdb</groupId>
    <artifactId>xtdb-metrics</artifactId>
    <version>1.23.2</version>
</dependency>

You will also need to include dependencies for your individual reporters - these are outlined below.

Metrics module

The XTDB metric reporters depend on a common metrics module:

  • JSON

  • Clojure

  • EDN

{
  "xtdb.metrics/metrics": { ... },
}
{:xtdb.metrics/metrics { ... }}
{:xtdb.metrics/metrics { ... }}

You can omit this if you’re happy with its default parameters.

Parameters

  • with-index-store-metrics? (boolean, default true): includes index-store metrics in the metrics registry

  • with-query-metrics? (boolean, default true): includes query metrics in the metrics registry

Prometheus reporter

This reporter pushes metrics to a Prometheus 'Pushgateway' at regular intervals.

You will need to add the following dependencies, alongside xtdb-metrics, in your project:

Additional dependencies

  • deps.edn

org.dhatim/dropwizard-prometheus {:mvn/version "2.2.0"}
io.prometheus/simpleclient_pushgateway {:mvn/version "0.8.1"}
io.prometheus/simpleclient_dropwizard {:mvn/version "0.8.1"}
io.prometheus/simpleclient_hotspot {:mvn/version "0.8.1"}
clj-commons/iapetos {:mvn/version "0.1.9"}

Node configuration

  • JSON

  • Clojure

  • EDN

{
  "xtdb.metrics.prometheus/reporter": { ... }
}
{:xtdb.metrics.prometheus/reporter { ... }}
{:xtdb.metrics.prometheus/reporter { ... }}

Parameters

  • registry (registry, defaults to shared registry)

  • push-gateway (string, required)

  • report-frequency (string/Duration, default "PT1S")

  • prefix (string): prefix all metric titles with this string

Prometheus HTTP exporter

The Prometheus HTTP exporter starts a standalone server hosting Prometheus metrics.

Additional dependencies

  • deps.edn

org.dhatim/dropwizard-prometheus {:mvn/version "2.2.0"}
io.prometheus/simpleclient_dropwizard {:mvn/version "0.8.1"}
io.prometheus/simpleclient_hotspot {:mvn/version "0.8.1"}
io.prometheus/simpleclient_httpserver {:mvn/version "0.8.1"}
clj-commons/iapetos {:mvn/version "0.1.9"}

Node configuration

  • JSON

  • Clojure

  • EDN

{
  "xtdb.metrics.prometheus/http-exporter": { ... }
}
{:xtdb.metrics.prometheus/http-exporter { ... }}
{:xtdb.metrics.prometheus/http-exporter { ... }}

Parameters

  • registry (registry, defaults to shared registry)

  • port (int, default 8080) port number for Prometheus client server.

  • jvm-metrics? (boolean, default false): includes JVM metrics in the metrics dump

AWS Cloudwatch

In addition to extra dependencies, you’ll need to ensure that your application has the cloudwatch:PutMetricData permission.

Additional dependencies

  • deps.edn

io.github.azagniotov/dropwizard-metrics-cloudwatch {:mvn/version "2.0.3"}
software.amazon.awssdk/cloudwatch {:mvn/version "2.10.61"}

Node configuration

  • JSON

  • Clojure

  • EDN

{
  "xtdb.metrics.cloudwatch/reporter": { ... }
}
{:xtdb.metrics.cloudwatch/reporter { ... }}
{:xtdb.metrics.cloudwatch/reporter { ... }}

Parameters

  • registry (registry, defaults to shared registry)

  • high-resolution? (boolean, default false): increase push rate from 1 minute to 1 second

  • dimensions (Map<String, String>): dimensions to include in the pushed metrics

  • jvm-metrics? (boolean, default false): includes JVM metrics in the pushed metrics

  • region (string): override default AWS region for uploading metrics

  • ignore-rules (List<String>): a list of metrics to ignore, in gitignore format. e.g. ["xtdb.tx" "!xtdb.tx.ingest"] would ignore xtdb.tx.*, except xtdb.tx.ingest

  • dry-run? (boolean, default false): reporter outputs to a local SLF4J logger instead

  • dry-run-report-frequency (string/Duration, default "PT1S")

JMX

Additional Dependencies

  • deps.edn

io.dropwizard.metrics/metrics-jmx {:mvn/version "4.1.2"}

Node configuration

  • JSON

  • Clojure

  • EDN

{
  "xtdb.metrics.jmx/reporter": { ... }
}
{:xtdb.metrics.jmx/reporter { ... }}
{:xtdb.metrics.jmx/reporter { ... }}

Parameters

  • registry (registry, defaults to shared registry)

  • domain (string): custom JMS domain

  • rate-unit (TimeUnit, default 'seconds'): unit to report rates

  • duration-unit (TimeUnit, default 'seconds'): unit to report durations

Console

The console reporter logs metrics to standard-out at regular intervals.

It has no additional dependencies.

Node configuration

  • JSON

  • Clojure

  • EDN

{
  "xtdb.metrics.console/reporter": { ... }
}
{:xtdb.metrics.console/reporter { ... }}
{:xtdb.metrics.console/reporter { ... }}

Parameters

  • registry (registry, defaults to shared registry)

  • report-frequency (string/Duration, default "PT1S")

  • rate-unit (TimeUnit, default 'seconds'): unit to report rates

  • duration-unit (TimeUnit, default 'seconds'): unit to report durations

CSV

The CSV reporter logs metrics to a CSV file at regular intervals.

It has no additional dependencies.

Node configuration

  • JSON

  • Clojure

  • EDN

{
  "xtdb.metrics.csv/reporter": { ... }
}
{:xtdb.metrics.csv/reporter { ... }}
{:xtdb.metrics.csv/reporter { ... }}

Parameters

  • registry (registry, defaults to shared registry)

  • output-file (string/File/Path, required)

  • report-frequency (string/Duration, default "PT1S")

  • rate-unit (TimeUnit, default 'seconds'): unit to report rates

  • duration-unit (TimeUnit, default 'seconds'): unit to report durations