Publishing metrics to Prometheus

Publishing DataSource metrics in Prometheus format

Version requirements:

  • DataSource for C 8.0.0+ (Liberator 8.0.0+, Transformer 8.0.0+, TREP Adapter 8.0.0+)

  • DataSource for Java 8.0.0+

DataSource metrics are disabled by default. To publish DataSource metrics, follow the steps below:

  1. Set prometheus-port to a free port on the DataSource’s host:

    prometheus-port 48557

  2. Restart the DataSource

  3. Open a web browser and navigate to http://<datasource_host>:<prometheus_port>/metrics. You will see output similar to the excerpt below:

    Example Liberator DataSource metrics in Prometheus format
    # HELP rttpd_user_global_login_fail counter
# TYPE rttpd_user_global_login_fail counter
rttpd_user_global_login_fail 0
# HELP rttpd_session_global_discards gauge
# TYPE rttpd_session_global_discards gauge
rttpd_session_global_discards 0

For the full list of published metrics, follow the links below:

Publishing custom metrics

The Java DataSource API uses Micrometer to publish Prometheus metrics. You can supplement the standard DataSource metrics with your own custom metrics by adding custom Meter objects to Micrometer’s global meter registry.

The Micrometer Metrics class provides static methods to create Counter and Gauge meters in the global meter registry. To create more advanced meters, access the global meter registry object directly via the static reference Metrics.globalRegistry.

Example 1. Adding a simple counter to a DataSource application’s metrics

To add a Counter to the global meter registry, use the Metrics.counter builder:

Counter myEventsCounter = Metrics.counter("my_events", Tags.empty());

The metrics published on the DataSource’s Prometheus interface (http://<datasource_host>:<prometheus_port>/metrics) now include a my_events_total metric:

# HELP my_events_total
# TYPE my_events_total counter
my_events_total 0.0

For more information on Prometheus and Micrometer, follow the links below:

Publishing JVM metrics in Prometheus format

DataSource metrics do not include metrics on the JVM associated with a DataSource. To publish JVM metrics, we recommend that you configure the DataSource’s JVM to load the Prometheus project’s JMX Exporter as a JVM Java agent.

Configuring a C DataSource to run the Prometheus JMX Exporter

This section describes how to configure a C DataSource’s embedded JVM to load the Prometheus JMX Exporter as a JVM Java agent.

The instructions in this section use Liberator as an example and assume that you have already deployed or activated a blade that enables Liberator’s embedded JVM. For example, the Liberator’s LiberatorJMX blade or Caplin’s PermissioningService module.

Follow the steps below:

  1. Download the latest Prometheus JMX Exporter Java Agent Jar file to your Deployment Framework’s tools directory.

  2. In global_config/environment.conf, define a convenience variable for the location of the JMX Exporter Jar file:

    File: global_config/environment.conf
    define JMX_EXPORTER_JAR ${BASE}/tools/jmx_prometheus_javaagent-0.17.2.jar

    The latest version of JMX Exporter at the time of writing is 0.17.2.

  3. Add the following line of configuration to Liberator’s java.conf override file:

    File: global_config/overrides/servers/Liberator/etc/java.conf
    jvm-options -javaagent:${JMX_EXPORTER_JAR}=jmx_exporter_port:${ccd}/jmx-exporter.yaml

    where jmx_exporter_port is a free port on the host. If you are running multiple DataSources on the host, then assign each JMX Exporter a unique port.

  4. Create a configuration file for the JMX Exporter in the overrides directory for Liberator:

    File: global_config/overrides/servers/Liberator/etc/jmx-exporter.yaml
    ---
rules:
- pattern: ".*"

  5. (Re)start Liberator

    $ ./dfw start

  6. Open a web browser and navigate to http://liberator_host:jmx_exporter_port. If the Java agent is running correctly, then you will see output similar to the (truncated) output below:

    # HELP jvm_threads_current Current thread count of a JVM
# TYPE jvm_threads_current gauge
jvm_threads_current 22.0
# HELP jvm_threads_daemon Daemon thread count of a JVM
# TYPE jvm_threads_daemon gauge
jvm_threads_daemon 19.0
# HELP jvm_threads_peak Peak thread count of a JVM
# TYPE jvm_threads_peak gauge
jvm_threads_peak 28.0

Configuring a Java DataSource to run the Prometheus JMX Exporter

This section describes how to configure a Java DataSource’s JVM to load the Prometheus JMX Exporter as a JVM Java agent.

Follow the steps below:

  1. Download the latest Prometheus JMX Exporter Java Agent jar file to your Deployment Framework’s tools directory.

  2. Make the following modifications (highlighted) to the Java DataSource’s startup script: kits/bladename/bladename-version/DataSource/bin/start-jar.sh. The example script below is adapted from Caplin’s Pricing Adapter Template (see start-jar.sh):

    #!/bin/bash

BLADENAME=MyPricingAdapter

# Prometheus JMX Exporter jar location, port, and config file location
PROM_AGENT=${CONFIG_BASE}../tools/jmx_prometheus_javaagent-0.17.2.jar (1)
PROM_AGENT_PORT=5580 (2)
PROM_AGENT_CONFIG=${CONFIG_BASE}overrides/${BLADENAME}/etc/jmx-exporter.yaml

if [ "$1" = "CONFREADER" ]; then
   shift
   confreading=1
   jar=`ls "$BINARY_ROOT"/lib/datasource*.jar|head -1`
else
   confreading=0
   jar=`ls "$BINARY_ROOT"/lib/$BLADENAME*.jar|head -1`
   classpath="${BINARY_ROOT}/lib/*"

   echo "Classpath: $jar"
fi

if [ $confreading = 1 ]; then
   java -jar "$jar" "$@"
   exit $?
else
   java \
    -javaagent:${PROM_AGENT}=${PROM_AGENT_PORT}:${PROM_AGENT_CONFIG} \
    -cp "$classpath" \
    -jar "$jar" "$@" 2> "$LOGDIR"/java-$BLADENAME.log >/dev/null &
   echo $!
fi
    1 The latest version of JMX Exporter at the time of writing is 0.17.2
    2 Set PROM_AGENT_PORT to a free port on the host. Each JMX Exporter running on the host requires its own unique port.

  3. Create a new configuration file for the JMX Exporter in the overrides directory for the adapter:

    File: global_config/overrides/<bladename>/etc/jmx-exporter.yaml
    ---
rules:
- pattern: ".*"

  4. (Re)start the Java DataSource:

    $ ./dfw start

  5. Open a web browser and navigate to http://<liberator-host>:5580/. If the Java agent is running correctly, then you will see output similar to the (truncated) output below:

    # HELP jvm_gc_collection_seconds Time spent in a given JVM garbage collector in seconds.
# TYPE jvm_gc_collection_seconds summary
jvm_gc_collection_seconds_count{gc="PS Scavenge",} 1.0
jvm_gc_collection_seconds_sum{gc="PS Scavenge",} 0.005
jvm_gc_collection_seconds_count{gc="PS MarkSweep",} 1.0
jvm_gc_collection_seconds_sum{gc="PS MarkSweep",} 0.013
# HELP jvm_classes_loaded The number of classes that are currently loaded in the JVM
# TYPE jvm_classes_loaded gauge
jvm_classes_loaded 2533.0

See also: