Configure DataSource Prometheus monitoring

Publishing DataSource metrics in Prometheus format

Version requirements:

  • DataSource for C 7.1.16+ (Liberator 7.1.15+, Transformer 7.1.10+, TREP Adapter 7.1.3+)

  • DataSource for Java 7.1.14+

DataSource metrics are disabled by default. To publish DataSource metrics, follow the steps below:

  1. Set prometheus-port to a free port on the DataSource’s host:

    prometheus-port 48557

  2. Restart the DataSource

  3. Open a web browser and navigate to http://<datasource_host>:<prometheus_port>/metrics. You will see output similar to the excerpt below:

    Example Liberator DataSource metrics in Prometheus format
    # HELP rttpd_user_global_login_fail counter
# TYPE rttpd_user_global_login_fail counter
rttpd_user_global_login_fail 0

# HELP rttpd_session_global_discards gauge
# TYPE rttpd_session_global_discards gauge
rttpd_session_global_discards 0

DataSource metric references:

Publishing JVM metrics in Prometheus format

DataSource metrics do not include metrics on the JVM associated with a DataSource. To publish JVM metrics, we recommend that you configure the DataSource’s JVM to load the Prometheus project’s JMX Exporter as a JVM Java agent.

Configuring a C DataSource to run the Prometheus JMX Exporter

This section describes how to configure a C DataSource’s embedded JVM to load the Prometheus JMX Exporter as a JVM Java agent.

The instructions in this section use Liberator as an example and assume that you have already deployed or activated a blade that enables Liberator’s embedded JVM. For example, the Liberator’s LiberatorJMX blade or Caplin’s PermissioningService module.

Follow the steps below:

  1. Download the latest Prometheus JMX Exporter Java Agent Jar file to your Deployment Framework’s tools directory.

  2. In global_config/environment.conf, define a convenience variable for the location of the JMX Exporter Jar file:

    File: global_config/environment.conf
    define JMX_EXPORTER_JAR ${BASE}/tools/jmx_prometheus_javaagent-0.17.2.jar

    The latest version of JMX Exporter at the time of writing is 0.17.2.

  3. Add the following line of configuration to Liberator’s java.conf override file:

    File: global_config/overrides/servers/Liberator/etc/java.conf
    jvm-options -javaagent:${JMX_EXPORTER_JAR}=jmx_exporter_port:${ccd}/jmx-exporter.yaml

    where jmx_exporter_port is a free port on the host. If you are running multiple DataSources on the host, then assign each JMX Exporter a unique port.

  4. Create a configuration file for the JMX Exporter in the overrides directory for Liberator:

    File: global_config/overrides/servers/Liberator/etc/jmx-exporter.yaml
    ---
rules:
- pattern: ".*"

  5. (Re)start Liberator

    $ ./dfw start

  6. Open a web browser and navigate to http://liberator_host:jmx_exporter_port. If the Java agent is running correctly, then you will see output similar to the (truncated) output below:

    # HELP jvm_threads_current Current thread count of a JVM
# TYPE jvm_threads_current gauge
jvm_threads_current 22.0
# HELP jvm_threads_daemon Daemon thread count of a JVM
# TYPE jvm_threads_daemon gauge
jvm_threads_daemon 19.0
# HELP jvm_threads_peak Peak thread count of a JVM
# TYPE jvm_threads_peak gauge
jvm_threads_peak 28.0

Configuring a Java DataSource to run the Prometheus JMX Exporter

This section describes how to configure a Java DataSource’s JVM to load the Prometheus JMX Exporter as a JVM Java agent.

Follow the steps below:

  1. Download the latest Prometheus JMX Exporter Java Agent jar file to your Deployment Framework’s tools directory.

  2. Make the following modifications (highlighted) to the Java DataSource’s startup script: kits/bladename/bladename-version/DataSource/bin/start-jar.sh. The example script below is adapted from Caplin’s Pricing Adapter Template (see start-jar.sh):

    #!/bin/bash

BLADENAME=MyPricingAdapter

# Prometheus JMX Exporter jar location, port, and config file location
PROM_AGENT=${CONFIG_BASE}../tools/jmx_prometheus_javaagent-0.17.2.jar (1)
PROM_AGENT_PORT=5580 (2)
PROM_AGENT_CONFIG=${CONFIG_BASE}overrides/${BLADENAME}/etc/jmx-exporter.yaml

if [ "$1" = "CONFREADER" ]; then
   shift
   confreading=1
   jar=`ls "$BINARY_ROOT"/lib/datasource*.jar|head -1`
else
   confreading=0
   jar=`ls "$BINARY_ROOT"/lib/$BLADENAME*.jar|head -1`
   classpath="${BINARY_ROOT}/lib/*"

   echo "Classpath: $jar"
fi

if [ $confreading = 1 ]; then
   java -jar "$jar" "$@"
   exit $?
else
   java \
    -javaagent:${PROM_AGENT}=${PROM_AGENT_PORT}:${PROM_AGENT_CONFIG} \
    -cp "$classpath" \
    -jar "$jar" "$@" 2> "$LOGDIR"/java-$BLADENAME.log >/dev/null &
   echo $!
fi
    1 The latest version of JMX Exporter at the time of writing is 0.17.2
    2 Set PROM_AGENT_PORT to a free port on the host. Each JMX Exporter running on the host requires its own unique port.

  3. Create a new configuration file for the JMX Exporter in the overrides directory for the adapter:

    File: global_config/overrides/<bladename>/etc/jmx-exporter.yaml
    ---
rules:
- pattern: ".*"

  4. (Re)start the Java DataSource:

    $ ./dfw start

  5. Open a web browser and navigate to http://<liberator-host>:5580/. If the Java agent is running correctly, then you will see output similar to the (truncated) output below:

    # HELP jvm_gc_collection_seconds Time spent in a given JVM garbage collector in seconds.
# TYPE jvm_gc_collection_seconds summary
jvm_gc_collection_seconds_count{gc="PS Scavenge",} 1.0
jvm_gc_collection_seconds_sum{gc="PS Scavenge",} 0.005
jvm_gc_collection_seconds_count{gc="PS MarkSweep",} 1.0
jvm_gc_collection_seconds_sum{gc="PS MarkSweep",} 0.013
# HELP jvm_classes_loaded The number of classes that are currently loaded in the JVM
# TYPE jvm_classes_loaded gauge
jvm_classes_loaded 2533.0