Configure DataSource Prometheus monitoring

Publishing DataSource metrics in Prometheus format

Version requirements:

  • DataSource for C 7.1.16+ (Liberator 7.1.15+, Transformer 7.1.10+, TREP Adapter 7.1.3+)

  • DataSource for Java 7.1.14+

DataSource metrics are disabled by default. To publish DataSource metrics, follow the steps below:

  1. Set prometheus-port to a free port on the DataSource’s host:

    prometheus-port 48557
  2. Restart the DataSource

  3. Open a web browser and navigate to http://<datasource_host>:<prometheus_port>/metrics. You will see output similar to the excerpt below:

    Example Liberator DataSource metrics in Prometheus format
    # HELP rttpd_user_global_login_fail counter
    # TYPE rttpd_user_global_login_fail counter
    rttpd_user_global_login_fail 0
    
    # HELP rttpd_session_global_discards gauge
    # TYPE rttpd_session_global_discards gauge
    rttpd_session_global_discards 0

DataSource metric references:

Publishing JVM metrics in Prometheus format

DataSource metrics do not include metrics on the JVM associated with a DataSource. To publish JVM metrics, we recommend that you configure the DataSource’s JVM to load the Prometheus project’s JMX Exporter as a JVM Java agent.

Configuring a C DataSource to run the Prometheus JMX Exporter

This section describes how to configure a C DataSource’s embedded JVM to load the Prometheus JMX Exporter as a JVM Java agent.

The instructions in this section use Liberator as an example and assume that you have already deployed or activated a blade that enables Liberator’s embedded JVM. For example, the Liberator’s LiberatorJMX blade or Caplin’s PermissioningService module.

Follow the steps below:

  1. Download the latest Prometheus JMX Exporter Java Agent Jar file to your Deployment Framework’s tools directory.

  2. In global_config/environment.conf, define a convenience variable for the location of the JMX Exporter Jar file:

    File: global_config/environment.conf
    define JMX_EXPORTER_JAR ${BASE}/tools/jmx_prometheus_javaagent-0.17.2.jar

    The latest version of JMX Exporter at the time of writing is 0.17.2.

  3. Add the following line of configuration to Liberator’s java.conf override file:

    File: global_config/overrides/servers/Liberator/etc/java.conf
    jvm-options -javaagent:${JMX_EXPORTER_JAR}=jmx_exporter_port:${ccd}/jmx-exporter.yaml

    where jmx_exporter_port is a free port on the host. If you are running multiple DataSources on the host, then assign each JMX Exporter a unique port.

  4. Create a configuration file for the JMX Exporter in the overrides directory for Liberator:

    File: global_config/overrides/servers/Liberator/etc/jmx-exporter.yaml
    ---
    rules:
    - pattern: ".*"
  5. (Re)start Liberator

    $ ./dfw start
  6. Open a web browser and navigate to http://liberator_host:jmx_exporter_port. If the Java agent is running correctly, then you will see output similar to the (truncated) output below:

    # HELP jvm_threads_current Current thread count of a JVM
    # TYPE jvm_threads_current gauge
    jvm_threads_current 22.0
    # HELP jvm_threads_daemon Daemon thread count of a JVM
    # TYPE jvm_threads_daemon gauge
    jvm_threads_daemon 19.0
    # HELP jvm_threads_peak Peak thread count of a JVM
    # TYPE jvm_threads_peak gauge
    jvm_threads_peak 28.0

Configuring a Java DataSource to run the Prometheus JMX Exporter

This section describes how to configure a Java DataSource’s JVM to load the Prometheus JMX Exporter as a JVM Java agent.

Follow the steps below:

  1. Download the latest Prometheus JMX Exporter Java Agent jar file to your Deployment Framework’s tools directory.

  2. Make the following modifications (highlighted) to the Java DataSource’s startup script: kits/bladename/bladename-version/DataSource/bin/start-jar.sh. The example script below is adapted from Caplin’s Pricing Adapter Template (see start-jar.sh):

    #!/bin/bash
    
    BLADENAME=MyPricingAdapter
    
    # Prometheus JMX Exporter jar location, port, and config file location
    PROM_AGENT=${CONFIG_BASE}../tools/jmx_prometheus_javaagent-0.17.2.jar (1)
    PROM_AGENT_PORT=5580 (2)
    PROM_AGENT_CONFIG=${CONFIG_BASE}overrides/${BLADENAME}/etc/jmx-exporter.yaml
    
    if [ "$1" = "CONFREADER" ]; then
       shift
       confreading=1
       jar=`ls "$BINARY_ROOT"/lib/datasource*.jar|head -1`
    else
       confreading=0
       jar=`ls "$BINARY_ROOT"/lib/$BLADENAME*.jar|head -1`
       classpath="${BINARY_ROOT}/lib/*"
    
       echo "Classpath: $jar"
    fi
    
    if [ $confreading = 1 ]; then
       java -jar "$jar" "$@"
       exit $?
    else
       java \
        -javaagent:${PROM_AGENT}=${PROM_AGENT_PORT}:${PROM_AGENT_CONFIG} \
        -cp "$classpath" \
        -jar "$jar" "$@" 2> "$LOGDIR"/java-$BLADENAME.log >/dev/null &
       echo $!
    fi
    1 The latest version of JMX Exporter at the time of writing is 0.17.2
    2 Set PROM_AGENT_PORT to a free port on the host. Each JMX Exporter running on the host requires its own unique port.
  3. Create a new configuration file for the JMX Exporter in the overrides directory for the adapter:

    File: global_config/overrides/<bladename>/etc/jmx-exporter.yaml
    ---
    rules:
    - pattern: ".*"
  4. (Re)start the Java DataSource:

    $ ./dfw start
  5. Open a web browser and navigate to http://<liberator-host>:5580/. If the Java agent is running correctly, then you will see output similar to the (truncated) output below:

    # HELP jvm_gc_collection_seconds Time spent in a given JVM garbage collector in seconds.
    # TYPE jvm_gc_collection_seconds summary
    jvm_gc_collection_seconds_count{gc="PS Scavenge",} 1.0
    jvm_gc_collection_seconds_sum{gc="PS Scavenge",} 0.005
    jvm_gc_collection_seconds_count{gc="PS MarkSweep",} 1.0
    jvm_gc_collection_seconds_sum{gc="PS MarkSweep",} 0.013
    # HELP jvm_classes_loaded The number of classes that are currently loaded in the JVM
    # TYPE jvm_classes_loaded gauge
    jvm_classes_loaded 2533.0