Activating the Persistence Service 6.2

The Persistence Service provides a persistent key-value store to Transformer modules and StreamLink clients. The key-value store is persistent across server restarts and shared between nodes in a Transformer cluster.

The service was introduced in Transformer 6.2.3 and rewritten for Transformer 7.0. Modules written using Transformer 6.2's Persistence API are not compatible with Transformer 7.0's Persistence API. For guidance on upgrading to Tranformer 7.0, see Upgrading to Caplin Platform 7.

The Persistence Service is required by the following Transformer modules: the Watchlist Service, the Alerts Service, and the High-Low Service.

Contents:

Activating the Persistence Service blades

The Persistence Service comprises two blades, which are deactivated by default:

  • PersistenceService: This blade provides persistence services to Transformer modules, including the PersistenceServiceClient module (below). See the Java Transformer Module (JTM) package com.caplin.transformer.module.persistence.
  • PersistenceServiceClient: This blade provides persistence services to StreamLink clients. It requires the PersistenceService module to be active.

To activate the Persistence Service blades, run the commands below from the root of your Deployment Framework:

$ ./dfw activate PersistenceService
$ ./dfw activate PersistenceServiceClient

Datastores

The Persistence Service supports two types of datastore:

  • File-backed persistence: Data is persisted to Transformer's local filesystem. This configuration is not supported for production use.
  • Database-backed persistence: Data is persisted to a database server.

Configuring file-backed persistence

File-backed persistence is configured by default.

Important: File-backed persistence is not supported for production use.

To configure file-backed persistence, edit the file <Framework-root>/global_config/overrides/PersistenceService/Transformer/etc/persistence.conf so that it is consistent with the values in the table below.

Configuration item Value Comment
enable-file-database TRUE Set to TRUE to enable file-backed persistence.
add-database-params/init-string ${ccd}/persistence The file-system directory that file-backed persistence should persist data to.

Configuring database-backed persistence

To configure the Persistence Service to use database-backed persistence:

  1. Create the Persistence Service's database table
  2. Add the database's JDBC driver to Transformer's classpath
  3. Configure the connection to the database
  4. Disable repeated database writes in a Transformer cluster

1. Create the Persistence Service's database table

Choose a SQL database server that has a JDBC driver and that supports authentication by username and password.

Create a table with two text columns. The first column stores persistence keys and the second column stores persistence values. Define the column that stores the persistence keys as the primary key of the table. You may choose any name for the table and its columns: the table name and column names are configurable in the Persistence Service's configuration file (see Defining the persistence configuration items).

Here's an example MySQL table definition that uses the default table and column names defined in the Persistence Service's configuration file:

CREATE TABLE persistence (
    pers_key VARCHAR(100),
    pers_val TEXT,
    PRIMARY KEY (pers_key)
);

You may need to increase the width of the 'pers_key' column if a module or StreamLink client uses keys greater than 100 characters in length.

2. Add the database's JDBC driver to Transformer's classpath

To add the database's JDBC driver to Transformer's classpath:

  1. Create the following directory: <Framework-root>/global_config/overrides/PersistenceService/Transformer/lib/java
  2. Copy the JDBC driver JAR file to the directory created in step 1.
  3. Make the following two changes to the file: <Framework-root>/global_config/overrides/PersistenceService/Transformer/etc/java.conf
    1. Add the following line (where jar_filename is the name of JDBC driver JAR file):
      1. jvm-global-classpath "${ccd}/../lib/java/jar_filename"
    2. Uncomment the following line:
      1. jvm-location ${JVM_LOCATION}

3. Configure the connection to the database

To configure database-backed persistence, edit the file <Framework-root>/global_config/overrides/PersistenceService/Transformer/etc/persistence.conf so that it is consistent with the values in the table below.

Configuration item Value Comment
enable-file-database FALSE Set to FALSE to use database-backed persistence.
persistence-classid com/​caplin/​transformer/​persistence/​JdbcPersistorImpl The implementation of Persistor used to write data to JDBC databases. Unless you write your own Persistor, use the value in this table.
add-database-params/init-string jdbc:mysql:/​/​localhost/​rtas The JDBC URL of the database. The example value in this table is for a local MySQL database called 'rtas'.
add-database-params/driver-name com.mysql.cj.jdbc.Driver The name of the JDBC driver's main class. The example value in this table is for Oracle's MySQL Connector/J 6
add-database-params/username user1 The username of the database account.
add-database-params/password password The password of the database account.
add-database-params/table-name persistence The name of the table used to store persistence data.
add-database-params/columns pers_key pers_val The names of the 'key' and 'value' columns, specified in that order and separated by a space.
add-database-params/extra-params DEBUG Extra parameters accepted by the Persistor. The com.caplin.transformer.persistence.JdbcPersistorImpl accepts one parameter: a log level.

4. Disable repeated database writes in a Transformer cluster

The Persistence Service 6.2 supports synchronisation of persisted data between nodes in a Transformer cluster. A change to a persisted value on one node will replicate to other nodes in the cluster.

By default, any change made to a persisted value results in a write to the database server. This includes changes made by the Persistence Service's cluster synchronisation. As a result, when a value is persisted on one cluster node, the value will be written to the database server as many times as there are nodes in the cluster. This is inefficient and unnecessary.

To stop each cluster node from persisting changes made during cluster synchronisation, add the following configuration option to <Framework-root>/global_config/overrides/PersistenceService/Transformer/etc/persistence.conf on each node in the cluster.

dont-persist-cluster-messages   TRUE

Configuring permissioning

To allow users to interact with the Persistence Service Client, configure Liberator's Permissioning Service according to the specification below. For more information about permissioning, see User Authentication and Permissioning.

Rules

Define rules that trigger when users contribute (publish) to the Persistence Service Client's CONTROL and RECORD subjects. The strings used for the permission namespace and for the action are arbitrary; you are free to customise them to your own naming scheme.

Message Subject Rule Type Permission Namespace Action Action Ref Product Ref
^/PRIVATE/PERSISTENCE/%u/CONTROL WRITE PERSISTENCE_SERVICE PERSISTENCE-CONTROL - ALL_PRODUCTS
^/PRIVATE/PERSISTENCE/%u/RECORD/.* WRITE PERSISTENCE_SERVICE PERSISTENCE-RECORD - ALL_PRODUCTS

Permissions

Assign users the following permissions.

Product Set Permission Namespace Action Authorisation
.* PERSISTENCE_SERVICE PERSISTENCE-CONTROL ALLOW
.* PERSISTENCE_SERVICE PERSISTENCE-RECORD ALLOW
^/PRIVATE/PERSISTENCE/%u/CONTROL - VIEW ALLOW
^/PRIVATE/PERSISTENCE/%u/RECORD/.* - VIEW ALLOW

See also: