Activating the Persistence Service 7

The Persistence Service provides a persistent key-value store to Transformer modules and StreamLink clients. The service can be configured to persist data to a database server or to a local SQLite file.

This page is for the Persistence Service included with Transformer 7.0. For the Persistence Service included with Transformer 6.2, see Activating the Persistence Service (Transformer 6.2).

Contents:

Dependent modules

The following Caplin-supplied Transformer modules are dependent on the Persistence Service:

  • Persistence Service Client
  • Watchlist Service
  • Alerts Service
  • High-Low Service

Upgrading from the Persistence Service 6.2

The Persistence Service 7.0 has a new API that is not compatible with modules written for the Persistence Service 6.2 API.

Custom modules written for the Persistence Service 6.2 API must be upgraded to the Persistence Service 6.2 API.

For a summary of breaking changes, see Upgrading to Caplin Platform 7: Persistence Service.

Activating the Persistence Service blades

The Persistence Service comprises two config-blades, which are deactivated by default:

  • PersistenceService: This blade provides persistence services to Transformer modules, including the PersistenceServiceClient module (below).
  • PersistenceServiceClient: This blade provides persistence services to StreamLink clients. It requires the PersistenceService module to be active.

To activate the Persistence Service blades, follow the steps below:

  1. From the root directory of your DFW, run the command below to activate the Persistence Service config-blades:

    ./dfw activate PersistenceService PersistenceServiceClient
  2. If you will be using database-backed persistence (required in production deployments), then enable Transformer's JVM. In the file global_config/overrides/PersistenceService/Transformer/etc/java.conf, uncomment the line below:

    jvm-location ${JVM_LOCATION}
  3. From the root directory of your DFW, run the command below to restart Transformer:

    ./dfw start Transformer

Persistence stores

The Persistence Service supports the following persistance stores:

Database-backed persistence

Data is persisted to database tables in a relational database server.

The database tables to which data is persisted are not created automatically. A database administrator must create the database tables required by the PersistenceServiceClient and any other modules that use the Persistence Service.

Database-backed persistence is the only form of persistence storage that is suitable for use in a Transformer 7 cluster and is the only form of persistence supported in production.

The Persistence Service supports the following database servers:

  • Oracle
  • Microsoft SQL Server
  • Sybase (tested with the Sybase jConnect for JDBC driver)
  • MySQL (tested with the MariaDB Connector/J driver)

    Note: the Oracle MySQL Connector/J driver is not supported.

  • Postgres
  • H2

File-backed persistence

Data is persisted to database tables in a local SQLite file. File-backed persistence is configured by default but is not supported for use in production deployments.

File-backed persistence cannot be used in a Transformer 7 cluster. Transformer 7, unlike Transformer 6.2, does not synchronise persisted data between cluster nodes.

Warning: file-backed persistence is unsupported in production environments, and, from version 7 of the Persistence Service, file-backed persistence cannot be used in Transformer clusters.

Because the persistence store is a local SQLite file, the Persistence Service has exclusive and administrator-level access to it. This allows the Persistence Service to automatically create the database tables required by the PersistenceServiceClient blade and other modules that use the Persistence Service.

When Transformer is started and file-backed persistence is enabled, the Transformer performs the following operations:

  1. Creates the SQLite database file (if it does not exist).
  2. Creates the database table (if it does not exist) required by the now deprecated JTM API, com.caplin.transformer.module.persistence.

    Note: The filename of the SQLite DDL script is specified by the persistence-filedb-bootstrap configuration option in the file <Framework-root>/active_blades/PersistenceService/Transformer/etc/persistence.conf

  3. Creates the database tables (if they do not exist) required by each Transformer module that uses the Persistence Service 7.

    Note: The filename of each module's SQLite DDL script is specified by the persistence-filedb-bootstrap configuration option in the module's configuration file Transformer/etc/persistence-extra.conf

Configuring database-backed persistence

To configure the Persistence Service to use database-backed persistence:

  1. Create database tables and triggers
  2. Add the database's JDBC driver to Transformer's classpath
  3. Configure the connection to the database

1. Create database tables and triggers

Use a database administrator account to create the tables and triggers.

For Caplin-supplied modules, adapt the SQLite scripts in the table below to the syntax of your database server:

  SQLite bootstrap script Notes
Persistence Service 7 /active_blades/PersistenceService/Transformer/etc/bootstrap.sql Creates the table required by the Persistence API in the deprecated JTM, com.caplin.transformer.module.persistence.
Persistence Service Client 7 /active_blades/PersistenceServiceClient/Transformer/etc/bootstrap.sql  
Watchlist Service 7 /active_blades/WatchlistService/Transformer/etc/bootstrap.sql  
Alerts Service 7 /active_blades/AlertsService/Transformer/etc/bootstrap.sql  
High-Low Service 6.2 - Version 6.2 uses the Persistence API in the deprecated JTM, com.caplin.transformer.module, and so does not have its own persistence table.

2. Add the database's JDBC driver to Transformer's classpath

To add the database's JDBC driver to Transformer's classpath:

  1. Create the following directory: <Framework-root>/global_config/overrides/PersistenceService/Transformer/lib/java
  2. Copy the JDBC driver JAR file to the directory created in step 1.
  3. Make the following two changes to the file: <Framework-root>/global_config/overrides/PersistenceService/Transformer/etc/java.conf
    1. Add the following line (where jar_filename is the name of JDBC driver JAR file):

      jvm-global-classpath "${ccd}/../lib/java/jar_filename"
    2. Uncomment the following line:

      jvm-location ${JVM_LOCATION}

3. Configure the connection to the database

To configure database-backed persistence, edit the file <Framework-root>/global_config/overrides/PersistenceService/Transformer/etc/persistence.conf

The example configuration below configures the Persistence Service to connect to a MySQL database on localhost.

enable-file-database FALSE

#
# Uncomment the jvm-location configuration in the java.conf file in this directory
# if the java persistence API is being used.
#
persistence-classid com/caplin/transformer/persistence/JdbcPersistorImpl
add-database-params
   init-string    jdbc:mysql://localhost:3306/rtas
   driver-name    org.mariadb.jdbc.Driver
   username       root
   password       password
   extra-params   5
end-database-params

For an explanation of the configuration items above, see the table below:

Configuration item Value Comment
enable-file-database FALSE Set to FALSE to use database-backed persistence.
persistence-classid com/​caplin/​transformer/​persistence/​JdbcPersistorImpl The implementation of the Persistor interface uses to persist data to the database. The example value in this table is the default JDBC Persistor that comes packaged with the Persistence Service.
init-string jdbc:mysql:/​/​localhost:3306/​rtas The JDBC URL of the database. The example value in this table specifies a local MySQL database called 'rtas'.
driver-name org.mariadb.jdbc.Driver

The name of the JDBC driver's main class. The example value in this table specifies the MariaDB Connector/J driver.

username root The username of the Persistence Service's database account.
password password The password of the Persistence Service's database account.
extra-params 5 Extra parameters accepted by the Persistor. The com.caplin.transformer.persistence.JdbcPersistorImpl accepts one parameter: the maximum time to wait, in seconds, for the operation used to check the validity of a database connection to complete. See Connection.isValid.

Configuring file-backed persistence

File-backed persistence is configured by default.

Support for SQLite is built-in to the Persistence Service. No external driver is required.

To revert to file-backed persistence after changing the configuration, edit the file <Framework-root>/global_config/overrides/PersistenceService/Transformer/etc/persistence.conf

The configuration below shows the default configuration for file-backed persistence:

enable-file-database  TRUE

#
# Uncomment the jvm-location configuration in the java.conf file in this directory
# if the java persistence API is being used.
#
#persistence-classid com/caplin/transformer/persistence/JdbcPersistorImpl
add-database-params
   init-string    ${ccd}/persistence.db
#  init-string    jdbc:mysql://localhost:3306/rtas
#  driver-name    org.mariadb.jdbc.Driver
#  username       root
#  password       password
end-database-params

For an explanation of the configuration items above, see the table below:

Configuration item Value Comment
enable-file-database TRUE Set to TRUE to enable file-backed persistence.
add-database-params/init-string ${ccd}/persistence.db The filename for the SQLite database. If the database does not exist, the Persistence Service will create it. For a list of valid filename formats, see URI Filenames in SQLite C Interface: Opening a new database.

Permissioning users to use the Persistence Service Client

To allow users to interact with the Persistence Service Client, configure Liberator's Permissioning Service according to the specification below. For more information about permissioning, see User Authentication and Permissioning.

Rules

Define rules that trigger when users contribute (publish) to the Persistence Service Client's CONTROL and RECORD subjects. The strings used for the permission namespace and for the action are arbitrary; you are free to customise them to your own naming scheme.

Message Subject Rule Type Permission Namespace Action Action Ref Product Ref
^/PRIVATE/PERSISTENCE/%u/CONTROL WRITE PERSISTENCE_SERVICE PERSISTENCE-CONTROL - ALL_PRODUCTS
^/PRIVATE/PERSISTENCE/%u/RECORD/.* WRITE PERSISTENCE_SERVICE PERSISTENCE-RECORD - ALL_PRODUCTS

Permissions

Assign users the following permissions.

Product Set Permission Namespace Action Authorisation
.* PERSISTENCE_SERVICE PERSISTENCE-CONTROL ALLOW
.* PERSISTENCE_SERVICE PERSISTENCE-RECORD ALLOW
^/PRIVATE/PERSISTENCE/%u/CONTROL - VIEW ALLOW
^/PRIVATE/PERSISTENCE/%u/RECORD/.* - VIEW ALLOW

See also: