Persist data to Transformer

Transformer's Persistence Service provides StreamLink clients and Transformer Modules with access to a persisted key-value store. This page compares the Persistence Service in Transformer 6.2 and Transformer 7.0, and provides examples of persisting data using the StreamLink API and the Java Transformer Module (JTM) API.

If you are reading this page as part of an upgrade to Caplin Platform 7, see Upgrading to Caplin Platform 7.

Contents:

Requirements

Transformer's Persistence Service must be activated.

For more information on activating the Persistence Service, see the page appropriate to your version of Transformer:

Breaking changes in the Persistence Service in Transformer 7

Breaking changes in Transformer 7:

  • Transformer 7's persistence API is not compatible with modules written for Transformer 6.2's persistence API
  • Transformer 7's persistence service uses a new database persistence model, which is incompatible with data stored in Transformer 6.2's persistence model.
  • Transformer 7's persistence service does not support Transformer 6.2's filedb format

The differences between the Persistence Service in Transformer 6.2 and Transformer 7.0 are summarised below:

  Transformer 6.2 Transformer 7
Number of database tables One table Multiple tables
When are database tables created Manually, during activation of the Persistence Service Manually, during installation of each dependent Transformer module
Persistence key One string, mapped to one primary-key column. One or more strings, each mapped to a primary-key column.
Persistence value One string, mapped to one value column. One or more strings, each mapped to a value column.
SQL data-types Character data-types.

Character, integer, and decimal data-types.

Table design Fixed design. Designed by the module or pipeline developer.
Mapping of data to tables and columns Specified by the Transformer administrator in the persistence.conf configuration file. Specified by the developer when persisting, reading, and removing key-value pairs.
Transaction support No No

Database transaction support

The persistence service does not support database transactions. To ensure that parallel writes are not made to the same persistence key, follow the guidance below:

  • In load-balanced deployments of Transformer, enable the source-affinity load-balancing algorithm to ensure that a user's persistence operations occur in sequence on the same instance of Transformer.
  • [Transformer 7] Do not share database tables between different Transformer modules.

Support for database transactions will be added in a later release of Transformer 7.

Persisting data using the Transformer 6.2 API

This section provides an example of using the legacy Java Transformer Module (JTM) API, com.caplin.transformer.module, to persist and retrieve values.

The Persistence API is in package com.caplin.transformer.module.persistence.

Configuration

No extra configuration is required for a Transformer module to use Transformer 6.2's Persistence Service.

Persisting a value

In Transformer 6.2, all persisted values are stored in the same database table. To ensure that the value's key is unique, prefix the key with any values required to distinguish it from other keys of the same name. In the example below, the key "defaultaccount" is prefixed by the module name and the the user's username.

To persist a value, use the Persistence.put method.

Persistence ps = new PersistenceImpl();

String key = moduleName + username + "defaultaccount";
String value = "account1"
ps.put(new PersistedValue(key, value));

Retrieving a value

To retrieve a value, use the Persistence.get method:

Persistence ps = new PersistenceImpl();

String key = moduleName + username + "defaultaccount";
String value = null;
PersistedValue pv = ps.get(key);
if (pv != null) {
    value = pv.getValue();
}

Retrieving all key-value pairs for a user

To retrieve all key-value pairs for a user, use the Persistence.query method. The example below retrieves all keys persisted by the module for a specific user:

Persistence ps = new PersistenceImpl();

String keyPrefix = moduleName + username;
PersistedValue[] persistedValues = ps.query(keyPrefix);

Persisting data using the Transformer 7 API

This section provides an example of using the Java Transformer Module (JTM) API, com.caplin.jtm, to persist and retrieve values.

Deprecation notice: Transformer 7.0's new JTM API, com.caplin.jtm, is a replacement for Transformer 6.2's JTM API, com.caplin.transformer.module. The original JTM API is now deprecated, and has an end-of-life scheduled for one year following the initial release of Transformer 7.

The Persistence API is in package com.caplin.jtm.persistence.

Creating database tables

In Transformer 7, you must create the tables that you want your module to persist data to. This is contrast to Transformer 6.2's Persistence Service, which stores all persisted values in one database table.

Warning: all Caplin supplied modules that use the Transformer 7 persistence service require a database administrator to create database tables as part of their installation. For detailed instructions, see Activating the Persistence Service 7, Deploying the Watchlist Service, and Deploying the Alerts Service.

The Transformer 7 persistence service supports the following features in database tables:

  • Multiple-column primary keys
  • Multiple value columns
  • Character, integer, and decimal column types

The SQL script below shows an example of a table suitable for use with Transformer 7's persistence service:

CREATE TABLE IF NOT EXISTS TRANSFORMER_EXAMPLE_MODULE (
    EXAMPLE_USER VARCHAR(250) NOT NULL,
    EXAMPLE_KEY VARCHAR(250) NOT NULL,
    EXAMPLE_VALUE VARCHAR(4096) NOT NULL,
    PRIMARY KEY (EXAMPLE_USER, EXAMPLE_KEY)
);

The table has a composite primary-key (EXAMPLE_USER and EXAMPLE_KEY) and a single value column (EXAMPLE_VALUE). The position of the EXAMPLE_USER column as the leftmost column of the primary key is deliberate; it optimises queries that return all the keys persisted for a single user (see Retrieving all key-value pairs for a user, below).

For further examples of table designs, see the SQLite reference script (Transformer/etc/bootstrap.sql) included in the Persistence Service Client, the Watchlist Service, and the Alerts Service. Caplin does not support SQLite in production; the scripts are provided as reference implementations for database administrators to adapt to their own RDBMS.

Persisting a value

To persist a value, use the Persistence.upsert method. Keys and values are passed to the method as java.util.Map collections of column values keyed by column names. The example below references the key and value columns created by the script, bootstrap.sql, above.

// transformerAccessor is passed to the module in the TransformerModule.initialise method
Persistence ps = transformerAccessor.getPersistence();

java.util.Map<String,String> keys = new java.util.HashMap<String,String>();
keys.put("EXAMPLE_USER", username);
keys.put("EXAMPLE_KEY", "defaultaccount");

java.util.Map<String,String> data = new java.util.HashMap<String,String>();
data.put("EXAMPLE_VALUE", "account1");

ps.upsert("TRANSFORMER_EXAMPLE_MODULE", keys, data);

Retrieving a value

To retrieve a value, use the Persistence.get method. The method returns a java.util.List of java.util.Map objects, where each Map object represents a resultset row. The example below references the key columns created by the script, bootstrap.sql, above.

// transformerAccessor is passed to the module in the TransformerModule.initialise method
Persistence ps = transformerAccessor.getPersistence();

java.util.Map<String,String> keys = new java.util.HashMap<String,String>();
keys.put("EXAMPLE_USER", username);
keys.put("EXAMPLE_KEY", "defaultaccount");

String value = null;
java.util.List<java.util.Map<String,String>> rows
    = ps.get("TRANSFORMER_EXAMPLE_MODULE", keys);
if (!rows.isEmpty()) {
    java.util.Map<String,String> row = rows.get(0);
    value = row.get("EXAMPLE_VALUE");
}

Retrieving all key-value pairs for a user

To retrieve all values persisted by this module for a specific user, use the Persistence.get method. The code example below is similar to that used for retrieving a single persisted value, except that this time the EXAMPLE_KEY column is omitted from the Map referenced by the keyColumns variable.

// transformerAccessor is passed to the module in the TransformerModule.initialise method
Persistence ps = transformerAccessor.getPersistence();

java.util.Map<String,String> keys = new java.util.HashMap<String,String>();
keys.put("EXAMPLE_USER", username);

String value = null;
java.util.List<java.util.Map<String,String>> rows
    = ps.get("TRANSFORMER_EXAMPLE_MODULE", keys);
for (java.util.Map<String,String> row : rows) {
    System.out.println(row.get("EXAMPLE_KEY") + "=" + row.get("EXAMPLE_VALUE"));
}

Persisting data using StreamLink

The Persistence Service Client module publishes a private namespace for each StreamLink user, where they can read and update persisted generic-records, and a control channel to allow each StreamLink user to create and delete records within their private namespace.

Subjects published by the Persistence Service Client module
Subject Description
/PRIVATE/PERSISTENCE/CONTROL

The control channel. You create and delete persisted records by contributing messages to this channel.

/PRIVATE/PERSISTENCE/RECORD/record_identifier Individual record channels. You write data to a persisted record by publishing data to it.

Security

A StreamLink user cannot access, change, or delete another StreamLink user's persisted records.

  • The record namespace, /PRIVATE/PERSISTENCE/RECORD, is mapped internally by Liberator to a namespace private to each StreamLink user: /PRIVATE/PERSISTENCE/username/RECORD.
  • The control subject, /PRIVATE/PERSISTENCE/CONTROL, is mapped internally by Liberator to a subject private to each StreamLink user: /PRIVATE/PERSISTENCE/username/CONTROL.
  • The Persistence Service Client persists each record under a persistence key that includes the StreamLink user's username.

Creating a persisted record

Before you can write to a persistence record, you must create it.

To create a persisted record, send the following command message to the control channel, /PRIVATE/PERSISTENCE/CONTROL.

Request message
Field Data Type Example Description
OperationId String 1

A unique identifier for this operation. The identifier will be returned in the control channel's response message, and will enable you to associate the response with the operation.

Important: Depending on Liberator's configuration and licence, it may be possible for a user to run concurrent sessions. The identifier you assign to OperationId must be unique across all the user's sessions.

CreateRecord String defaultblotter An identifier for the record.

The Persistence Service Client will respond with the following message on the control channel:

Response message
Field Data Type Example Description
OperationId String 1 The unique identifier for this operation, as provided by you in the request message.
CreateResult Integer 1 [Transformer 7 only] 1 if the operation succeeded, or -1 if the operation failed.
OperationFinished Boolean true true if the operation succeeded, or false if the operation failed.

Reading a persisted record

Subscribe to the record's subject. For example, to read the record with a record identifier of 'defaultblotter', subscribe to '/PRIVATE/PERSISTENCE/RECORD/defaultblotter'.

Writing to a persisted record

To write data to record, it must exist first. You can test whether a record exists by subscribing to it.

Writing data to a record does not return a success value. To confirm that a write was successful, subscribe to the subject beforehand and listen for the update that occurs after a successful write.

To write data to a record, follow the steps below:

  1. Subscribe to the persisted record's subject.
    • If nodata is returned, then the record does not exist. Create the record (see Creating a persisted record) and re-subscribe to the record's subject.
  2. To write data to the record, publish fields to the persisted record's subject.
    • Your fields are merged with any previously contributed fields in the record.
    • You cannot delete a previously contributed field, but you can set the field to an empty value.
    • If the contribution is successful, your subscription to the record's subject will receive an update.

Deleting a persisted record

To delete a persisted record, send the following command message to the control channel, /PRIVATE/PERSISTENCE/CONTROL.

Request message
Field Data Type Example Description
OperationId String 1

A unique identifier for this operation. The identifier will be returned in the control channel's response message, and will enable you to associate the response with the operation.

Important: Depending on your Liberator's configuration and licence, it may be possible for a user to run concurrent sessions. The identifier you assign to OperationId must be unique across all the user's sessions.

DeleteRecord String blotter The record's identifier.

The Persistence Service Client will respond with the following message on the control channel:

Response message
Field Data Type Example Description
OperationId String 1 The unique identifier for this operation, as provided by you in the request message.
DeleteResult Integer 1 [Transformer 7 only] 1 if the operation succeeded, or -1 if the operation failed.
OperationFinished Boolean true true if the operation succeeded, or false if the operation failed.