Persist data to Transformer

Transformer's Persistence Service provides StreamLink clients and Transformer Modules with access to a persisted key-value store. This page compares the Persistence Service in Transformer 6.2 and Transformer 7.0, and provides examples of persisting data using the StreamLink API and the Java Transformer Module (JTM) API.

Contents:

Requirements

Transformer's Persistence Service is disabled by default.

To persist data using Transformer's module APIs, you must first activate the Transformer blades below:

  • PersistenceService

To persist data using StreamLink, you must first activate the Transformer blades below:

  • PersistenceService
  • PersistenceServiceClient

For more information on activating the Persistence Service, see the page appropriate to your version of Transformer:

Persisting data using Transformer's APIs

This section describes how to persist data in Transformer 6.2 and Transformer 7.0.

Changes to the Persistence Service API in Transformer 7

The Persistence Service APIs were rewritten for Transformer 7, and are not backward compatible with the Persistence Service APIs for Transformer 6.2.

The new Persistence Service API in Transformer 7 offers performance improvements through normalisation. Each module and pipeline can now persist data to its own set of database tables rather than sharing a single table, as was the case in Transformer 6.2. As a result, database queries can now operate on smaller datasets. Persistence tables are designed by the module developer, and the table specification can be tailored to the nature of the data stored by the module.

The differences between the Persistence Service in Transformer 6.2 and Transformer 7.0 are summarised below:

  Transformer 6.2 Transformer 7
Number of tables One table. One or more tables.
Persistence key One string, mapped to one primary-key column. One or more strings, each mapped to a primary-key column.
Persistence value One string, mapped to one value column. One or more strings, each mapped to a value column.
SQL data-types Character data-types.

Character, integer, and decimal data-types.

Table design Fixed design. Designed by the module or pipeline developer.
Mapping of key-value pairs to tables and columns Specified by the Transformer administrator in the persistence.conf configuration file. Specified by the developer when persisting, reading, and removing key-value pairs.

Persisting data in Transformer 6.2

This section provides an example of using the Java Transformer Module (JTM) API, com.caplin.transformer.module, to persist and retrieve values.

The Persistence API is in package com.caplin.transformer.module.persistence.

Blade configuration

You do not need to add any configuration to the module's blade to use Transformer 6.2's Persistence Service.

Persisting a value

In Transformer 6.2, all persisted values are stored in the same database table. To ensure that the value's key is unique, prefix the key with any values required to distinguish it from other keys of the same name. In the example below, the key "defaultaccount" is prefixed by the module name and the the user's username.

To persist a value, use the Persistence.put method.

Persistence ps = new PersistenceImpl();

String key = moduleName + username + "defaultaccount";
String value = "account1"
ps.put(new PersistedValue(key, value));

Retrieving a value

To retrieve a value, use the Persistence.get method:

Persistence ps = new PersistenceImpl();

String key = moduleName + username + "defaultaccount";
String value = null;
PersistedValue pv = ps.get(key);
if (pv != null) {
    value = pv.getValue();
}

Retrieving all key-value pairs for a user

To retrieve all key-value pairs for a user, use the Persistence.query method. The example below retrieves all keys persisted by the module for a specific user:

Persistence ps = new PersistenceImpl();

String keyPrefix = moduleName + username;
PersistedValue[] persistedValues = ps.query(keyPrefix);

Persisting data in Transformer 7.0

This section provides an example of using the Java Transformer Module (JTM) API, com.caplin.jtm, to persist and retrieve values.

Deprecation notice: Transformer 7.0's new JTM API, com.caplin.jtm, is a replacement for Transformer 6.2's JTM API, com.caplin.transformer.module. The original JTM API is now deprecated, and has an end-of-life scheduled for one year following the initial release of Transformer 7.

The Persistence API is in package com.caplin.jtm.persistence.

Blade configuration

In Transformer 7.0, modules and pipelines can each use their own set of persistence tables. Whether the creation of these tables is automatic or must be done manually depends on the persistence store:

  • JDBC database: table creation cannot be automated because the Persistence Service should not have table-creation privileges to a shared database.
  • Local file (SQLite): table creation can be automated because the Persistence Service has full privileges to its own SQLite database.
    • Note: Persistence by local file (SQLite) is not supported for use in production and cannot be used in Transformer 7 clusters.

To automate the creation of persistence tables when local file persistence is enabled, include two files in the blade's Transformer/etc directory:

  • persistence-extra.conf: the Persistence Service reads this file on start up. The file contains a single configuration item that defines the filename of the SQLite DDL script for creating this module's persistence table:
    • persistence-filedb-bootstrap  ${ccd}/bootstrap.sql
  • bootstrap.sql: the file containing the SQLite DDL script to create this module's persistence table.
    • Contents of Transformer/etc/bootstrap.sql in this example:
    • CREATE TABLE IF NOT EXISTS TRANSFORMER_EXAMPLE_MODULE (
          EXAMPLE_USER VARCHAR(250) NOT NULL,
          EXAMPLE_KEY VARCHAR(250) NOT NULL,
          EXAMPLE_VALUE VARCHAR(4096) NOT NULL,
          PRIMARY KEY (EXAMPLE_USER, EXAMPLE_KEY)
      );
    • Note: the table has a composite primary-key (EXAMPLE_USER and EXAMPLE_KEY) and a single value column (EXAMPLE_VALUE). The composite primary key makes it possible to perform queries that return all the keys persisted for a user by this module (see later).

When deploying your blade to production, your system administrator will create a variant of the bootstrap.sql file suitable for your production database server, and will run the script when the module's blade is deployed to production.

Persisting a value

To persist a value, use the Persistence.upsert method. Keys and values are passed to the method as java.util.Map collections of column values keyed by column names. The example below references the key and value columns created by the script, bootstrap.sql, above.

// transformerAccessor is passed to the module in the TransformerModule.initialise method
Persistence ps = transformerAccessor.getPersistence();

java.util.Map<String,String> keyColumns = new java.util.HashMap<String,String>();
keys.put("EXAMPLE_USER", username);
keys.put("EXAMPLE_KEY", "defaultaccount");

java.util.Map<String,String> data = new java.util.HashMap<String,String>();
data.put("EXAMPLE_VALUE", "account1");

ps.upsert("TRANSFORMER_EXAMPLE_MODULE", keyColumns, data);

Retrieving a value

To retrieve a value, use the Persistence.get method. The method returns a java.util.List of java.util.Map objects, where each Map object represents a resultset row. The example below references the key columns created by the script, bootstrap.sql, above.

// transformerAccessor is passed to the module in the TransformerModule.initialise method
Persistence ps = transformerAccessor.getPersistence();

java.util.Map<String,String> keyColumns = new java.util.HashMap<String,String>();
keys.put("EXAMPLE_USER", username);
keys.put("EXAMPLE_KEY", "defaultaccount");

String value = null;
java.util.List<java.util.Map<String,String>> rows = ps.get("TRANSFORMER_EXAMPLE_MODULE", keyColumns);
if (!rows.isEmpty()) {
    java.util.Map<String,String> row = rows.get(0);
    value = row.get("EXAMPLE_VALUE");
}

Retrieving all key-value pairs for a user

To retrieve all values persisted by this module for a specific user, use the Persistence.get method. The code example below is similar to that used for retrieving a single persisted value, except that this time the EXAMPLE_KEY column is omitted from the Map referenced by the keyColumns variable.

// transformerAccessor is passed to the module in the TransformerModule.initialise method
Persistence ps = transformerAccessor.getPersistence();

java.util.Map<String,String> keyColumns = new java.util.HashMap<String,String>();
keys.put("EXAMPLE_USER", username);

String value = null;
java.util.List<java.util.Map<String,String>> rows = ps.get("TRANSFORMER_EXAMPLE_MODULE", keyColumns);
for (java.util.Map<String,String> row : rows) {
    System.out.println(row.get("EXAMPLE_KEY") + "=" + row.get("EXAMPLE_VALUE"));
}

Persisting data using StreamLink

The Persistence Service Client module publishes a private namespace for each StreamLink user, where they can read and update persisted generic-records, and a control channel to allow each StreamLink user to create and delete records within their private namespace.

Subjects published by the Persistence Service Client module
Subject Description
/PRIVATE/PERSISTENCE/CONTROL

The control channel. You create and delete persisted records by contributing messages to this channel.

/PRIVATE/PERSISTENCE/RECORD/record_identifier Individual record channels. You write data to a persisted record by publishing data to it.

Security

A StreamLink user cannot access, change, or delete another StreamLink user's persisted records.

  • The record namespace, /PRIVATE/PERSISTENCE/RECORD, is mapped internally by Liberator to a namespace private to each StreamLink user: /PRIVATE/PERSISTENCE/username/RECORD.
  • The control subject, /PRIVATE/PERSISTENCE/CONTROL, is mapped internally by Liberator to a subject private to each StreamLink user: /PRIVATE/PERSISTENCE/username/CONTROL.
  • The Persistence Service Client persists each record under a persistence key that includes the StreamLink user's username.

Creating a persisted record

You must create a persistence record before you can write data to it.

To create a persisted record, send the following command message to the control channel, /PRIVATE/PERSISTENCE/CONTROL.

Request message
Field Data Type Example Description
OperationId String 1

A unique identifier for this operation. The identifier will be returned in the control channel's response message, and will enable you to associate the response with the operation.

Important: Depending on Liberator's configuration and licence, it may be possible for a user to run concurrent sessions. The identifier you assign to OperationId must be unique across all the user's sessions.

CreateRecord String defaultblotter An identifier for the record.

The Persistence Service Client will respond with the following message on the control channel:

Response message
Field Data Type Example Description
OperationId String 1 The unique identifier for this operation, as provided by you in the request message.
CreateResult Integer 1 [Transformer 7 only] 1 if the operation succeeded, or -1 if the operation failed.
OperationFinished Boolean true true if the operation succeeded, or false if the operation failed.

Reading a persisted record

Subscribe to the record's subject. For example, to read the record with a record identifier of 'defaultblotter', subscribe to '/PRIVATE/PERSISTENCE/RECORD/defaultblotter'.

Writing to a persisted record

To write data to record, it must exist first. You can test whether a record exists by subscribing to it.

Writing data to a record does not return a success value. To confirm that a write was successful, subscribe to the subject beforehand and listen for the update that occurs after a successful write.

To write data to a record, follow the steps below:

  1. Subscribe to the persisted record's subject.
    • If nodata is returned, then the record does not exist. Create the record (see Creating a persisted record) and re-subscribe to the record's subject.
  2. To write data to the record, publish fields to the persisted record's subject.
    • Your fields are merged with any previously contributed fields in the record.
    • You cannot delete a previously contributed field, but you can set the field to an empty value.
    • If the contribution is successful, your subscription to the record's subject will receive an update.

Deleting a persisted record

To delete a persisted record, send the following command message to the control channel, /PRIVATE/PERSISTENCE/CONTROL.

Request message
Field Data Type Example Description
OperationId String 1

A unique identifier for this operation. The identifier will be returned in the control channel's response message, and will enable you to associate the response with the operation.

Important: Depending on your Liberator's configuration and licence, it may be possible for a user to run concurrent sessions. The identifier you assign to OperationId must be unique across all the user's sessions.

DeleteRecord String blotter The record's identifier.

The Persistence Service Client will respond with the following message on the control channel:

Response message
Field Data Type Example Description
OperationId String 1 The unique identifier for this operation, as provided by you in the request message.
DeleteResult Integer 1 [Transformer 7 only] 1 if the operation succeeded, or -1 if the operation failed.
OperationFinished Boolean true true if the operation succeeded, or false if the operation failed.