Persistence configuration

Transformer's built-in Persistence Service provides a persistent key-value store, shareable between nodes in a Transformer cluster. The configuration items here are used to define how persistence databases and files are accessed and controlled.

The Persistence Service is disabled by default. To activate the Persistence Service, see Activating the Persistence Service 7.

The Persistence Service 7 persists data to a JDBC-compliant relational database server or to the local filesystem. From version 7 of the Persistence Service, a JDBC database server is the only persistence store that can be used in a Transformer cluster. File-backed persistence (persisting data to the local filesystem) is provided as an option of convenience for development environments, but it is unsupported in production environments.

Warning: file-backed persistence is unsupported in production environments, and, from version 7 of the Persistence Service, file-backed persistence cannot be used in Transformer clusters.

To configure persistence to a database, besides defining the relevant configuration items that are documented here, you'll also need to set up the persistence database, giving it a name, defining the table and columns needed to store the persisted data, and defining the usernames(s) and access permissions that will allow Transformer to write and read the data.

Add your persistence configuration to the file <Framework-root>/global_config/overrides/PersistenceService/Transformer/etc/persistence.conf, where <Framework-root> is your Deployment Framework's top-level directory.

Note: In the following documentation, the term ​"this Transformer" means the Transformer for which you are defining the configuration, and "this node" means the cluster node that "this Transformer" represents.

Here's a list of all the persistence-related configuration items.

add-database-params

add-database-params specifies parameters for the database or file used to persist data.

Syntax:

add-database-params
   init-string    [string]
   driver-name    [string]
   username       [string]
   password       [string]
   table-name     [string]
   columns        [array of strings]
   extra-params   [array of strings]
end-add-database-params

The options for add-database-params are:

Name Type Default Description
columns array of strings [none]

A space-separated list of the names of the database columns where the persisted data is stored.

This option is not needed when persisting to a file.

Each piece of data is stored as a key-value pair. The names of the database columns should be listed in this order: <key column> <value column>. For example, to persist keys to a column called 'pers_key' and values to a column called 'pers_val', use the configuration below:

columns pers_key pers_val

Warning: If you're implementing your own Transformer module that persists data to the persistence database, make sure the keys you store have names that are unique to your module and don't clash with the keys used by other modules, such as the Watchlist Service.

driver-name string [none]

The name of the JDBC driver. For example: com.mysql.jdbc.Driver

This option is not needed when persisting to a file.

extra-params array of strings [implementation dependent]

A list of additional parameters that are used by this Transformer's implementation of the com.caplin.transformer.persistence.Persistor interface.

This option is not needed when persisting to a file.

The type, number, usage and default values of the parameters in the list are dependent on the implementation of Persistor that's used. For example, the basic JDBC implementation of Persistor (JdbcPersistorImpl) that's provided with the Persistence Service 7 takes one additional parameter: the timeout (in seconds) for the JDBC call Connection.isValid().

Also see persistence-classid.

init-string string [none]

Initialisation string for the persistence database or file.

  • If data is persisted to a file, this is the directory name and path of the file. In the Persistence Config blade the default file path and name is ${ccd}/persistence, where ${ccd} is the Transformer's current configuration directory.
  • If data is persisted to a database, this is the JDBC URL of the database; for example:
    jdbc:mysql://localhost/mydatabase
password string [none]

The database access password associated with username.

This option is not needed when persisting to a file.

table-name string [none]

The name of the database table where the persisted data is stored.

This option is not needed when persisting to a file.

username string [none]

The username that allows access to the database.

This option is not needed when persisting to a file.

disable-cluster-replication

Applicability: Persistence Service 6.2 only. The Persistence Service 7 does not replicate persisted values between nodes in a cluster.

disable-cluster-replication when TRUE, prevents this Transformer from sending to the other Transformers in the cluster data that it has already persisted.

When persistence is file-based, don't specify disable-cluster-replication. The default value of FALSE then applies, and, although each Transformer in the cluster maintains its own persistence file, this Transformer sends a copy of its persisted data to every other Transformer so that they can also write it to their own persistence files.

Conversely, when a (shared) persistence database is used, you should set disable-cluster-replication to TRUE on every node in the cluster to prevent multiple copies of the same data being persisted to the (shared) database. You should set dont-persist-cluster-messages to TRUE as well.

Syntax:

disable-cluster-replication
or
disable-cluster-replication TRUE
 

Type: boolean

Default value: FALSE

dont-persist-cluster-messages

Applicability: Persistence Service 6.2 only. The Persistence Service 7 does not replicate persisted values between nodes in a cluster.

dont-persist-cluster-messages when TRUE, ensures that messages received from another Transformer in the cluster are not persisted on this Transformer.

When persistence is file-based, don't specify dont-persist-cluster-messages. The default value of FALSE then applies, so that this Transformer saves to its own persistence file all the persisted data that other Transformers in the cluster send to it.

Conversely, when a (shared) persistence database is used, you should set dont-persist-cluster-messages to TRUE on every node in the cluster, to prevent the same data being persisted to the database multiple times (the data's then persisted solely by the Transformer where it originated). You should set ​disable-cluster-replication to TRUE as well.

Syntax:

dont-persist-cluster-messages
​or
dont-persist-cluster-messages TRUE

Type: boolean

Default value: FALSE (messages from another cluster node are always persisted on this node)

enable-file-database

enable-file-database when TRUE, enables file-based persistence.

Warning: file-backed persistence is unsupported in production environments, and, from version 7 of the Persistence Service, file-backed persistence cannot be used in Transformer clusters.

Syntax:

enable-file-database
or
enable-file-database TRUE

Type: boolean

Default value: FALSE (data is persisted to a database)

logfile

logfile specifies the name of the file in which events about persisting data are recorded.

The filename can contain the parameters %a and %h  At run time, %a is replaced by the Transformer's application name (see the DataSource configuration item application-name), and %h is replaced by the host name of the machine on which the Transformer is running.

Syntax:

logfile <log-file-name>

Type: string

Default value: persistence.log

log-level​

log-level specifies the severity of the errors and events that Transformer's persistence module writes to its log file (see logfile​).

Syntax:

log-level <log-level-name>

Type: string

Default value: INFO

Values accepted:

log-level-name DESCRIPTION
DEBUG Reports all errors and events.
INFO Reports events and information regarding normal operation of the persistence module, and all errors included in the WARN, NOTIFY, ERROR and CRIT log levels.
WARN Reports minor errors and all errors included in the NOTIFY, ERROR and CRIT log levels.
NOTIFY Reports errors regarding data corruptions and all errors included in the ERROR and CRIT log levels.
ERROR Reports serious errors and all errors included in the CRIT log level.
CRIT Reports critical errors that prevent the persistence module from running.

persistence-classid​

persistence-classid​ specifies the Java class ID of the implementation of the Java Persistor interface that provides this Transformer's database persistence capability. There's a basic JDBC  implementation of Persistor that's provided with, and used by, Transformer's Persistence Config blade; it's called JdbcPersistorImpl. (For more about Persistor and JdbcPersistorImpl, see the Transformer Persistence API for Java documentation.)

The class ID is conventionally the fully-qualified name of the implementation of Persistor, with periods (.) replaced by foward slashes (/).

Syntax:

persistence-classid​ <java-class-id>

Type: string

Default value: (none; but persistence-classid is only required if enable-file-database is FALSE)

Example: 

persistence-classid com/caplin/transformer/persistence/JdbcPersistorImpl

If you've created your own implementation of Persistor, you must also specify your implementation class in an add-javaclass configuration item. Set the add-javaclass::class-id option to the same value that you used for persistence-classid.


See also: