Upgrading to Caplin Platform 7

This topic describes the steps to required to upgrade Caplin Platform 6.2 to Caplin Platform 7.

For an overview of what is new in Caplin Platform 7, see Caplin Platform 7 Release Highlights.

Contents:

Server requirements

For Caplin Platform 7's server requirements, see Caplin Platform System Requirements.

Compatibility with StreamLink 6.2 network clients

Caplin Platform 7 is backward compatible with StreamLink 6.2 clients.

Upgrade clients to StreamLink 7 to take advantage of the following new features in Caplin Platform 7:

  • Re-ordering of watchlists
  • Repeatable fields in contribs
  • Cancelling triggers in Tranformer's Alerts Service module

Compatibility with DataSource 6.2 network peers

Caplin Platform 7 components are compatible with Caplin Platform 6.2 DataSource peers, subject to the following caveats:

  • Liberator 7 servers cannot be clustered with Liberator 6.2 servers.
  • Transformer 7 servers cannot be clustered with Transformer 6.2 servers.
  • DataSource 6.2 peers do not support IPv6, but Caplin Platform 7 components do. Do not bind the DataSource server sockets of Liberator 7 or Transformer 7 to an IPv6 address.

    Warning: on dual-stack networks, in which hosts have dual IPv4 and IPv6 addresses, you can inadvertently bind a DataSource 7 component to an IPv6 address if you assign a hostname to datasrc-interface. By default, name resolution queries on modern operatings grant IPv6 addresses higher precedence than IPv4 addresses. To avoid network-connectivity issues between Caplin Platform 7 components and Caplin Platform 6.2 components, see DataSource > IPv6 Networking.

Compatibility with the Deployment Framework 6.2

All servers in your Caplin stack must use the same version of the Deployment Framework. When upgrading to Caplin Platform 7, all your servers must be upgraded to the Deployment Framework 7.

The Deployment Framework 7 is compatible with:

  • Liberator 7
  • Transformer 7
  • Adapters written using DataSource 7.0 libraries
  • Adapters written using DataSource 6.2 libraries (to avoid connectivity issues with Caplin Platform 7 components, see DataSource: IPv6 networking)

Liberator 7, Transformer 7, and adapters written using DataSource 7 libraries cannot be deployed on the Deployment Framework 6.2.

Deployment Framework compatibility matrix
  Deployment Framework 6.2 Deployment Framework 7
Liberator 6.2
Transformer 6.2
Adapters (DataSource 6.2)
Liberator 7
Transformer 7
Adapters (DataSource 7)

Compatibility with Caplin FX Suite

The minimum version of Caplin FX Sales that is compatible with Transformer 7 is Caplin FX Sales 1.11.

Upgrade strategy

It is not possible to perform an in-place upgrade of the Deployment Framework from version 6.2 to version 7. Instead, install the Deployment Framwork 7 in parallel, and then migrate data and configuration overrides from your Caplin Platform 6.2 installation.

To migrate from version 6.2 to version 7.0 of the Deployment Framework, follow the instructions below:

  1. Install the Deployment Framework 7 in parallel to your existing installation.
  2. Deploy components to the Deployment Framework 7:
    1. See Upgrading to Liberator 7, below.
    2. See Upgrading to Transformer 7, below.
  3. Manually merge configuration overrides from your Deployment Framework 6.2 installation to your Deployment Framework 7.0 installation.

    Warning: Do not overwrite Caplin Platform 7 override files with Caplin Platform 6.2 override files. Port 6.2 configuration to 7.0 configuration files manually.

  4. Perform a test migration of data persisted by Transformer 6.2.

    See Upgrading to Transformer 7, below.

  5. Test your Deployment Framework 7.0 installation.
  6. To cut over to the Deployment Framework 7.0:
    1. Shutdown both your 6.2 and 7.0 installations.
    2. Migrate data persisted by Transformer 6.2 (tested in step 4 above).
    3. Start the Deployment Framework 7.0.

Upgrading to Liberator 7

Summary of important changes in Liberator 7:

  • Liberator 7 supports IPv6.

    On dual-stack networks, hostnames in Caplin Platform configuration files that previously resolved to an IPv4 address may now resolve to an IPv6 address. To avoid connectivity issues in dual-stack (IPv4 and IPv6) networks, see DataSource: IPv6 Networking.

  • Liberator 7's Auth API is not compatible with auth modules written for Liberator 6.2.

    If your deployment uses a custom Liberator auth module, you must migrate the module to the Liberator 7 Auth API.

    If your deployment uses Caplin's Permissioning Auth Module (PAM), you must deploy the Caplin Permissioning Service 7.

Upgrading to Transformer 7

Summary of important changes in Transformer 7:

  • Transformer 7 supports IPv6.

    On dual-stack networks, hostnames in Caplin Platform configuration files that previously resolved to an IPv4 address may now resolve to an IPv6 address. To avoid connectivity issues in dual-stack (IPv4 and IPv6) networks, see DataSource: IPv6 Networking.

  • Transformer 7's Persistence Service has a new API, new data storage formats, and no longer synchronises persisted data between Transformer cluster nodes.

    To migrate your modules to the Persistence Service API, see Persistence Service below.

  • Transformer 7 includes a new Java Transformer Module (JTM) API.

    The original JTM, com.caplin.transformer.module, is now deprecated and has an end-of-life scheduled for one year from the release of Transformer 7.0. For details, see Java Transformer Module (JTM) API below.

To migrate data persisted under Transformer 6.2 to Transformer 7, follow the instructions below:

  1. Copy Transformer 6.2's memory file to Transformer 7. See Transformer memory file, below.
  2. Copy the Charting Service's cache directory to Transformer 7. See Caplin Module: Charting Service, below.
  3. Migrate data persisted by Transformer's Persistence Service 6.2:
    1. For data persisted by your own modules, see Persistence Service, below.
    2. For data persisted by StreamLink clients, see Persistence Service Client and Persisted-data migration tool, below.
    3. For data persisted by Caplin's Watchlist Service, see Caplin Module: Watchlist Service and Persisted-data migration tool, below.
    4. For data persisted by Caplin's Alerts Service, see Caplin Module: Alerts Service and Persisted-data migration tool, below.
    5. For data persisted by Caplin's High-Low Service, see Caplin Module: High-Low Service and Persisted-data migration tool, below.

Persistence Service

The Persistence Service in Transformer 7 has been rewritten to improve query performance, and is incompatible with the Persistence Service in Transformer 6.2.

Summary of important changes:

  • The Persistence Service in Transformer 7 has a new API:
    • C modules that use the Transformer 6.2 Persistence API will not work with Transformer 7. Update affected modules to the new Persistence API.
    • Lua pipeline scripts that use the Transformer 6.2 Persistence API will not work with Transformer 7. Update affected modules to the new Persistence API.
    • Java modules that use the com.caplin.transformer.module.persistence package of the now deprecated JTM API, com.caplin.transformer.module, will continue to be supported, with caveats, for one year from the release of Transformer 7.0. See Java Transformer Module (JTM) API below.
  • The Persistence Service in Transformer 7 has a new, more flexible database schema:
    • Multiple database tables are supported, and a table must be specified when persisting data using the new Persistence API. This allows Transformer modules to persist data to separate tables.

      Caplin-supplied modules create the database tables they require when file-backed persistence is used, but not when database-backed persistence is used.

      Note: in production environments, where database-backed persistence must be used, a database administrator must create the persistence tables required by Caplin-supplied modules. For details, see Activating the Persistence Service 7
    • The API supports database tables with composite (multi-column) primary keys and multiple value columns
    • Migrating a module's persisted data to a Transformer 7 installation requires an extra step to reformat the data for the new table schema.
  • The Persistence Service in Transformer 7 does not synchronise persisted data between Transformer cluster nodes.
    • File-backed (local) persistence cannot be used in Transformer 7 clusters. Use database-backed (shared) persistence instead.

      Note: File-backed persistence remains unsupported for use in production environments.

To update a custom module to use the new Persistence Service 7:

  1. Design and create one or more database tables to store the module's persisted data.

    The table design will vary depending on the needs of your module. The Caplin supplied modules Persistence Service Client, WatchList Service, and Alerts Service use tables that have a composite primary-key comprising a username and persistence-key, followed by one or more value columns. For example, the SQLite DDL for the table used by the Persistence Service Client is:

    CREATE TABLE IF NOT EXISTS TF_RECORD_PERSISTENCE (
        PERSISTENCE_USER VARCHAR(250) NOT NULL,
        PERSISTENCE_ID VARCHAR(100) NOT NULL,
        PERSISTENCE_DATA VARCHAR(4096) NOT NULL,
        PRIMARY KEY (PERSISTENCE_USER, PERSISTENCE_ID)
    );
  2. Update the module's code to use the new Persistence API.
  3. Write a script to read data persisted by the module under Transformer 6.2 and write it to the module's table or tables in your new Transformer 7 installation.

Java Transformer Module (JTM) API

Transformer 7 includes a new JTM API, com.caplin.jtm, that is closely modelled on the design of the DataSource for Java API.

The original JTM API, com.caplin.transformer.module, is now deprecated. Support for the original JTM API will end one year from the first release of Transformer 7, after which time the original JTM API may be removed from Transformer. During the support period, modules written using the original JTM API will work with Transformer 7, subject to the following caveats:

  • If you are using database-backed persistence, you must manually create the database table used to emulate the single table used by the Persistence Service 6.2: TF_LEGACY_PERSISTENCE.

    The file <Framework-root>/active_blades/PersistenceService/Transformer/etc/bootstrap.sql contains example DDL to create the table in SQLite 3.

  • Persisted data is not synchronised between nodes in a Transformer 7 cluster:

Transformer memory file

Transformer's object cache is persisted to disk on shutdown. The location of the memory file is defined by the configuration item memory-file.

To optionally warm-start Transformer 7 with the object cache from your Transformer 6.2 installation, copy the memory file from Transformer 6.2 to Transformer 7.

Caplin module: Persistence Service Client

The Persistence Service Client, included with Transformer, is used by StreamLink clients to persist data to Transformer. Examples of dependent applications include:

To migrate Persistence Service Client data persisted under Transformer 6.2 to Transformer 7:

  1. Activate the Persistence Service Client blade in Transformer 7.
  2. Create the database table and database trigger required by the Persistence Service Client 7 in the Persistence Service's database.

    The file <Framework-root>/active_blades/PersistenceServiceClient/Transformer/etc/bootstrap.sql contains example DDL to create the table and trigger in SQLite 3. Adapt this script to the SQL syntax of your database.

  3. Write a script to read the data persisted by the Watchlist Service 6.2 and write the data to the new database table for the Watchlist Service 7.0

    The serialisation format for persisted data in the Persistence Service Client 7 is the same as the serialisation format in the Persistence Service Client 6.2. No conversion is required.

    Tip: Caplin provide a migration tool that you can use as a starting point for your migration. See Persisted-data migration tool.

Caplin module: Watchlist Service

The Watchlist Service 6.2 is not compatible with Transformer 7. Deploy the Watchlist Service 7 instead.

To migrate Watchlist Service data persisted under Transformer 6.2 to Transformer 7:

  1. Deploy the Watchlist Service 7 blade to Transformer 7.
  2. Create the database table and database trigger required by the Watchlist Service 7 in the Persistence Service's database.

    The file <Framework-root>/active_blades/WatchlistService/Transformer/etc/bootstrap.sql contains example DDL to create the table and trigger in SQLite 3. Adapt this script to the SQL syntax of your database.

    Warning: The database trigger defined in the bootstrap.sql script is required. The Watchlist Service 7 will not operate correctly without it.

  3. Write a script to read the data persisted by the Watchlist Service 6.2 and write the data to the new database table for the Watchlist Service 7.0

    Although the Watchlist Service 7 uses a new database table design, the serialisation format for watchlist data has not changed in the Watchlist Service 7.0.

    Warning: The values in the WATCHLIST_POSITION table column must form a contiguous sequence of integers (1–n). If this requirement is not met, the Watchlist Service 7 will not operate correctly.

    Tip: Caplin provide a migration tool that you can use as a starting point for your migration. See Persisted-data migration tool, below.

Caplin module: Alerts Service

The Alerts Service 6.2 is not compatible with Transformer 7. Deploy the Alerts Service 7 instead.

To migrate Alerts Service data persisted under Transformer 6.2 to Transformer 7:

  1. Deploy the Alerts Service 7 blade to Transformer 7.
  2. Create the two database tables required by the Alerts Service 7 in the Persistence Service's database.

    The file <Framework-root>/active_blades/AlertsService/Transformer/etc/bootstrap.sql contains example DDL to create the table and trigger in SQLite 3. Adapt this script to the SQL syntax of your database.

  3. Write a script to read the data persisted under the Alerts Service 6.2 and write the data to the new database tables for the Alerts Service 7.0

    The serialisation format for triggers and notifications in the Alerts Service 7 is different from the serialisation format in the Alerts Service 6.2. You will need to convert serialised triggers and notifications to the new serialisation format before writing them to the new database tables for the Alerts Service 7.0

    Tip: Caplin provide a migration tool that you can use as a starting point for your migration. See Persisted-data migration tool, below.

Caplin module: Refiner

Refiner 6.2 uses the now deprecated JTM API, com.caplin.transformer.module. Deploy the Refiner 7 module instead.

The Refiner module does not persist data. No data migration is required.

The Java configuration item add-javaclass for Refiner 7 has been relocated to Refiner 7's overrides. If you have added new classes to Refiner 6.2's classpath, this configuration now belongs in Refiner 7's java.conf override file. 

Caplin module: High-Low Service 6.2

The High-Low Service uses the Persistence Service API in the now deprecated JTM API, com.caplin.transformer.module. For details of the duration of support for the deprecated JTM, see Java Transformer Module (JTM) API, above.

To migrate High-Low Service data persisted under Transformer 6.2 to Transformer 7:

  1. Deploy the High-Low Service 6.2 blade to Transformer 7.
  2. If you have not already done so, create the database table required by the deprecated Persistence API used by High-Low Service 6.2. For details, see Java Transformer Module (JTM) API, above.
  3. Write a script to read data persisted under the High-Low Service 6.2 and write the data to the TRANSFORMER_LEGACY_PERSISTENCE table.

    Tip: Caplin provide a migration tool that you can use as a starting point for your migration. See Persisted-data migration tool, below.

Caplin module: Charting Service

To migrate Charting Service data persisted under Transformer 6.2 to Transformer 7:

  1. Deploy the Charting Service 6.2 to Transformer 7
  2. Copy the contents of the charts directory under the Deployment Framework 6.2 to the Deployment Framework 7

    Note: The name of directory to which charting data is persisted is defined by the configuration item cache-directory.

Persisted-data migration tool

Caplin provide a peristed-data migration tool to migrate data persisted by Caplin-supplied modules from the single database table of the Persistence Service 6.2 to the multiple-table schema of the Persistence Service 7.

The data migration tool is not a supported product. The source code for the tool is supplied 'as is', and you are free to modify it and use it as you see fit. Caplin are not responsible for any loss or damage arising directly or indirectly from the use of this tool. As with any data-migration operation, backup your data first.

To obtain a copy of the source code, see Caplin's persistence-upgrade repository on GitHub.