In Transformer, data transformation services are implemented in Transformer Modules. Caplin provides a Transformer module called Refiner that allows large lists of data (containers) to be efficiently filtered and sorted in real time on behalf of client applications. Refiner is supplied as a Service blade that’s deployed in the Caplin Platform Deployment Framework. This tutorial shows you how to deploy the Refiner blade and use it to sort and filter container responses.
Once you’ve completed the steps on this page, you should be familiar with how to deploy Refiner and how to request filtered and sorted data via Transformer.
In these instructions you’ll be using the
Refiner is a Transformer Java module and has the following requirements:
Refiner must be deployed to a Transformer that is licensed to run Java Transformer modules. Check your Transformer licence contains the string below:
module jtm transformer
For questions regarding licensing, contact Caplin Support.
Transformer’s Java Virtual Machine (JVM) must be configured with a maximum heap size that allows Refiner to run your data transformation workloads comfortably without the JVM running out of memory. See Tuning Transformer’s JVM heap for Refiner.
To install Refiner, follow the instructions below:
1. Install a Deployment Framework, Liberator and Transformer
The Framework has a built-in demonstration Adapter blade called DemoDataSource that you’ll use to show that Refiner is working. Make sure the DemoDataSource blade is active, by running the following dfw command from the
./dfw activate DemoDataSource
You can verify that the blade has been activated, by running the command:
2. Deploy the Refiner Service blade
The Refiner Service blade is supplied in a zip file with a name like
Testing the installation
Follow the instructions below to check the Refiner Service is available:
Start the Liberator, Transformer and DemoDataSource adapter, by running
./dfw startfrom the
The Liberator should recognise the following data services and DataSources. In particular, the status page should show a data service for the Refiner Service, as highlighted in red here:
Stop the Liberator, Transformer and DemoDataSource adapter, by running
./dfw stopfrom the
<Framework-root>directory. This is so you can examine various log files to check Transformer’s working correctly.
Examine the log file
If the Transformer is working correctly, this log contains the version number of the Transformer Module library:
<YYYY/MM/DD-HH:MM:SS.NNN> +0000: INFO: Java Transformer module (6.2.0-297741) starting up <YYYY/MM/DD-HH:MM:SS.NNN> +0000: INFO: Loading user class with identifier jtm <YYYY/MM/DD-HH:MM:SS.NNN> +0000: CRIT: Thread [main/1]: Product: Java Transformer Module Version : 6.2.3-bbbbbb Build Date : dd-Mon-yyyy Build Time : hh:mm Build Number : bbbbbb Copyright : Copyright 1995-2014 Caplin Systems Ltd
In the same log file, you should also see Caplin Refiner registering as a provider for the namespace it uses:
2011/12/16-15:41:05.703 +0000: INFO: Registering as a provider of </FILTER/*>
Examine the log file
If Caplin Refiner is working correctly, this log contains its version number:
<YYYY/MM/DD-HH:MM:SS.NNN> +0000 - Thread [main/1]: SEVERE: Refiner 6.2.3-bbbbbb
If everything looks ok, restart the Liberator, Transformer and DemoDataSource adapter, by running:
Log on to Liberator from a web browser and navigate to the Liberator Explorer page:
From the menu on the Liberator’s home page, select Diagnostics
On the Diagnostics page that’s displayed, select the Liberator Explorer link.
|For more about using Liberator Explorer, see How can I… Use Liberator Explorer to request and send data.|
Requesting an unfiltered container
The DemoDataSource adapter can supply a container called
/EXAMPLES/PRICING/CONTAINERS/EQUITIES that’s populated with stock prices.
Use Liberator Explorer to request
/EXAMPLES/PRICING/CONTAINERS/EQUITIES. This returns the un-filtered and un-sorted container:
Requesting a filtered container
In Liberator Explorer, request
Liberator routes this request to Transformer, because its RefinerService1 data service contains the
include-pattern ^/FILTER/ (have a look in the Liberator configuration file for the Refiner Service blade at
/FILTER prefix directs Transformer to pass the request on to its Refiner module, which then returns just those instruments whose
BestBid field is greater than
Requesting a sorted container
In Liberator Explorer, request
Now Refiner sorts the instruments in ascending alphabetical order of the
|Avoid filtering or sorting on record fields that are subject to frequent updates. Frequent updates can have an adverse affect on the performance of Caplin Refiner, Liberator, and the requesting clients.|
Tuning Transformer’s JVM heap for Refiner
The default maximum heap size of Transformer’s JVM is 256MB. Before deploying Refiner to production, tune the maximum heap size of Transformer’s JVM to a size that can comfortably accommodate your data transformation workloads without running out of memory.
Follow the steps below:
In the file
<framework_root>/global_config/overrides/servers/Transformer/etc/java.conf, adjust the start and maximum JVM heap size to 2048MB or higher:
# The size of the Java heap (in megabytes) define JVM_XMS_HEAP_SIZE 2048 define JVM_XMX_HEAP_SIZE 2048
Oracle recommend that the start and maximum size of the JVM are set to the same value to minimise garbage collection.
Start (or restart) Transformer:
./dfw start Transformer
From the same user account used to start Transformer, run a JVM profiler and connect to Transformer’s embedded JVM (which has the same PID as Transformer). The JVM profiler records heap memory allocation over time, which you refer to in step 6.
Test Transformer with Refiner against your expected production workloads.
If Transformer raises out-of-memory errors during your tests, then increase the maximum heap size and begin again from step 2.
Referring to the JVM profiler’s graph of free heap memory, increase/decrease the JVM heap size to a size that can run your expected Refiner workloads while retaining a comfortable margin of free memory to cope with unexpected spikes in workload. After adjusting the JVM heap size, begin again from step 2.
About Liberator Explorer
How can I… Use Liberator Explorer to request and send data
How can I… Access Refiner through StreamLink
How can I… Filter and Sort in Refiner
How can I… Configure Refiner
How can I… Configure sorting and filtering (in Caplin Trader 4 grids).
Reference: Refiner configuration properties