Record latency in record updates

How does latency recording work?

A record typically passes through a number of DataSource applications running on the Caplin Platform; for example, a record containing an update to the price of an FX instrument could originate from a Pricing Adapter that’s connected to an external pricing system, the record may pass through a Transformer (perhaps to be filtered), and then through a Liberator before ending up in a client application such as a Caplin Trader application.

The update takes a finite time to pass from the Pricing Adapter to Transformer and on to Liberator. Each of these components adds a processing delay, and the update also takes time to travel through the network connecting one component to another. This is illustrated in the following diagram:

Latency chain example: FX Pricing Adapter to Transformer to Liberator to StreamLink in client

Here an FX Pricing Adapter (Integration Adapter) receives an FX price record from a pricing system at time T1. The record exits the FX Pricing Adapter at time T2 and enters a Transformer at time T3. The Transformer sends the record on to a Liberator at time T4, which the Liberator receives at time T5.

You can configure each of the DataSource applications in the chain (Pricing Adapter, Transformer, Liberator) to add timestamps to the update record, so the record contains a history of how long it spent at each significant point - this is called the latency data chain. The timestamps have a resolution of 1 millisecond (1 nanosecond from Caplin Platform 7.1).

To accurately record latency, the system clocks of all servers in the chain must be synchronised with a common time source.

Latency recording is only meant to be used as a tool for tuning your system, so although you can embed latency data in your "live" data, this can add unwanted overhead. Therefore, we recommend that you restrict the addition of latency data to a few dummy subjects used specifically for latency measurement. For example, assume you want to measure the latency of FX data originating from an FX Pricing Adapter. Rather than coding the adapter to add latency data to all updates for subjects starting with /FX/, instead code the adapter to add latency data for updates to a dummy subject, /FX/LAT, that you use only for latency measurement.

By default, your adapter should record the initial timestamp in a field called LTY_INIT_TS, but you can choose any field name, providing that you configure all the DataSource applications that handle the update to recognise this name (see the configuration item latency-chain-init-ts-field).

Here’s an example of an update record containing (only) latency chain data:

Example update containing only latency chain data (Caplin Platform 6.2)
Field name Field value Description



Initial timestamp (milliseconds since 1 January 1970).



List of events



List of timestamp deltas

The FX Pricing Adapter records the initial timestamp in the field LTY_INIT_TS.

The format of the initial timestamp has changed from Caplin Platform 7.1:

  • Caplin Platform < 7.1: milliseconds from 1 January 1970

  • Caplin Platform >= 7.1: seconds from 1 January 1970, with a fractional component that runs to nine digits (nanosecond resolution). For example: 1125062541.880462579

Then the Adapter adds an event called fxadapter1_E to the events list field (LTY_LIST_EVENT), and to the timestamp deltas list field (LTY_LIST_TS) it adds the time of the event in milliseconds (nanoseconds in Caplin Platform 7.1) since the initial timestamp. The event name consists of the DataSource application name (fxadapter1) with an E appended. E means "Entry" - the time when the record "entered" the DataSource application (in this case its the time when the record was created, which is the same as the initial timestamp).

When the Adapter sends the update record on to the Transformer, it adds another event called fxadapter1_X to the events list field, and adds the time of this event in milliseconds (nanoseconds in Caplin Platform 7.1) since the initial timestamp to the timestamp deltas list field. The X suffix of course means "Exit", as it’s the delta time when the record left the Adapter.

Transformer and Liberator add their own delta times as the update record passes through them, so the list of events and corresponding delta times is built up like this:

Latency chain example for Caplin Platform 6.2
Data event Time since initial timestamp (ms*) Meaning



Time elapsed between initial timestamp and entering fxadapter1



Time elapsed between initial timestamp and exiting fxadapter1



Time elapsed between initial timestamp and entering transformer1



Time elapsed between initial timestamp and exiting transformer1



Time elapsed between initial timestamp and entering liberator1 (the default name for a Liberator is 'rttpd')

* Nanoseconds from Caplin Platform 7.1

What record types can hold latency data?

You can put latency data into records of type 1 data format, where the fields are defined by a numeric id and named through a fields.conf file (see Field definition format).

In deployments using Liberator 6.2.10 or later, you can also put latency data in generic data records.

How do I record the initial timestamp?

The initial timestamp in a latency chain must be set manually; subsequent timestamps in the latency chain are set automatically by all DataSource components that have latency-chain-enable set.

Recording the initial timestamp using a DataSource API

To start off the chain of latency data you must record an initial timestamp in the relevant data records within the first DataSource application in the chain. This would usually be done in the Integration Adapter from where the data is sent into the Caplin Platform.

You create the timestamp by calling a DataSource API function or method on the record, which records the time in the record field whose name is defined by the configuration item latency-chain-init-ts-field, as explained in How does latency recording work? above.

The method you call depends on the language in which you’ve implemented your DataSource application, as follows.

Liberator 6.2.10 or later is required to record timestamps in generic records.
DataSource API Version API call to record initial timestamp

C DataSource API

Timestamping type 1 format records: 6.x and later

Timestamping generic data records: 6.2.8 and later (but see warning above)

Use the ds_add_data() function to set the initial timestamp in the field LTY_INIT_TS (by field number).

See Creating Data Objects in the C DataSource API documentation.

Java DataSource API

6.2.7 and later (but see warning above)


.NET DataSource API

6.2.8 and later (but see warning above)



Recording the initial timestamp in a Transformer pipeline

If you want to start recording the latency chain from Transformer (for example, because the data whose latency you want to track has been created in Transformer), you can set the initial timestamp from within a Transformer pipeline.

To do this in Lua, follow the steps below:

  1. Create the record:

    myrecord = DSRecord createrecord (/MYSUBJECT)
  2. Set the timestamp field LTY_INIT_TS to a millisecond value (pre Transformer 7.1) or a nanosecond timestamp in the format 'S.sssssssss', where 'S' is the number of seconds from 1 January 1970 and 'ssssssssss' is a fractional component that runs to 9 digits (Transformer 7.1 and higher):

    myrecord:setfield("LTY_INIT_TS", timestampmsec)

This example assumes that the name of the initial timestamp field is the default one. If you’ve defined the name as something different (see latency-chain-init-ts-field in How do I change the names of latency chain fields? below), you’ll have to change the Lua code accordingly. Alternatively, in the setfield() call, you can specify the field name by number, rather than by name.

Adding additional timestamps to a latency chain

You can add additional instrumentation points to a latency chain. For example, you may want to record how long it takes the DataSource application to perform a specific calculation.

Before you write additional instrumentation points to a record, check the following is true:

  • Latency chaining is enabled (check the value of the latency-chain-enable configuration item).

  • The initial latency-chain timestamp has been set (indicating that this is a record for which latency is being measured).

DataSource API Available in versions API call to add a custom timestamp

C DataSource API

Timestamping type 1 format records: 6.x and later

Timestamping generic data records: 6.2.8 and later (but see warning above)

void ds_add_latency_chain

void ds_add_latency_chain2

Java DataSource API

6.2.7 and later (but see warning above)


.NET DataSource API

6.2.8 and later (but see warning above)



How do I turn latency recording on and off?

By default, latency recording is turned off. To turn it on, define the configuration item latency-chain-enable in each DataSource application in the required chain (you’ll have to stop and restart the applications for the change to take effect):

latency-chain-enable  TRUE

You’ll also need to code the first DataSource application in the latency chain (typically an Integration Adapter) to add the initial timestamp field to the records whose latency you want to measure. The other DataSource applications in the chain only add latency times to records that contain an initial timestamp field whose name is defined by the latency-chain-init-ts-field configuration item.

Since latency recording is really only meant to be used as a tool for tuning your system, a flexible way to implement it is to implement the Integration Adapter so that you can dynamically control whether or not it creates records with an initial timestamp. You can then permanently turn on latency recording for all your DataSource applications by setting latency-chain-enable. At run-time, latency records are only sent through the Platform components if you’ve turned on their generation at the Integration Adapter. When latency record generation is turned off, the rest of the components in the chain will still check every record for the presence of the timestamp field, but the overhead of doing this is trivial.

When you deploy a Liberator in the Caplin Deployment Framework, the Liberator fields configuration item requested-fields-only is set to TRUE by default. The effect of this is that, even when latency recording is turned on, Liberator doesn’t send the latency chain data back to clients unless they explicitly request the latency timestamp fields. If you want to always pass the latency data through to clients, you must set requested-fields-only to FALSE; however, this means that all other record updates sent to clients may contain fields that the client is not interested in, increasing the Liberator to client traffic unnecessarily.

Regardless of whether the latency data is sent to clients, you can always view it using the Caplin Management Console - see How can I…​ Monitor latency statistics using the CMC.

How do I change the names of latency chain fields?

To change the names of the latency chain fields from their defaults (LTY_INIT_TS, and so on), define the following configuration items in each of the DataSource applications in the chain. Each application must have the same definitions.

Configuration item Default value Description



Initial timestamp



List of events



List of timestamp deltas

Timestamping in Liberator

You can add timestamp fields within Liberator in addition to, or instead of, the standard latency chain timestamps described above. This Liberator timestamp is a lightweight way for a client application to calculate the time taken for a record to be transmitted to it from the Liberator.

This facility is independent of the latency chain timestamps, and the recording of the Liberator timestamp isn’t controlled by the setting of the latency-chain-enable configuration item described in How do I turn latency recording on and off? above.

You set up the timestamp field(s) by specifying the timestamp-field configuration item and/or the timestamp-field option of add-object (an add-object timestamp option overrides the global timestamp-field item for the object in question). The Liberator only adds the timestamp-field, and the timestamp within it, to objects that don’t already contain a field of that name.

For information on the precision and format of the timestamp, see the documentation for the global timestamp-field configuration item.

How does Liberator throttling and bursting affect latency measurement?

Latency measurements are affected by some Liberator configuration settings. The two main features that can delay messages are throttling and bursting.

Object throttling by default is set to 1 second, which means that an update could be delayed by up to 1 second as a result. Bursting on client session output is set by default to 0.5 seconds, so an update could get delayed by a further 0.5 seconds on top of the throttling delay. Both these features have their benefits; throttling prevents sending out multiple updates to the same object in a short space of time, while bursting can improve overall performance in a system with a large number of clients by batching together small messages when output to a client. You can turn throttling off if it has a detrimental effect on latency, but you should always have a burst setting, even if it’s a small value such as 0.1 seconds.

See also: