Data services configuration

These DataSource configuration items define a DataSource application’s data services.

When the DataSource application requests an object programmatically (such as subscribing to a subject), the data services system determines, through the configuration, which peers the request goes to instead of you having to specify this explicitly in the application’s code. It also prevents a request from going to any peer more than once.

Of course, if your DataSource application is an Integration Adapter that only sends data to its peers, you don’t need to define any data services for it.

Data services can’t be configured in Java-based DataSource applications.
"this DataSource application" means the DataSource application for which you are defining the configuration. "DataSource peer" or "peer" means a DataSource application that this DataSource application communicates with.

For some examples of how data services are configured, see the data services features and concepts page.

add-data-service

add-data-service specifies a data service that this DataSource application can use.Through the add-source-group option and the add-priority option within add-source-group, it allows you to specify multiple peers that provide data for the service, load balance the provision of data across multiple peers, and configure alternate peers that your DataSource application can failover to. For examples of these setups, see the data services features and concepts page.

Also see the note below about default behaviour.

Use in: C

Syntax:

add-data-service
   add-source-group
      add-priority
         remote-label [value]
      end-priority
      ...
      add-priority
         remote-label [value]
      end-priority
   end-source-group
   ...
   add-source-group
      ...
   end-source-group

   affinity            [array of strings]
   disable-auto-status [boolean]
   discard-timeout     [float]
   exclude-pattern     [array of strings]
   include-pattern     [array of strings]
   nodata-ignore-request-timeout [boolean]
   request-timeout     [float]
   required-state      [integer/string]
   service-name        [string]
   service-type        [integer/string]
end-data-service
Option Type Default Description

add-source-group

group item

See Default behaviour.

The add-source-group group item specifies, via the add-priority option, one or more DataSource peers that provide the data for the data service.

  • By using multiple add-source-group items, you can specify multiple sources of data for the data service.

  • By using multiple add-priority options within an add-source-group item, you can specify alternative sets of peers to failover to.

For examples of these setups, see the data services features and concepts page

affinity

array of strings

[none]

Deprecated from version 6.2.6+ of Liberator and version 6.2.5+ of Transformer; use the affinity option within the add-source-group group item instead.

disable-auto-status

boolean

FALSE

If you don’t want this DataSource application to send subject status changes for this data service to subscribing peers, set disable-auto-status to TRUE You’d typically do this in a Transformer, to prevent it from sending subject status changes unnecessarily on to subscribing Liberators.

discard-timeout

float

For DataSource applications other than Liberator:

-1.0 (no timeout)

For Liberator only:

Value of

active-discard-timeout

Specifies the time in seconds for which this DataSource application holds on to the data for a subscribed subject once the last client application has unsubscribed from that data. After this time, the object is deleted from the DataSource application’s cache, and the DataSource application sends a discard instruction to the data service’s peer(s) to cancel the subscription.

For Liberator only:

  • This option overrides the setting of the global time out for active objects, active-discard-timeout.

  • If an object obtained by this data service was specified through the Liberator’s add-object configuration item, the discard-timeout option of the add-object (if any) overrides this discard-timeout.

exclude-pattern

array of strings

[none]

One or more regular expressions that determine subjects for which this data service must not provide data.

This option is optional. Use it in conjunction with the include-pattern option.

For example, include-pattern ^/FX/MAJOR/ specifies that this data service is to provide data for subjects that begin with /FX/MAJOR/. Then specifying exclude-pattern ^/FX/MAJOR/AUDUSD ^/FX/MAJOR/AUDGBP excludes the currency pairs AUDUSD and AUDGBP from the prices returned by the data service.

You can have multiple occurences of exclude-pattern in the same add-source-group item, so the above example could be written as:

exclude-pattern ^/FX/MAJOR/AUDUSD
exclude-pattern  ^/FX/MAJOR/AUDGBP

include-pattern

array of strings

^/

One or more regular expressions that determine the subjects for which this data service provides data.

For example: include-pattern ^/FX/MAJOR/ ^/FX/MINOR/ specifies that this data service is to provide data for subjects that begin with /FX/MAJOR/ or /FX/MINOR/

You can have multiple occurences of include-pattern in the same add-source-group item, so the above example could be written as:

include-pattern ^/FX/MAJOR/
include-pattern  ^/FX/MINOR/

You should normally specify an include-pattern option, otherwise the data service will return updates for all subjects (that is, subjects starting with /) that the service’s DataSource peers can provide.

Also see the exclude-pattern option.

nodata-ignore-request-timeout

boolean

FALSE

A DataSource peer that provides a data service can send back a "no data" response if there is no data available for a requested subject. The data service then discards any outstanding requests for the subject.

The nodata-ignore-request-timeout option applies when multiple peers can supply requested data for a service.

When this option is TRUE, if any peer providing the service sends back a "no data" response, this response is immediately propagated from the DataSource API to the custom code of the DataSource application - the request-timeouts (or source-request-timeouts) for the remaining peers are ignored.

When this option is FALSE, if a "no data" response is received from a peer providing the service, the data service continues wait for the data from any of the other peers, applying the request-timeouts (or source-request-timeouts), until either the data is returned by at least one peer, or all the peers have timed out or returned a "no data" response.

request-timeout

float

Value of

service-request-timeout

Specifies the timeout in seconds for all subscription requests sent to this data service. If the DataSource peers providing the service don’t respond within this time, the requested subject is assumed to not be available and the application sends a discard message for the subject to the peers.

A value of 0 means no timeout is set and this DataSource application then waits indefinitely for this data service to respond to a request. Use this setting in Liberator if you want to allow all Liberator’s subscription requests for this data service to be open subscriptions (see How can I…​ Keep subscriptions open when the peer is down).

request-timeout is the master timeout; it overrides the global service-request-timeout configuration item, the retry-time option of add-source-group, and the timeouts on individual peer requests (request-timeout option of add-peer, and the global source-request-timeout item).

required-state

integer/

string

0 (=down)

This option only applies to Liberator.

Specifies the state that the data service must be in when Liberator starts up, in order for Liberator to accept client connections.

Possible values are:

  • 0 or down or optional: Liberator accepts client connections regardless of the state of the data service.

  • 1 or limited: Liberator accepts client connections only if the data service has status 1 (limited) or 2 (ok or up).

  • 2 or ok or up: Liberator accepts client connections only if the data service has status 2 (ok or up).

The default value of down means that, if no required-state options are specified for any data services, Liberator accepts client connections at start up, regardless of the state of the services.

Liberator checks service status against the required-state option only at start up. For example, if required-state is ok and the data service status is "ok" at start up, client connections are accepted. Then if the service status subsequently changes to "down", the Liberator will continue to accept client connections.

service-name

string

[none]

The name of the data service.

service-type

integer/

string

3 (= active|broadcast)

Specifies whether the DataSource peers that supply data for the service are of type active, broadcast, or both.

Possible values are:

  • active or 1: This data service is provided by active DataSource peers. An active DataSource keeps track of which subjects have been requested and sends updates for those subjects only (see Active subscription model).

  • broadcast or 2: This data service is provided by broadcast DataSource peers.

  • active|broadcast or 3: This data service is provided by both active and broadcast peers.

You can use this option to override the local-type option of a peer’s add-peer configuration. For example, assume a peer that supplies data for a data service is configured as an active peer and also is configured to broadcast data (that is, its add-peer's local-type option is set to broadcast|active). If you don’t want your data service to receive the peer’s broadcast data, set the add-data-service's service-type option to active.

add-source-group

add-source-group specifies, via the add-priority option one or more DataSource peers that provide the data for the data service defined by add-data-service. By using multiple add-source-group items, you can specify multiple sources of data for the data service. By using multiple add-priority options within an add-source-group item, you can specify alternative sets of peers to failover to. If you specify more than one peer within an add-priority item, subscriptions are load-balanced across them. For examples of these setups, see the data services features and concepts page.

Use in: C

Syntax:

add-data-service
   ...
   add-source-group
      add-priority
         remote-label [value]
      end-priority
      ...
      add-priority
         remote-label [value]
      end-priority

      affinity [array of strings]
      required [boolean]
      retry-time [float]
   end-source-group
end-data-service
Option Type Default Description

add-priority

group item

See Default behaviour.

Specifies one or more DataSource peers that this DataSource application can use to provide data for a data service. If you specify more than one peer in an add-priority option, subscriptions are load-balanced across those peers.

You can also use multiple instances of add-priority to implement failover to alternative peers. (But, because the remote-label option of add-priority points to an add-peer configuration item, this DataSource application will first try to fail over to any alternate peers that you’ve defined in the addr option of the add-peer item.)

For examples of load balancing and failover configurations, see the data services features and concepts page.

affinity

array of strings

[none]

Enables source affinity for load-balanced adapters in the source group. Source affinity distributes channel subscriptions evenly across a load-balanced set of adapters while maintaining the integrity of user sessions.

Some integration adapters are designed to serve data on one channel as a result of activity on another. For example, trading adapters are designed to write entries to a user’s blotter channel as a result of trading activity on the user’s trade channel. A load-balancer can inadvertently break the operation of such adapters by distributing the user’s channels to different adapter instances.

Source affinity stops a user’s channels from being distributed to different adapters in a load-balanced set by establishing a strong affinity between the user and one adapter in the set.

To ensure that two source groups, under different data services, each select an identical adapter for a user:

  • both source groups should be assigned the same affinity key (see below).

  • both source groups should have be defined with an identical arrangement of adapters: the same adapters, in the same priority groups and in the same order.

Usage: affinity [affinity_key] [regular_expression]

affinity_key: an arbitrary string identifying a set of user-to-adapter affinities. Source groups with identical affinity keys will select the same adapter instance for any of a user’s channels served by them. Source groups with identical affinity keys must have the same number and order of adapters.

regular_expression: a regular expression used to extract the username from the mapped subject of a channel served by a source group. Enables the source affinity algorithm to identify the owner (source) of a channel.

Example: affinity trading-adapters /PRIVATE/([^-]+).*/

For more information, see the following links:

required

boolean

FALSE

When set to TRUE, if one of the peers that provides the service (see add-priority option) is down, updates from this service are marked as stale and the status of the service is changed to stale.

retry-time

float

30.0

After finding that none of the peers defined by add-priority options in the source group have responded to a request, this DataSource application waits retry-time seconds before reissuing the request to the group.

The minimum value allowed is 1.0 seconds.

The DataSource application issues the request to each of the peers in the source group in turn. Each request is timed out according to the setting of request-timeout in the add-peer configuration item. If none of the peers in the source group replies, the DataSource application waits retry-time and then tries each connection again in turn. It repeats this sequence until the master timeout for the service, defined by service-request-timeout, or the request-timeout option of add-data-service, expires. If the request-timeout out of the add-data-service is set to -1, the DataSource reissues the request indefinitely.

add-priority

add-priority specifies one or more DataSource peers that this DataSource application can use to provide data for a data service. By using multiple add-priority options within an add-source-group item, you can specify alternative sets of peers to failover to. If you specify more than one peer within an add-priority item, subscriptions are load-balanced across them; for an example of this, see the data services features and concepts page.

Use in: C

Syntax:

add-data-service
   ...
   add-source-group
      ...
      add-priority
         remote-label [array of strings]
         remote-label [array-of-strings]
         ...
         label        [array-of-strings]
         label        [array-of-strings]
         ...
      end-priority
   end-source-group
end-data-service
Option Type Default Description

label

array of strings

See

Default behaviour.

Deprecated. Use remote-label instead.

One or more labels identifying DataSource peers that supply data for this DataService. Each label must match the label option of an add-peer configuration item that defines a DataSource peer. If you specify more than one peer, either through multiple values of a label option or by multiple label options, subscriptions to the referenced peers are load-balanced across the peers.

Also see Note 1.

remote-label

array of strings

See

Default behaviour.

One or more labels identifying DataSource peers that supply data for this DataService. Each label must match the remote-label option of an add-peer configuration item that defines a DataSource peer. If you specify more than one peer, either through multiple values of a remote-label option or by multiple remote-label options, subscriptions to the referenced peers are load-balanced across the peers.

Also see Note 1 below, and the example in DataSource peers configuration (part 1).

Note 1: Platform blades, and Liberator and Transformer, that run under the Caplin Platform Deployment Framework use the deprecated label option in their configuration, rather than remote-label. This will change in a future release of the Framework.

broadcast-cleanup-age

broadcast-cleanup-age specifies how old in minutes broadcast objects provided by this DataSource application need to be before they’re removed from the DataSource application’s cache. Every broadcast-cleanup-period minutes past broadcast-cleanup-time, the application checks the cache and deletes any broadcast objects that are older than broadcast-cleanup-age.

This configuration item is ignored if broadcast-cleanup-time isn’t set.

Use in: C

Syntax: broadcast-cleanup-age <age-in-minutes>

Type: integer

Default value: -1 (no timeout set, so broadcast items are never removed from the cache)

broadcast-cleanup-period

broadcast-cleanup-period specifies how often in minutes, relative to broadcast-cleanup-time, broadcast objects provided by this DataSource application are removed from the DataSource application’s cache. Every broadcast-cleanup-period minutes past broadcast-cleanup-time, the application checks the cache and deletes any broadcast objects that are older than broadcast-cleanup-age.

This configuration item is ignored if broadcast-cleanup-time isn’t set.

Use in: C

Syntax: broadcast-cleanup-period <time-interval-in-minutes>

Type: integer

Default value: 1440 minutes (= 24 hours; that is, clean up every day at the time set by broadcast-cleanup-time.)

broadcast-cleanup-time

broadcast-cleanup-time specifies when broadcast objects provided by this DataSource application are first removed from the DataSource application’s cache. This time is the number of minutes after midnight (local time) on the day the Liberator was started, or if it’s greater than or equal to 1440 (24 hours), it’s the number of minutes past midnight on a Sunday. At broadcast-cleanup-time, and subsequently every broadcast-cleanup-period minutes past broadcast-cleanup-time, the application checks the cache and deletes any broadcast objects that are older than broadcast-cleanup-age.

By default, the DataSource application’s cache is not cleaned up on a timed basis.

Use in: C

Syntax: broadcast-cleanup-time <time-past-midnight-in-minutes>

Default value: -1 (Don’t clean up the DataSource application’s cache on a timed basis, so broadcast-cleanup-age and broadcast-cleanup-period are ignored.)

We recommend that you configure a daily cleanup of broadcast objects. For example, set broadcast-cleanup-time to 0 and broadcast-cleanup-period to 1440 mins (= 24 hrs), so that broadcast objects are cleaned up at midnight every day. Use broadcast-cleanup-age to control how old the broadcast objects should be before they’re actually removed from the cache.

cleanup-stale-timeout

cleanup-stale-timeout specifies when to clean up stale objects (such as subscriptions). When all the DataSource peers in a data service have been down for cleanup-stale-timeout seconds, all the objects that can no longer be updated by the service are deleted from this DataSource application’s cache.

Use in: C

Syntax: cleanup-stale-timeout <timeout-in-seconds>

Type: float

Default value: -1.0 (no timeout)

service-request-timeout

service-request-timeout specifies the global request timeout in seconds for all data services.

This is the time that this DataSource application waits for a data service to respond to a request (for example, a subscription request). If the DataSource peers providing the service don’t respond within this time, the requested object is assumed to not be available and the application sends a discard message for the object to the peers.

A value of 0 or -1 means no timeout is set and this DataSource application then waits indefinitely for a data service to respond to a request. Use this setting in Liberator if you want to allow all of the Liberator’s subscription requests to be open subscriptions (see How can I…​ Keep subscriptions open when the peer is down).

service-request-timeout is a global setting applied to all data services. You can override this timeout for a particular data service by setting the request-timeout option of add-data-service.

Use in: C

Syntax: service-request-timeout <timeout-in-seconds>

Type: float

Default value: 10.0 seconds

Data service default behaviour

If you don’t define a data service for your DataSource application, it’ll act as if you defined the following configuration and will therefore send each subscription request to all of its active DataSource peers at once.

Default data service configuration

add-data-service
   service-name         default
   required-state       down
   include-pattern      ^/
   add-source-group
      required         FALSE
      add-priority
         remote-label  source1
      end-priority
   end-source-group
   add-source-group
      required         FALSE
      add-priority
         remote-label  source2
      end-priority
   end-source-group
   ...
   add-source-group
      required         FALSE
      add-priority
         remote-label  sourceN
      end-priority
   end-source-group
end-data-service

See also: