DataSource peers configuration (part 2)

Here are the rest of the DataSource configuration items that define a DataSource application’s connections to its DataSource peers.

"this DataSource application" means the DataSource application for which you are defining the configuration. "DataSource peer" or "peer" means a DataSource application that this DataSource application communicates with.

More DataSource-peer configuration items can be found in DataSource peers configuration (part 1).

add-apikey

Introduced in platform 8, DataSource supports using an API key to authenticate and authorise adapters. This allows the Liberator or Transformer to control whether to accept a peer connection and which dynamic services to allow it access to.

The adapter uses the add-peer:apikey configuration option to specify an API key for a connection, and the Liberator or Transformer uses add-apikey to assign roles to that API key.

The first value provided to add-apikey is the SHA-256 hash of the API key. A value for this can be generated using the sha256sum command-line tool available on Linux machines. The second value is a comma-separated list of security roles to grant to the API key.

The following security roles are available:

Role Description

ds-connect

Allow a peer to connect

ds-dynamic-fields

Allow a peer to supply and receive field codes

ds-dynamic-peer

Allow a peer to connect dynamically

ds-dynamic-service

Allow a peer to supply data-service configuration

Since: DataSource 8

Use in: C

Syntax: add-apikey <SHA256_hash> "<role>[,…​]"

Type: string array

Default value: none

Example:

add-apikey f9912b266474a1805205454548fa6e4f80f5f688a379481d14976bdbea25e047 "ds-connect,ds-dynamic-fields,ds-dynamic-peer,ds-dynamic-service"

datasrc-batch-time

datasrc-batch-time specifies the time period in seconds for this DataSource application to batch together messages sent to a peer.

As an example, if datasrc-batch-time time is 0.2 seconds, when the DataSource application needs to send a message to the peer and there are no outgoing messages already in the queue, it queues the message and waits 0.2 seconds. Any subsequent outgoing messages are also queued until 0.2 seconds have elapsed, and then all the queued messages are sent to the peer in one batch. Batching messages in this way can improve performance when there is a high rate of data updates being sent to the peer.

If datasrc-batch-time is set to 0.0, messages are not batched.

Use in: C

Syntax: datasrc-batch-time <time-period>

Type: float

Default value: 0.01 (seconds)

Values accepted: 0.0 — 10.0

datasrc-default-obj-hash-size

datasrc-default-obj-hash-size specifies the default number of entries in the hashtable that holds information about this DataSource application’s active subscriptions.

Use in: C

Syntax: datasrc-default-obj-hash-size <number-of-objects>

Type: integer

Default value: 16384 objects

datasrc-heartbeat-slack-time

When this DataSource application doesn’t receive an expected DataSource heartbeat from a peer, it waits datasrc-heartbeat-slack-time seconds before disconnecting from the peer and trying to reconnect to it.

datasrc-heartbeat-slack-time applies to all dynamic peers, and static peers that do not have heartbeat-slack-time configured. If add-peer::heartbeat-slack-time is set, it takes precedence over datasrc-heartbeat-slack-time.

After modifying any of the heartbeat configuration items, restart both the DataSource application and its peers for the change to take effect.

Available from: DSDK 7.1.16 (Liberator 7.1.15, Transformer 7.1.10), DS.NET 7.1.16, DataSource for Java 7.1.13

Use in: C, Java

Syntax: datasrc-heartbeat-slack-time <time_in_seconds>

Type: float

Default value: 5.0

datasrc-heartbeat-time

The two peers involved in a DataSource connection exchange heartbeat messages at regular intervals so that each of the DataSource applications can check that the other is still present, and take appropriate action if it is not (such as attempt to reconnect to the peer, or fail over to another peer).

datasrc-heartbeat-time defines the time in seconds between each DataSource heartbeat. When two DataSource applications connect, they compare their configured heartbeat times and use the lowest.

datasrc-heartbeat-time applies to all dynamic peers, and static peers that do not have heartbeat-time configured. If add-peer::heartbeat-time is set, it takes precedence over datasrc-heartbeat-time.

After modifying any of the heartbeat configuration items, restart both the DataSource application and its peers for the changes to take effect.

Available from: DSDK 7.1.16 (Liberator 7.1.15, Transformer 7.1.10), DS.NET 7.1.16, DataSource for Java 7.1.13

Use in: C, Java

Syntax: datasrc-heartbeat-time <time_in_seconds>

Type: integer

Default value: 15

datasrc-id

The use of this configuration item is discouraged. Use datasrc-local-label instead.

datasrc-id specifies the ID number of this DataSource application. You can override this ID for a particular peer connection by adding a local-id option to the add-peer configuration item relating to the connection.

Use in: C, Java

Syntax: datasrc-id <ID-number>

Type: integer

Default value: 0

datasrc-interface

datasrc-interface binds the DataSource server socket to a single IP address or to a single IP address wildcard.

Only one DataSource server-socket is supported per DataSource application.

If no value is specified (default), then the DataSource server socket is available on all IPv4 and IPv6 addresses on the host. Some system settings affect this functionality; see the two tables below.

Do not set datasrc-interface directly for Liberator. Instead, use the configuration macros LIBERATOR${THIS_LEG}_DATASRC_INTERFACE and LIBERATOR${OTHER_LEG}_DATASRC_INTERFACE. For more information, see Deployment Framework’s configuration macros.

The behaviour of datasrc-interface differs slightly between C-based and Java-based DataSource applications:

DataSource for C
Value Example Value Result

<empty_string>

""

[Default] The DataSource server-socket is available on all available IPv4 and IPv6 addresses on the host. IPv6 addresses are supported from DataSource 7. See footnote 1.

IPv4 address

192.0.2.1

The DataSource server-socket is bound to the specified IPv4 address.

IPv4 wildcard

0.0.0.0

The DataSource server-socket is bound to all available IPv4 addresses on the host.

IPv6 address

2001:db8:0:0:0:0:0:1

The DataSource server-socket is bound to the specified IPv6 address. Supported from DataSource 7.

IPv6 wildcard

::

The DataSource server-socket is bound to all available IPv6 addresses on the local machine. Supported from DataSource 7.

Hostname

server.example.com

The DataSource server-socket is bound to the first IP address the hostname resolves to. IPv6 addresses are supported from DataSource 7. See footnote 2.

  1. On dual-stack hosts, setting datasrc_interface to "" causes DataSource for C to open an IPv6 server socket that accepts IPv4-mapped addresses. IPv4 address mapping is enabled by default on Linux, but it can be disabled by the system administrator. See your operating system’s documentation for /proc/sys/net/ipv6/bindv6only.

  2. From version 7, DataSource for C uses the POSIX function getaddrinfo for name resolution, which can return multiple IP addresses for a host. On Linux, getaddrinfo returns IP addresses in the order defined in RFC 3484, which gives IPv6 addresses precedence over IPv4 addresses of equal suitability. For more information, see your operating system’s manual entry for the POSIX function getaddrinfo.

DataSource for Java
Value Example Value Result

<empty_string>

""

[Default] The DataSource server-socket is available on all available IPv4 and IPv6 addresses on the host. IPv6 addresses are supported from DataSource 7. See footnote 1.

IPv4 address

192.0.2.1

The DataSource server-socket is bound to the specified IPv4 address.

IPv4 wildcard

0.0.0.0

The DataSource server-socket is bound to all available IPv4 and IPv6 addresses on the host. IPv6 addresses are supported from DataSource 7. See footnote 1.

IPv6 address

2001:db8:0:0:0:0:0:1

The DataSource server-socket is bound to the specified IPv6 address. Supported from DataSource 7.

IPv6 wildcard

::

The DataSource server-socket is bound to all available IPv4 and IPv6 addresses on the host. Supported from DataSource 7. See footnote 1.

Hostname

server.example.com

The DataSource server-socket is bound to the first IP address the hostname resolves to. IPv6 addresses are supported from DataSource 7. See footnote 2.

  1. On dual-stack hosts, by default Java creates IPv6 server sockets that accept IPv4-mapped addresses. To force Java to create IPv4 server sockets instead, set the Java system property java.net.preferIPv4Stack to true. See Oracle’s Networking IPv6 User Guide for more details.

  2. DataSource for Java uses Java’s java.net.InetAddress class for name resolution. In the interests of compatibility, Java gives IPv4 addresses precedence over IPv6 addresses of equal suitability. To force a Java virtual machine (JVM) to give precedence to IPv6 addresses, set the JVM option java.net.preferIPv6Addresses to true. See Oracle’s Networking IPv6 User Guide for more details.

Use in: C, Java

Syntax: datasrc-interface <address_or_hostname>

Type: string

Default value: "" (empty string)

datasrc-local-flags

Specifies flags determining the behaviour of this DataSource application when it starts up and establishes connections to peers, or when it reconnects to peers.

Overridden by add-peer:local-flags for specific peer connections.

Syntax: datasrc-local-flags flag[|flag …​]

Flags
  • none or 0: No special restart/reconnection behaviour.

  • recvautoreplay or 4: When restarting, this DataSource application must accept replay updates from the peer. This flag is deprecated.

  • noconnect or 8: When this DataSource application starts up, it must not automatically connect to the peer.

  • nofields: Do not send Dynamic Fields data in the DataSource handshake. Required for a DataSource 8 application to connect to Liberator/DSDK < 7.1.31 or Transformer < 7.1.19.

Example: datasrc-local-flags nofields

datasrc-local-label

datasrc-local-label sets a name that uniquely defines this DataSource application within the Caplin Platform installation. Together with the remote-label option of add-peer, it replaces datasrc-id and the remote-id option of add-peer.

You can override this label for a particular peer connection by adding a local-label option to the add-peer configuration item relating to the connection (this allows you to implement multiple connections for the same peer).

Here’s an example of how datasrc-local-label and remote-label are used together.

Use in: C, Java

Syntax: datasrc-local-label <label-name>

Type: string

Default value: [none]

datasrc-name

The name of this DataSource application, which the application’s DataSource peers use to identify it.

You can override this name by adding a local-name option to an add-peer entry - the local-name is then the name that the DataSource peer defined by the add-peer entry uses to identify this application.

The name can contain the parameters %a and %h. At run time, %a is replaced by the DataSource application-name, and %h is replaced by the host name of the machine on which the DataSource application is running.

Use in: C, Java

Syntax: datasrc-name <name-of-this-DataSource-application>

Type: string

Default value: %a-%h

datasrc-pkt-log

datasrc-pkt-log specifies the file path of this DataSource application’s packet log file. It can also be used to turn off packet logging.

The file path can be relative or absolute. If relative, the file path is relative to the value of log-dir.

To disable the log, set the file path to /dev/null.

Supported substitution tokens
Token Description

%a

The DataSource application-name.

%h

The host name of the machine on which this DataSource application is running.

Use in: C, Java

Syntax: datasrc-pkt-log <file_path>

Type: string

Default value: packet-%a.log

datasrc-port

datasrc-port specifies the network port to listen on for non-WebSocket connections from DataSource peers. The default of 0 means that no connections can be made to this DataSource application (in which case, you’d configure the application so that it initiates the connections to its peers; see the addr and port options of add-peer).

To listen for secure (SSL) connections from DataSource peers, you must specify the configuration item datasrc-ssl-port.

Use in: C, Java

Syntax: datasrc-port <port-number>

Type: integer

Default value: 0 (No connections can be made.)

datasrc-public-addresses

datasrc-public-addresses specifies a list of network addresses at which this DataSource component’s DataSource port (datasrc-port) is available to peers and Discovery. These addresses, together with the DataSource component’s hostname as reported by the OS or as set by datasrc-interface, are registered by the DataSource with Discovery during the Discovery-DataSource handshake. Discovery and peers attempting to connect to this DataSource try addresses in datasrc-public-addresses first before trying the DataSource component’s hostname.

Network addresses may be specified as hostnames, fully-qualified domain names, or IP addresses.

Use in: C (DSDK 7.1.27+, Liberator 7.1.27+, Transformer 7.1.17+), Java (DataSource for Java 7.1.20+)

Compatibility: Discovery 7.1.4

Syntax: datasrc-public-addresses address …​

Type: string array

Default value: none

Example: datasrc-public-addresses liberator1 liberator1.example.com

datasrc-ws-port

datasrc-ws-port specifies the network port to listen on for WebSocket connections from DataSource peers. The default of 0 means that no WebSocket connections can be made to this DataSource application (in which case, you’d configure the application so that it initiates the WebSocket connections to its peers; see the addr, port, websocket and websocket-path options of add-peer).

Unlike datasrc-port there is no separate port for secure connections. This port listens for secure (SSL) connections from DataSource peers if datasrc-ssl-enable is TRUE.

Use in: C, Java

Since: DSDK 7.1.30 (Liberator 7.1.30, Transformer 7.1.18), DataSource for Java 7.1.22, Discovery 7.1.5

Syntax: datasrc-ws-port <port-number>

Type: integer

Default value: 0 (No connections can be made.)

datasrc-ws-path

datasrc-ws-path specifies the path to listen on for WebSocket connections from DataSource peers. See the websocket-path option of add-peer when configuring applications initiating the connection.

Use in: C, Java

Since: DSDK 7.1.30 (Liberator 7.1.30, Transformer 7.1.18), DataSource for Java 7.1.22, Discovery 7.1.5

Syntax: datasrc-ws-path <path>

Type: string

Default value: /

datasrc-reject-new-peers

datasrc-reject-new-peers when set to TRUE, prevents a peer from connecting to this DataSource application when there’s already a peer connected that has the same label (or the same ID if there’s no label defined for the peer).

Setting datasrc-reject-new-peers prevents abnormal behaviour if the system is wrongly configured and two peers are inadvertently deployed with the same labels or IDs. By default, if a DataSource peer tries to connect to this DataSource application but there is already one connected with the same label or ID, the current peer is disconnected and the new one is allowed to connect. The first peer could then attempt to reconnect, which would cause the second peer to be disconnected. The second peer would then in turn attempt to reconnect, and so on.

This configuration item turns off this default behaviour, so the second DataSource peer is never allowed to connect.

The peer’s label is defined in the remote-label option of this DataSource application’s add-peer configuration entry. The peer’s ID is defined in the remote-id option of the add-peer.

Use in: C, Java

Syntax: datasrc-reject-new-peers <boolean>

Type: boolean

Default value: FALSE (a new peer with the same label or ID as an existing one is allowed to connect)

datasrc-tcp-nodelay-off

datasrc-tcp-nodelay-off specifies whether the DataSource application’s peer connection sockets should have the TCP_NODELAY feature turned off. The default is FALSE, which means TCP_NODELAY is enabled. Setting this configuration item to TRUE disables TCP_NODELAY.

Use in: C, Java

Syntax: datasrc-tcp-nodelay-off <boolean>

Type: boolean

Default value: FALSE (TCP_NODELAY is disabled)

peer-accept-wait-time

peer-accept-wait-time specifies the time in seconds to wait for an information packet when accepting a connection from a peer.

Use in: C

Syntax: peer-accept-wait-time <time-interval>

Type: float

Default value: 10.0 seconds

peer-debug-metrics-interval

peer-debug-metrics-interval specifies the interval in seconds between updates of the following Prometheus metrics:

Use in: Java

Syntax: peer-debug-metrics-interval <time-interval>

Type: int

Default value: 10 seconds

peer-monitor-interval

peer-monitor-interval specifies the interval in seconds at which statistics about this DataSource application’s connection to its DataSource peers are read and transferred to the application’s JMX monitoring module.

You can override this item for a specific peer by adding a monitor-interval option to this DataSource application’s add-peer configuration item for the peer.

Use in: C

Syntax: peer-monitor-interval <time-interval>

Type: float

Default value: 30.0 seconds

peer-request-timeout

peer-request-timeout is a higher-precedence alias of source-request-timeout.

Peer-request timeout precedence (highest first): add-peer::request-timeout > peer-request-timeout > source-request-timeout.

Use in: C

Syntax: peer-request-timeout <timeout-in-seconds>

Type: float

Default value: -1.0 (no timeout)

peer-thread-pool-size

peer-thread-pool-size specifies the maximum number of pooled connection threads allocated to handle the DataSource application’s peer connections that don’t have an explicitly named thread.

If you configure more than 30 peers, you should define peer-thread-pool-size. For more about peer-thread-pool-size and the thread-name option of add-peer, see Configure threads for peer communication.

If this item is configured, the DataSource application creates a pool of threads that are used to handle peer connections. The item specifies the maximum number of threads in the pool.

When a peer connection doesn’t have a named thread configured for it (see the thread-name option of the add-peer item), communication with the peer is handled on one of the pool threads. However, if peer-thread-pool-size is 0 or not defined, an unnamed thread is created and dedicated to that peer. If peer-thread-pool-size is less than the number of peer connections with no explicitly named thread, these connections share threads from the pool.

Use in: C

Syntax: peer-thread-pool-size <max-number-of-threads>

Type: integer

Default value: -1 (no pool configured - see explanation below)

source-request-timeout

source-request-timeout specifies a timeout in seconds for a DataSource peer’s response to a request (for example, a subscription request) sent to it by this DataSource application. If the peer doesn’t respond within this time, the requested object is assumed to be not available and the application sends a discard message for the object to the peer.

Peer-request timeout precedence (highest first): add-peer::request-timeout > peer-request-timeout > source-request-timeout.

A request to a peer in the context of a data-service is also subject to the data-service’s request timeout. For more information, see add-data-service::request-timeout and service-request-timeout.

Use in: C

Syntax: source-request-timeout <timeout-in-seconds>

Type: float

Default value: -1.0 (no timeout)


See also: