Logging configuration

These DataSource configuration items define what information a DataSource application can log and how the log files are automatically cycled.

For an overview of how logging works, including more about log levels and log file cycling, see Logging in DataSource applications.

See also, datasrc-pkt-log in DataSource peers configuration (part 2), and log-monitor-level in DataSource Monitoring Configuration.

"this DataSource application" means the DataSource application for which you are defining the configuration. "DataSource peer" or "peer" means a DataSource application that this DataSource application communicates with.

add-log

add-log provides configuration options that, for a particular type of log file, override one or more of the global logging configuration items. You specify the type of log file in the name option, and add other options to customise the file’s log level, log monitor level, and log file cycling characteristics.

Use in: C, Java

Syntax:

add-log
   name                [string]
   level               [string]
   maxsize             [integer]
   monitor-level       [string]
   period              [integer]
   suffix              [string]
   syslog-level        [string]
   time                [integer]
   use-parent-handlers [boolean]
end-log

The options for add-log are:

Name Type Default Description

level

string

[value of log-level]

This option only applies to the event log (when the name option is set to 'event_log').

Sets the logging level for the event log, overriding the log-level item. Only events of this severity or greater are written to the event log. For a list of valid severity levels, see log-level.

Also see the syslog-level option.

maxsize

integer

[value of log-maxsize]

The maximum size in bytes for this type of log file. The file size is checked every log-cycle-period (or every period option of this add-log, if defined), and the log file is cycled if it exceeds maxsize. A value of 0 means there’s no maximum size limit, but then the log file is always cycled every period / log-cycle-period regardless of its size.

This option overrides the log-maxize item.

monitor-level

string

[value of log-monitor-level]

This option only applies to the event log (when the name option is set to 'event_log')

It specifies the threshold at which log messages about this DataSource application’s events and errors are published to the monitoring subsystem. It overrides the log-monitor-level item.

For the list of valid values, see log-monitor-level.

This option is not available in Java-based DataSource applications.

name

string

[none]

The name (type) of log to which the rest of this add-log's options apply.

Note that the log cycle filename for the log is constructed from a combination of the run-time filename implied by name and the value of suffix/log-cycle-suffix.

If you’re writing your own C-based DataSource application, using the C DataSource API, you can define your own log files in addition to the standard ones. In that case, name must correspond to the name of a log object that you’ve created in the code. (In Java-based DataSource applications you can either log to the standard DataSource event log, or create your own logger when you create an instance of the DataSource class.)

Caplin-supplied DataSource applications, such as Liberator have a set of built-in log files with standard types that you can specify in name, as follows:

  • event_log: (All DataSource applications) The event log file. The run-time filename is defined by event-log.

  • http_access_log: (Liberator only) The HTTP access log file. The run-time filename is defined by Liberator’s http-access-log configuration item.

  • http_error_log: (Liberator only) The HTTP error log file. The run-time filename is defined by Liberator’s http-error-log configuration item.

  • javaauth_log: (Liberator only) The Java auth log file. The run-time filename is auth-rttpd.log (this is the log file produced by Java-based Liberator Permissioning Service blades).

  • news_log: (Liberator only) The log file that stores news headlines. The run-time filename is defined by Liberator’s news-log configuration item.

  • object_log: (Liberator only) The object log file that keeps a record of all request and discard commands for objects. The run-time filename is defined by Liberator’s object-log configuration item.

  • packet_log: (All DataSource applications) The log file that records each packet received from the application’s DataSource peers. The run-time filename is defined by the datasrc-pkt-log configuration item.

  • request_log: (Liberator only) The log file that records RTTP requests made to Liberator. The run-time filename is defined by the Liberator’s request-log configuration item.

  • rttp_log: (Liberator only) The RTTP message log file. The run-time filename is defined by Liberator’s rttp-log configuration item.

  • session_log: (Liberator only) The log file that records messages about client connections, disconnections, logins and logouts. The run-time filename is defined by Liberator’s session-log configuration item.

  • xmlauth_log: (Liberator only) The XMLauth log file. The run-time filename is defined by Liberator’s auth-logfile configuration item in the configuration for the XMLauth auth module.

period

integer

[value of log-cycle-period]

The interval in minutes for cycling this type of log file. Each day, at every interval after the first log cycle defined by log-cycle-time (or the time option of this add-log), the log is checked to see if it should be cycled. If the maxsize option of this add-log is defined, or log-maxsize is defined, the log file is cycled if it exceeds the maximum allowed size. If neither of maxsize or log-maxsize is defined, the log file is always cycled. regardless of its size.

This option overrides the log-cycle-period item.

For how to set up weekly log cycling for a particular type of log file, see the time option.

For log files that grow rapidly, a sensible configuration would be to set a suitable maximum size for the file (see the maxsize option), and set the period option to 5, so the file is checked every five minutes.

suffix

string

[value of log-cycle-suffix]

A suffix format that’s passed to the strftime() system function; the resulting string is appended to the name of the cycled log file. For information about the format specification, see the strftime documentation on the host OS.

This option overrides the log-cycle-suffix item.

syslog-level

string

[value of

log-syslog-level]

Since: DataSource 6.2.4

This option only applies to the event log (when name is set to 'event_log').

syslog-level specifies the severity of the errors and events that this DataSource application reports to the operating system’s log for this type of log message. It overrides the log-syslog-level item.

The log-level item, and level option (if specified) take precedence over syslog-level. For example, if syslog-level is set to DEBUG but log-level is set to INFO, only log messages with severity INFO or higher go to the syslog.

For the list of valid values, see log-syslog-level.

This option is not available in Java-based DataSource applications

time

integer

[value of log-cycle-time]

The time in minutes from midnight (local time) at which this DataSource application first checks this type of log file for cycling.

This option overrides the log-cycle-time item.

The subsequent times at which the application checks the log file for cycling is determined by log-cycle-period (or the period option of this add-log). If neither the maxsize option of this add-log nor log-maxsize is defined, the log file is always cycled, regardless of its size.

If time is greater than 1440 (24 hours), the cycle time is taken from the start of the week (midnight on Sunday night). For example, a value of 1460 sets cycle time to 00:20 hours on Monday. This allows you to cycle the log file weekly on a specific day if the period option is set to 10080 (7 days). For example, set time to 2880 (48 hours) and period to 10080, and the log file is checked for cycling at midnight every Monday.

To set weekly log cycling for all files, see log-cycle-time.

If you don’t specify this option and you don’t specify log-cycle-time, the default value of 240 minutes (4 hours) applies, so the log file is first checked for cycling at 04:00 hours (04:00 am) each day.

use-parent-handlers

boolean

FALSE

This option applies to Java-based DataSource applications only.

When set to TRUE, this log’s java.util.logging.Logger instance posts log messages to its own java.util.logging.Handler instances and to the Handler instances of all ancestor Logger instances in the Logger hierarchy.

This option overrides the log-use-parent-handlers item.

event-log

event-log specifies the file name of this DataSource application’s event Log. This log contains messages about starting up, shutting down, connections to DataSource peers, and other general and debug information.

Supported substitution tokens
Token Description

%a

The DataSource application-name.

%h

The host name of the machine on which this DataSource application is running.

By convention, log filenames have the format: <log type>-<application name>.log

Example for Liberator
event-log event-%a.log

In this example, because Liberator’s application name is "rttpd", the configuration item names the Liberator’s event log file as event-rttpd.log.

Log files are written to the location specified by the log-dir configuration item.

Use in: C, Java

Syntax: event-log <log-file-name>

Type: string

Default value: event-%a.log (in Transformer it’s just %a.log)

log-category-filter

Enable or disable categorised log content.

To enable a category, list it as an argument to log-category-filter. To disable a category, prefix the category with a negative operator (-).

Log category names are case-sensitive.

Since: DSDK 7.1.25

Use in: C

Syntax: log-category-filter [-]category …​

Type: array of strings

Available categories:

Category Default Description

SSL.Detail

Disabled

Log the cipher lists presented by TLS clients of Liberator. To collect the cipher list information required by this logging category, enable https-log-hello-ciphers and/or directssl-log-hello-ciphers.

Examples:

Enable the SSL.Detail category
log-category-filter SSL.Detail

log-cycle-period

log-cycle-period specifies the interval in minutes for cycling logs. Each day, at every log-cycle-period after the first log cycle defined by log-cycle-time (or by the time option of an add-log item), each log file is checked to see whether it should be cycled. If the maxsize option of the add-log for the file is defined, or log-maxsize is defined, the log file is cycled if it exceeds the maximum allowed size. If neither of maxsize or log-maxsize is defined, the log file is always cycled, regardless of its size.

You can override the log-cycle-period setting for a particular type of log file by specifying the period option of the file’s add-log configuration item.

For log files that grow rapidly, a sensible configuration would be to set a suitable maximum size for the file (see the maxsize option of add-log), and set the period option of add-log to 5, so the file is checked every five minutes.

For how to set up weekly log cycling, see log-cycle-time.

Use in: C, Java

Syntax: log-cycle-period <interval-in-minutes>

Type: integer

Default value: 1440 minutes (= 24 hours)

log-cycle-suffix

log-cycle-suffix defines a suffix format that’s passed to the strftime() system function; the resulting string is appended to cycled log files. You can override this setting for a particular type of log file by specifying the suffix option of the add-log configuration item.

For information on the strftime format specification, see the strftime documentation on the host platform. Avoid using strftime conversion specifiers that render characters unsuitable for filenames, such as directory separators (/ and \) and path separators (: and ;). Examples of conversion specifiers to avoid include %D, %X, and %+.

We recommend that you don’t use a suffix format that includes seconds, such as %T. This is because if the server is busy when the log file cycles, the create time of the log file may be out by a second, and this can prevent old (unwanted) log files from being overwritten. This can be a particular problem for packet log files, which can grow very large.

Use in: C, Java

Syntax: log-cycle-suffix <strftime-format-string>

Type: string

Default value: .%u (This appends the log filenames with a day number between 1and 7, representing Monday [1] to Sunday [7].)

log-cycle-time

log-cycle-time specifies the time in minutes from midnight (local time) at which log files are first checked for cycling. You can override this setting for a particular type of log file by specifying the time option of the add-log configuration item.

The subsequent times at which this DataSource application checks log files for cycling is determined by log-cycle-period (or the period option of add-log). If neither the maxsize option of the add-log configuration item for the log file nor log-maxsize is defined, the log file is always cycled, regardless of its size.

You can override the log-cycle-time setting for a particular type of log file by specifying the time option of the file’s add-log configuration item.

If log-cycle-time is greater than 1440 (24 hours), the cycle time is taken from the start of the week (midnight on Sunday night). For example, a value of 1460 sets cycle time to 00:20 hrs on Monday. This allows you to set weekly log cycling for all log files on a specific day if log-cycle-period is set to 10080 (7 days). For example, set log-cycle-time to 2880 (48 hours) and log-cycle-period to 10080, and all log files are checked for cycling at midnight every Monday.

To set weekly log cycling for an individual file, see the time option of add-log.

Use in: C, Java

Syntax: log-cycle-time <minutes-past-midnight>

Type: integer

Default value: 240 minutes (= 4 hours) So the log files are first checked for cycling at 04:00 hrs (04:00 am) every day.

log-dir

log-dir specifies the default directory (as an absolute or relative path) where this DataSource application’s log files are created.

Supported substitution tokens
Token Description

%r

The root directory (application-root) under which this DataSource application runs

%a

The DataSource application-name

Use in: C, Java

Syntax: log-dir <directory-path>

Type: string

Default value: %r/var

log-level

log-level specifies the global logging-level for logs written by this DataSource. Only events of log-level severity or greater are written to the DataSource’s logs.

The logging level for the event log can be set separately from the global logging-level. Specify a separate level for the event log by including the level option in the add-log block for the event log. The level option overrides log-level:

log-level INFO

add-log
  name event_log
  ⋮
  # Overrides log-level for the event log only
  level WARN
  ⋮
end-log

The logging level for events written to the operating system log can be raised to a higher threshold than the global logging-level. Specify a higher level for the operating system log by setting a value for the configuration item log-syslog-level or setting a value for the configuration option syslog-level of the add-log block for the event log. The syslog-level option overrides the log-syslog-level item. Setting log-syslog-level or syslog-level to a lower severity than log-level has no effect.

log-level INFO

# Raises global logging-level for the operating system log
log-syslog-level WARN

add-log
  name event_log
  ⋮
  # Overrides log-syslog-level for the event log only
  syslog-level INFO
  ⋮
end-log

The log level for the DataSource’s monitoring interface is set separately by log-monitor-level and is not affected by the value of log-level.

Events logged by a DataSource for Java application are logged with java.util.logging.Level severities.

Events logged by a Java module embedded in a DataSource for C application (for example, a Liberator or Transformer Java module), are logged with DataSource for C severities. The java.util.logging.Level severities map to DataSource for C severities according to the table below:

Java module level Severity logged by host DataSource for C application

Level.FINEST

FINEST

Level.FINER

FINER

Level.FINE

DEBUG

Level.CONFIG

CONFIG

Level.INFO

INFO

Level.WARNING

WARN

Level.SEVERE

CRIT

Notes on the CONFIG severity level:

  • CONFIG is not a valid value for log-level or level.

  • CONFIG has a different severity level in DataSource for Java than it does in DataSource for C. In DataSource for Java: CONFIG < INFO. In DataSource for C: CONFIG > CRIT. In a DataSource for Java application, setting log-level to INFO or higher excludes events of CONFIG severity. In a Java module within a DataSource for C application, setting the C application’s log-level to INFO or higher does not exclude Level.CONFIG events logged by the Java module.

Use in: C, Java

Syntax: log-level <log-level-name>

Type: string

Default value: INFO

Values accepted:

Values accepted by DataSource for C
Value Description

FINEST

Reports highly detailed tracing messages.

FINER

Reports fairly detailed tracing messages.

DEBUG

Reports tracing messages.

INFO

Reports events and information regarding normal operation, and all errors included in the WARN, NOTIFY, ERROR and CRIT log levels.

WARN

Reports minor errors and all errors included in the NOTIFY, ERROR and CRIT log levels.

NOTIFY

Reports errors regarding data corruptions and all errors included in the ERROR and CRIT log levels.

ERROR

Reports serious errors regarding network connections and all errors included in the CRIT log level.

CRIT

Reports critical errors that prevent the DataSource application from running.
Values accepted by DataSource for Java
Value Synonyms Description

FINEST

Reports highly detailed tracing messages.

FINER

Reports fairly detailed tracing messages.

FINE

DEBUG

Reports tracing messages.

INFO

Reports events and information regarding normal operation, and all errors included in the WARNING and SEVERE log levels.

WARNING

WARN, NOTIFY

Reports minor errors and all errors included in the SEVERE log level.

SEVERE

ERROR, CRIT

Reports serious errors that prevent the DataSource application from running.

log-maxfiles

log-maxfiles sets the total number of files (the active file + cycled files) that a log may comprise. For example, if log-maxfiles is set to 7, then each log can comprise a maximum of 7 files (the active log file + 6 cycled files).

Specify log-maxfiles together with log-maxsize to enable you to calculate the maximum disk space each log can consume.

To specify no limit to the number of files a log may comprise, set log-maxfiles to -1 (the default value).

Use in: Java (since Java DataSource API 7.1.2)

Type: Integer

Syntax: log-maxfiles <max-num-files>

Default: -1

log-max-history

log-max-history specifies the maximum number of log lines that are retained for viewing via JMX monitoring tools such as the Caplin Management Console. At any one time, up to log-max-history of the most recently generated log lines can be viewed.

Use in: C

Syntax: log-max-history <max-number-of retained-log-lines>

Type: integer

Default value: 10

log-maxsize

log-maxsize specifies the maximum log file size in bytes. The size of each log file is checked every log-cycle-period (or every period of the add-log for the log file type, if defined), and a log file is cycled if it exceeds log-maxsize. A value of 0 means there’s no maximum size limit, but each log file is always cycled every period / log-cycle-period regardless of its size.

You can override this setting for a particular type of log file by specifying the maxsize option of the add-log configuration item.

Use in: C, Java

Syntax: log-maxsize <max-size-in-bytes>

Type: integer

Default value: 0 bytes (meaning that log files are cycled every time they are checked)

log-syslog-level

log-syslog-level specifies the threshold severity for the events a DataSource for C application writes to the operating system’s log. On Linux, the operating system log is Syslog. On Windows, the operating system log is the Windows Application Log.

The threshold severities specified by log-level and level are applied before the level specified by log-syslog-level; thus, log-syslog-level can only raise, not lower, the level set by log-level and level.

You can override log-syslog-level by specifying the syslog-level option of the add-log configuration item.

DataSource for Java applications do not log to the operating system log, but Java modules embedded in DataSource for C applications do (indirectly). Java modules log to the operating system log via their host application’s logger. For a table illustrating how a Java module’s java.util.logging severity levels are mapped to its host’s DataSource for C severity levels, see the documentation for log-level, above.

The log level for the DataSource’s monitoring interface is set separately by log-monitor-level and is not affected by the value of log-syslog-level.

On Linux, the DataSource for C logger translates DataSource severities to Syslog event severities (see RFC 5424) as illustrated in the table below:

DataSource to Syslog mapping
DataSource level Syslog level

FINEST

Debug

FINER

Debug

DEBUG

Debug

INFO

Informational

WARN

Warning

NOTIFY

Warning

ERROR

Error

CRIT

Error

On Windows, the DataSource for C logger translates DataSource severities to Microsoft Windows event severities as illustrated in the table below:

DataSource to Windows mapping
DataSource level Windows level

FINEST

Information

FINER

Information

DEBUG

Information

INFO

Information

WARN

Warning

NOTIFY

Warning

ERROR

Error

CRIT

Error

Also see syslog-facility.

Use in: C

Since: DataSource for C (DSDK) 6.2.4

Syntax: log-syslog-level <log-level-name>

Type: string

Default value: ERROR

Values accepted:

Log level Description

FINEST

Reports highly detailed tracing messages.

FINER

Reports fairly detailed tracing messages.

DEBUG

Reports tracing messages.

INFO

Reports events and information regarding normal operation and all errors included in the WARN, NOTIFY, ERROR and CRIT debug levels.

WARN

Reports minor errors and all errors included in the NOTIFY, ERROR and CRIT debug levels.

NOTIFY

Reports errors regarding data corruptions and all errors included in the ERROR and CRIT debug levels.

ERROR

Reports serious errors regarding network connections and all errors included in the CRIT debug level.

CRIT

Reports critical errors that prevent the DataSource application from running.

log-use-parent-handlers

log-use-parent-handlers specifies that a Java DataSource application’s java.util.logging.Logger instance should post log messages to the java.util.logging.Handler objects of ancestors in the Logger hierarchy in addition to its own Handler instances.

Set this configuration item to TRUE in the following use cases:

  • You are running a DataSource application in your IDE and you want to output the application’s log messages to your IDE’s integrated console (by default the root Logger has a ConsoleHandler that writes to the standard error stream).

  • You want your DataSource application to use a logging library other than java.util.logging and the library integrates with java.util.logging by registering a java.util.logging.Handler with the root Logger.

You can override this setting for a particular type of log file by specifying the use-parent-handlers option of the add-log configuration item.

For more information, see the documentation for the method Logger.setUseParentHandlers.

Use in: Java

Syntax: log-use-parent-handlers TRUE

Type: boolean

Default value: FALSE

syslog-facility

syslog-facility specifies the Linux syslog facility to use when this this DataSource application’s error and event messages are recorded in Linux’s system log (syslog). The message severities to be logged are determined by log-syslog-level and the syslog-level option of the add-log configuration item.

Refer to your operating system manual for instructions on configuring the destination of messages logged to a syslog facility.

Since: DataSource 6.2.4

Use in: C

Syntax: syslog-facility <facility-name>

Type: string

Default value: local6

Default log-cycling behaviour

If you don’t specify any log-cycling configuration, then by default all log files are cycled at 04:00 hours each day, so that a separate log file for each type of log is created for each day. You can of course override this default setting for a particular type of log file by specifying the relevant options in an add-log …​ end-log configuration entry.

The default settings are equivalent to the following configuration:

# Cycle logs every time they are checked:
log-maxsize       0

# Cycle logs for the first time at 4 hrs after midnight:
log-cycle-time    240

# Cycle logs every 24 hrs after log-cycle-time:
log-cycle-period  1440

# Cycled log filename has suffix 1 - 7 (representing Mon - Sun):
log-cycle-suffix  .%u
This default configuration can create very large log files if your system has lots of fast moving data. It’s useful to have as much log data as possible in an individual file, but you should change the logging configuration appropriately if the files grow too large.

See also: