Transformer Pipeline Module API Reference
Transformer Pipeline Module API Reference Documentation

About the Pipeline Module

The pipeline module is a Transformer module which embeds a powerful, yet intuitive, scripting language. This permits rapid development and deployment of business rules.


Embedded scripting language

The pipeline module embeds version 5.1 of Lua, and the API defined in this document is written in Lua. Lua is a powerful light-weight language that is dynamically typed and has automatic memory management with garbage collection. For more information about Lua please see

Library support

The pipeline module is supplied with a library of commonly required business rules arranged in packages. These functions implement common business rules such as:

  • Calculation of High price
  • Calculation of Low price
  • Hold and halt status
  • Inactivity monitoring
  • Update counter

The use of packages permits partitioning of the symbol name space and reduces symbol name clashes.

Native function support

The pipeline module can integrate with existing transformer modules, calling their member functions directly, thus ensuring that the previous development can be reutilised.

Compilation free deployment

Pipeline scripts can be deployed without the need for a separate compilation step. Scripts are compiled into byte code as they are required.

The monitoring interface to the pipeline module allows updates to take place dynamically, a script can be uploaded whilst the Transformer is running and the current one will be replaced. Should the new script fail due to a syntax error, the change will be reverted until the next Transformer restart.

Opaque data storage

The pipeline module provides a mechanism to store and retrieve data between invocations. This permits data to be held between updates. Access to this storage is provided through the database package.

Data Dictionary Support

The pipeline module supports the concept of using a de-reference data dictionary, see Pipeline Concepts: Data Dictionary for more details.

Simple pipeline example

The example below calculates the Low and High prices (and times that they occured at) for all symbols under the /DEMO directory. It uses the calc Library package which provides the calc.high() and calc.low() functions.

For clarity in this example we bypass the Data Dictionary functionality, calling methods on the dsdata objects directly.

Append this to the end of etc/pipeline.conf:

    id              simple
    listener-regex  ^/DEMO/
    pipeline-file   simple.lua
    update-func     update

Place this in etc/pipeline/simple.lua:.

function update(input)
   local output = createrecord(input:getsubject())

   local price = input:getfield("TradePrice")
   local price_time = input:getfield("TradeTime")

   if price ~= nil then
      calc.high(input, output, price, price_time)
      calc.low(input, output, price, price_time)

      if output:numfields() > 0 then

Generated on Sat Aug 26 2017 12:36:33 for Transformer Pipeline Module API Reference