Refiner subject syntax

This page describes the syntax of subjects that are processed by Refiner.

If you are system administrator, this page will help you decipher Refiner subjects in log files.

If you are a software developer and you’re using StreamLink, see Access Refiner through StreamLink.

If you are a software developer and you’re using Caplin Trader’s Grid module, see Configure sorting and filtering.

Refiner limitations

Avoid filtering or sorting on record fields that are subject to frequent updates. Frequent updates can have an adverse affect on the performance of Caplin Refiner, Liberator, and the requesting clients.

Refiner subject format

To use Refiner to filter, sort, and group data in a container, prepend the container’s subject with /FILTER and append the container’s subject with ? followed by a string of Refiner parameters.

Syntax diagram:

railroad refiner subject format

At least one of the following parameters is required: filter, sort, groupby

The mapsuffix parameter is normally added dynamically, when it is required, by a permissioning adapter. See The 'mapsuffix' parameter.

Multiple instances of the same parameter are not allowed, with the exception of the sort parameter, which from Refiner 7 may be specified multiple times to construct a multi-column sort. See The 'sort' parameter.

An example Refiner subject is shown below. It requests the /CTR/FI/ALL container, sorted and grouped by the Description field.

/FILTER/CTR/FI/ALL?sort=Description:text:ascending;groupby=Description

The 'filter' parameter

The filter parameter comprises one or more field-matching criteria.

A single field-matching criterion has the following syntax:

railroad refiner filter parameter
Operators
Meaning Text fields Numeric fields Notes

=

equal to

Defaults to text comparison. For a numeric comparison, prefix the field name with number: or use the == operator.

!=

not equal to

Defaults to text comparison. For a numeric comparison, prefix the field name with number: or use the !== operator.

==

equal to (numeric)

!==

not equal to (numeric)

<

less than

Refiner 7.1.0+

StreamLink’s ContainerFilterFactory supports use of this operator with text fields from StreamLink 7.1.8. Defaults to a numeric comparison. For a text comparison, prefix the field name with text:

>

greater than

Refiner 7.1.0+

StreamLink’s ContainerFilterFactory supports use of this operator with text fields from StreamLink 7.1.8. Defaults to a numeric comparison. For a text comparison, prefix the field name with text:

<=

less than or equal to

Refiner 7.1.0+

StreamLink’s ContainerFilterFactory supports use of this operator with text fields from StreamLink 7.1.8. Defaults to a numeric comparison. For a text comparison, prefix the field name with text:

>=

more than or equal to

Refiner 7.1.0+

StreamLink’s ContainerFilterFactory supports use of this operator with text fields from StreamLink 7.1.8. Defaults to a numeric comparison. For a text comparison, prefix the field name with text:

~

regular expression match

Usually used for text fields

#

regular expression match
(case insensitive)

Usually used for text fields

You can join multiple field-matching criteria with logical AND ('&') and logical OR ('|'). There’s no implicit precedence between the logical operators, so you need to establish precedence using parenthesis ('(' and ')').

Example: field 'CpnRate' is greater than 6
CpnRate>6
Example: field 'CpnRate' is between 6 and 10 (non-inclusive)
CpnRate>6&CpnRate<10
Example: field 'MaturityDate' equals 20130328 OR field 'CpnRate' is between 6 and 10 (non-inclusive)
MaturityDate=20130328|(CpnRate>6&CpnRate<10)

If the field name or comparison value in a filter criterion includes a Refiner comparison operator (see table above), a Refiner logical operator ('&' or '|'), or the Refiner parameter separator (';'), then these characters must be encoded using a Refiner escape sequence. See Special characters.

Special characters

The following characters have a special meaning in Refiner filter expressions. To include these characters as literals in filter expressions (in a field name or in a value), use the following escape sequences:

Character Escape sequence

@

@0

(

@1

)

@2

&

@3

|

@4

!

@5

=

@6

>

@7

<

@8

~

@9

#

@a

:

@b

,

@c

?

@d

;

@e

For example, to filter a ccy field using the regular expression USD(CHF|CAD), use the filter parameter below (escape sequences highlighted):

filter=ccy#USD@1CHF@4CAD@2
StreamLink contains classes that make building the Refiner parameter string easier. See Access Refiner through StreamLink.

The 'sort' parameter

The sort parameter comprises a field name, a field data type, and a sort order:

railroad refiner sort parameter

For example, to sort the 'MaturityDate' field in ascending numerical order, use the sort parameter below:

MaturityDate:number:ascending

From Refiner 7, you can specify more than one sort parameter in a Refiner parameter string. The sort parameters are applied in the order they are supplied, and create a multi-column sort:

Specifying a multi-column sort in a Refiner subject (Refiner 7 and later)
/FILTER/CTR/FI/ALL?sort=MaturityDate:number:ascending;sort=CpnRate:number:descending

If two or more records have the same value in the sort field(s), the records are sorted into their natural order (the order in which the records' integration adapter would return the records unsorted to the client).

The 'groupby' parameter

The groupby parameter comprises a single value: the field name to group records by.

railroad refiner groupby parameter

Grouping adds extra records to the container that act as headers for blocks of records with the same field value. For example, if you sort the /CTR/FI/All container by the Description field and group the records by the Description field, the returned container looks like this:

Container: /FILTER/CTR/FI/ALL?sort=Description:text:ascending;groupby=Description
/FILTER/META/CTR/FI/ALL/GROUPHEADER/groupby=Description/AUSTRIA
SubHeaderText=AUSTRIA
/FI/EU/A Description=AUSTRIA CpnRate=4.35 MaturityDate=20140715 BidPrice=103.948 ...
/FI/EU/B Description=AUSTRIA CpnRate=6.25 MaturityDate=20270715 BidPrice=122.890 ...

/FILTER/META/CTR/FI/ALL/GROUPHEADER/groupby=Description/GERMANY
SubHeaderText=GERMANY
Description=GERMANY /FI/EU/C CpnRate=4.92 MaturityDate=20130328 BidPrice=101.859 ...

/FILTER/META/CTR/FI/ALL/GROUPHEADER/groupby=Description/US TREASURY
SubHeaderText=US TREASURY
/FI/US/A Description=US TREASURY CpnRate=11.25 MaturityDate=20141115 BidPrice=...
/FI/US/B Description=US TREASURY CpnRate=11.75 MaturityDate=20150215 BidPrice=...
/FI/US/C Description=US TREASURY CpnRate=10.62 MaturityDate=20150815 BidPrice=...

Each of the header records (/FILTER/META/CTR/FI/ALL/GROUPHEADER/ and so on) contains one field, SubHeaderText, which contains the value of the field that you are grouping by. When the client application receives the sorted and grouped container, it typically displays the header rows in the grid with different styling to the data rows.

Within Streamlink, the exact way in which record grouping is added to a container subscription depends on which StreamLink library you are using. For details, consult the API Reference document for your particular StreamLink implementation.

The 'mapsuffix' parameter

The 'mapsuffix' parameter is added to a Refiner subject dynamically by a permissioning integration adapter.

The parameter comprises a single value:

railroad refiner mapsuffix parameter

The suffix is appended to the subjects of the items in the container and is used to implement price tiers. For more information on 'mapsuffix', see Set Refiner user permissions and subject mappings.

The container placeholder

The sequence of subscribing to a container, subscribing to all the records in it, and then processing each of the records in Caplin Refiner can take longer than the active request timeout configured in Liberator (see source-request-timeout, add-peer: request-timeout and add-data-service: request-timeout). If Refiner were to wait until it had processed all the results before sending a response, the active request timeout might trigger first and Liberator would then discard the container. To prevent this happening, as soon as Caplin Refiner receives a filter or sort request, it sends back a container response with a single special subject name in it: /FILTER/META/PLACEHOLDER. This is called the container placeholder record. It tells the Liberator that Refiner has received the request, preventing the container subscription from being discarded.

Handling container placeholders in the client

Typically a client application displays a loading indicator while waiting for a response to a sort or filter request, and it removes the loading indicator when a response is received. If the first response received is a container of size 1 that contains a placeholder record, the client should ignore the response and leave the loading indicator in place. The next container update received will either be a container image of the results, or an explicit removal of the placeholder record that indicates there are no matching results. The client should respond by removing the loading indicator.

Caplin Refiner explicitly removes the placeholder record when the time taken to filter the container exceeds the timeout defined in the Refiner configuration property placeholder.timeout. In this case, the client can still receive results from the filter request until such time as it discards the filter.


See also: