Performance Testing with Benchsrc and Benchrttp

Benchsrc and Benchrttp are performance testing tools for the Caplin Platform. Benchsrc is used to generate Record and Container updates. Benchrttp is used to generate client sessions, exerting load on the Platform. Using either or both of these it is possible to performance test the Caplin Platform under varying message and client load.

Overview

In this tutorial, you will learn how to:

  • deploy the Benchsrc blade;
  • configure Benchsrc to provide record updates;
  • send UDP commands to change the update rate from Benchsrc;
  • set up and configure a Benchrttp client;
  • send UDP commands to Benchrttp to add more clients;
  • send UDP commands to record the current latency status.

Benchsrc

In previous tutorials, you learnt how to set up the Deployment Framework and how to deploy Liberator and Transformer. These are necessary prerequisites. Deploying the Benchsrc blade is a similar process.

Steps

  1. Copy the Benchsrc blade to the kits directory of your Deployment Framework. Run ./dfw deploy to deploy it.
  2. To configure Benchsrc to run on the localhost run command ./dfw hosts Benchsrc localhost
  3. Start the Deployment Framework (./dfw start) and try requesting the subject /BENCHSRC/A from Liberator Explorer. (Ensure your Permissioning Adapter is running if you have deployed the Permission Sevice).
  4. You should see a record updating in real-time.

Configure a data set in Benchsrc

You can configure the records provided by the Benchsrc, tailoring the data provided to the Platform.

 

Understand

Open global_config/overrides/Benchsrc/DataSource/etc/benchsrc.conf. Here you will find the following configuration:

add-object-set
        id                        ACTIVESET
        updates-rate-per-object   1
        prefix                    /

        add-field
            fieldnum            1
            type microseconds
        end-field

        add-field
            fieldnum            2
            type seqnum
        end-field

        add-field
                fieldnum        22
                type            price
                min-value       428.10
                max-value       428.30
                format          "%.2f"
        end-field

        add-field
                fieldnum        25
                min-value       428.40
                max-value       428.60
                type            price
                format          "%.2f"
        end-field
end-object-set

Analysis:

  • id: This option is used to set the ID for the object set. This will be used when sending UDP messages to change the update rate.
  • prefix: This option is used to set the prefix of the object set. E.g. If you set the prefix to be /BENCHSRC/TUTORIAL/ that means any request subject starting with /BENCHSRC/TUTORIAL/ will apply this object set.
  • updates-rate-per-object: This option is used to set the update rate for the object set. E.g. When set to 4, this means each record from this object set will update 4 times per second. (Note: This doesn’t mean that for each second it will update 4 times. Rather, it means that after running for a long time, the average number of  updates per second is 4.)
  • add-field / end-field: These options are used to configure the fields of the object. The options to configure the fields are:
    • field-num: This option is used to set the field number with which the field name is encoded (where the field name is found in fields.conf).
    • type: This option describes the kind of updates which this field will send. The possible types are:
      • milliseconds: When the record updates the field will contain the current timestamp in milliseconds
      • microseconds: When the record updates the field will contain the current timestamp in microseconds
      • seqnum: Record updates are numbered in sequence i.e. seqnum is 1 on the first update, 2 on the second update … n on the nth update.
      • price: This field's value will be a random decimal number in the format defined by the format option
      • list: When the record updates, it will update the next item in the list defined by the values option. E.g. If we define values as A B C then on the first update the field's value will be A, on the second B, on the third C, on the fourth A again, and so on.
      • fixed: The field will have the same value on each record update.
    • format: This option is used to define format of the decimal number for a field. E.g. %.2f, %d
    • values: This option is used to define the possible values of a field.
    • image-only: When this option is set, the field will only appear in the first update.
 

Try this Yourself

Define an object set that meets the following requirements:

  • It applies to any record starting with /BENCHSRC/TUTORIAL
  • The record updates once every 2 seconds
  • The record contains following fields:
    • A millisecond timestamp
    • A list with options: Cat, Dog, Pig, Cow
    • A random price between 0 and 20

Hint: Add the fields to global_config/fields.conf

You can try it out by requesting /BENCHSRC/TUTORIAL/0 in Liberator Explorer.

Send UDP commands to change update rate

In the previous section you've just seen how the update rate is set in the config file. However, on every configuration change Benchsrc must be restarted for the changes to take effect. This is inefficient when running multiple test cases. Benchsrc also supports a UDP interface so that commands (such as changing the update rate) can be sent at runtime. However, note that UDP messages are not guaranteed to be received by the Benchsrc, especially under heavy load. Therefore you should always check the logs to make sure that the UDP command is received.

The UDP command tool can be found here: active_blades/Benchsrc/DataSource/bin/udpsend.exe.

 

Understand

This is the command to change the update rate at runtime:

udpsend.exe -s <ipaddress> -p <udp-port> updates-rate-per-object <new-update-rate> <id>

So if you want to change the update rate to one update every 10 seconds for the object set defined in the previous section:

  • ipaddress: Since the benchsrc is running on your local machine, the ip address will be 127.0.0.1
  • udp-port: You can find the udp-port defined in global_config/overrides/Benchsrc/DataSource/etc/benchsrc.conf as ${THIS_LEG}5010. You will also find in global_config/environment.conf that THIS_LEG is 1. Thus, the udp-port is 15010
  • new-update-rate: This should be the update rate that we want to change to. It should be 0.1
  • id: This should be the id if the object set e.g. ACTIVESET.

 

 

Try this Yourself

  1. While Liberator, Transformer and Benchsrc are running use the udpsend tool to change the update rate to one update every 10 seconds. You should notice that the updates for the /BENCHSRC/TUTORIAL/0 object suddenly slow down in Liberator Explorer. You should also notice the following two lines in the benchsrc log file:
    INFO: Processing UDP Command 'updates-rate-per-object' with arguments '0.1 ACTIVESET'
    INFO: UDP:updates-rate-per-object ACTIVESET 0.1
  2. Next try to change the update rate to 10 times per second.
  3. You should notice that after you change the update rate, the record begins updating once every second, not 10 times per second as expected. Can you tell why?
    ...
    It is because the default throttling for Liberator is once per second. To turn it off, add this line to global_config/overrides/Benchsrc/Liberator/etc/rttpd.conf
    object-throttle-off
  4. Restart the environment, refresh Liberator Explorer and request /BENCHSRC/TUTORIAL/0. Change the update rate to 10 updates per second and you should notice the very fast update rate in the Liberator Explorer.

Benchrttp

The Benchrttp client generator is not shipped as a Platform blade and cannot be deployed in the Deployment Framework.

Steps

  1. Copy the Benchrttp package to convenient a location on your hard drive. Open up a new cygwin console window and unzip the Benchrttp package using cygwin unzip command.
  2. Open the Benchrttp configuration located in etc/benchrttp.conf.
  3. Ensure that it is configured to connect to the correct Liberator host and port:
    server           localhost
    port             18080
 

Understand

Notice the following default configuration in this file:

clients          1

You don’t need to change this at the moment, but this configuration option defines how many users you want to create.

 

connect-time   0.1

This option defines the interval between user logins. When set to 0.1, there will be a 0.1 second interval between a user's login and the next.

  1. Next you need change the object-set that you would like to request. Replace:
    add-object-set
            id              BCASTSET1
            num             20
            max             100
            format          /A/Xd
    end-object-set
    with
    add-object-set
            id             TEST1
            num            10
            max            1000
            format         /BENCHSRC/TUTORIAL/Xd
    end-object-set
    For each client 10 objects will be chosen randomly from 1000 with names /BENCHSRC/TUTORIAL/X 00000, /BENCHSRC/TUTORIAL/X X00001, etc.
  2. Then remove this line (Java is not needed for Benchrttp):

    include-file      java.conf
  3. The last thing is to add this line:

    microsecond-latency

    This means the latency is calculated at the microsecond level and also the timestamp sent from the benchsrc is microsecond level.

  4. Restart the Deployment Framework and start Benchrttp by running etc/benchrttp start (You can stop Benchrttp by running etc/benchrttp stop)

  5. Open the logs: var/event-benchrttp.log. Here you should see the indication that all clients have logged in and the total amount of clients is 1:

    INFO: All Clients 1

Send UDP commands to add more clients

The client option only allows you to set the number of users before you start the Benchrttp. When you want to add more users during the test, you should use UDP commands to add more users.

 

Understand

The syntax of the UDP message to add more clients should be:

./udpsend.exe -s <ipaddress> -p <benchrttp-udp-port> clients <number-of-clients>

For example, if we want to set the number of clients to 10 clients.

  • ipaddress: Should be 127.0.0.1 as Benchrttp is running on the localhost
  • benchrttp-udp-port: Should be the udp-port set in the benchrttp.conf which is 13001
  • number-of-clients: Should be 10.
 

Try this Yourself

Use udpsend (in the bin directory) to change the number of clients to 10

Check the logs and wait until you see:

INFO: All Clients 10

 

Send UDP commands to record the current latency

Benchrttp and Benchsrc are predominately used for performance testing where an important metric to measure is the message latency. Benchrttp calculates latency automatically from an embedded timestamp but to achieve this you must make sure your Benchsrc provides a field called Time with the timestamp in it. The timestamp can be either in milliseconds or microseconds. If it is in microsecond make sure you have added microsecond-latency in your benchrttp.conf.

The syntax of the udp message to check the latency status should be:

./udpsend.exe -s <ipaddress> -p <benchrttp-udp-port> stats <time-in-seconds>

Steps

  1. If we want to get the average latency in 30 seconds, the command should be:
    ./udpsend.exe -s 127.0.0.1 -p 13001 stats 30
  2. When you execute this command the logs should show:
    INFO: Processing UDP Command 'stats' with arguments '30'
  3. Wait for around 30 seconds, when the logs should show something similar to:
    STATS AVERAGE    30     5.06   10 1674 - min 0 max 10000 sd 946 nummax 24 sum 1361074620 num 1519 bw 210 2101 contrib 0 max 0
    Where:
    30 – The amount of time that we used to check the latency
    5.06 – The average update rate per client per second during that 30 seconds. It means each client got an average of 5.06 updates every second. (Note: that is not per subscription)
    10 –The number of clients
    1674 – The average latency in 30 seconds in microseconds 
    min 0 - The minimum latency in 30 seconds was 0 microseconds
    max 10000 – The maximum latency in the 30 seconds was 10000 microseconds
    sd 946 – This means the standard deviation is 946.

Review

Having completed this tutorial you should be familiar with how to use Benchsrc and Benchrttp to create a performance testing environment for Liberator.

Just a reminder that performance tests should be run in Linux rather than Windows due to performance considerations - Windows kits are for development purposes only.