Troubleshooting Checklist


The Caplin RET Troubleshooting Checklist provides a systematic approach to troubleshooting RET Adapter issues. It outlines a series of steps to follow when debugging the RET Adapter Toolkit stack. 

An example of an issue could be "rates aren't being received", or "blotters aren't populated", rather than "a particular trade has failed".

 

  1. Preliminaries

    Have you identified what feature is broken, and which adapter provides that feature?

    See the Adapters Feature List for a list of available features.

    Make sure logging is enabled for debugging:

    • Enable StreamLinkJS logs in the client by adding ?debug=finer to the query
    • Enable TrAPI logs by adding ret_log_enabled=true to the trapi-connection.properties for the relevant adapter
  2. Connectivity

    • Check the Liberator Status page - is the Adapter connected to Liberator? If you can't access this page you will need to open a console and navigate to you Deployment Framework's topmost directory, then enter the following command:

       ./dfw status 
    • If the output from the above status command indicates that there is no connection, check the adapter logs. Start with javaAdapterName.log (which is the Java standard output log) to see if the Adapter has crashed or if any Java classes are missing.
    • Check the all-AdapterName.log and observe any TrAPI connection errors (these are usually trading exceptions thrown by TrAPI or AdminAPI). Search keyword: TradingException.
    • Ensure that only one instance of the Adapter is configured on one machine to connect to the Liberator. 
    • Ensure there are no infrastructure issues that may affect connectivity (e.g., check the SCS Relay status).
  3. Configuration

    Check the configuration relating to the broken feature. The Liberator Configuration Reference explains how to correctly configure Caplin Platform components.

  4. Log review

    Start with the all-AdapterName.log. Observe the time stamps within the logs and assess the importance of any exceptions thrown that are logged during the action in question. Some exceptions may be routine, so it is important to identify the relevant exceptions. This can also highlight any connectivity problems - you will need to investigate any connection errors such as checking the status of the RET SCS Relay.
     
     It is likely that there may be nothing unusual during this first analysis of the logs.
  5. Full stack logs

      With regards to log levels - an ERROR does not necessarily mean a fatal error; instead it could refer to something routine such as a bad response from RET, or the Client. While investigating Adapter issues, it is advised that you investigate things that are relevant, i.e. do not waste effort observing error lines if they do not appear significant or relevant to the issue at hand.

Often, there is no failure that would indicate a system wide problem with the Adapters. In this case it is necessary to trace the failing action right the way through the entire stack. In other words, from the Client -> Liberator and Transformer ->  Adapter -> and then to TrAPI. For example, a blotter subscription is sent by the Client: we would then expect to see it pass through Liberator, Transformer, and then one of the Blotter Adapters.

First identify the subscription by its Subject and the time it was requested (for Trading and Orders issues, follow the order id):

  • Does the subscription appear in the Liberator event log? If not, was it rejected in the Auth log? See Permissioning Issues further down.
  • Does the subscription appear in the Transformer event log? If not, was any connectivity established?
  • Does it appear in the Adapter log? If not, is the Liberator or Transformer service correctly configured? Are there any connectivity issues present?
  • If the failing action appears in the all-AdapterName.log, is it possible to follow how it was processed?
  • Each step of processing the action is logged, and therefore any missing steps would be suspicious. Was the action explicitly logged as failed?
  • If a response is sent to the client, does it arrive as expected? Trace back the steps to confirm how it is sent.
  1. Package configuration and logs for Caplin

    • Use the ./dfw dump command to create a configuration dump
    • Use the ./dfw logs command to create a log dump
    • zip up configs and logs rather than RAR or p7 - as an unzip tool is universal
    • A screenshot of the Client can always be useful


      Selective or specific log snippets are unhelpful at this point, as it is likely that logs will need to be looked at end-to-end. In general, logs provided as text files are most useful as they can be processed quickly using a grep command-line utility and other tools.

  2. Permissioning Issues

    If a particular permissioning issue is suspect, review the Liberator auth.log. This must be set to DEBUG before any useful information is captured (configuration details are explained in the Liberator Configuration Reference).
    If the failing action is rejected by the Auth module, it will log an 'Authentication result: Failure: Denied' message. Review the debug level messages logged during start up. Are the correct permissions being received? If not, then review the Permissioning Adapter logs and configuration.

    • Is the Permissioning Adapter UP and communicating with Liberator? Check this by Navigating to the Liberator status page http://localhost:18080/status
    • Are the correct permissions being sent? Check the Permissioning Adapter all-AdapterName.log if permissions where loaded correctly.
    • Is the underlying source of the permissioning config correctly set up? And is it providing the right permissions?
    • Are any group or user permissions missing?