Skip to content

Overview

Concepts and Principles

Development

Overview

IDEs

API Explorer

Releases

Release Notes

TORO Integrate

Coder Studio

Coder Cloud

Bug Reports

Search

The Tracker Search Index

In TORO Integrate, you are given the option to log data consumed and produced by your integrations via a feature called Tracker. Tracker is the term used to refer to the family of services and resources wired together to record service invocation data. By logging data coming in and out of TORO Integrate, you are granted access to these data again so you could use them when performing audits, analysis, troubleshooting, and when generating reports.

Use cases

Suppose you have exposed a REST API using TORO Integrate for use in your e-commerce web site. Every time a customer orders something on the website, it contacts TORO Integrate behind the scenes to have the order created in your warehouse. By indexing order data, you can easily create reports about the:

  • Average orders received in the previous months
  • Orders received per day over the last month
  • Orders received which costs greater than $50 in the last week
  • Orders the contained certain products
  • Orders shipped to a certain postal state
  • Orders that have resulted in an error
  • Best-selling products in the previous month, or year

You can also do other integrations that compute:

  • How often your partners invoke TORO Integrate services
  • How often your suppliers send their data for your systems
  • The number of products published over the Google Marketplace
  • How many new customers have signed up to receive your newsletter in the previous month
  • The average value of all invoices, graphed per day for the last month

Tracker is a fully searchable, queryable logging engine that's great for looking up the data that your code has dealt with in the past.

How It Works

Under the hood, Tracker is backed by a Solr core of the same name (tracker)1 linked to your instance's core package. When a service invocation is triggered by an endpoint or HTTP call, Tracker will log the details of the service invocation by adding an entry to the tracker database2 and then creating a Lucene document from that entry and indexing it to the aforementioned Solr core's index, which we call the Tracker Search Index. Querying data against the index instead of the database allows for faster, finer-grained searches.

Tracking is on by default

TORO Integrate automatically adds a record to tracker every time code is invoked over HTTP/S or via endpoints, unless configured to not do so.

Tracker Versus Monitor

Although both Tracker and Monitor log service invocation data, there are still a few key differences between them:

  • Tracker logging is optional; Monitor logging is not. If you can get what you need in Monitor, it is better to not use Tracker as this will mean less processes for TORO Integrate to handle.
  • Unlike that of the Monitor Search Index, there are no restrictions in modifying the contents of the Tracker Search Index. You can add, remove, and modify Tracker documents as you wish.
  • With Tracker, you can specify which service invocation data to index (request data only, response data only, or both). Monitor will not provide you such options; the fields it will index cannot be changed.

Documents

We refer to entries in the index as documents. Documents have a particular set of fields. In the Tracker Search Index, documents are configured to have the following fields:

Field Name Java Data Type Description SQL Table SQL Column SQL Data Type
internalId String A unique string used to identify documents apart. This is assigned by Tracker and is used to retrieve the document in future sessions, such as when you would like to update it by adding a new state to it. toro_tracker_document internal_id VARCHAR(40)
externalId String A unique string used to identify service invocations apart. A service call made during a certain date at 02:43 PM is different from the call made during 11:45 PM. This ID is assignable by users although by default, this is the current Unix time at which the service was called. toro_tracker_document external_id VARCHAR(40)
documentType DocumentType The name of the integration you want the document to be associated with. By default, when a service is invoked by an Integrate Endpoint, the document type is the type of that Integrate Endpoint; services invoked via HTTP will have a document type after the name of leading path the call was directed at. toro_tracker_document doc_type_id VARCHAR(100)
stateName String The name of the last recorded state of the integration call. This should indicate the progress of the invocation; for example, from Received -> Processing -> Processed or Error. toro_tracker_document state_name VARCHAR(40)
senderId String The name of the system, user, or address that triggered the call. This can be the IP address where the HTTP inbound request came from or which Integrate Endpoint invoked the service. toro_tracker_document sender_id VARCHAR(255)
receiverId String The name of the system, user, or address that is the eventual recipient of the invocation. If the invocation is triggered by an endpoint, then that endpoint's service is marked as the receiver. For HTTP endpoints, this is their relative request URL mapping. toro_tracker_document receiver_id VARCHAR(255)
userName String The user who initiated the call. This can be set to "Anonymous" if the caller is anonymous, or the name of a Marketplace User or Integrate User, if known. toro_tracker_document user_name VARCHAR(150)
timeReceived Timestamp The time the document was created in Tracker. toro_tracker_document time_received TIMESTAMP
logMessages Set<DocumentLog> Log messages that should be associated with the document. toro_tracker_document_log
properties Set<DocumentProperty> Key-value pairs that define additional or custom properties associated to a document. toro_tracker_document_property
states Set<DocumentState> A list of states the document was in throughout the integration. toro_tracker_document_state
childDocuments Set<DocumentRelationship> Parent-child relationship listings wherein the document of interest is the child. toro_tracker_document_relationship
parentDocuments Set<DocumentRelationship> Parent-child relationship listings wherein the document of interest is the parent. toro_tracker_document_relationship

Tracker documents are represented internally by Document objects. This is the same type you'll have to work with if you plan to fetch or set Tracker document properties in your code (e.g. adding your own log messages, properties, or states), in which you'll use the aforementioned class's getter and setter methods.

Customizable field values

Whilst TORO Integrate tries its best to automatically populate document fields, most, if not all, of the values it has set can still be overridden so that it conforms to your organization's preference and most importantly, when doing so, it is important to relay or document those changes.

Log Messages

Log messages are represented by DocumentLog objects, which are composed of the following fields:

Field Name Java Data Type Description SQL Table SQL Column SQL Data Type
documentLogId Integer The identifier for the log message. toro_tracker_document_log document_log_id INT(10)
document Document The document which this log message belongs to. toro_tracker_document_log internal_id VARCHAR(40), FK
timeReceived Timestamp The time at which the message was logged. toro_tracker_document_log time_received TIMESTAMP
logType String The type of log message. toro_tracker_document_log log_type VARCHAR(10)
logMessage String The content of the log message itself. toro_tracker_document_log log_message VARCHAR(1024)

Ideally, document log messages should tell us about the service invocation's condition during certain time frames.

Properties

Key-value pairs that define additional or custom properties associated to a document.

Field Name Data Type Description SQL Table SQL Column SQL Data Type
document Document The document which this property belongs to. toro_tracker_document_property internal_id VARCHAR(40), FK
propertyKey String The name of the property. toro_tracker_document_property property_key VARCHAR(30)
propertyValue String The value of the property. toro_tracker_document_property property_value VARCHAR(100)

States

A list of states the document was in throughout the integration. States are so called because they let us know the different, transitioning states of the document (or the service invocation, rather) whilst TORO Integrate was processing it. With states, the data present at certain points of the process (of invoking the service) is made known to us, like when it encounters an error, is starting the invocation, or has successfully completed the task.

States are represented internally by DocumentState objects. This type is comprised of the following fields:

Field Name Data Type Description SQL Table SQL Column SQL Data Type
documentStateId Integer The auto-generated, unique number used to identify document states. toro_tracker_document_state document_state_id INT(10)
document Document The document which owns this state. toro_tracker_document_state internal_id VARCHAR(40)
stateName String The name of the state; does not have to be unique. toro_tracker_document_state time_received TIMESTAMP
timeReceived Timestamp The time at which this state was achieved. toro_tracker_document_state state_name VARCHAR(50)
documentSize Long The size of the state's content. toro_tracker_document_state document_size BIGINT
content DocumentStateContent The payload brought by the state. toro_tracker_doc_stcontent

Document states are an important feature of Tracker. Document state records allow you to store binary and textual data within a document. This means you can save JSON or XML data, spreadsheets, or even PDF and word processor files. What's more, the data is all indexed in a search engine. This means that you can literally do a search within the Tracker Search Index and see which services are being used, as well as search for the data that's going into them.

Tracker uses Apache Tika to extract data from binary files, which Tracker then indexes to make such data queryable. Tracker will be able to index the data as long as its format is supported by Tika.

A document state's content typically comes in the form of message bodies. A message body is the payload of the message received and sent by TORO Integrate. For service invocations triggered by HTTP, this is the HTTP request and HTTP response3 bodies, by default4. For services triggered by Integrate Endpoints, state contents will vary by type. For example, with Email Endpoints, the starting state can be the files attached in the received email or the body of the email, itself, whilst the final state may contain the endpoint's reply to the email.

Discrepancies

As with the case in Monitor, there are a few discrepancies depending on how the service is exposed. To show you these differences, the following examples below5 are provided:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
{
  "internalId": "fa30f599-6030-4ad5-836e-2a18926ee1a6",
  "externalId": "1534393691669",
  "type": {
    "id": "hello",
    "name": "hello"
  },
  "stateName": "Processed",
  "senderId": "0:0:0:0:0:0:0:1",
  "receiverId": "/api/hello/world",
  "username": "Anonymous",
  "timestamp": 1534393691629,
  "logs": [],
  "properties": [
    {
      "key": "Integrate_Server",
      "value": "TORO Integrate@TORO-MM64:8080 (fe80:0:0:0:42cd:e074:a3d4:4fdc%utun0)"
    },
    {
      "key": "Duration(ms)",
      "value": "29"
    }
  ],
  "states": [
    {
      "id": 1,
      "name": "Received",
      "timeReceived": 1534393691629,
      "canBeResubmitted": false,
      "contentFileName": "request.xml",
      "contentSize": 5347
    },
    {
      "id": 2,
      "name": "Processed",
      "timeReceived": 1534393691793,
      "canBeResubmitted": false,
      "contentFileName": "response.txt",
      "contentSize": 13
    }
  ]
}
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
{
  "internalId": "7ae04eee-d419-404d-ada6-da623e3e9ed5",
  "externalId": "1534394377149",
  "type": {
    "id": "hello",
    "name": "hello"
  },
  "stateName": "Processed",
  "senderId": "0:0:0:0:0:0:0:1",
  "receiverId": "/api/hello/world",
  "username": "Anonymous",
  "timestamp": 1534394377149,
  "logs": [],
  "properties": [
    {
      "key": "Integrate_Server",
      "value": "TORO Integrate@TORO-MM64:8080 (fe80:0:0:0:42cd:e074:a3d4:4fdc%utun0)"
    },
    {
      "key": "Duration(ms)",
      "value": "208"
    }
  ],
  "states": [
    {
      "id": 3,
      "name": "Received",
      "timeReceived": 1534394377149,
      "canBeResubmitted": false,
      "contentFileName": "request.xml",
      "contentSize": 5195
    },
    {
      "id": 4,
      "name": "Processed",
      "timeReceived": 1534394377366,
      "canBeResubmitted": false,
      "contentFileName": "response.txt",
      "contentSize": 15
    }
  ]
}
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
{
  "internalId": "d3d21653-e75c-46c1-a6a3-dfc82b0db9a6",
  "externalId": "1534394572980",
  "type": {
    "id": "hello",
    "name": "hello"
  },
  "stateName": "Processed",
  "senderId": "0:0:0:0:0:0:0:1",
  "receiverId": "/api/hello/world",
  "username": "Anonymous",
  "timestamp": 1534394572979,
  "logs": [],
  "properties": [
    {
      "key": "Integrate_Server",
      "value": "TORO Integrate@TORO-MM64:8080 (fe80:0:0:0:42cd:e074:a3d4:4fdc%utun0)"
    },
    {
      "key": "Duration(ms)",
      "value": "303"
    }
  ],
  "states": [
    {
      "id": 5,
      "name": "Received",
      "timeReceived": 1534394572979,
      "canBeResubmitted": false,
      "contentFileName": "request.xml",
      "contentSize": 5195
    },
    {
      "id": 6,
      "name": "Processed",
      "timeReceived": 1534394573295,
      "canBeResubmitted": false,
      "contentFileName": "response.txt",
      "contentSize": 15
    }
  ]
}
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
{
  "internalId": "97bdc10b-237f-4c27-97f9-867145ee7ad7",
  "externalId": "1534400984618",
  "type": {
    "id": "Scheduler",
    "name": "Scheduler"
  },
  "stateName": "Processed",
  "senderId": "SendScheduledEmail",
  "receiverId": "gloop:endpointServices.gloop.schedulerExamples.SendScheduledEmail/endpointServices.gloop.schedulerExamples.SendScheduledEmail",
  "username": "",
  "timestamp": 1534400984618,
  "logs": [],
  "properties": [
    {
      "key": "Duration(ms)",
      "value": "6350"
    }
  ],
  "states": [
    {
      "id": 7,
      "name": "Started",
      "timeReceived": 1534400984618,
      "canBeResubmitted": false,
      "contentFileName": "",
      "contentSize": 0
    },
    {
      "id": 8,
      "name": "Processed",
      "timeReceived": 1534400990995,
      "canBeResubmitted": false,
      "contentFileName": "",
      "contentSize": 0
    }
  ]
}

As you may notice, Groovy Service, Gloop API, and Ad Hoc Gloop Service endpoints have fairly similar fields.

  • By default, these types have set the leading URL mapping as the name and ID of the type of each document. In the example, all endpoints are accessible via the path /hello/{name}, wherein the leading URL path is hello.
  • The senderId is the IP address where the request came from.
  • The receoverId is the URL mapping where the request was received at.

Integrate Endpoints service invocations differ in that:

  • The default name of their starting state is "Started" instead of "Received".
  • The senderId is the name of the endpoint itself.
  • The receiverId is the qualified name of the service of the endpoint.

User Interface

To display the Tracker UI where you can search and see the details of documents:

Click on the Toggle Tracker View button on Coder's toolbar. You should be presented with a list of documents present in Tracker. Clicking on one of these documents will display more details about that document.

Accessing the Tracker UI via the *Toggle Tracker View* button

From the tabs below, click on the Tracker tab to display the documents present in the Tracker Search Index:

Accessing the Tracker UI via the *Tracker* tab

Click on the search button with Tracker selected in the static search bar above each page:

Accessing the Tracker UI via the search button

REST API

As with the other search indices, the Tracker Search Index can be queried and managed via the Search API, which is based off on Solr's API. Simply ensure to substitute the following path parameters with their respective values:

Path Parameters

Parameter Name Value
package core
core tracker

For your ease of use, there are also tracker endpoints exposed in TORO Integrate's own REST API.


  1. Out-of-the-box, the tracker Solr core resides in an embedded Solr instance and should be migrated to a remote Solr server instead for performance boost. 

  2. TORO Integrate, by default, uses an embeded HSQL database for tracker. It is recommended that you migrate your data there in a more reliable database when running on production. 

  3. Before enabling request and/or response data logging, there are privacy and security concerns that should be addressed by the integration developers. 

  4. It's possible to add your own states to a document, but TORO Integrate, by default, puts states of its own from the data received and sent by the endpoint. 

  5. Displayed JSON data are obtained via the /tracker/document/{id} endpoint.