Skip to content

Overview

Concepts and Principles

Development

Overview

IDEs

API Explorer

Releases

Release Notes

TORO Integrate

Coder Studio

Coder Cloud

Bug Reports

Search

The Invoke Monitor Search Index

The Invoke Monitor Search Index, typically shortened to Monitor Search Index, is an index of all the service calls that were made within your instance – specifically those triggered by external requests or configured events. It is used internally by the system to enforce API throttling and monetization if you have configured rules.

Under the hood, all this takes to work is a Solr core called invoke_monitor1 linked to your instance's core package. When a request is received by TORO Integrate, it logs the service invocation by adding a document to the aforementioned Solr core's index. Recording data this way allows the Invoke Monitor Service2 to do faster, fine-grained searches for you.

Unlike custom search indices or the Tracker Search Index, clients are not allowed to add their own documents to the Invoke Monitor Search Index. This is to ensure the accuracy of the residing statistical data. Disabling the index is also not allowed for the same reason3.

Document Structure

TORO Integrate produces a document per service invoke which contains the particulars of the service call. Every document is later on indexed to make it queryable. The following table describes the searchable properties of these documents:

Field Name Data Type Description
id String The auto-generated identifier for the document.
ruleId long The identifier of the rule that first matched the request4. This is set to -1 if there is no positive match.
timeReceived Date The time the service call was made in yyyy-MM-dd'T'HH:mm:ss.SSSZ format.
trackerId String The identifier of the document's counterpart in the Tracker Search Index, if it exists5.
packageName String The name of the package which handled the request.
endpointName String The name of the endpoint that invoked the service.
endpointType String The type of endpoint over which the service was invoked.
apiGroup String The name of the group the caller (user) belongs to.
serviceName String The name of the service that was invoked.
url String The invoke URL.
user String The username of the user who invoked the service.
success boolean Flag indicating whether the service was successfully executed or not.
cost Double The cost of invoking the service.
responseTime long The duration of the service call.
location String The originating request's coordinates.

Indexing Documents

As mentioned beforehand, the direct alteration of the Invoke Monitor Search Index is not permitted. Instead, your instance automatically does the indexing for you whenever it gets notified about a client or event call to any of your endpoints.

Unstandardized Properties

However, depending on how your endpoints are exposed and how they are invoked, there will be slight variations in the values of certain document properties. The following subsections will explain these discrepancies.

To give you get a head start, here are snippets from a search API response that let you compare the contents of the service call documents – all varying on how the services were exposed:

Documents of services invoked normally

Below are sample service call documents produced when the services are invoked normally. Invoking requests via the Invoker yields slightly different results.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
{
    "id": "d8d311cd-1229-44f2-87b7-f2918d2552ed",
    "ruleId": -1,
    "timeReceived": "2018-05-29T08:25:02.028Z",
    "trackerId": "42ac0c77-e5d3-40a0-9ac8-b396555194a1",
    "packageName": "speak",
    "endpointType": "http",
    "serviceName": "sayHello",
    "url": "/api/hello/world",
    "user": "anonymous",
    "success": true,
    "cost": 0,
    "responseTime": 1527582302027,
    "_version_": 1601786139950186496
}
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
{
    "id": "88cc2fcc-7eac-46db-8c9c-1a01cdec1e91",
    "ruleId": -1,
    "timeReceived": "2018-05-29T06:36:15.785Z",
    "trackerId": "20221586-4de0-44d5-ba9e-3609c57b56f0",
    "packageName": "speak",
    "endpointName": "org.speak.English",
    "endpointType": "api",
    "serviceName": "org.speak.Hello",
    "url": "/api/hello/world",
    "user": "user@your.org",
    "success": true,
    "cost": 0,
    "responseTime": 1527575775513,
    "_version_": 1601779296683163648
}
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
{
    "id": "5f4a561d-2933-43ef-a218-0a506c6697d0",
    "ruleId": -1,
    "timeReceived": "2018-05-29T06:04:14.869Z",
    "trackerId": "55e70e29-8176-4ad3-a82c-18e5cc1ad8bd",
    "packageName": "speak",
    "endpointType": "http",
    "serviceName": "org.speak.Hello",
    "url": "/api/hello/world",
    "user": "anonymous",
    "success": true,
    "cost": 0,
    "responseTime": 1527573854054,
    "_version_": 1601777282827943936
}
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
{
    "id": "98806c08-767f-409a-9c55-1488d54a125d",
    "ruleId": 0,
    "timeReceived": "2018-05-29T08:37:23.640Z",
    "packageName": "examples",
    "endpointName": "SendScheduledEmail",
    "endpointType": "scheduler",
    "serviceName": "gloop:endpointServices.gloop.schedulerExamples.SendScheduledEmail/endpointServices.gloop.schedulerExamples.SendScheduledEmail",
    "user": "anonymous",
    "success": true,
    "responseTime": 1527583043290,
    "_version_": 1601786918021890048,
    "cost": 1
}

trackerId

  • The trackerId only gets set when the service is configured to log the calls to Tracker.

endpointName

  • This property is only present in services triggered by Integrate Endpoints or Gloop API endpoints.
  • If a Gloop API endpoint invoked the service, it'll be equivalent to the namespace of the Gloop API.
  • If the service is called by an Integrate Endpoint, endpointName will be set to the name of that Integrate Endpoint.

endpointType

  • If the call came from a Gloop API endpoint, it'll have the value of api.
  • If the call was from an Integrate Endpoint instead, endpointType will be set to the actual type of the Integrate Endpoint (e.g. scheduler, email, ftp-server) that wrapped the service.

serviceName

  • In the case of Spring-based Groovy REST controllers, this is the name of the method that handled the call.
  • For Gloop API endpoints and ad hoc Gloop service endpoints, this is the namespace of the service that was invoked.
  • For Gloop Services invoked via the Service Invoker, though, the service name is consistently set to runGloop.
  • Integrate Endpoints will set serviceName to the service namespace of the service they are configured to call.

url

  • Not set if service is called by an Integrate Endpoint.
  • The invocation URLs of services invoked via the Service Invoker (invoked via browser) are different and thus will be reflected in the service call document.

    If the same Groovy service above was called via the Service Invoker, its service call document's url property will be:

    1
    2
    3
    4
    5
    {
        ...
        "url": "/invoke/speak/groovy/org.speak.Greeter/public+org.springframework.http.ResponseEntity%3C%3F%3E+org.speak.Greeter.sayHello%28java.lang.String%29",
        ...
    }
    

    With the Gloop service, though, we'll get something like:

    1
    2
    3
    4
    5
    {
        ...
        "url": "/invoke/speak/gloop/org.speak.Hello",
        ...
    }
    

Searching Documents

Despite being a system component, it is still possible to make use of the data in the Invoke Monitor Search Index in order to generate statistical accounts of the service invokes that have occurred. In order to fetch figures, you may use the search API or simply take advantage of the report builder which creates reports that retrieve the data on your behalf.

Via the UI

As of Version 3.0, only the Server Admin UI exposes an interface for querying against the Invoke Monitor Search Index graphically.

The static bar on top of the UI will contain a search bar which can query documents from the Tracker or Monitor indices, depending on which is selected.

The *Monitor Search Bar*

Simply click on the search button (decorated with a magnifying glass) to execute your query. If you want to narrow down your search results, supply the fields in the form for specific, property-based searches or the primary search input bar for a generic search that matches any document property.

The page displaying the results will roughly look like this:

The results of a query against the Invoke Monitor Search Index displayed in the Admin Server UI

The UI also allows you to filter or break down your results by selecting various facets at the top. Next to each facet, a number will appear. This number is the number of documents in the Invoke Monitor Search Index that match the facet.

Facets of Monitor search results

Via Code

To query the Invoke Monitor Search Index in Gloop, create an Invoke Code Step (preferred) or Gloovy Step calling the utility search method of the MonitorMethods class. Your step should roughly look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
SearchResult searchMonitor(query,
                           monitorDocumentIds,
                           trackerDocumentIds,
                           packageNames,
                           endpointTypes,
                           users,
                           endpointNames,
                           serviceNames,
                           urls,
                           successes,
                           startDate,
                           endDate,
                           minimumCost,
                           maximumCost,
                           pageSize,
                           page)

MonitorMethods has other utility methods for your Invoke Monitor-related needs

MonitorMethods contains convenience methods for:

  • Managing and retrieving Monitor Rules
  • Fetching the billing details of your users
  • Setting or fetching HTTP request costs

Via the REST API

TORO Integrate has REST endpoints exposed for querying Solr cores associated with Integrate Packages. The specifications of the REST API are described in Search API. Simply remember to substitute the following path parameters with their respective values:

Path Parameters

Parameter Name Value
package core
core invoke_monitor

With this said, all your queries should be directed to paths beginning in:

1
/esbapi/v1/solr/core/invoke_monitor/

  1. Out-of-the-box, the invoke_monitor Solr core resides in an embedded Solr instance and should be migrated to a remote Solr server instead for performance boost. 

  2. Refers to the family of internal services responsible for managing the Invoke Monitor Search Index. Also called Monitor for brevity. 

  3. There is no exposed configuration to stop the Invoke Monitor Service from creating and indexing service call documents. However, TORO Integrate is internally capable of conditionally disabling this behavior. The Invoke Monitor Service is on if (1) TORO Integrate's assigned license supports throttling and monetization or (2) the license imposes limits on the number of service invokes per minute, hour, day, week, or month. 

  4. Because rules can have overlapping scopes, there are times when multiple rules are applicable to a single service invocation. However, TORO Integrate will check all rules (based on their priority), and use the first match. 

  5. Because logging with tracker is optional.