Skip to content

Overview

Concepts and Principles

Development

Overview

IDEs

API Explorer

Releases

Release Notes

TORO Integrate

Coder Studio

Coder Cloud

Bug Reports

Search

Search API

With your integrations in place, it is expected that data will flow in and out of your TORO Integrate instance. But what if you want to run queries on those bits and pieces of information that have passed through? Fret not, TORO Integrate ships with an out-of-the-box support for Solr.

To query indexed data, TORO Integrate comes with its own REST endpoints derived from Apache Solr's REST API. If you have familiarized yourself with their REST API, then using TORO Integrate's REST API should be a walk in the park! In addition to the derived search endpoint, TORO Integrate offers other endpoints to make running queries even more convenient for you.

TORO Integrate's REST API for searching is simply a wrapper for Solr's REST API. You can always just query your Solr servers directly (unless they are embedded). However, there are plenty of reasons to choose the former over the latter:

  • Support for searching in Embedded Solr cores via REST API
  • No need to authenticate your requests to Solr if you're already an authenticated user of TORO Integrate
  • Additional endpoints that make querying a lot more easier

The following sections will discuss the specifications of TORO Integrate's Solr-inspired search API.

Path Parameters

All endpoints in TORO Integrate's REST API for searching begin with the path:

1
/esbapi/v1/solr/<package>/<core>/

... wherein <package> and <core> must be substituted. These path parameters are described in the table below:

Common Path Parameters

Parameter Name Data Type Description
package string The name of the package where the Solr instance is configured.
core string The name of the target Solr core instance.

Headers

Aside from specifying the package and core names in your request path, you must also include a header in your request in order to identify as an authorized user. To access TORO Integrate's search REST API, one must be either an ECC user or a TORO Integrate user under ESBAdminGroup.

Common Headers

Header Name Data Type Description
Authorization string The access token of a TORO Integrate user.

Query Parameters

A good chunk of TORO Integrate's search endpoints rely on or support Solr's REST API parameters. Therefore, in order to fully benefit from what TORO Integrate's REST API has to offer, one must have a good understanding of the Apache Solr REST API.

Below are some sources that might help you get familiar with the allowed parameters in Solr and TORO Integrate's REST APIs:

  • Common Query Parameters discusses the query parameters common to all Solr parsers.
  • Faceting Query Parameters lists the query parameters used for faceting (that is, arranging search results into categories based on indexed terms). You will want to check out Range Faceting and Pivot Faceting as they are two of the most common ways to do faceting.
  • Result Grouping describes the parameters you can use for grouping documents together and getting the top documents for each group.
  • Stats Component explains the parameters you can add to your request in order to get the statistics of the numeric, string, or date fields of your document set.

Warning

The wt query parameter is not supported due to some limitations. However, the response type can be configured by specifying the Accept header where the options are:

  • text/plain
  • application/json
  • application/xml

Endpoints

As said beforehand, the TORO Integrate search REST API exposes a replica of Apache Solr's search API. In addition to this endpoint, however, it also offers other search endpoints to make running queries a breeze for you:

Solr's Search API

GET /esbapi/v1/solr/<package>/<core>/<request-handler>

This is the primary search endpoint in TORO Integrate's REST API, derived from Apache Solr's one and only search endpoint. This mirrors the functions of the latter and thus, to fully understand the capabilities and restrictions of this endpoint, one must be familiar with Apache Solr's REST API.

The Search Request Handler (select) is among the most commonly used request handlers. This handler queries the Solr index. It will be used in the succeeding examples below. To learn more about other request handlers, please refer to Solr's documentation for request handlers.

Request

Path Parameters

Parameter Name Data Type Description
request-handler string The request handler to use during the search.

Note

Make sure the request handler is properly configured in solrconfig.xmlfor the REST API to work as expected.

For other path parameters, see Path Parameters. For possible query parameters, see Query Parameters.

Example

1
2
3
curl -X GET \
  'http://<host>:<port>/esbapi/v1/solr/core/invoke_monitor/select?q=*%3A*&rows=2&cache=false' \
  -H 'Authorization: Bearer <access-token>'

Response

Content Type

  • application/json
  • application/xml
  • text/plain

Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
{
    "responseHeader": {
        "status": 0,
        "QTime": 1,
        "params": {
            "q": "*:*",
            "cache": "false",
            "rows": "2",
            "wt": "json",
            "version": "2.2"
        }
    },
    "response": {
        "numFound": 29,
        "start": 0,
        "docs": [
            {
                "id": "64db237c-7523-4ba1-8bf7-404b30c59a5e",
                "ruleId": -1,
                "timeReceived": "2018-01-24T00:48:15.780Z",
                "packageName": "examples",
                "endpointName": "public java.lang.Object SnoopScript.snoopRequest(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)",
                "endpointType": "http",
                "serviceName": "public java.lang.Object SnoopScript.snoopRequest(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)",
                "url": "/invoke/examples/groovy/SnoopScript/public%20java.lang.Object%20SnoopScript.snoopRequest(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)",
                "user": "user",
                "success": false,
                "cost": 1,
                "responseTime": 20,
                "_version_": 1590432781769375744
            },
            {
                "id": "ffad11c0-3669-4afe-844b-68413cd22134",
                "ruleId": -1,
                "timeReceived": "2017-08-18T03:07:31.263Z",
                "trackerId": "37bf1014-09c0-427f-89a5-9ef54f5157ab",
                "packageName": "examples",
                "endpointName": "public java.lang.Object SQLExamples.getEmployees()",
                "endpointType": "http",
                "serviceName": "public java.lang.Object SQLExamples.getEmployees()",
                "url": "/invoke/examples/groovy/SQLExamples/public%20java.lang.Object%20SQLExamples.getEmployees()",
                "user": "user@example.com",
                "success": true,
                "cost": 1,
                "responseTime": 7,
                "_version_": 1576036625380540416
            }
        ]
    }
}

List Facet Count

GET /esbapi/v1/solr/<package>/<core>/facets

Queries the Solr index and generates a facet count using the facet.field term.

Request

For the required path parameters, see Path Parameters.

Query Parameters

Parameter Name Data Type Required Description
facet.field string true The document field that should be treated as facet. You can define multiple facet.field parameters.

Aside from specifying facet.fields, other Solr API query parameters can be used to further customize the results to your liking — faceting query parameters, in particular.

Example

1
2
3
curl -X GET \
  'http://<host>:<port>/esbapi/v1/solr/core/invoke_monitor/facets?q=endpointType%3Aftp-server&facet.field=packageName&facet.field=endpointName' \
  -H 'authorization: Bearer <access-token>'

The curl request above specified the q query parameter in order to only do a facet count on documents that have an endpointType field with a value of ftp-server. The facet.fields requested are endpointName and packageName and so, we should expect the response to show the facet counts of these fields.

Response

Content Type

  • application/json
  • application/xml
  • text/plain

Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
{
    "facet_field": {
        "packageName": {
            "name": "packageName",
            "facet_count": 3,
            "count": 895,
            "facet": [
                {
                    "name": "Package_00",
                    "count": 335
                },
                {
                    "name": "Package_01",
                    "count": 282
                },
                {
                    "name": "Package_02",
                    "count": 278
                }
            ]
        },
        "endpointName": {
            "name": "endpointName",
            "facet_count": 3,
            "count": 895,
            "facet": [
                {
                    "name": "Endpoint_02",
                    "count": 309
                },
                {
                    "name": "Endpoint_00",
                    "count": 306
                },
                {
                    "name": "Endpoint_01",
                    "count": 280
                }
            ]
        }
    }
}

Get Facet Count

GET /esbapi/v1/solr/<package>/<core>/facet/<facet-field>

This endpoint is pretty much similar to the List Facet Count endpoint, however, this API only limits facet counting to a single facet field.

Request

Path Parameters

Parameter Name Data Type Required Description
facet-field string true The document field that should be treated as facet.

Aside from specifying the facet.field, other Solr API query parameters can be used to further customize the results to your liking — faceting query parameters, in particular. For other required path parameters, see Path Parameters.

Example

1
2
3
curl -X GET \
  'http://<host>:<port>/esbapi/v1/solr/core/invoke_monitor/facet/packageName?q=user%3AUser_016' \
  -H 'authorization: Bearer <access-token>'

The curl request above has the q query parameter specified in order to do a search for documents whose user fields have a value of User_016. By doing so, the results will then be filtered so we only get the facet count of the facet field packageName of documents that have a user field value of User_016.

Response

Content Type

  • application/json
  • application/xml
  • text/plain

Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
{
    "name": "packageName",
    "facet_count": 3,
    "count": 895,
    "facet": [
        {
            "name": "Package_00",
            "count": 335
        },
        {
            "name": "Package_01",
            "count": 282
        },
        {
            "name": "Package_02",
            "count": 278
        }
    ]
}

List Facet Terms

GET /esbapi/v1/solr/<package>/<core>/facet/<facet-field>/names

Lists down all the terms under the specified facet field.

Request

Path Parameters

Parameter name Data Type Required Description
facet_field string true The document field that should be treated as facet.

Aside from specifying the facet.field, other Solr API query parameters can be used to further customize the results to your liking — faceting query parameters, in particular. For the required path parameters, see Path Parameters.

Example

1
2
3
curl -X GET \
  'http://<host>:<port>/esbapi/v1/solr/core/invoke_monitor/facet/packageName/names?q=*%3A*' \
  -H 'authorization: Bearer <access-token>'

The q query parameter value is *:* which means the search will match all documents. By doing so, we will be able to get the terms of the facet field packageName of all documents.

Response

Content Type

  • application/json
  • application/xml
  • text/plain

Example

1
2
3
4
5
[
    "Package_01",
    "Package_00",
    "Package_02"
]

List Statistics

GET /esbapi/v1/solr/<package>/<core>/stats

Queries the Solr index and generates statistics for numeric, string, and date fields within the document set by specifying stats.field.

Request

Query Parameters

Parameter Name Data Type Required Description
stats.field string true The document field for which statistics should be generated. You can define multiple stats.field parameters to generate multiple statistics.
stats.calcdistinct boolean false If true, the countDistinct and distinctValues statistics will be computed and included in the response. These calculations can be very expensive for fields that do not have a tiny cardinality, so they are disabled by default.

Aside from specifying stats.field, other Solr API query parameters can be use to customize the resulting statistics - statistics query parameters, in particular. For the required path parameters, see Path Parameters.

Example

1
2
3
curl -X GET \
  'http://<host>:<port>/esbapi/v1/solr/core/invoke_monitor/stats?stats.field=%7B!func%7Dtermfreq('\''packageName'\''%2C'\''Package_01'\'')&stats.field=%7B!func%7Dtermfreq('\''endpointType'\''%2C'\''ftp-server'\'')&q=*%3A*' \
  -H 'authorization: Bearer <access-token>'

The value of stats.field may seem unusual compared to the previous example but this is just another way to generate statistics. In this request, we are using the function query instead of just plainly specifying the document field in stats.field. The above request will generate statistics that check for frequency of ftp-server as a value of endpointType and Package_01 as a value of packageName in the Solr index.

Response

Content Type

  • application/json
  • application/xml
  • text/plain

Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
{
    "stats": {
        "termfreq('endpointType','ftp-server')": {
            "name": "termfreq('endpointType','ftp-server')",
            "min": 0,
            "max": 1,
            "sum": 895,
            "count": 10032,
            "missing": 0,
            "mean": 0.089214513556619,
            "sumOfSquares": 895,
            "stddev": 0.28506733335231
        },
        "termfreq('packageName','Package_01')": {
            "name": "termfreq('packageName','Package_01')",
            "min": 0,
            "max": 1,
            "sum": 3365,
            "count": 10032,
            "missing": 0,
            "mean": 0.33542663476874,
            "sumOfSquares": 3365,
            "stddev": 0.47216292752283
        }
    }
}

Get Statistics

GET /esbapi/v1/solr/<package>/<core>/stats/<stats-field>

Similar to List Statistics, however, this endpoint is only limited to getting the statistics of a single field.

Warning

Unlike that of List Statistics, function queries are not supported here due to character limitations in request paths.

Request

Path Parameters

Parameter Name Data Type Required Description
stats-field string true The document field for which statistics should be generated.

For other required path parameters, see Path Parameters.

Query Parameters

Parameter Name Data Type Required Description
stats.calcdistinct boolean false If true, the countDistinct and distinctValues statistics will be computed and included the response. These calculations can be very expensive for fields that do not have a tiny cardinality, so they are disabled by default.

Aside from specifying stats.field, other Solr API query parameters can be use to customize the resulting statistics - statistics query parameters, in particular.

Example

1
2
3
curl -X GET \
  'http://<host>:<port>/esbapi/v1/solr/core/invoke_monitor/stats/cost?q=*%3A*' \
  -H 'authorization: Bearer <access-token>'

Response

Content Type

  • application/json
  • application/xml
  • text/plain
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
{
    "name": "cost",
    "min": 321,
    "max": 59482,
    "sum": 984732,
    "count": 10032,
    "missing": 0,
    "mean": 3,
    "sumOfSquares": 3245,
    "stddev": 2
}

Range Faceting

GET /esbapi/v1/solr/<package>/<core>/rangefacet/<facet-range>

According to the Solr's documentation on Range Faceting:

You can use Range Faceting on any date field or any numeric field that supports range queries. This is particularly useful for stitching together a series of range queries (as facet by query) for things like prices.

Tip

If you are going to work with date range facets, it's a good idea to read Solr's guide for working with dates.

Request

Path Parameters

Parameter Name Data Type Required Description
facet_range string true Specifies the field to facet by range.

For other required path parameters, see Path Parameters.

Query Parameters

Parameter Name Data Type Required Description
facet.range.start date true Specifies the lower bound of the ranges.
facet.range.end date true Specifies the upper bound of the ranges.
facet.range.gap string true The span of each range expressed as a value to be added to the lower bound. For date fields, this should be expressed using DateMathParser's syntax.

Aside from the query parameters specified above, you can use other Solr API query parameters to further refine the statistics to be generated from your request - range faceting query parameters, in particular.

Example

1
2
3
curl -X GET \
  'http://<host>:<port>/esbapi/v1/solr/core/invoke_monitor/rangefacet/timeReceived?facet.range.start=2020-01-01T00%3A00%3A00Z&facet.range.end=2020-01-02T00%3A00%3A00Z&facet.range.gap=%2B8HOUR&q=*%3A*' \
  -H 'authorization: Bearer <access-token>'

The following parameters are present in the request above:

  • The facet-range path parameter is set to timeReceived. This is the field we ought to facet by range.
  • The facet.range.start query parameter is set to 2020-01-01T00:00:00Z. This means we'll only do range faceting for values of timeReceived that are on or after this date.
  • The facet.range.end query parameter is set to 2020-01-02T00:00:00Z. This means we'll only do range faceting for values of timeReceived that are on or before this date.
  • The facet.range.gap query parameter is set to +8HOURS which will be the interval gap between each range facet.
  • The q query parameter is set to *:* which means this will match all documents and we'll be applying range faceting for all of them.

Response

Content Type

  • application/json
  • application/xml
  • text/plain

Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
    "name": "timeReceived",
    "start": 1577836800000,
    "end": 1577923200000,
    "gap": "+8HOUR",
    "range_value": [
        "2020-01-01T00:00:00Z",
        "2020-01-01T08:00:00Z",
        "2020-01-01T16:00:00Z"
    ],
    "range_count": [
        19,
        19,
        24
    ],
    "counts": [
        {
            "value": "2020-01-01T00:00:00Z",
            "count": 19
        },
        {
            "value": "2020-01-01T08:00:00Z",
            "count": 19
        },
        {
            "value": "2020-01-01T16:00:00Z",
            "count": 24
        }
    ]
}

Pivot Faceting

GET /esbapi/v1/solr/<package>/<core>/pivots

Queries the Solr index and generates a table summarizing the results. More accurately, according to the Solr's documentation:

Pivoting is a summarization tool that lets you automatically sort, count, total, or average data stored in a table. The results are typically displayed in a second table showing the summarized data. Pivot faceting lets you create a summary table of the results [from faceting] documents by multiple fields.

Request

Query Parameters

Parameter name Data type Required Description
facet.pivot boolean true Defines the fields to use for the pivot. Defining multiple fields in a single facet.pivot parameter values will create multiple "facet_pivot" sections in the response, this is achieved by separating each field with a comma. You can also define multiple facet.pivot parameters as well to create multiple pivots.

Aside from the query parameters specified above, you can use other Solr API query parameters to further refine the statistics to be generated from your request - pivot faceting query parameters, in particular. For the required path parameters, see Path Parameters.

Examples

1
2
3
curl -X GET \
  'http://<host>:<port>/esbapi/v1/solr/core/invoke_monitor/pivots?facet.pivot=packageName%2CendpointName&q=*' \
  -H 'authorization: Bearer <access-token>'

The following query parameters are present in the request above:

  • facet.pivot's value is set to packageName,endpointName and because of this, the endpoint (as will be seen in the response) will generate a more detailed summary by showing the inner pivot of the parent pivot.
  • q's value is * which means it'll match all documents without filtering.

Response

Content Type

  • application/json
  • application/xml
  • text/plain

Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
{
    "facet_pivot": {
        "packageName,endpointName": {
            "field_name": "packageName,endpointName",
            "pivot": [
                {
                    "value": "Package_01",
                    "field": "packageName",
                    "pivot": [
                        {
                            "value": "Endpoint_03",
                            "field": "endpointName",
                            "count": 3205
                        },
                        {
                            "value": "Endpoint_02",
                            "field": "endpointName",
                            "count": 2304
                        }
                    ],
                    "count": 5509
                }
            ]
        }
    }
}

From the response above, we can then conclude that:

  • The total number of documents that have a packageName field with a value of Package_01 is 5509.
  • The total number of documents that have a packageName field with a value of Package_01 and an endpointName field with a value of Endpoint_02 is 2304.
  • The total number of documents that have a packageName field with a value of Package_01 and an endpointName field with a value of Endpoint_03 is 3205.