Skip to content

Home

Introduction

Development

Overview

IDEs

Services

Service manager

API Explorer

Extended Solr Search API

With your integrations deployed, it is expected for data to flow in and out of your TORO Integrate instance. But what if you want to query those bits and pieces of information? This is where TORO Integrate's out-of-the-box support for Solr comes handy.

To query indexed data, you can use Apache Solr's API; extended and re-exposed in TORO Integrate. You can direct queries meant for Solr to TORO Integrate, and use additional, convenient endpoints for performing document searches. You can still query your Solr servers directly (unless they are embedded), but there are plenty of reasons to choose TORO Integrate's Solr API over the native Solr API such as:

  • Support for document searches in embedded Solr cores via REST
  • No need to authenticate your requests to Solr if you're already an authenticated user of TORO Integrate
  • Additional endpoints that make querying a lot more easier

The Solr Search API is NOT just for custom search indices

You can use this API to query documents in Tracker and Monitor as well.

Headers

You must include a header in your request to be able to use the API. To access the TORO Integrate's Solr API, one must either be an ECC user or an Integrate user under the ESBAdminGroup group.

Header name Data type Description
Authorization string The access token of an Integrate user.

Path parameters

All endpoints in TORO Integrate's Solr API begin with the path:

1
/esbapi/v1/solr/<package>/<core>/

The <package> and <core> path parameters must be substituted, as described in the table below:

Parameter name Data type Description
package string The name of the Integrate package where the Solr instance is configured.
core string The name of the target Solr core instance.

Query parameters

TORO Integrate's Solr Search API simply re-exposes a part of Solr's REST API, hence supported parameters are also shared between the two. In order to fully benefit from what the re-exposed API has to offer, one must have a good understanding of the Apache Solr REST API.

Below are resources that can help you get familiar with allowed parameters in both Solr REST APIs:

  • Common query parameters discusses the query parameters common to all Solr parsers.
  • Faceting query parameters lists the query parameters used for faceting (that is, arranging search results into categories based on indexed terms). You will want to check out Range faceting and Pivot faceting as they are two of the most common ways to do faceting.
  • Result grouping describes the parameters you can use for grouping documents together and getting the top documents for each group.
  • Stats component explains the parameters you can add to your request in order to get the statistics of the numeric, string, or date fields of your document set.

wt query parameter is unsupported

The wt query parameter is unsupported, but the response type can be configured by specifying the Accept header. Commonly used options are:

  • text/plain
  • application/json
  • application/xml

Endpoints

TORO Integrate's Solr API exposes a replica of Apache Solr's Search API. In addition to this API, however, it also exposes other endpoints for faster searches:

Solr API

  • Method: GET
  • Path: /esbapi/v1/solr/<package>/<core>/<request-handler>

Perhaps the most important endpoint in the API, this endpoint mirrors the functionality of Solr's native REST API. Thus to fully understand the capabilities and restrictions of this endpoint, one must be familiar with Apache Solr's REST API.

The SearchHandler (select) is among the most commonly used request handlers. This handler queries the Solr index. It will be used in the succeeding examples below. To learn more about other request handlers, please refer to Solr's documentation for request handlers.

Only Solr's SearchHandler is mirrored by TORO Integrate

Other handlers such as the UpdateHandler, are not supported. Requests that use unsupported handlers have to be sent directly to the Solr server as TORO Integrate cannot act as a proxy.

Request

Path parameters

Parameter name Data type Description
request-handler string The request handler to use during the search.
Global parameters and headers

For other parameters and headers, see:

Hander configuration

Ensure the handler you want to use is properly configured in solrconfig.xml for the API to work as expected.

Example

1
2
3
curl -X GET \
  'http://<host>:<port>/esbapi/v1/solr/core/invoke_monitor/select?q=*%3A*&rows=2&cache=false' \
  -H 'Authorization: Bearer <access-token>'

Response

Content type

  • application/json
  • application/xml
  • text/plain

Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
{
    "responseHeader": {
        "status": 0,
        "QTime": 1,
        "params": {
            "q": "*:*",
            "cache": "false",
            "rows": "2",
            "wt": "json",
            "version": "2.2"
        }
    },
    "response": {
        "numFound": 29,
        "start": 0,
        "docs": [
            {
                "id": "64db237c-7523-4ba1-8bf7-404b30c59a5e",
                "ruleId": -1,
                "timeReceived": "2018-01-24T00:48:15.780Z",
                "packageName": "examples",
                "endpointName": "public java.lang.Object SnoopScript.snoopRequest(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)",
                "endpointType": "http",
                "serviceName": "public java.lang.Object SnoopScript.snoopRequest(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)",
                "url": "/invoke/examples/groovy/SnoopScript/public%20java.lang.Object%20SnoopScript.snoopRequest(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)",
                "user": "user",
                "success": false,
                "cost": 1,
                "responseTime": 20,
                "_version_": 1590432781769375744
            },
            {
                "id": "ffad11c0-3669-4afe-844b-68413cd22134",
                "ruleId": -1,
                "timeReceived": "2017-08-18T03:07:31.263Z",
                "trackerId": "37bf1014-09c0-427f-89a5-9ef54f5157ab",
                "packageName": "examples",
                "endpointName": "public java.lang.Object SQLExamples.getEmployees()",
                "endpointType": "http",
                "serviceName": "public java.lang.Object SQLExamples.getEmployees()",
                "url": "/invoke/examples/groovy/SQLExamples/public%20java.lang.Object%20SQLExamples.getEmployees()",
                "user": "user@example.com",
                "success": true,
                "cost": 1,
                "responseTime": 7,
                "_version_": 1576036625380540416
            }
        ]
    }
}

List facet count

  • Method: GET
  • Path: /esbapi/v1/solr/<package>/<core>/facets

This endpoint queries the Solr index and generates a facet count using the facet.field term.

Request

Query parameters

Parameter name Data type Required Description
facet.field string true The document field that should be treated as facet. You can define multiple facet.field parameters.

Aside from specifying facet.fields, other parameters can be used to further customize the results to your liking — faceting query parameters in particular.

Global parameters and headers

For other parameters and headers, see:

Example

1
2
3
curl -X GET \
  'http://<host>:<port>/esbapi/v1/solr/core/invoke_monitor/facets?q=endpointType%3Aftp-server&facet.field=packageName&facet.field=endpointName' \
  -H 'authorization: Bearer <access-token>'

The cURL request above specified the q query parameter in order to only do a facet count on documents that have an endpointType field with a value of ftp-server. The facet.fields requested are endpointName and packageName and so, we should expect the response to show the facet counts of these fields.

Response

Content type

  • application/json
  • application/xml
  • text/plain

Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
{
    "facet_field": {
        "packageName": {
            "name": "packageName",
            "facet_count": 3,
            "count": 895,
            "facet": [
                {
                    "name": "Package_00",
                    "count": 335
                },
                {
                    "name": "Package_01",
                    "count": 282
                },
                {
                    "name": "Package_02",
                    "count": 278
                }
            ]
        },
        "endpointName": {
            "name": "endpointName",
            "facet_count": 3,
            "count": 895,
            "facet": [
                {
                    "name": "Endpoint_02",
                    "count": 309
                },
                {
                    "name": "Endpoint_00",
                    "count": 306
                },
                {
                    "name": "Endpoint_01",
                    "count": 280
                }
            ]
        }
    }
}

Get facet count

  • Method: GET
  • Path: /esbapi/v1/solr/<package>/<core>/facet/<facet-field>

This endpoint is similar to the List facet count endpoint, but facet counting is limited to a single facet field.

Request

Path parameters

Parameter name Data type Required Description
facet-field string true The document field that should be treated as facet.

Aside from specifying a facet.field parameter, other parameters can be used to further customize the results to your liking — faceting query parameters in particular.

Global parameters and headers

For other parameters and headers, see:

Example

1
2
3
curl -X GET \
  'http://<host>:<port>/esbapi/v1/solr/core/invoke_monitor/facet/packageName?q=user%3AUser_016' \
  -H 'authorization: Bearer <access-token>'

The cURL request above has the q query parameter specified in order to do a search for documents whose user fields have a value of User_016. By doing so, the results will be filtered so we only get the facet count of the facet field packageName of documents that have a user field value of User_016.

Response

Content type

  • application/json
  • application/xml
  • text/plain

Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
{
    "name": "packageName",
    "facet_count": 3,
    "count": 895,
    "facet": [
        {
            "name": "Package_00",
            "count": 335
        },
        {
            "name": "Package_01",
            "count": 282
        },
        {
            "name": "Package_02",
            "count": 278
        }
    ]
}

List facet terms

  • Method: GET
  • Path: /esbapi/v1/solr/<package>/<core>/facet/<facet-field>/names

Lists down all the terms under the specified facet field.

Request

Path parameters

Parameter name Data type Required Description
facet_field string true The document field that should be treated as facet.

Aside from specifying a facet.field parameter, other parameters can be used to further customize the results to your liking — faceting query parameters in particular.

Global parameters and headers

For other parameters and headers, see:

Example

1
2
3
curl -X GET \
  'http://<host>:<port>/esbapi/v1/solr/core/invoke_monitor/facet/packageName/names?q=*%3A*' \
  -H 'authorization: Bearer <access-token>'

The q query parameter value is *:* which means the search will match all documents. By doing so, we will be able to get the terms of the facet field packageName of all documents.

Response

Content type

  • application/json
  • application/xml
  • text/plain

Example

1
2
3
4
5
[
    "Package_01",
    "Package_00",
    "Package_02"
]

List statistics

  • Method: GET
  • Path: /esbapi/v1/solr/<package>/<core>/stats

Queries the Solr index and generates statistics for numeric, string, and date fields within the document set by specifying stats.field.

Request

Query parameters

Parameter name Data type Required Description
stats.field string true The document field for which statistics should be generated. You can define multiple stats.field parameters to generate multiple statistics.
stats.calcdistinct boolean false If true, the countDistinct and distinctValues statistics will be computed and included in the response. These calculations can be very expensive for fields that do not have a tiny cardinality, so they are disabled by default.

Aside from specifying stats.field, other parameters can be use to customize the resulting statistics - statistics query parameters in particular.

Global parameters and headers

For other parameters and headers, see:

Example

1
2
3
curl -X GET \
  'http://<host>:<port>/esbapi/v1/solr/core/invoke_monitor/stats?stats.field=%7B!func%7Dtermfreq('\''packageName'\''%2C'\''Package_01'\'')&stats.field=%7B!func%7Dtermfreq('\''endpointType'\''%2C'\''ftp-server'\'')&q=*%3A*' \
  -H 'authorization: Bearer <access-token>'

The value of stats.field may seem unusual compared to the previous example but this is just another way to generate statistics. In this request, we are using the function query instead of just plainly specifying the document field in stats.field. The above request will generate statistics that check for frequency of ftp-server as a value of endpointType and Package_01 as a value of packageName in the Solr index.

Response

Content type

  • application/json
  • application/xml
  • text/plain

Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
{
    "stats": {
        "termfreq('endpointType','ftp-server')": {
            "name": "termfreq('endpointType','ftp-server')",
            "min": 0,
            "max": 1,
            "sum": 895,
            "count": 10032,
            "missing": 0,
            "mean": 0.089214513556619,
            "sumOfSquares": 895,
            "stddev": 0.28506733335231
        },
        "termfreq('packageName','Package_01')": {
            "name": "termfreq('packageName','Package_01')",
            "min": 0,
            "max": 1,
            "sum": 3365,
            "count": 10032,
            "missing": 0,
            "mean": 0.33542663476874,
            "sumOfSquares": 3365,
            "stddev": 0.47216292752283
        }
    }
}

Get statistics

  • Method: GET
  • Path: /esbapi/v1/solr/<package>/<core>/stats/<stats-field>

This endpoint is similar to List statistics; the difference is it only gathers the statistics of a single field.

Function queries

Unlike that of List statistics, function queries are not supported here due to character limitations in request paths.

Request

Path parameters

Parameter name Data type Required Description
stats-field string true The document field for which statistics should be generated.

Query parameters

Parameter name Data type Required Description
stats.calcdistinct boolean false If true, the countDistinct and distinctValues statistics will be computed and included the response. These calculations can be very expensive for fields that do not have a tiny cardinality, so they are disabled by default.

Aside from specifying stats.field, other parameters can be use to customize the resulting statistics - statistics query parameters in particular.

Global parameters and headers

For other parameters and headers, see:

Example

1
2
3
curl -X GET \
  'http://<host>:<port>/esbapi/v1/solr/core/invoke_monitor/stats/cost?q=*%3A*' \
  -H 'authorization: Bearer <access-token>'

Response

Content type

  • application/json
  • application/xml
  • text/plain
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
{
    "name": "cost",
    "min": 321,
    "max": 59482,
    "sum": 984732,
    "count": 10032,
    "missing": 0,
    "mean": 3,
    "sumOfSquares": 3245,
    "stddev": 2
}

Range faceting

  • Method: GET
  • Path: /esbapi/v1/solr/<package>/<core>/rangefacet/<facet-range>

According to the Solr's documentation on range faceting:

You can use range faceting on any date field or any numeric field that supports range queries. This is particularly useful for stitching together a series of range queries (as facet by query) for things like prices.

Working with dates

If you are going to work with date range facets, it's a good idea to read Solr's guide for working with dates.

Request

Path parameters

Parameter name Data type Required Description
facet_range string true Specifies the field to facet by range.

Query parameters

Parameter name Data type Required Description
facet.range.start date true Specifies the lower bound of the ranges.
facet.range.end date true Specifies the upper bound of the ranges.
facet.range.gap string true The span of each range expressed as a value to be added to the lower bound. For date fields, this should be expressed using DateMathParser's syntax.

Aside from the parameters specified above, you can use other Solr parameters to further refine your request - range faceting query parameters in particular.

Global parameters and headers

For other parameters and headers, see:

Example

1
2
3
curl -X GET \
  'http://<host>:<port>/esbapi/v1/solr/core/invoke_monitor/rangefacet/timeReceived?facet.range.start=2020-01-01T00%3A00%3A00Z&facet.range.end=2020-01-02T00%3A00%3A00Z&facet.range.gap=%2B8HOUR&q=*%3A*' \
  -H 'authorization: Bearer <access-token>'

The following parameters are present in the request above:

  • The facet-range path parameter is set to timeReceived. This is the field we indend to facet by range.
  • The facet.range.start query parameter is set to 2020-01-01T00:00:00Z. This means we'll only do range faceting for values of timeReceived that are on or after this date.
  • The facet.range.end query parameter is set to 2020-01-02T00:00:00Z. This means we'll only do range faceting for values of timeReceived that are on or before this date.
  • The facet.range.gap query parameter is set to +8HOURS which will be the interval gap between each range facet.
  • The q query parameter is set to *:* which means this will match all documents and we'll be applying range faceting for all of them.

Response

Content type

  • application/json
  • application/xml
  • text/plain

Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
    "name": "timeReceived",
    "start": 1577836800000,
    "end": 1577923200000,
    "gap": "+8HOUR",
    "range_value": [
        "2020-01-01T00:00:00Z",
        "2020-01-01T08:00:00Z",
        "2020-01-01T16:00:00Z"
    ],
    "range_count": [
        19,
        19,
        24
    ],
    "counts": [
        {
            "value": "2020-01-01T00:00:00Z",
            "count": 19
        },
        {
            "value": "2020-01-01T08:00:00Z",
            "count": 19
        },
        {
            "value": "2020-01-01T16:00:00Z",
            "count": 24
        }
    ]
}

Pivot faceting

  • Method: GET
  • Path: /esbapi/v1/solr/<package>/<core>/pivots

Queries the Solr index and generates a table summarizing the results. More accurately, according to the Solr's documentation:

Pivoting is a summarization tool that lets you automatically sort, count, total, or average data stored in a table. The results are typically displayed in a second table showing the summarized data. Pivot faceting lets you create a summary table of the results [from faceting] documents by multiple fields.

Request

Query parameters

Parameter name Data type Required Description
facet.pivot boolean true Defines the fields to use for the pivot. Defining multiple fields in a single facet.pivot parameter values will create multiple "facet_pivot" sections in the response, this is achieved by separating each field with a comma. You can also define multiple facet.pivot parameters as well to create multiple pivots.

Aside from the query parameter specified above, you can use other parameters to further refine your request - pivot faceting query parameters, in particular.

Global parameters and headers

For other parameters and headers, see:

Examples

1
2
3
curl -X GET \
  'http://<host>:<port>/esbapi/v1/solr/core/invoke_monitor/pivots?facet.pivot=packageName%2CendpointName&q=*' \
  -H 'authorization: Bearer <access-token>'

The following query parameters are present in the request above:

  • facet.pivot's value is set to packageName,endpointName and because of this, the endpoint (as will be seen in the response) will generate a more detailed summary by showing the inner pivot of the parent pivot.
  • q's value is * which means it'll match all documents without filtering.

Response

Content type

  • application/json
  • application/xml
  • text/plain

Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
{
    "facet_pivot": {
        "packageName,endpointName": {
            "field_name": "packageName,endpointName",
            "pivot": [
                {
                    "value": "Package_01",
                    "field": "packageName",
                    "pivot": [
                        {
                            "value": "Endpoint_03",
                            "field": "endpointName",
                            "count": 3205
                        },
                        {
                            "value": "Endpoint_02",
                            "field": "endpointName",
                            "count": 2304
                        }
                    ],
                    "count": 5509
                }
            ]
        }
    }
}

From the response above, we can then conclude that:

  • The total number of documents that have a packageName field with a value of Package_01 is 5509.
  • The total number of documents that have a packageName field with a value of Package_01 and an endpointName field with a value of Endpoint_02 is 2304.
  • The total number of documents that have a packageName field with a value of Package_01 and an endpointName field with a value of Endpoint_03 is 3205.