Skip to content

Overview

Concepts and Principles

Development

Overview

IDEs

API Explorer

Releases

Release Notes

TORO Integrate

Coder Studio

Coder Cloud

Bug Reports

Search

Creating, Reading, Updating, and Deleting Documents in your Custom Search Index

After you have successfully linked your Solr core to your Integrate Package, perhaps your next question is, "How do I add, edit, or delete documents in the index?"

In this guide, we will show you how to add, edit, or delete documents in a custom search index. We will be creating scripts that index and un-index movie data and we will discuss the objects and methods interacting in those scripts that make indexing and un-indexing possible. For simplicity's sake, the data we're going to index will be manually entered via parameters.

Stuff you need to know...

This guide assumes that you have gone through the basics of creating a custom Solr core or collection and Gloop/Groovy Services.

Get the code!

The scripts mentioned in this guide are available in the examples package.

Preparation

Before we get to indexing and un-indexing documents, we must ensure that our custom Solr core is already exposed to the Integrate Package we're going to use. These steps were discussed in here.

Here's the outline of our set-up:

  • Our package is called examples. This is where our scripts will reside.
  • Our target Solr core is embedded and named movie-core. As such, the directory structure of the examples package is:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    examples
    ├── classes
    ├── code
    ├── conf
    ├── web
    └── solr
        └── movie-core
            └── core.properties
            └── conf
                └── schema.xml
                └── solrconfig.xml
    
  • The examples package's esb-package.xml file was already edited to make the embedded Solr core known:

    1
    2
    3
    4
    5
    6
    7
    <esb-package>
        <!-- ... -->
        <solr-cores>
            <solr-core name="movie-core" enabled="true" />
            <!-- ... -->
        </solr-cores>
    </esb-package>
    

Procedures

Now that our package and custom Solr core are set up, we can proceed to creating our scripts. This guide is split into four sub-sections:

  • The first sub-section is about creating the model necessary to contain the data we need to index.
  • The second sub-section will show us how to actually index the data of our populated model.
  • The third sub-section will be about updating previously added documents.
  • The fourth and final sub-section will discuss how to remove documents from the index.

We will use Gloop and Groovy Services to do all those things for us.

Other examples...

Check out the examples package's search indices. There, you will see services that demonstrate the use of SolrMethods and other Solr related functionality.

Creating the Model

You can manually create your Gloop Model from scratch or you can extract the fields defined in schema.xml to create a model based from it. We will do the later in the SchemaToGloopModelGenerator service:

Model-generating Gloop service

We'll place this script in examples's code directory, under the solr.customSolrCore.model package. You should be able to use this script to parse your own schema.xml file but depending on your set up, you may need to tweak it a little more. Here's a breakdown of its steps:

  • In Line 1, we have a Map Step that calls GroovyMethods.getPackage() to get the Integrate Package the script is in. The return value is then stored in a variable called esbPackage.

  • In Line 2, we have another Map Step that declares and initializes a Path variable that points to schema.xml's location. We'll use esbPackage#getHome() as the base path and from there, we can traverse to schema.xml's actual location, like so:

    1
    Paths.get(esbPackage.getHome(), 'solr', 'movie-core', 'conf', 'schema.xml')
    
  • In Line 3, we have added a third Map Step but this time, we use it to declare and initialize a String variable containing schema.xml's content. We did that this way:

    1
    Files.readAllBytes(movieCorePath);
    
  • In Line 4, we create an Invoke Step that calls SolrMethods.solrSchemaToGloopModel(String, String, String, String, List<GloopModel>). This method will create the Gloop Model Movie, based on the schema.xml file, in solr.customSolrCore.model.

    1
    SolrMethods.solrSchemaToGloopModel("MovieDocument", schemaContent, null, "solr.customSolrCore.model", null)
    

All you have to do is run this service and viola! you now have your schema.xml-based Gloop Model! If you're following through our example, this will produce MovieDocument.model in solr.customSolrCore.model. We'll use this model later.

1
2
3
4
5
6
7
<package-name>
└── code
   └── solr
       └── customSolrCore
           └── model
               └── MovieDocument.model
               └── SchemaToGloopModelGenerator.gloop

The MovieDocument Gloop Model should have the following fields:

  • id (String)
  • movieTitle (String)
  • director (String)
  • cast (String[])

In this case, the Groovy Bean class MovieDocument.groovy will hold the movie data we want to index. We'll place it under the solr.customSolrCore.model package. Its content will be:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
package solr.customSolrCore.model

import org.apache.solr.client.solrj.beans.Field

class MovieDocument {

    @Field
    String movieTitle;

    @Field
    String director;

    @Field
    String[] cast;

}

The @Field annotations indicate which fields we want to index.

Indexing the Model's Data

Since our model is ready, we can now create a service that gets and indexes the model's data. We'll populate our models manually to make things simpler.

The MovieIndexer Gloop Service will be responsible for indexing our MovieDocument's data. Here's a preview of the steps we will have in this service:

The MovieIndexer service's steps

MovieIndexer's sole input parameter is called movieDocument which is based on the MovieDocument Gloop Model we created earlier. Because of this, we will be prompted to enter four fields when we run the service: id, movieTitle, director and casts. TORO Integrate will build the movieDocument parameter from our inputs and from there, we can index movieDocument via SolrMethods.index(String, String, GloopModel).

The bullet points below explain each step in the service:

  • In Line 1, we have a Block Step of type Try Catch. This allows Gloop to mirror Java's try-catch where it wraps the code that could possibly throw an exception in try block and performs a rescue in the catch block.
  • In Line 3, under the try block, we have an Invoke Step that calls SolrMethods.index(String, String, GloopModel). This is where the actual indexing will happen. It'll index movieDocument so that it will be available for querying in examples's movie-core Solr core later.
  • In Line 5, we have another Invoke Step that calls LoggerMethods.error(String) - this time, under the catch block. This'll just log the exception if anything goes wrong whilst indexing.

Running the service will prompt you to populate the required MovieIndexer model. You can enter whatever values you want to index. The service, if invoked successfully, should return a response similar to below:

MovieIndexer's sample response

This time, we'll create an endpoint whose parameters are to be mapped to the MovieDocument bean's fields. We can just call this Spring controller-based endpoint and the indexing will take place.

Simply create a Groovy file named MovieSolrAPI in solr.customSolrCore and edit it so that it contains the code below.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
package solr.customSolrCore

import java.util.Map
import javax.servlet.http.HttpServletRequest

import org.springframework.web.bind.annotation.*
import org.springframework.web.bind.annotation.RequestMethod
import org.springframework.http.MediaType
import org.apache.solr.servlet.SolrRequestParsers

import io.toro.integrate.core.service.annotation.InputType
import io.toro.integrate.core.service.annotation.InputTypeField

@RestController
@RequestMapping('/solr-package')
class MovieSolrAPI {
    @RequestMapping(produces = MediaType.TEXT_PLAIN_VALUE, method = RequestMethod.POST)
    def addDocument(@RequestParam String movieTitle,
                    @RequestParam String director, 
                    @RequestParam String[] casts) {
        def document = new MovieDocument(movieTitle: movieTitle, director: director, cast: casts)
        'movie-core'.index( null, document ).toString()
    }
}

As you may notice, in:

  • Line 21, we constructed a MovieDocument object (document variable) from the parameters of our request.
  • Line 22, we used SolrMethods.index(String, String, GloopModel) method, a one-liner, to index the data for us. We subsequently called the GloopMethod#toString() method so that our endpoint's response is the indexed MovieDocument model.

With that said, a call to the endpoint will trigger the indexing of your movie data.

1

Updating Documents

There are three ways to update a Solr document in TORO Integrate:

  • The update-document one-liner method (preferred)

    True to its purpose, using the one-liner method is the easiest way to update an existing document.

  • SolrJ's SolrClient

    A call to the one-liner method SolrMethods.solr(String) returns a SolrClient object which you can use to directly interact with the Solr core tied to it (specified by passing the name of the core as the argument). However, to use SolrClient, one must be familiar with SolrJ and Groovy.

    1
    // Insert example using SolrClient to update a document.
    
  • The Search API

    Alternatively, you can use TORO Integrate's Search REST API to update your document. This will involve the use of Solr's UpdateRequestHandlers via the native or Solr-derived endpoint.

    1
    // Insert same request here.
    

Removing Documents

There are three ways to delete a Solr document in TORO Integrate:

  • The delete-document one-liner method (preferred)

    True to its purpose, using the one-liner method is the easiest way to update an existing document.

  • SolrJ's SolrClient

    A call to the one-liner method SolrMethods.solr(String) returns a SolrClient object which you can use to directly interact with the Solr core tied to it (specified by passing the name of the core as the argument). However, to use SolrClient, one must be familiar with SolrJ and Groovy.

    1
    // Insert example using SolrClient to update a document.
    
  • The Search API

    Alternatively, you can use TORO Integrate's Search REST API to delete your document. This will involve the use of Solr's UpdateRequestHandlers via the native or Solr-derived endpoint.

    1
    // Insert same request here.