Skip to content

Gloop Model Array Icon

Gloop Cursors

Occasionally when writing software you need to deal with millions of rows of data, not thousands. When dealing with this much data, it's very easy to run out of memory and/or reduce the performance of your code. To get around this, software sometimes uses the concept of a Cursor, or Iterator, which is a pointer to a single record in a list of data. This way all the data isn't stored in memory. Instead, as the data is processed, old records in memory are replaced with new records. Gloop is no different to this. Gloop Cursors will appear as an array in Coder so you can still use it the same way as any other Gloop Object array, but it's important to know that Gloop Cursors come with some limitations:

  • They can only be iterated over once
  • Some cursors can only be read from
  • Some cursors can only be written to

Currently, Gloop uses cursors under the following conditions only:

  • Models returned from Gloop Batch SQL Services are cursors that can only be written to
  • Models returned from Gloop Multi-Select SQL Services are cursors that can only be read from
  • Models returned from flat file reading and writing services
  • Reading large plain-text files via FileMethods.getTextFileInputCursor(...)
  • Reading large XML files via XmlMethods.getInputCursorFromFile(...) or XmlMethods.getInputCursorFromInputStream(...)
  • Reading large JSON files via JsonMethods.getInputCursorFromFile(...) or JsonMethods.getInputCursorFromInputStream(...)
  • Writing large JSON files via JsonMethods.openJsonOutputCursor(...)

Below is a screenshot showing a service that returns a Gloop Cursor, but still shows as an array in Coder:

Gloop Cursor Output in Coder Studio

Gloop Cursor Output in Coder

When cursors are used as the input and outputs of Iterate and While Steps, they will automatically read and write from the cursors for you. However, if you need to use the cursors manually, you can use the Gloop Cursor-based methods in the io.toro.integrate.GloopMethods class (which is in the core package). This class includes methods for reading, writing, and closing Gloop Cursors.

Examples

There are Gloop Services in the examples package that demonstrate how to use Gloop Cursors, they are:

  • All the services in the sqlServices.invoke.multiple package
  • flatfile.ReadFile
  • flatfile.WriteFile
  • jsonCursor.input.ReadAndWriteSimpleJson
  • jsonCursor.output.WriteComplexJson
  • xmlCursor.ReadLargeXmlFile