Tuesday January 27, 2015 - tags:    node.js, grunt, elasticsearch

Elasticsearch bulk api with Grunt

The following post will demonstrate working with the elasticsearch bulk api, via grunt and grunt-elasticsearch-bulk. This is useful if; for example, you would like to perform integration testing against an elasticsearch instance, or for simply seeding data for a development environment.

grunt-elasticsearch-bulk uses the official elasticsearch.js low-level Elasticsearch client.

Getting Started

npm install grunt-elasticsearch-bulk --save-dev  

Once the plugin has been installed, it may be enabled inside your Gruntfile with this line of JavaScript:

grunt.loadNpmTasks('grunt-elasticsearch-bulk');  

In your project's Gruntfile, add a section named elasticsearchbulk to the data object passed into grunt.initConfig().

The src element will run through the files in sequence, so for example, if you wanted to run an integration test, you could as an example, delete any existing data, and then recreate records into elasticsearch to run your tests against.

grunt-elasticsearch-bulk uses elasticsearch.js; so simply provide the options used there; where relevant for bulk api:

http://www.elasticsearch.org/guide/en/elasticsearch/client/javascript-api/current/configuration.html

grunt.initConfig({  
    elasticsearchbulk: {
      src: {
        src : [ 'delete.json', 'create.json', 'update.json' ]
      },
      options: {      
        elastic: {
          host: "http://127.0.0.1:9200",
          client: {
            apiVersion: "1.0",
            log: "info"
          }
        }
      }
    }
});

Working with the elasticsearch bulk api

The elasticsearch bulk api allows you to add, update and remove data.

Create record

The below json file create.json, contains a dataset which creates three records.

[
{ "create" : { "_index" : "user", "_type" : "editor", "_id" : "e1" } },
{
    "name": "Indie",
    "created": "2014-08-06T13:51:19+0100",
    "role": "editor"
},
{ "create" : { "_index" : "user", "_type" : "contributor", "_id" : "c1" } },
{
    "name": "Rima",
    "created": "2014-08-06T13:51:19+0100",
    "role": "contributor"
}
]

Delete record

The following dataset delete.json deletes an existing record.

[
{ "delete" : { "_index" : "user", "_type" : "contributor", "_id" : "c1" } }
]

Update record

The following dataset update.json updates an existing record.

[
{ "update" :  { "_index" : "user", "_type" : "editor", "_id" : "e1" } }
{
    "name": "Indie",
    "created": "2014-08-06T13:51:19+0100",
    "role": "admin"
}
]

More Information

Please refer to elasticsearch.js documentation here:
http://www.elasticsearch.org/guide/en/elasticsearch/client/javascript-api/current/api-reference.html

And the docs for bulk api here:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/1.4/docs-bulk.html