Indexing Documents

1. Indexing Single Document

Field	Description	Example
index	Name of the index to which the document will be added	`my-index`
id	Unique identifier for the document (optional)	`12345`
body	Document content as JSON object	`{ "title": "Sample Document", "content": "This is a sample document content.", "tags": ["elk", "stack"] }`

Example: Indexing a Single Document

POST /my-index/_doc/12345
{
  "title": "Sample Document",
  "content": "This is a sample document content.",
  "tags": ["elk", "stack"]
}

2. Indexing with Auto-Generated ID

Field	Description	Example
_id	Unique identifier for the document in the index	`"5f8d3c4b1a2b3c4d5e6f7a8b"`
index	Name of the index where the document is stored	`"logs-2024-05"`
type	Document type (deprecated in Elasticsearch 7.x+)	`"doc"`

Example: Indexing a Document with Auto-Generated ID

POST /my_index/_doc
{
  "message": "Application started successfully",
  "timestamp": "2024-05-10T12:00:00Z",
  "level": "INFO"
}

Note: When no _id is provided, Elasticsearch auto-generates a unique ID for the document.

Warning: Auto-generated IDs are not guaranteed to be sequential or predictable.

3. Indexing with Custom ID

Field	Description	Example
_id	Unique identifier for the document in the index	`"my-custom-id"`
index	Name of the index where the document is stored	`"my-index"`
body	Document content to be indexed	`{ "name": "John Doe", "email": "john@example.com" }`

Example: Indexing a Document with Custom ID

const { Client } = require('@elastic/elasticsearch');
const client = new Client({ node: 'http://localhost:9200' });

async function indexWithCustomId() {
  const result = await client.index({
    index: 'users',
    id: 'user-123',
    body: {
      name: 'John Doe',
      email: 'john@example.com'
    }
  });

  console.log(result);
}

indexWithCustomId();

Note: If _id is not provided, Elasticsearch will auto-generate a unique ID.

4. Using Bulk Indexing

Bulk indexing allows you to index multiple documents in a single request, improving performance and reducing overhead.

Field	Description	Values
index	Name of the index to which documents are added	`my-index`
id	Unique identifier for the document (optional)	`12345`
doc	Document content to be indexed	`{ "name": "John Doe", "age": 30 }`

Example: Bulk Indexing Documents

const { Client } = require('@elastic/elasticsearch');
const client = new Client({ node: 'http://localhost:9200' });

async function bulkIndex() {
  const body = [
    { index: { _index: 'my-index', _id: '1' } },
    { name: 'John Doe', age: 30 },
    { index: { _index: 'my-index', _id: '2' } },
    { name: 'Jane Smith', age: 25 }
  ];

  try {
    const response = await client.bulk({ body });
    console.log('Bulk index response:', response);
  } catch (error) {
    console.error('Error during bulk indexing:', error);
  }
}

bulkIndex();

Note: Bulk operations are atomic per batch, but individual document failures do not stop the entire operation.

5. Updating Documents

Operation	Description	Notes
index	`POST /{index}/_doc/{id}`	Creates or updates a document with the specified ID
update	`POST /{index}/_update/{id}`	Updates the document using a script or partial document
upsert	`POST /{index}/_doc/{id}?doc_as_upsert=true`	Creates a new document if it doesn't exist, otherwise updates it

Example: Update Document with Script

{
  "script": {
    "source": "ctx._source.views += 1"
  },
  "upsert": {
    "views": 1
  }
}

Example: Partial Document Update

{
  &quot doc": {
    "status": "published"
  }
}

6. Using Scripted Updates

Scripted updates in the ELK Stack allow you to modify documents using Painless scripting language. These scripts can update fields, perform calculations, or conditionally change document content.

Field	Description	Example
script	Defines the Painless script to execute	`"script": { "source": "ctx._source.field += 1" }`
lang	Scripting language (default is Painless)	`"lang": "painless"`
params	Parameters to pass into the script	`"params": { "increment": 5 }`

Example: Incrementing a Field Value

{
  "script": {
    "source": "ctx._source.counter += params.increment",
    "params": {
      "increment": 10
    }
  }
}

Example: Conditional Update Based on Field Value

{
  "script": {
    "source": """
      if (ctx._source.status == params.oldStatus) {
        ctx._source.status = params.newStatus;
        ctx._source.updatedAt = params.timestamp;
      }
    """,
    "params": {
      "oldStatus": "pending",
      "newStatus": "completed",
      "timestamp": "2026-04-05T10:00:00Z"
    }
  }
}

Note: Painless scripts run in the context of the document being updated and have access to ctx._source for modifying fields.

Warning: Be cautious with complex scripts as they may impact performance or cause unexpected behavior. Always test scripts in a development environment.

7. Upserting Documents

Operation	Description	Notes
index	`POST /_doc/{id}`	Creates or updates a document with the specified ID
upsert	`POST /_update/{id}`	Updates a document if it exists, or creates it if it doesn't

Example: Upsert a Document

{
  "doc": {
    "name": "John Doe",
    "age": 35
  },
  "doc_as_upsert": true
}

Note: If doc_as_upsert is true and the document does not exist, it will be created with the content from doc.

Example: Using External ID

POST /my_index/_upsert/my_external_id
{
  "doc": {
    "name": "Jane Smith",
    "department": "Engineering"
  },
  "doc_as_upsert": true
}

8. Deleting Documents

Operation	Description	Notes
delete_by_query	Deletes documents matching a query	Can delete multiple documents in one operation
delete_by_id	Deletes a document by its ID	Fastest way to delete a single document

Example: Delete by ID

DELETE /my-index/_doc/123

Example: Delete by Query

POST /my-index/_delete_by_query
{
  "query": {
    "match": {
      "status": "inactive"
    }
  }
}

Note: Deletion is not immediate; it's marked as deleted and removed during merge operations.

9. Deleting by Query

Option	Values	Notes
query	`match_all`	Deletes all documents in the index
query	`term`	Deletes documents matching a specific term
query	`range`	Deletes documents within a range of values

Example: Delete all documents with a specific field value

{
  "query": {
    "term": {
      "status": "inactive"
    }
  }
}

Example: Delete documents older than a specific date

{
  "query": {
    "range": {
      "created_date": {
        "lt": "2023-01-01"
      }
    }
  }
}

Note: The delete-by-query API is deprecated in favor of the delete_by_query API in newer versions of Elasticsearch.

Warning: Be cautious when using match_all as it will delete all documents in the index.

10. Setting Document Versioning

Property	Description	Values	Notes
_version	Document version number	`integer`	Used for optimistic concurrency control
if_seq_no	Sequence number for document update	`integer`	Must match current sequence number to update
if_primary_term	Primary term for document update	`integer`	Must match current primary term to update

Example: Update Document with Version Check

{
  "doc": {
    "title": "Updated Document"
  },
  "if_seq_no": 5,
  "if_primary_term": 1
}

Note: Versioning helps prevent overwrites when multiple updates occur simultaneously.

11. Configuring Routing

Property	Description	Default	Notes
index.routing.allocation.enable	Controls allocation of shards based on routing	`all`	Valid values: `none`, `primaries`, `replicas`, `all`
index.routing.rebalance.enable	Enables rebalancing of shards based on routing	`all`	Valid values: `none`, `primaries`, `replicas`, `all`
index.routing.allocation.include._ip	Includes nodes matching the IP address in routing	`null`	Can be used to route documents to specific nodes by IP
index.routing.allocation.include._id	Includes nodes matching the node ID in routing	`null`	Can be used to route documents to specific nodes by ID

Example: Routing with Custom Field

{
  "settings": {
    "index.routing.allocation.include._ip": "192.168.1.10",
    "index.routing.rebalance.enable": "primaries"
  },
  "mappings": {
    "properties": {
      "user_id": {
        "type": "keyword"
      }
    }
  }
}

Note: Routing is particularly useful for performance optimization when you have a large number of documents and want to ensure related documents are stored on the same shard.

12. Handling Conflicts

Conflict Type	Description	Solution
Version Conflicts	When a document is updated with an older version number, Elasticsearch rejects the update.	Use `if_seq_no` and `if_primary_term` to handle versioning.
Concurrent Updates	Multiple updates to the same document can cause conflicts if not managed properly.	Implement retry logic with exponential backoff or use optimistic concurrency control.

Example: Handling Version Conflicts

const updateDoc = {
  script: {
    source: "ctx._source.field = params.newValue",
    params: {
      newValue: "updated_value"
    }
  },
  upsert: {
    field: "updated_value"
  },
  if_seq_no: 123,
  if_primary_term: 456
};

Note: If a conflict occurs, Elasticsearch throws a VersionConflictEngineException. Always handle this in your application logic.