Understanding Elastic Common Schema

1. Understanding ECS Purpose

Aspect	Description	Benefit
Standardization	Provides a common structure for log data across different sources and systems.	Enables unified analysis and correlation of logs from various platforms.
Flexibility	Supports both structured and unstructured data with extensible fields.	Allows organizations to adapt ECS to their specific use cases without losing interoperability.
Mapping	Maps fields from various sources (e.g., logs, metrics) to ECS schema.	Facilitates seamless integration with Elastic Stack components like Kibana and Elasticsearch.
Versioning	Uses versioned schemas to maintain backward compatibility.	Ensures that updates don't break existing workflows or dashboards.

Example: ECS Field Mapping

{
  "event": {
    "category": "web",
    "type": "access"
  },
  "url": {
    "full": "https://example.com/page?query=123"
  },
  "user_agent": {
    "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
  }
}

Note: ECS is not a replacement for your existing logging systems but rather a way to enrich and standardize the data they produce.

2. Using Core Field Sets

Field Set	Description	Common Fields	Usage
agent	Information about the agent (a piece of software responsible for collecting and reporting data).	`agent.name`, `agent.version`, `agent.type`	Used to identify the agent responsible for generating logs or metrics.
event	Information about the event itself, such as its type and category.	`event.category`, `event.type`, `event.action`	Used to categorize and classify events for filtering and analysis.
host	Information about the host machine where the event originated.	`host.name`, `host.id`, `host.os.name`	Used to identify the host and its operating system for context.
user	Information about the user associated with the event.	`user.name`, `user.id`, `user.email`	Used to track user activity and identify individuals in logs.
source	Information about the source of the event, such as IP address or port.	`source.ip`, `source.port`, `source.bytes`	Used to identify the origin of network traffic or data sources.

Example: Using Core Field Sets in a Log Entry

{
  "agent": {
    "name": "filebeat",
    "version": "8.10.0"
  },
  "event": {
    "category": "file",
    "type": "access"
  },
  "host": {
    "name": "web-server-01",
    "os": {
      "name": "Linux"
    }
  },
  "user": {
    "name": "john.doe",
    "id": "12345"
  },
  "source": {
    "ip": "192.168.1.100",
    "port": 54321
  }
}

Note: Core field sets are designed to be consistent across different types of data, making it easier to correlate and analyze logs from various sources.

3. Using Event Fields

Field	Type	Description	Example
event.action	keyword	Identifies the action being performed	`"create"`
event.category	keyword	Category of the event (e.g., network, file, process)	`"file"`
event.id	keyword	Unique identifier for the event	`"1234567890"`
event.severity	integer	Numeric severity level (1-10)	`5`
event.timestamp	date	Timestamp of the event in ISO 8601 format	`"2025-04-05T14:30:00Z"`

Example: Log Entry with Event Fields

{
  "event": {
    "action": "create",
    "category": "file",
    "id": "1234567890",
    "severity": 5,
    "timestamp": "2025-04-05T14:30:00Z"
  },
  "file": {
    "name": "example.txt",
    "path": "/var/log/example.txt"
  }
}

Note: The Elastic Common Schema (ECS) provides a standardized structure for logs and events, ensuring consistency across different data sources.

4. Using Network Fields

Field	Type	Description	Example
network.direction	`string`	Direction of the network traffic (e.g., `inbound`, `outbound`)	`"inbound"`
network.transport	`string`	Transport protocol used (e.g., `tcp`, `udp`)	`"tcp"`
network.type	`string`	Network type (e.g., `ipv4`, `ipv6`)	`"ipv4"`
network.community_id	`string`	Community ID for tracking flows across systems	`"1:z5XhIuV2d8pQk9vLmN3oR4sT7wY8zA1b2c3d4e5f6g7h8i9j0kLmN"`

Example: Log Entry with Network Fields

{
  "network": {
    "direction": "inbound",
    "transport": "tcp",
    "type": "ipv4",
    "community_id": "1:z5XhIuV2d8pQk9vLmN3oR4sT7wY8zA1b2c3d4e5f6g7h8i9j0kLmN"
  },
  "source": {
    "ip": "192.168.1.100",
    "port": 54321
  },
  "destination": {
    "ip": "10.0.0.1",
    "port": 80
  }
}

Note: The network.community_id field is useful for correlating network events across different systems in a distributed environment.

5. Using Host Fields

Field	Type	Description	Example
host.id	keyword	Unique identifier of the host machine	`4e3d9e8b-2a5c-4f1e-a7b2-c9d0e1f2g3h4`
host.name	keyword	Hostname of the host machine	`web-server-01`
host.ip	ip	IP address of the host machine	`192.168.1.100`
host.mac	keyword	MAC address of the host machine	`00:1A:2B:3C:4D:5E`
host.architecture	keyword	Architecture of the host machine (e.g., x86_64)	`x86_64`
host.os.name	keyword	Name of the operating system (e.g., Linux, Windows)	`Linux`
host.os.version	keyword	Version of the operating system	`5.4.0-123-generic`
host.os.platform	keyword	Platform of the operating system (e.g., ubuntu, centos)	`ubuntu`

Example: Log Entry with Host Fields

{
  "host": {
    "id": "4e3d9e8b-2a5c-4f1e-a7b2-c9d0e1f2g3h4",
    "name": "web-server-01",
    "ip": "192.168.1.100",
    "mac": "00:1A:2B:3C:4D:5E",
    "architecture": "x86_64",
    "os": {
      "name": "Linux",
      "version": "5.4.0-123-generic",
      "platform": "ubuntu"
    }
  },
  "message": "User login successful",
  "timestamp": "2025-04-05T12:34:56Z"
}

Note: Host fields are essential for correlating logs across systems and identifying the source of events in the ELK stack.

6. Using User Fields

Field	Type	Description	Example
user.id	keyword	Unique identifier for the user	`"12345"`
user.name	keyword	Username or login name of the user	`"john_doe"`
user.email	keyword	Email address of the user	`"john.doe@example.com"`
user.full_name	text	Full name of the user	`"John Doe"`
user.domain	keyword	Domain or realm the user belongs to	`"example.com"`

Example: User Fields in Log Event

{
  "user": {
    "id": "12345",
    "name": "john_doe",
    "email": "john.doe@example.com",
    "full_name": "John Doe",
    "domain": "example.com"
  }
}

Note: User fields should be populated consistently across all logs to ensure accurate user-based analytics and monitoring.

7. Using File Fields

Field	Type	Description	Example
file.name	keyword	Name of the file (including extension)	`app.log`
file.path	keyword	Full path to the file	`/var/log/app.log`
file.size	long	Size of the file in bytes	`1024`
file.extension	keyword	Extension of the file	`log`
file.created	date	Timestamp when the file was created	`2023-10-01T12:00:00Z`
file.modified	date	Timestamp when the file was last modified	`2023-10-05T14:30:00Z`

Example: Log file metadata

{
  "file": {
    "name": "application.log",
    "path": "/var/log/application.log",
    "size": 5120,
    "extension": "log",
    "created": "2023-10-01T12:00:00Z",
    "modified": "2023-10-05T14:30:00Z"
  }
}

8. Using Process Fields

Field	Type	Description	Example
process.name	keyword	Name of the process	`java`
process.pid	integer	Process ID	`12345`
process.ppid	integer	Parent Process ID	`1234`
process.args	keyword array	Command line arguments	`["-jar", "app.jar"]`
process.executable	keyword	Path to the executable file	`/usr/bin/java`
process.title	keyword	Title of the process	`java -jar app.jar`

Example: Process Fields in Log Entry

{
  "process": {
    "name": "java",
    "pid": 12345,
    "ppid": 1234,
    "args": ["-jar", "app.jar"],
    "executable": "/usr/bin/java",
    "title": "java -jar app.jar"
  }
}

9. Mapping Custom Data to ECS

Field	ECS Mapping	Description
custom.field.name	`custom.field.name`	Custom field added to ECS document
event.category	`event.category`	Category of event (e.g., "process", "network")
event.type	`event.type`	Type of event (e.g., "start", "end")

Example: Custom Field Mapping

{
  "custom": {
    "field": {
      "name": "example_value"
    }
  },
  "event": {
    "category": "process",
    "type": "start"
  }
}

Note: When mapping custom data to ECS, ensure that field names follow ECS naming conventions and do not conflict with existing ECS fields.

10. Using ECS in Beats

Field	Description	Example Value
ecs.version	Version of the ECS used in the event	`1.12.0`
event.dataset	Identifier for the dataset that generated the event	`apache.access`
event.module	Name of the module that generated the event	`apache`
event.type	Type of event (e.g., "info", "error")	`access`
host.name	Hostname of the system generating the event	`web01.example.com`
source.ip	IP address of the source system	`192.168.1.100`
destination.ip	IP address of the destination system	`10.0.0.50`
user.name	Username of the user associated with the event	`johndoe`

Example: ECS Mapping in Apache Beat

output.elasticsearch:
  hosts: ["localhost:9200"]
  index: "ecs-apache-%{+yyyy.MM.dd}"

processors:
  - add_fields:
      target: ecs
      fields:
        version: "1.12.0"

Note: Ensure that the Beat configuration includes ECS field mappings to align with the schema.

11. Using ECS in Logstash

Field	Description	Example
ecs.version	Version of the ECS used in the event	`1.12.0`
event.dataset	Dataset name for the event	`apache.access`
event.module	Module that generated the event	`apache`
host.name	Name of the host generating the event	`web01.example.com`
log.level	Log level of the event	`INFO`

Example: ECS Mapping in Logstash

filter {
  grok {
    match => { "message" => "%{IP:clientip} %{WORD:method} %{URIPATH:uri}" }
  }
  mutate {
    add_field => { "ecs.version" => "1.12.0" }
    add_field => { "event.module" => "apache" }
    add_field => { "event.dataset" => "apache.access" }
  }
}

Note: Ensure that all ECS fields are properly mapped in your Logstash configuration to maintain consistency across logs.

12. Validating ECS Compliance

Validation Method	Description	Tool/Command	Notes
ECS Schema Validation	Checks if logs or events conform to ECS field definitions	`ecs-validate`	Available as a CLI tool from Elastic
Elasticsearch Ingest Pipeline	Validate ECS compliance during log ingestion using ingest pipelines	`PUT /_ingest/pipeline/ecs-validation`	Can be configured to reject non-compliant documents
ECS Validator Plugin	Plugin for Logstash to validate ECS fields	`logstash-filter-ecs-validator`	Can be used in a Logstash pipeline to validate ECS fields

Example: Using ecs-validate CLI tool

ecs-validate --schema-version 1.12.0 --input-file /path/to/log.json

Note: The ecs-validate tool requires a schema version to validate against. Ensure that the schema version matches your ECS version.