Skip to content

Commit 4ebef2e

Browse files
AlexRuiz7AlexRuiz7
committed
Add ECS mappings generator (#36)
* Add ECS mappings generator, documentation and files for vulnerability detector * Add event generator script * Update template settings --------- Signed-off-by: Álex Ruiz <[email protected]>
1 parent 435b0df commit 4ebef2e

File tree

7 files changed

+484
-0
lines changed

7 files changed

+484
-0
lines changed

‎ecs/.gitignore

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
**/mappings
2+
*.log
3+
generatedData.json

‎ecs/README.md

+104
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
## ECS mappings generator
2+
3+
This script generates the ECS mappings for the Wazuh indices.
4+
5+
### Requirements
6+
7+
- ECS repository clone. The script is meant to be launched from the root level of that repository.
8+
- Python 3.6 or higher
9+
- jq
10+
11+
### Folder structure
12+
13+
There is a folder for each module. Inside each folder, there is a `fields` folder with the required
14+
files to generate the mappings. These are the inputs for the ECS generator.
15+
16+
### Usage
17+
18+
**Copy the `generate.sh` script to the root level of the ECS repository.**
19+
20+
Use the `generate.sh` script to generate the mappings for a module. The script takes 3 arguments,
21+
plus 2 optional arguments to upload the mappings to the Wazuh indexer (using **composable** indexes).
22+
23+
```plaintext
24+
Usage: ./generate.sh <ECS_VERSION> <INDEXER_SRC> <MODULE> [--upload <URL>]
25+
* ECS_VERSION: ECS version to generate mappings for
26+
* INDEXER_SRC: Path to the wazuh-indexer repository
27+
* MODULE: Module to generate mappings for
28+
* --upload <URL>: Upload generated index template to the OpenSearch cluster. Defaults to https://localhost:9200
29+
Example: ./generate.sh v8.10.0 ~/wazuh-indexer vulnerability-detector --upload https://indexer:9200
30+
```
31+
32+
For example, to generate the mappings for the `vulnerability-detector` module using the
33+
ECS version `v8.10.0` and the Wazuh indexer in path `~/wazuh/wazuh-indexer`:
34+
35+
```bash
36+
./generate.sh v8.10.0 ~/wazuh/wazuh-indexer vulnerability-detector
37+
```
38+
39+
### Output
40+
41+
A new `mappings` folder will be created inside the module folder, containing all the generated files.
42+
The files are versioned using the ECS version, so different versions of the same module can be generated.
43+
For our use case, the most important files are under `mappings/<ECS_VERSION>/generated/elasticsearch/legacy/`:
44+
45+
- `template.json`: Elasticsearch compatible index template for the module
46+
- `opensearch-template.json`: OpenSearch compatible index template for the module
47+
48+
The original output is `template.json`, which is not compatible with OpenSearch by default. In order
49+
to make this template compatible with OpenSearch, the following changes are made:
50+
51+
- the `order` property is renamed to `priority`.
52+
- the `mappings` and `settings` properties are nested under the `template` property.
53+
54+
The script takes care of these changes automatically, generating the `opensearch-template.json` file as a result.
55+
56+
### Upload
57+
58+
You can either upload the index template using cURL or the UI (dev tools).
59+
60+
```bash
61+
curl -u admin:admin -k -X PUT "https://indexer:9200/_index_template/wazuh-vulnerability-detector" -H "Content-Type: application/json" -d @opensearch-template.json
62+
```
63+
64+
Notes:
65+
- PUT and POST are interchangeable.
66+
- The name of the index template does not matter. Any name can be used.
67+
- Adjust credentials and URL accordingly.
68+
69+
### Adding new mappings
70+
71+
The easiest way to create mappings for a new module is to take a previous one as a base.
72+
Copy a folder and rename it to the new module name. Then, edit the `fields` files to
73+
match the new module fields.
74+
75+
The name of the folder will be the name of the module to be passed to the script. All 3 files
76+
are required.
77+
78+
- `fields/subset.yml`: This file contains the subset of ECS fields to be used for the module.
79+
- `fields/template-settings-legacy.json`: This file contains the legacy template settings for the module.
80+
- `fields/template-settings.json`: This file contains the composable template settings for the module.
81+
82+
### Event generator
83+
84+
For testing purposes, the script `generate_events.py` can be used to generate events for a given module.
85+
Currently, it is only able to generate events for the `vulnerability-detector` module. To support other
86+
modules, please extend or refactor the script.
87+
88+
The script prompts for the required parameters, so it can be launched without arguments:
89+
90+
```bash
91+
./event_generator.py
92+
```
93+
94+
The script will generate a JSON file with the events, and will also ask whether to upload them to the
95+
indexer. If the upload option is selected, the script will ask for the indexer URL and port, credentials,
96+
and index name.
97+
98+
The script uses a log file. Check it out for debugging or additional information.
99+
100+
#### References
101+
102+
- [ECS repository](https://github.com/elastic/ecs)
103+
- [ECS usage](https://github.com/elastic/ecs/blob/main/USAGE.md)
104+
- [ECS field reference](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html)

‎ecs/generate.sh

+83
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
#!/bin/bash
2+
3+
# Function to display usage information
4+
show_usage() {
5+
echo "Usage: $0 <ECS_VERSION> <INDEXER_SRC> <MODULE> [--upload <URL>]"
6+
echo " * ECS_VERSION: ECS version to generate mappings for"
7+
echo " * INDEXER_SRC: Path to the wazuh-indexer repository"
8+
echo " * MODULE: Module to generate mappings for"
9+
echo " * --upload <URL>: Upload generated index template to the OpenSearch cluster. Defaults to https://localhost:9200"
10+
echo "Example: $0 v8.10.0 ~/wazuh-indexer vulnerability-detector --upload https://indexer:9200"
11+
}
12+
13+
# Function to generate mappings
14+
generate_mappings() {
15+
ECS_VERSION="$1"
16+
INDEXER_SRC="$2"
17+
MODULE="$3"
18+
UPLOAD="$4"
19+
URL="$5"
20+
21+
IN_FILES_DIR="$INDEXER_SRC/ecs/$MODULE/fields"
22+
OUT_DIR="$INDEXER_SRC/ecs/$MODULE/mappings/$ECS_VERSION"
23+
24+
# Ensure the output directory exists
25+
mkdir -p "$OUT_DIR" || exit 1
26+
27+
# Generate mappings
28+
python scripts/generator.py --strict --ref "$ECS_VERSION" \
29+
--subset "$IN_FILES_DIR/subset.yml" \
30+
--template-settings "$IN_FILES_DIR/template-settings.json" \
31+
--template-settings-legacy "$IN_FILES_DIR/template-settings-legacy.json" \
32+
--out "$OUT_DIR" || exit 1
33+
34+
# Replace "match_only_text" type (not supported by OpenSearch) with "text"
35+
echo "Replacing \"match_only_text\" type with \"text\""
36+
find "$OUT_DIR" -type f -exec sed -i 's/match_only_text/text/g' {} \;
37+
38+
# Transform legacy index template for OpenSearch compatibility
39+
cat "$OUT_DIR/generated/elasticsearch/legacy/template.json" | jq '{
40+
"index_patterns": .index_patterns,
41+
"priority": .order,
42+
"template": {
43+
"settings": .settings,
44+
"mappings": .mappings
45+
}
46+
}' >"$OUT_DIR/generated/elasticsearch/legacy/opensearch-template.json"
47+
48+
# Check if the --upload flag has been provided
49+
if [ "$UPLOAD" == "--upload" ]; then
50+
upload_mappings "$OUT_DIR" "$URL" || exit 1
51+
fi
52+
53+
echo "Mappings saved to $OUT_DIR"
54+
}
55+
56+
# Function to upload generated composable index template to the OpenSearch cluster
57+
upload_mappings() {
58+
OUT_DIR="$1"
59+
URL="$2"
60+
61+
echo "Uploading index template to the OpenSearch cluster"
62+
for file in "$OUT_DIR/generated/elasticsearch/composable/component"/*.json; do
63+
component_name=$(basename "$file" .json)
64+
echo "Uploading $component_name"
65+
curl -u admin:admin -X PUT "$URL/_component_template/$component_name?pretty" -H 'Content-Type: application/json' -d@"$file" || exit 1
66+
done
67+
}
68+
69+
# Check if the minimum required arguments have been provided
70+
if [ $# -lt 3 ]; then
71+
show_usage
72+
exit 1
73+
fi
74+
75+
# Parse command line arguments
76+
ECS_VERSION="$1"
77+
INDEXER_SRC="$2"
78+
MODULE="$3"
79+
UPLOAD="${4:-false}"
80+
URL="${5:-https://localhost:9200}"
81+
82+
# Generate mappings
83+
generate_mappings "$ECS_VERSION" "$INDEXER_SRC" "$MODULE" "$UPLOAD" "$URL"

0 commit comments

Comments
 (0)