Skip to content

Commit 479fbbf

Browse files
AlexRuiz7f-galland
AlexRuiz7
and
f-galland
committed
Add Python module to accomplish OCSF compliant events (#159)
* Adding Python script that receives a continuous json stream over stdin and outputs parquet to Security Lake * Adding logstash pipeline for python script * encode_parquet() function fixed to handle lists of dictionaries * Correct error in encode_parquet() * Avoid storing the block ending in the output buffer * Add comments on handling files and streams with pyarrow for future reference * Add s3 handling reference links * Write parquet directly to bucket * Added basics of map_to_ocsf() function * Minor fixes * Map alerts to OCSF as they are read * Add script to convert Wazuh events to OCSF Also adds a simple test script * Add OCSF converter + Parquet encoder + test scripts * Update .gitignore * Include the contents of the alert under unmapped * Add support for different OCSF schema versions * Use custom ocsf module to map alerts * Modify script to use converter class * Code polish and fix errors * Remove unnecessary type declaration from debug flag * Improved parquet encoding * Initial commit for test env's docker-compose.yml * Remove sudo references from docker-compose.yml * Add operational Python module to transform events to OCSF * Create minimal Docker environment to test and develop the integration. * Fix events-generator's Inventory starvation * Remove files present in #147 * Cleanup * Add FQDN hostnames to services for certificates creation * Add S3 Ninja (Mock) (#165) * Setup certificates in Wazuh Indexer and Logstash containers (#166) * Add certificate generator service * Add certificate config to docker compose file * Use secrets for certificates * Disable permission handling inside cert's generator entrypoint.sh * Back to using a bind mount for certs * Have entrypoint.sh generate certs with 1000:1000 ownership * Correct certificate permissions and bind mounting * Add security initialization variable to compose file * Fix permissions on certs generator entrypoint * Add cert generator config file * Remove old cert generator dir * Set indexer hostname right in pipeline file * Roll back commented code --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Fix Logstash pipelines * Remove unused file * Implement OCSF severity normalize function --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Federico Gustavo Galland <[email protected]>
1 parent 137a987 commit 479fbbf

26 files changed

+1044
-12
lines changed

‎integrations/.gitignore

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
elastic
2+
opensearch
3+
splunk
4+
common
5+
config
6+
docker/certs

‎integrations/README.md

+37
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
## Wazuh indexer integrations
2+
3+
This folder contains integrations with third-party XDR, SIEM and cybersecurity software.
4+
The goal is to transport Wazuh's analysis to the platform that suits your needs.
5+
6+
### Amazon Security Lake
7+
8+
Amazon Security Lake automatically centralizes security data from AWS environments, SaaS providers,
9+
on premises, and cloud sources into a purpose-built data lake stored in your account. With Security Lake,
10+
you can get a more complete understanding of your security data across your entire organization. You can
11+
also improve the protection of your workloads, applications, and data. Security Lake has adopted the
12+
Open Cybersecurity Schema Framework (OCSF), an open standard. With OCSF support, the service normalizes
13+
and combines security data from AWS and a broad range of enterprise security data sources.
14+
15+
##### Usage
16+
17+
A demo of the integration can be started using the content of this folder and Docker.
18+
19+
```console
20+
docker compose -f ./docker/amazon-security-lake.yml up -d
21+
```
22+
23+
This docker compose project will bring a *wazuh-indexer* node, a *wazuh-dashboard* node,
24+
a *logstash* node and our event generator. On the one hand, the event generator will push events
25+
constantly to the indexer. On the other hand, logstash will constantly query for new data and
26+
deliver it to the integration Python program, also present in that node. Finally, the integration
27+
module will prepare and send the data to the Amazon Security Lake's S3 bucket.
28+
<!-- TODO continue with S3 credentials setup -->
29+
30+
For production usage, follow the instructions in our documentation page about this matter.
31+
(_when-its-done_)
32+
33+
As a last note, we would like to point out that we also use this Docker environment for development.
34+
35+
### Other integrations
36+
37+
TBD

‎integrations/amazon-security-lake/.dockerignore

+180
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,180 @@
1+
wazuh-event.ocsf.json
2+
*.parquet
3+
Dockerfile
4+
5+
# Created by https://www.toptal.com/developers/gitignore/api/python
6+
# Edit at https://www.toptal.com/developers/gitignore?templates=python
7+
8+
### Python ###
9+
# Byte-compiled / optimized / DLL files
10+
__pycache__/
11+
*.py[cod]
12+
*$py.class
13+
14+
# C extensions
15+
*.so
16+
17+
# Distribution / packaging
18+
.Python
19+
build/
20+
develop-eggs/
21+
dist/
22+
downloads/
23+
eggs/
24+
.eggs/
25+
lib/
26+
lib64/
27+
parts/
28+
sdist/
29+
var/
30+
wheels/
31+
share/python-wheels/
32+
*.egg-info/
33+
.installed.cfg
34+
*.egg
35+
MANIFEST
36+
37+
# PyInstaller
38+
# Usually these files are written by a python script from a template
39+
# before PyInstaller builds the exe, so as to inject date/other infos into it.
40+
*.manifest
41+
*.spec
42+
43+
# Installer logs
44+
pip-log.txt
45+
pip-delete-this-directory.txt
46+
47+
# Unit test / coverage reports
48+
htmlcov/
49+
.tox/
50+
.nox/
51+
.coverage
52+
.coverage.*
53+
.cache
54+
nosetests.xml
55+
coverage.xml
56+
*.cover
57+
*.py,cover
58+
.hypothesis/
59+
.pytest_cache/
60+
cover/
61+
62+
# Translations
63+
*.mo
64+
*.pot
65+
66+
# Django stuff:
67+
*.log
68+
local_settings.py
69+
db.sqlite3
70+
db.sqlite3-journal
71+
72+
# Flask stuff:
73+
instance/
74+
.webassets-cache
75+
76+
# Scrapy stuff:
77+
.scrapy
78+
79+
# Sphinx documentation
80+
docs/_build/
81+
82+
# PyBuilder
83+
.pybuilder/
84+
target/
85+
86+
# Jupyter Notebook
87+
.ipynb_checkpoints
88+
89+
# IPython
90+
profile_default/
91+
ipython_config.py
92+
93+
# pyenv
94+
# For a library or package, you might want to ignore these files since the code is
95+
# intended to run in multiple environments; otherwise, check them in:
96+
# .python-version
97+
98+
# pipenv
99+
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
100+
# However, in case of collaboration, if having platform-specific dependencies or dependencies
101+
# having no cross-platform support, pipenv may install dependencies that don't work, or not
102+
# install all needed dependencies.
103+
#Pipfile.lock
104+
105+
# poetry
106+
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
107+
# This is especially recommended for binary packages to ensure reproducibility, and is more
108+
# commonly ignored for libraries.
109+
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
110+
#poetry.lock
111+
112+
# pdm
113+
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
114+
#pdm.lock
115+
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
116+
# in version control.
117+
# https://pdm.fming.dev/#use-with-ide
118+
.pdm.toml
119+
120+
# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
121+
__pypackages__/
122+
123+
# Celery stuff
124+
celerybeat-schedule
125+
celerybeat.pid
126+
127+
# SageMath parsed files
128+
*.sage.py
129+
130+
# Environments
131+
.env
132+
.venv
133+
env/
134+
venv/
135+
ENV/
136+
env.bak/
137+
venv.bak/
138+
139+
# Spyder project settings
140+
.spyderproject
141+
.spyproject
142+
143+
# Rope project settings
144+
.ropeproject
145+
146+
# mkdocs documentation
147+
/site
148+
149+
# mypy
150+
.mypy_cache/
151+
.dmypy.json
152+
dmypy.json
153+
154+
# Pyre type checker
155+
.pyre/
156+
157+
# pytype static type analyzer
158+
.pytype/
159+
160+
# Cython debug symbols
161+
cython_debug/
162+
163+
# PyCharm
164+
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
165+
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
166+
# and can be added to the global gitignore or merged into this file. For a more nuclear
167+
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
168+
#.idea/
169+
170+
### Python Patch ###
171+
# Poetry local configuration file - https://python-poetry.org/docs/configuration/#local-configuration
172+
poetry.toml
173+
174+
# ruff
175+
.ruff_cache/
176+
177+
# LSP config files
178+
pyrightconfig.json
179+
180+
# End of https://www.toptal.com/developers/gitignore/api/python

0 commit comments

Comments
 (0)