Skip to content

Commit 0819e41

Browse files
AlexRuiz7AlexRuiz7
committed
Add tools to assemble DEB packages (#96)
* Add tools to assemble DEB packages * Move wazuh-indexer-performance-analyzer.service to common * Enable assembly of DEB packages * Enable full set of plugins * Actually skip tar assembly * Add installation of dependencies for DEB assembly * Install dependencies using sudo * Format files * Refactor assemble script
1 parent 37cc9dd commit 0819e41

File tree

9 files changed

+439
-186
lines changed

9 files changed

+439
-186
lines changed

‎.github/workflows/build.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ jobs:
5050
exclude:
5151
# skip arm64 until we have arm runners
5252
- architecture: arm64
53-
- distribution: [tar, deb] # Exclude deb assembly until it's implemented
53+
- distribution: tar
5454

5555
uses: ./.github/workflows/r_assemble.yml
5656
with:

‎.github/workflows/r_assemble.yml

+5-1
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,11 @@ jobs:
3939
name: ${{ inputs.min }}
4040
path: artifacts/dist
4141

42+
- name: Provision
43+
if: ${{ inputs.distribution == 'deb' }}
44+
run: |
45+
sudo bash scripts/provision.sh
46+
4247
- name: Run `assemble.sh`
4348
run: |
4449
bash scripts/assemble.sh -v ${{ vars.OPENSEARCH_VERSION }} -p linux -a ${{ inputs.architecture }} -d ${{ inputs.distribution }}
@@ -55,4 +60,3 @@ jobs:
5560
name: ${{ inputs.name }}
5661
path: artifacts/dist/${{ steps.get_name.outputs.name }}
5762
if-no-files-found: error
58-

‎distribution/packages/src/rpm/wazuh-indexer-performance-analyzer.service ‎distribution/packages/src/common/wazuh-indexer-performance-analyzer.service

File renamed without changes.

‎distribution/packages/src/deb/Makefile

+19
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# Copyright OpenSearch Contributors
2+
# SPDX-License-Identifier: Apache-2.0
3+
#
4+
# The OpenSearch Contributors require contributions made to
5+
# this file be licensed under the Apache-2.0 license or a
6+
# compatible open source license.
7+
8+
# deb opensearch Makefile
9+
10+
all: install
11+
12+
install:
13+
./debmake_install.sh $(CURDIR)
14+
15+
clean: ;
16+
17+
distclean: clean
18+
19+
.PHONY: all clean distclean install

‎distribution/packages/src/deb/debmake_install.sh

+46
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
#!/bin/bash
2+
3+
# Copyright OpenSearch Contributors
4+
# SPDX-License-Identifier: Apache-2.0
5+
#
6+
# The OpenSearch Contributors require contributions made to
7+
# this file be licensed under the Apache-2.0 license or a
8+
# compatible open source license.
9+
10+
# debmake opensearch install script
11+
12+
set -ex
13+
14+
if [ -z "$1" ]; then
15+
echo "Missing curdir path"
16+
exit 1
17+
fi
18+
19+
curdir=$1
20+
product_dir=/usr/share/wazuh-indexer
21+
# config_dir=/etc/wazuh-indexer
22+
data_dir=/var/lib/wazuh-indexer
23+
log_dir=/var/log/wazuh-indexer
24+
pid_dir=/var/run/wazuh-indexer
25+
buildroot=${curdir}/debian/wazuh-indexer
26+
27+
# Create necessary directories
28+
mkdir -p "${buildroot}"
29+
mkdir -p "${buildroot}${pid_dir}"
30+
mkdir -p "${buildroot}${product_dir}/plugins"
31+
32+
# Install directories/files
33+
cp -a "${curdir}"/etc "${curdir}"/usr "${curdir}"/var "${buildroot}"/
34+
chmod -c 0755 "${buildroot}${product_dir}"/bin/*
35+
if [ -d "${buildroot}${product_dir}"/plugins/opensearch-security ]; then
36+
chmod -c 0755 "${buildroot}${product_dir}"/plugins/opensearch-security/tools/*
37+
fi
38+
39+
# Symlinks (do not symlink config dir as security demo installer has dependency, if no presense it will switch to rpm/deb mode)
40+
ln -s ${data_dir} "${buildroot}${product_dir}/data"
41+
ln -s ${log_dir} "${buildroot}${product_dir}/logs"
42+
43+
# Change Permissions
44+
chmod -Rf a+rX,u+w,g-w,o-w "${buildroot}"/*
45+
46+
exit 0

‎docker/README.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Frontend development environments
1+
# Indexer development environments
22

33
Install [Docker Desktop][docker-desktop] as per its instructions, available for Windows, Mac
44
and Linux (Ubuntu, Debian & Fedora).

‎scripts/README.md

+142-42
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,9 @@
22

33
The packages' generation process consists on 2 steps:
44

5-
* **Build**: compiles the Java application and bundles it into a package.
6-
* **Assembly**: uses the package from the previous step and inflates it with plugins and
7-
configuration files, ready for production deployment.
5+
- **Build**: compiles the Java application and bundles it into a package.
6+
- **Assembly**: uses the package from the previous step and inflates it with plugins and
7+
configuration files, ready for production deployment.
88

99
We usually generate the packages using GitHub Actions, however, the process is designed to
1010
be independent enough for maximum portability. GitHub Actions provides infrastructure, while
@@ -27,7 +27,6 @@ act -j build -W .github/workflows/build.yml --artifact-server-path ./artifacts
2727
[Build slim packages/build] 🏁 Job succeeded
2828
```
2929

30-
3130
#### Running in Docker
3231

3332
Within the [Docker environment](../docker):
@@ -38,60 +37,162 @@ bash scripts/build.sh -v 2.11.0 -s false -p linux -a {x64|arm64} -d {rpm|deb|tar
3837

3938
The generated package is sent to `artifacts/`
4039

41-
4240
## Assemble
4341

44-
<!--
42+
<!--
4543
### TAR
46-
### DEB
4744
-->
4845

46+
### DEB
47+
48+
The script will:
49+
50+
- Extract the deb package using `ar` and `tar` tools.
51+
52+
> By default, `ar` and `tar` tools expect the package to be in `wazuh-indexer/artifacts/tmp/deb`. The script takes care of creating the required folder structure, copying also the min package and the Makefile.
53+
54+
Current folder loadout at this stage:
55+
56+
```
57+
artifacts/
58+
|-- dist
59+
| |-- wazuh-indexer-min_4.9.0_amd64.deb
60+
`-- tmp
61+
`-- deb
62+
|-- Makefile
63+
|-- data.tar.gz
64+
|-- debmake_install.sh
65+
|-- etc
66+
|-- usr
67+
|-- var
68+
`-- wazuh-indexer-min_4.9.0_amd64.deb
69+
```
70+
71+
`usr`, `etc` and `var` folders contain `wazuh-indexer` files, extracted from `wazuh-indexer-min-*.deb`.
72+
`Makefile` and the `debmake_install` are copied over from `wazuh-indexer/distribution/packages/src/deb`.
73+
The `wazuh-indexer-performance-analyzer.service` file is also copied from the same folder. It is a dependency of the SPEC file.
74+
75+
- Install the plugins using the `opensearch-plugin` CLI tool.
76+
- Set up configuration files.
77+
78+
> Included in `min-package`. Default files are overwritten.
79+
80+
- Bundle a DEB file with `debmake` and the `Makefile`.
81+
82+
> `debmake` and other dependencies can be installed using the provision.sh script. The
83+
> script is invoked by the GitHub Workflow.
84+
85+
Current folder loadout at this stage:
86+
87+
```
88+
artifacts/
89+
|-- artifact_name.txt
90+
|-- dist
91+
| |-- wazuh-indexer-min_4.9.0_amd64.deb
92+
| `-- wazuh-indexer_4.9.0_amd64.deb
93+
`-- tmp
94+
`-- deb
95+
|-- Makefile
96+
|-- data.tar.gz
97+
|-- debmake_install.sh
98+
|-- etc
99+
|-- usr
100+
|-- var
101+
`-- wazuh-indexer-min_4.9.0_amd64.deb
102+
```
103+
104+
### Running in Act
105+
106+
```console
107+
act -j assemble -W .github/workflows/build.yml --artifact-server-path ./artifacts --matrix distribution:deb --matrix architecture:x64 --var OPENSEARCH_VERSION=2.11.0
108+
109+
[Build slim packages/build] 🏁 Job succeeded
110+
```
111+
112+
#### Running in Docker
113+
114+
Pre-requisites:
115+
116+
- Current directory: `wazuh-indexer/`
117+
- Existing deb package in `wazuh-indexer/artifacts/dist/deb`, as a result of the _Build_ stage.
118+
119+
```console
120+
MIN_PKG_PATH="./artifacts"
121+
docker run --rm \
122+
-v ./scripts/:/home/wazuh-indexer/scripts \
123+
-v $MIN_PKG_PATH:/home/wazuh-indexer/artifacts \
124+
-v ./distribution/packages/src:/home/wazuh-indexer/distribution/packages/src \
125+
-w /home/wazuh-indexer \
126+
-it ubuntu:jammy /bin/bash
127+
128+
# https://github.com/opensearch-project/opensearch-build/blob/2.11.1/docker/ci/dockerfiles/current/build.ubuntu2004.opensearch.x64.arm64.dockerfile
129+
130+
# Install necessary packages
131+
apt-get update -y && apt-get upgrade -y && apt-get install -y curl build-essential curl &&
132+
apt-get install -y debmake debhelper-compat &&
133+
apt-get install -y libxrender1 libxtst6 libasound2 libxi6 libgconf-2-4 &&
134+
apt-get install -y libatk1.0-0 libatk-bridge2.0-0 libcups2 libdrm2 libatspi2.0-dev libxcomposite-dev libxdamage1 libxfixes3 libxfixes-dev libxrandr2 libgbm-dev libxkbcommon-x11-0 libpangocairo-1.0-0 libcairo2 libcairo2-dev libnss3 libnspr4 libnspr4-dev freeglut3 &&
135+
apt-get clean -y
136+
137+
# Install aptly and required changes to debmake
138+
# Remove lintian for now due to it takes nearly 20 minutes for OpenSearch as well as nearly an hour for OpenSearch-Dashboards during debmake
139+
curl -o- https://www.aptly.info/pubkey.txt | apt-key add - &&
140+
echo "deb http://repo.aptly.info/ squeeze main" | tee -a /etc/apt/sources.list.d/aptly.list &&
141+
apt-get update -y && apt-get install -y aptly && apt-get clean -y &&
142+
dpkg -r lintian
143+
144+
bash scripts/assemble.sh -v 2.11.0 -p linux -a x64 -d deb
145+
```
146+
49147
### RPM
50148

51-
The `assemble.sh` script will use the output from the `build.sh` script and use it as a
52-
base to bundle together a final package containing the plugins, the production configuration
149+
The `assemble.sh` script will use the output from the `build.sh` script and use it as a
150+
base to bundle together a final package containing the plugins, the production configuration
53151
and the service files.
54152

55153
The script will:
56154

57155
- Extract the rpm package using `rpm2cpio` and `cpio` tools.
58-
59-
> By default, `rpm2cpio` and `cpio` tools expect the package to be in `wazuh-indexer/artifacts/tmp/rpm`. The script takes care of creating the required folder structure, copying also the min package and the SPEC file.
60-
61-
Current folder loadout at this stage:
62-
```
63-
/rpm/$ARCH
64-
/etc
65-
/usr
66-
/var
67-
wazuh-indexer-min-*.rpm
68-
wazuh-indexer.rpm.spec
69-
```
70-
71-
`usr`, `etc` and `var` folders contain `wazuh-indexer` files, extracted from `wazuh-indexer-min-*.rpm`.
72-
`wazuh-indexer.rpm.spec` is copied over from `wazuh-indexer/distribution/packages/src/rpm/wazuh-indexer.rpm.spec`.
73-
The `wazuh-indexer-performance-analyzer.service` file is also copied from the same folder. It is a dependency of the SPEC file.
156+
157+
> By default, `rpm2cpio` and `cpio` tools expect the package to be in `wazuh-indexer/artifacts/tmp/rpm`. The script takes care of creating the required folder structure, copying also the min package and the SPEC file.
158+
159+
Current folder loadout at this stage:
160+
161+
```
162+
/rpm/$ARCH
163+
/etc
164+
/usr
165+
/var
166+
wazuh-indexer-min-*.rpm
167+
wazuh-indexer.rpm.spec
168+
```
169+
170+
`usr`, `etc` and `var` folders contain `wazuh-indexer` files, extracted from `wazuh-indexer-min-*.rpm`.
171+
`wazuh-indexer.rpm.spec` is copied over from `wazuh-indexer/distribution/packages/src/rpm/wazuh-indexer.rpm.spec`.
172+
The `wazuh-indexer-performance-analyzer.service` file is also copied from the same folder. It is a dependency of the SPEC file.
74173

75174
- Install the plugins using the `opensearch-plugin` CLI tool.
76175
- Set up configuration files.
77176

78-
> Included in `min-package`. Default files are overwritten.
79-
177+
> Included in `min-package`. Default files are overwritten.
178+
80179
- Bundle an RPM file with `rpmbuild` and the SPEC file `wazuh-indexer.rpm.spec`.
81-
- `rpmbuild` is part of the `rpm` OS package.
82180

83-
> `rpmbuild` is invoked from `wazuh-indexer/artifacts/tmp/rpm`. It creates the {BUILD,RPMS,SOURCES,SRPMS,SPECS,TMP} folders and applies the rules in the SPEC file. If successful, `rpmbuild` will generate the package in the `RPMS/` folder. The script will copy it to `wazuh-indexer/artifacts/dist` and clean: remove the `tmp\` folder and its contents.
181+
- `rpmbuild` is part of the `rpm` OS package.
84182

85-
Current folder loadout at this stage:
86-
```
87-
/rpm/$ARCH
88-
/{BUILD,RPMS,SOURCES,SRPMS,SPECS,TMP}
89-
/etc
90-
/usr
91-
/var
92-
wazuh-indexer-min-*.rpm
93-
wazuh-indexer.rpm.spec
94-
```
183+
> `rpmbuild` is invoked from `wazuh-indexer/artifacts/tmp/rpm`. It creates the {BUILD,RPMS,SOURCES,SRPMS,SPECS,TMP} folders and applies the rules in the SPEC file. If successful, `rpmbuild` will generate the package in the `RPMS/` folder. The script will copy it to `wazuh-indexer/artifacts/dist` and clean: remove the `tmp\` folder and its contents.
184+
185+
Current folder loadout at this stage:
186+
187+
```
188+
/rpm/$ARCH
189+
/{BUILD,RPMS,SOURCES,SRPMS,SPECS,TMP}
190+
/etc
191+
/usr
192+
/var
193+
wazuh-indexer-min-*.rpm
194+
wazuh-indexer.rpm.spec
195+
```
95196

96197
### Running in Act
97198

@@ -105,8 +206,8 @@ act -j assemble -W .github/workflows/build.yml --artifact-server-path ./artifact
105206

106207
Pre-requisites:
107208

108-
* Current directory: `wazuh-indexer/`
109-
* Existing rpm package in `wazuh-indexer/artifacts/dist/rpm`, as a result of the _Build_ stage.
209+
- Current directory: `wazuh-indexer/`
210+
- Existing rpm package in `wazuh-indexer/artifacts/dist/rpm`, as a result of the _Build_ stage.
110211

111212
```console
112213
MIN_PKG_PATH="./artifacts"
@@ -121,4 +222,3 @@ apt-get update
121222
apt-get install -y rpm2cpio rpm cpio
122223
bash scripts/assemble.sh -v 2.11.0 -p linux -a x64 -d rpm
123224
```
124-

0 commit comments

Comments
 (0)