From 2b1ce829f2331bf9931e924b60cea4047c166013 Mon Sep 17 00:00:00 2001 From: Artur Czepiel Date: Sun, 9 Mar 2025 13:33:50 +0100 Subject: [PATCH 1/2] add some basic docs --- README.md | 72 +++++++++++++++++++-- docs/architecture.md | 90 ++++++++++++++++++++++++++ docs/deployment.md | 146 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 304 insertions(+), 4 deletions(-) create mode 100644 docs/architecture.md create mode 100644 docs/deployment.md diff --git a/README.md b/README.md index 9a7e8c9..6b4f0ae 100644 --- a/README.md +++ b/README.md @@ -50,7 +50,16 @@ The bot's main goal is to make life easier for the conference teams by streamlin * Managed with `uv`. All Makefiles, Docker images, and related configs are set up to use `uv`. * **CI/CD** - * Built with GitHub Actions. + * Built with GitHub Actions + * Runs tests and linters on every push to pull requests + * Automatically deploys to production when code is merged to `main` + +## Documentation + +Check the `docs/` directory for additional information: + +* [Architecture Overview](docs/architecture.md) - Basic system design and integration workflow +* [Deployment Guide](docs/deployment.md) - Server setup and deployment process ## Local Development @@ -71,13 +80,68 @@ $ pyenv install 3.12 ## Contributing -... +### Setting Up Development Environment + +1. Clone the repository +2. Install `uv` - all the dependencies and Makefile targets are using `uv`. +3. Set up environment variables in `.env` file +4. Run the database with `docker-compose up -d` +5. Apply migrations with `make migrate` +6. Run the development server with `make server` +7. Run the bot with `make bot` +8. Run the worker with `make worker` + +You can check the Django admin at http://localhost:4672/admin/ to see webhooks and tasks. + +### Adding a New Integration + +1. Create a module in `intbot/core/integrations/` +2. Define Pydantic models to organize the data +3. Add a webhook endpoint in `core/endpoints/webhooks.py` +4. Add security checks (signature verification) +5. Update `process_webhook` in `core/tasks.py` +6. If it will send Discord messages, add routing logic in `channel_router.py` +7. Add the new URL in `urls.py` + +### Testing + +The project uses pytest with Django: +- Run all tests with `make test` +- Run single tests with `make test/k K=your_keyword` +- Run fast tests with `make test/fast` +- Check test coverage with `make test/cov` + +When testing webhooks, make sure to test: +- Security checks (signature verification) +- Data parsing +- Channel routing (if using Discord) +- Message formatting (if using Discord) + +### Code Quality + +We use ruff and mypy to lint and format the project. +Both of those are run on CI for every Pull Request. + +- Use type hints for all functions and classes +- Run `make format` before committing +- Run `make lint` and `make type-check` to check for issues +- Follow the same error handling patterns as existing code ## Operations -... +TODO: Expand on this part :) + +### Monitoring + +- Currently not configured + +### Debugging + +- Logs can be viewed on the intbot_user with `make logs` +- The Django admin interface is available at `/admin/` ## Deployment -... +Currently deployed to a separate VPS, using ansible (for both provisioning and deployment). +See deployment doc for more details: [Deployment Guide](docs/deployment.md) diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..4fc42f9 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,90 @@ +# Internal Bot Architecture + +## Overview + +The internal-bot helps EuroPython Society teams communicate better. Right now it connects GitHub and Zammad with Discord, but it's built to do more in the future. + +The app has three main parts: +1. **Django Web App**: Receives webhooks, provides an admin panel, and will support more features later +2. **Discord Bot**: Sends messages to Discord and responds to commands +3. **Background Worker**: Handles tasks in the background without blocking the web app + +## System Architecture + +``` +┌─────────────────┐ ┌──────────────────┐ ┌────────────────┐ +│ External │ │ │ │ │ +│ Services │────▶│ Django App │────▶│ Discord │ +│ (GitHub, │ │ (Webhook API) │ │ Channels │ +│ Zammad) │ │ │ │ │ +└─────────────────┘ └──────────────────┘ └────────────────┘ + │ ▲ + ▼ │ + ┌──────────────────┐ + │ │ + │ Database │ + │ (PostgreSQL) │ + │ │ + └──────────────────┘ +``` + +### Data Flow + +1. External services send webhooks to our app +2. We verify and save these webhooks to the database +3. Our background worker processes these webhooks: + - For some webhooks (like GitHub), we need to fetch more data to make them useful + - We then turn the webhook data into a format we can use +4. If a Discord message needs to be sent, the channel router picks the right channel +5. Discord messages are saved to the database +6. The Discord bot checks for new messages and sends them + +### Using the Admin Panel + +The Django Admin panel lets you: +- See all webhooks and filter them by type or date +- Look at the raw webhook data +- Check if tasks worked or failed +- Manually trigger processing for webhooks +- View and manage Discord messages + +## Key Components + +### Models + +- **Webhook**: Stores webhook data including source, event type, and content +- **DiscordMessage**: Represents a message to be sent to Discord +- **Task**: (django-tasks) Stores background task execution history + +### Integrations + +#### GitHub Integration +- Receives webhooks at `/webhooks/github/` +- Verifies signatures using HMAC-SHA256 +- Currently handles project item events and issues +- Routes messages based on project ID or repository + +**Why GitHub Needs Extra Steps:** +1. **The Problem**: GitHub webhooks often just contain IDs, not useful information +2. **Getting More Data**: + - We need to ask GitHub's GraphQL API for details like item names and descriptions + - Without this extra step, notifications would just say "Item 12345 was moved" instead of what actually happened +3. **How It Works**: + - First, we save the webhook and get the extra data from GitHub + - Then, we process it into a readable message + - Finally, we send it to the right Discord channel + +This approach lets us send helpful messages with real information instead of just ID numbers. + +#### Zammad Integration +- Receives webhooks at `/webhooks/zammad/` +- Verifies signatures using HMAC-SHA1 +- Processes ticket and article information +- Figures out what happened (new ticket, reply to ticket) +- Routes messages based on ticket group (billing, helpdesk) + +### Channel Router +- Decides which Discord channel should receive messages +- Uses different routing rules for each source (GitHub, Zammad) +- Channel mappings are set in configuration +- Can be extended for new message sources in the future diff --git a/docs/deployment.md b/docs/deployment.md new file mode 100644 index 0000000..212c69f --- /dev/null +++ b/docs/deployment.md @@ -0,0 +1,146 @@ +# Deployment Guide + +This document explains how to deploy the internal-bot to a production environment. + +## Overview + +The deployment process uses: +- Docker for containerization +- Docker Compose for container orchestration +- Ansible for automation +- Nginx for web server and SSL termination + +The application is deployed as several containers: +- Django web app (handles webhooks) +- Discord bot (sends messages to Discord) +- Background worker (processes tasks) +- PostgreSQL database + +## Prerequisites + +- A server running Ubuntu +- SSH access to the server +- Domain name pointing to the server +- uv installed on your local machine (it will automatically download and install ansible, if you run it from the `make deploy/*` targets. + +## Deployment Process + +The deployment is done in three stages using Ansible playbooks: + +### 1. Server Setup + +```bash +make deploy/provision +``` + +This runs the first two playbooks: +- `01_setup.yml`: Sets up server, installs Docker, creates users (nginx_user and intbot_user) +- `02_nginx.yml`: Configures Nginx with SSL certificates + +### 2. Application Deployment + +```bash +make deploy/app +``` + +This runs the `03_app.yml` playbook which: +- Builds Docker images +- Sets up environment variables +- Creates Docker Compose configuration +- Runs database migrations +- Starts all services + +## Ansible Templates and Separate User Environments + +The deployment uses a separated approach with different users for different responsibilities: + +### Separate Users and Docker Compose Files + +1. **Nginx Environment** (managed by `nginx_user`): + - Uses its own Docker Compose file generated from `docker-compose.nginx.yml.j2` + - Handles SSL termination and proxying + - Has access to port 80/443 for web traffic + +2. **Application Environment** (managed by `intbot_user`): + - Uses its own Docker Compose file generated from `docker-compose.app.yml.j2` + - Runs Django app, Discord bot, worker, and database + - Doesn't need direct public internet access + +Both environments are connected via a shared Docker network called "shared_with_nginx_network". This architecture provides several benefits: +- **Security**: Each component runs with minimal required permissions +- **Access Control**: Different teams can have access to different parts (some only to app, some to both) +- **Separation of Concerns**: Nginx configuration changes don't affect the application +- **Maintenance**: Either component can be updated independently + +### Custom Makefiles for Each Environment + +Ansible generates two specialized Makefiles: + +1. **Nginx Makefile** (`Makefile.nginx.j2`): + - Focused on SSL certificate management + - Key targets: + - `certbot/init-staging`: Set up staging certificates (for testing) + - `certbot/upgrade-to-prod`: Upgrade to production certificates + - `certbot/renew`: Renew existing certificates + - `certbot/force-reissue-PROD-certificate`: Force reissue production certificates + +2. **Application Makefile** (`Makefile.app.j2`): + - Focused on application management + - All commands use the `prod/` prefix + - Key targets: + - `prod/migrate`: Run database migrations + - `prod/shell`: Access Django shell + - `prod/db_shell`: Access database shell + - `prod/manage`: Run Django management commands + - `logs`: View application logs + +## Version Control + +Deployments are tied to specific Git commits: + +```bash +# Deploy specific version +make deploy/app V=abcd1234 +``` + +If no version is specified, the current Git commit hash is used. + +## Environment Variables + +The application needs these environment variables in `intbot.env`: + +- `DJANGO_SECRET_KEY`: Secret key for Django +- `DJANGO_ALLOWED_HOSTS`: Comma-separated list of allowed hosts +- `DATABASE_URL`: PostgreSQL connection string +- `DISCORD_BOT_TOKEN`: Discord bot authentication token +- `DISCORD_GUILD_ID`: Discord server ID +- `GITHUB_WEBHOOK_SECRET`: Secret for GitHub webhook verification +- `GITHUB_TOKEN`: GitHub API token +- `ZAMMAD_WEBHOOK_SECRET`: Secret for Zammad webhook verification + +An example file is available at `deploy/templates/app/intbot.env.example`. + +## Monitoring + +- Logs can be viewed with `docker compose logs` +- The Django admin interface is available at `/admin/` +- Server monitoring should be set up separately (not included) + +## Troubleshooting + +Common issues: +- **Webhook verification failures**: Check secret keys in environment variables +- **Database connection errors**: Verify DATABASE_URL is correct +- **Discord messages not being sent**: Check DISCORD_BOT_TOKEN and permissions + +For more detailed logs: +```bash +docker compose logs -f bot +docker compose logs -f web +docker compose logs -f worker +``` + +Or even simpler if you want to see all of them at once +```bash +make logs +``` From 5f78b8e00027f5530cf0283b66a356dc0ddf7002 Mon Sep 17 00:00:00 2001 From: Artur Czepiel Date: Sun, 9 Mar 2025 13:44:56 +0100 Subject: [PATCH 2/2] add a project layout overview --- README.md | 14 ++++++++++++++ docs/architecture.md | 39 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 53 insertions(+) diff --git a/README.md b/README.md index 6b4f0ae..77ef847 100644 --- a/README.md +++ b/README.md @@ -61,6 +61,20 @@ Check the `docs/` directory for additional information: * [Architecture Overview](docs/architecture.md) - Basic system design and integration workflow * [Deployment Guide](docs/deployment.md) - Server setup and deployment process +## Project Structure + +This project follows a simple, focused organization: + +``` +deploy/ - Deployment configuration and Ansible playbooks +docs/ - Documentation files +intbot/ - Main application code + ├─ core/ - Core functionality (bot, integrations, endpoints) + └─ tests/ - Test suite +``` + +See the [Architecture Overview](docs/architecture.md) for more details. + ## Local Development diff --git a/docs/architecture.md b/docs/architecture.md index 4fc42f9..afbb7b6 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -9,6 +9,45 @@ The app has three main parts: 2. **Discord Bot**: Sends messages to Discord and responds to commands 3. **Background Worker**: Handles tasks in the background without blocking the web app +Besides handling webhooks, the bot can also: +- Respond to user commands (like `!ping`) +- Answer questions in channels and threads +- React to messages and reactions from users + +The bot code and supported commands can be found in `intbot/core/bot/main.py`. + +## Project Structure + +The codebase is organized as follows: + +``` +deploy/ # Deployment configuration +├── playbooks/ # Ansible playbooks +└── templates/ # Templates for Docker Compose and Makefiles + +docs/ # Project documentation +├── architecture.md # System design and structure +└── deployment.md # Deployment guide + +intbot/ +├── core/ # Main Django app with all the business logic +│ ├── bot/ # Discord bot implementation +│ ├── endpoints/ # API endpoints for webhooks +│ ├── integrations/ # Integration modules (GitHub, Zammad) +│ ├── management/ # Django management commands +│ └── models.py # Database models +├── intbot/ # Django project settings +│ ├── settings.py # Configuration +│ └── urls.py # URL routing +└── tests/ # Test suite (mirrors the application structure) +``` + +This structure was chosen because: +- It's simple and has a single `core` app instead of many small apps +- It clearly separates the integration logic from the bot logic +- Tests mirror the application structure, making them easy to find +- It supports multiple entry points (web server, bot, worker) from one codebase + ## System Architecture ```