Basic docs

README.md

@@ -50,7 +50,30 @@ The bot's main goal is to make life easier for the conference teams by streamlin
   * Managed with `uv`. All Makefiles, Docker images, and related configs are set up to use `uv`.
 
 * **CI/CD**
-  * Built with GitHub Actions.
+  * Built with GitHub Actions
+  * Runs tests and linters on every push to pull requests
+  * Automatically deploys to production when code is merged to `main`
+
+## Documentation
+
+Check the `docs/` directory for additional information:
+
+* [Architecture Overview](docs/architecture.md) - Basic system design and integration workflow
+* [Deployment Guide](docs/deployment.md) - Server setup and deployment process
+
+## Project Structure
+
+This project follows a simple, focused organization:
+
+```
+deploy/    - Deployment configuration and Ansible playbooks
+docs/      - Documentation files
+intbot/    - Main application code
+ ├─ core/  - Core functionality (bot, integrations, endpoints)
+ └─ tests/ - Test suite
+```
+
+See the [Architecture Overview](docs/architecture.md) for more details.
 
 ## Local Development
 
@@ -71,13 +94,68 @@ $ pyenv install 3.12
 
 ## Contributing
 
-...
+### Setting Up Development Environment
+
+1. Clone the repository
+2. Install `uv` - all the dependencies and Makefile targets are using `uv`.
+3. Set up environment variables in `.env` file
+4. Run the database with `docker-compose up -d`
+5. Apply migrations with `make migrate`
+6. Run the development server with `make server`
+7. Run the bot with `make bot`
+8. Run the worker with `make worker`
+
+You can check the Django admin at http://localhost:4672/admin/ to see webhooks and tasks.
+
+### Adding a New Integration
+
+1. Create a module in `intbot/core/integrations/`
+2. Define Pydantic models to organize the data
+3. Add a webhook endpoint in `core/endpoints/webhooks.py`
+4. Add security checks (signature verification)
+5. Update `process_webhook` in `core/tasks.py`
+6. If it will send Discord messages, add routing logic in `channel_router.py`
+7. Add the new URL in `urls.py`
+
+### Testing
+
+The project uses pytest with Django:
+- Run all tests with `make test`
+- Run single tests with `make test/k K=your_keyword`
+- Run fast tests with `make test/fast`
+- Check test coverage with `make test/cov`
+
+When testing webhooks, make sure to test:
+- Security checks (signature verification)
+- Data parsing
+- Channel routing (if using Discord)
+- Message formatting (if using Discord)
+
+### Code Quality
+
+We use ruff and mypy to lint and format the project.
+Both of those are run on CI for every Pull Request.
+
+- Use type hints for all functions and classes
+- Run `make format` before committing
+- Run `make lint` and `make type-check` to check for issues
+- Follow the same error handling patterns as existing code
 
 ## Operations
 
-...
+TODO: Expand on this part :)
+
+### Monitoring
+
+- Currently not configured
+
+### Debugging
+
+- Logs can be viewed on the intbot_user with `make logs`
+- The Django admin interface is available at `/admin/`
 
 ## Deployment
 
-...
+Currently deployed to a separate VPS, using ansible (for both provisioning and deployment).
 
+See deployment doc for more details: [Deployment Guide](docs/deployment.md)

docs/architecture.md

@@ -0,0 +1,129 @@
+# Internal Bot Architecture
+
+## Overview
+
+The internal-bot helps EuroPython Society teams communicate better. Right now it connects GitHub and Zammad with Discord, but it's built to do more in the future.
+
+The app has three main parts:
+1. **Django Web App**: Receives webhooks, provides an admin panel, and will support more features later
+2. **Discord Bot**: Sends messages to Discord and responds to commands
+3. **Background Worker**: Handles tasks in the background without blocking the web app
+
+Besides handling webhooks, the bot can also:
+- Respond to user commands (like `!ping`)
+- Answer questions in channels and threads
+- React to messages and reactions from users
+
+The bot code and supported commands can be found in `intbot/core/bot/main.py`.
+
+## Project Structure
+
+The codebase is organized as follows:
+
+```
+deploy/                  # Deployment configuration
+├── playbooks/           # Ansible playbooks
+└── templates/           # Templates for Docker Compose and Makefiles
+
+docs/                    # Project documentation
+├── architecture.md      # System design and structure
+└── deployment.md        # Deployment guide
+
+intbot/
+├── core/                # Main Django app with all the business logic
+│   ├── bot/             # Discord bot implementation
+│   ├── endpoints/       # API endpoints for webhooks
+│   ├── integrations/    # Integration modules (GitHub, Zammad)
+│   ├── management/      # Django management commands
+│   └── models.py        # Database models
+├── intbot/              # Django project settings
+│   ├── settings.py      # Configuration
+│   └── urls.py          # URL routing
+└── tests/               # Test suite (mirrors the application structure)
+```
+
+This structure was chosen because:
+- It's simple and has a single `core` app instead of many small apps
+- It clearly separates the integration logic from the bot logic
+- Tests mirror the application structure, making them easy to find
+- It supports multiple entry points (web server, bot, worker) from one codebase
+
+## System Architecture
+
+```
+┌─────────────────┐     ┌──────────────────┐     ┌────────────────┐
+│  External       │     │                  │     │                │
+│  Services       │────▶│  Django App      │────▶│  Discord       │
+│  (GitHub,       │     │  (Webhook API)   │     │  Channels      │
+│  Zammad)        │     │                  │     │                │
+└─────────────────┘     └──────────────────┘     └────────────────┘
+                               │   ▲
+                               ▼   │
+                        ┌──────────────────┐
+                        │                  │
+                        │  Database        │
+                        │  (PostgreSQL)    │
+                        │                  │
+                        └──────────────────┘
+```
+
+### Data Flow
+
+1. External services send webhooks to our app
+2. We verify and save these webhooks to the database
+3. Our background worker processes these webhooks:
+   - For some webhooks (like GitHub), we need to fetch more data to make them useful
+   - We then turn the webhook data into a format we can use
+4. If a Discord message needs to be sent, the channel router picks the right channel
+5. Discord messages are saved to the database
+6. The Discord bot checks for new messages and sends them
+
+### Using the Admin Panel
+
+The Django Admin panel lets you:
+- See all webhooks and filter them by type or date
+- Look at the raw webhook data
+- Check if tasks worked or failed
+- Manually trigger processing for webhooks
+- View and manage Discord messages
+
+## Key Components
+
+### Models
+
+- **Webhook**: Stores webhook data including source, event type, and content
+- **DiscordMessage**: Represents a message to be sent to Discord
+- **Task**: (django-tasks) Stores background task execution history
+
+### Integrations
+
+#### GitHub Integration
+- Receives webhooks at `/webhooks/github/`
+- Verifies signatures using HMAC-SHA256
+- Currently handles project item events and issues
+- Routes messages based on project ID or repository
+
+**Why GitHub Needs Extra Steps:**
+1. **The Problem**: GitHub webhooks often just contain IDs, not useful information
+2. **Getting More Data**: 
+   - We need to ask GitHub's GraphQL API for details like item names and descriptions
+   - Without this extra step, notifications would just say "Item 12345 was moved" instead of what actually happened
+3. **How It Works**:
+   - First, we save the webhook and get the extra data from GitHub
+   - Then, we process it into a readable message
+   - Finally, we send it to the right Discord channel
+
+This approach lets us send helpful messages with real information instead of just ID numbers.
+
+#### Zammad Integration
+- Receives webhooks at `/webhooks/zammad/`
+- Verifies signatures using HMAC-SHA1
+- Processes ticket and article information
+- Figures out what happened (new ticket, reply to ticket)
+- Routes messages based on ticket group (billing, helpdesk)
+
+### Channel Router
+- Decides which Discord channel should receive messages
+- Uses different routing rules for each source (GitHub, Zammad)
+- Channel mappings are set in configuration
+- Can be extended for new message sources in the future

docs/deployment.md

@@ -0,0 +1,146 @@
+# Deployment Guide
+
+This document explains how to deploy the internal-bot to a production environment.
+
+## Overview
+
+The deployment process uses:
+- Docker for containerization
+- Docker Compose for container orchestration
+- Ansible for automation
+- Nginx for web server and SSL termination
+
+The application is deployed as several containers:
+- Django web app (handles webhooks)
+- Discord bot (sends messages to Discord)
+- Background worker (processes tasks)
+- PostgreSQL database
+
+## Prerequisites
+
+- A server running Ubuntu
+- SSH access to the server
+- Domain name pointing to the server
+- uv installed on your local machine (it will automatically download and install ansible, if you run it from the `make deploy/*` targets.
+
+## Deployment Process
+
+The deployment is done in three stages using Ansible playbooks:
+
+### 1. Server Setup
+
+```bash
+make deploy/provision
+```
+
+This runs the first two playbooks:
+- `01_setup.yml`: Sets up server, installs Docker, creates users (nginx_user and intbot_user)
+- `02_nginx.yml`: Configures Nginx with SSL certificates
+
+### 2. Application Deployment
+
+```bash
+make deploy/app
+```
+
+This runs the `03_app.yml` playbook which:
+- Builds Docker images
+- Sets up environment variables
+- Creates Docker Compose configuration
+- Runs database migrations
+- Starts all services
+
+## Ansible Templates and Separate User Environments
+
+The deployment uses a separated approach with different users for different responsibilities:
+
+### Separate Users and Docker Compose Files
+
+1. **Nginx Environment** (managed by `nginx_user`):
+   - Uses its own Docker Compose file generated from `docker-compose.nginx.yml.j2`
+   - Handles SSL termination and proxying
+   - Has access to port 80/443 for web traffic
+
+2. **Application Environment** (managed by `intbot_user`):
+   - Uses its own Docker Compose file generated from `docker-compose.app.yml.j2`
+   - Runs Django app, Discord bot, worker, and database
+   - Doesn't need direct public internet access
+
+Both environments are connected via a shared Docker network called "shared_with_nginx_network". This architecture provides several benefits:
+- **Security**: Each component runs with minimal required permissions
+- **Access Control**: Different teams can have access to different parts (some only to app, some to both)
+- **Separation of Concerns**: Nginx configuration changes don't affect the application
+- **Maintenance**: Either component can be updated independently
+
+### Custom Makefiles for Each Environment
+
+Ansible generates two specialized Makefiles:
+
+1. **Nginx Makefile** (`Makefile.nginx.j2`):
+   - Focused on SSL certificate management
+   - Key targets:
+     - `certbot/init-staging`: Set up staging certificates (for testing)
+     - `certbot/upgrade-to-prod`: Upgrade to production certificates
+     - `certbot/renew`: Renew existing certificates
+     - `certbot/force-reissue-PROD-certificate`: Force reissue production certificates
+
+2. **Application Makefile** (`Makefile.app.j2`):
+   - Focused on application management
+   - All commands use the `prod/` prefix
+   - Key targets:
+     - `prod/migrate`: Run database migrations
+     - `prod/shell`: Access Django shell
+     - `prod/db_shell`: Access database shell
+     - `prod/manage`: Run Django management commands
+     - `logs`: View application logs
+
+## Version Control
+
+Deployments are tied to specific Git commits:
+
+```bash
+# Deploy specific version
+make deploy/app V=abcd1234
+```
+
+If no version is specified, the current Git commit hash is used.
+
+## Environment Variables
+
+The application needs these environment variables in `intbot.env`:
+
+- `DJANGO_SECRET_KEY`: Secret key for Django
+- `DJANGO_ALLOWED_HOSTS`: Comma-separated list of allowed hosts
+- `DATABASE_URL`: PostgreSQL connection string
+- `DISCORD_BOT_TOKEN`: Discord bot authentication token
+- `DISCORD_GUILD_ID`: Discord server ID
+- `GITHUB_WEBHOOK_SECRET`: Secret for GitHub webhook verification
+- `GITHUB_TOKEN`: GitHub API token
+- `ZAMMAD_WEBHOOK_SECRET`: Secret for Zammad webhook verification
+
+An example file is available at `deploy/templates/app/intbot.env.example`.
+
+## Monitoring
+
+- Logs can be viewed with `docker compose logs`
+- The Django admin interface is available at `/admin/`
+- Server monitoring should be set up separately (not included)
+
+## Troubleshooting
+
+Common issues:
+- **Webhook verification failures**: Check secret keys in environment variables
+- **Database connection errors**: Verify DATABASE_URL is correct
+- **Discord messages not being sent**: Check DISCORD_BOT_TOKEN and permissions
+
+For more detailed logs:
+```bash
+docker compose logs -f bot
+docker compose logs -f web
+docker compose logs -f worker
+```
+
+Or even simpler if you want to see all of them at once
+```bash
+make logs
+```