Janus

Janus is a Bayesian A/B Testing application that supports multivariant experiments. It's designed to help you make data-driven decisions by analyzing conversion rates, revenue, and ARPU (Average Revenue Per User) across multiple variants.

Features

• Multivariant Testing: Compare multiple variants simultaneously (not just A vs B)
• Bayesian Statistics: Get more insightful results faster than traditional frequentist methods
• Key Metrics Analysis:
  • Conversion rate
  • Revenue for conversions
  • Average revenue per impression (ARPU)
• Modern Web Interface: Clean, responsive UI built with Bootstrap
• FastAPI Backend: High-performance API for experiment analysis

Installation

1. Clone this repository:

git clone https://github.com/lgabs/janus.git
cd janus

2. Create a virtual environment (optional but recommended):

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

3. Install dependencies:

pip install -r requirements.txt

Running the Application

Using Python directly:

uvicorn main:app --reload

Then open your browser and navigate to http://localhost:8000

Using Docker:

# Build the Docker image
docker build -t janus .

# Run the container
docker run -p 8000:8000 janus

Using Docker Compose (Recommended):

# Start the application
docker-compose up -d

# View logs
docker-compose logs -f

# Stop the application
docker-compose down

Then open your browser and navigate to http://localhost:8000

How to Use

1. Enter your baseline variant name (e.g., "A" or "Control")

2. Add your variants with their respective data:

   • Name: A unique identifier for the variant
   • Impressions: Total number of users/sessions exposed to this variant
   • Conversions: Number of successful conversions
   • Revenue: Total revenue generated by this variant

3. Click "Run Analysis" to see the results:

   • Summary statistics for each variant
   • Conversion statistics with probability of being the best variant
   • ARPU statistics with probability of being the best variant

4. Export your results as CSV if needed

API Documentation

The API documentation is available at /docs when the application is running.

Technical Details

This application uses:

• FastAPI for the backend
• Bayesian-Testing for statistical calculations
• Bootstrap 5 for the frontend UI
• Jinja2 for HTML templating

References

• What is A/B Testing
• The bayesian calculations were implemented based on this VWO white paper
• Agile A/B testing with Bayesian Statistics and Python
• Understanding Bayesian A/B testing (using baseball statistics)
• It's time to re-think A/B testing
• Conjugate Priors
• Curso de Teste A/B Bayesiano do Lazy Programmer
• Binomial Distributions
• Bayes theorem
• The quick proof of Bayes Theorem