Scrape Autolab

Scrape Autolab for submissions, writeups, and handouts from all of your courses

The result will look something like this

├── 15-122-principles-of-imperative-computation-f16
│   ├── administrative
│   │   └── academic-integrity-form
│   │       ├── handout.pdf
│   │       └── [email protected]_academicintegrityform_1_acad.pdf
│   ├── labs
│   │   ├── lab-1-setup
│   │   │   ├── handout-01.tgz
│   │   │   └── lab01.pdf
│   │   ├── lab-10-this-one-s-a-treet
│   │   │   ├── handout-10.tgz
│   │   │   └── lab10.pdf
│   │   ├── lab-11-pq-puns-are-too-hard
│   │   │   ├── handout-11.tgz
│   │   │   └── lab11.pdf
│   │   ├── lab-12-once-you-c1-you-c
│   │   │   └── lab12.pdf
│   │   ├── lab-13-all-sort-of-sorts
│   │   │   ├── handout-13.tgz
│   │   │   └── lab13.pdf
│   │   ├── lab-14-passwordlab
│   │   │   ├── handout-14.tgz
│   │   │   └── lab14.pdf
│   │   ├── lab-15-spend-some-cycles
│   │   │   ├── handout-15.tgz
│   │   │   └── lab15.pdf
│   │   ├── lab-3-a-reversal-of-fortunes
│   │   │   ├── handout-03.tgz
│   │   │   └── lab-03.pdf
│   │   ├── lab-4-time-of-your-life
│   │   │   ├── handout-04.tgz
│   │   │   └── lab04.pdf
│   │   ├── lab-5-fibonacci-has-bad
│   │   │   ├── handout-05.tgz
│   │   │   └── lab05.pdf
│   │   ├── lab-6-misclaculation
│   │   │   └── lab06.pdf
│   │   ├── lab-7-list-en-up
│   │   │   └── lab07.pdf
│   │   ├── lab-8-pun
│   │   │   └── lab08.pdf
│   │   ├── lab-9-legacy-of-the-void
│   │   │   ├── handout-09.tgz
│   │   │   └── lab09.pdf
│   │   └── no-lab-2-labor-day
│   │       └── lab02.pdf

Prerequisites

• Python 3.8.3

Setup

Install python requirements

pip3 install -r requirements.txt

Copy over user authentication information

Login to autolab, reload the page, and copy the network request for autolab.andrew.cmu.edu as curl

Convert curl to python requests code with this

You should see code that looks something like this

import requests

cookies = {
    'browser.timezone': 'America/Los_Angeles',
    '_autolab3_session': '_autolab3_session_value',
    '_session_id': '_session_id_value',
}

headers = {
    'Connection': 'keep-alive',
    'Upgrade-Insecure-Requests': '1',
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
    'Sec-Fetch-Site': 'same-origin',
    'Sec-Fetch-Mode': 'navigate',
    'Sec-Fetch-User': '?1',
    'Sec-Fetch-Dest': 'document',
    'Referer': 'https://autolab.andrew.cmu.edu/courses/10405-s18/course_user_data/112931',
    'Accept-Language': 'en-US,en;q=0.9',
    'If-None-Match': 'W/"d10db62142119c47702085c94c3effc5"',
}

response = requests.get('https://autolab.andrew.cmu.edu/', headers=headers, cookies=cookies)

Copy the cookies = {...} and headers = {...} parts to scrape_autolab.py#L21

Specify the local output directory for downloaded files

Set OUTPUT_PATH at scrape_autolab.py#L50 to the directory where you want to download assignments to

Run

python3 scrape_autolab.py