NOTE: Run all of the following steps from <project_dir>/extractors.
pip install -r ../requirements.txt
To perform the preprocessing of QMSum necessary to reproduce the experiments, follow the instructions in the preprocessing directory.
git clone https://github.com/huggingface/transformers.git
cd transformers
git checkout 65659a29cf5a079842e61a63d57fa24474288998
cd ..
# Data prep, training, inference, postprocessing on utterance-level input
# switch to 1 for segment-level; outputs files for seq2seq training to output-relreg-utt
bash run_relreg.sh 0 output-relreg-utt
# switch to 1 for segment-level; outputs files to output-relregTT-utt
bash run_relreg_tt.sh 0 output-relregTT-utt
bash train_qmsum_bart.sh
Select best checkpoints from runs in the previous step, where NAME is taken from the train_qmsum_bart.sh script:
python ../multiencoder/select_checkpoints.py NAME
To generate predictions on the validation set:
bash predict_qmsum_bart.sh
python ../rouge/report_rouge.py --ref-path PATH_TO_REFERENCES --pred-paths PATH_TO_PREDICTIONS
We have included checkpoints for all 5 training runs of the RelReg-W model used in the final evaluation, along with their performance on the validation set:
| Run | ROUGE-1 | ROUGE-2 | ROUGE-L | Checkpoint |
|---|---|---|---|---|
| 1 | 37.03 | 12.47 | 32.47 | download |
| 2 | 36.44 | 12.27 | 32.18 | download |
| 3 | 37.10 | 12.47 | 32.61 | download |
| 4 | 36.45 | 12.11 | 32.30 | download |
| 5 | 36.82 | 11.91 | 32.43 | download |
To generate predictions using these models, please download the above checkpoints and replace the --model_name_or_path line in predict_qmsum_bart.sh accordingly.