Adding Minimal Reproducible Usage Example For TPU support on examples/seq2seq

[AdityaSoni19031997]

Attempt to resolve #5895.

Minimal Working Colab Example.

For using more than a single core, one needs to ensure that enough RAM is available else wait for PyTorch-XLA to release a stable version. They have also released a fix way back that prevent excessive memory usage for nightly.

[AdityaSoni19031997]

The test will obviously break, right?

[marton-avrios]

Any progress on merging this?

[AdityaSoni19031997]

they won't accept it as the checks have failed?
But it's expected for the checks to fail as I have modified modeling_bart and it's gonna have one more Param count.

[marton-avrios]

Is that so @sshleifer?

[sshleifer]

I think you need to call this again after line 952 to make the tests pass.

[sshleifer]

Thanks for the contribution, this looks awesome!

We can't merge with failing tests, but I think the tests can pass.

Could you also check

RUN_SLOW=1 pytest tests/test_modeling_bart.py
RUN_SLOW=1 pytest tests/test_modeling_marian.py
RUN_SLOW=1 pytest test_modeling_mbart.py

add the USE_CUDA=1 prefix to make them run faster on GPU.

[sshleifer]

Actually can we add a support_tpu flag to BartConfig, init it to False, and only allocate lm_head if it's set to True. I'm concerned that we are wasting RAM when we train on GPU. (I would happily change my mind if I encountered evidence that this change doesn't change GPU RAM consumption.)

[marton-avrios]

I tried this version and it seems to work but it stucks at "Validation sanity check". Working colab here

[AdityaSoni19031997]

Well I removed the validation check altogether by passing in the concerned flag to 0. Tried debugging to find out what's causing it, but I couldn't figure it out. If you will train and validate, it will work.

…

On Mon, 27 Jul 2020, 17:46 marton-avrios, ***@***.***> wrote: I tried this version and it seems to work but it stucks at "Validation sanity check". — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#5960 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFNPJJQYIPQDG5ZPYXDTTT3R5VV23ANCNFSM4PEHWJYA> .

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[sshleifer]

This is now supported by Seq2SeqTrainer. Use that if you want TPU support!