Varnet training default parameters

[asaksena98]

Hello, I have been training the e2e varnet on 1 GPU with all the arguments set as default, except the sample rate is changed to 0.2. While the model performs quite well on the validation set (around 0.95 SSIM), it performs quite poorly on the public leaderboard for brain data (SSIM of 0.86 for 4x acceleration and 0.76 for 8x), and the images produced have lots of artifacts. My guess is that this occurs because the default acceleration rate is 4x and I should change it to either 8x or [4, 8], but I feel like there is more to this issue.

Apart from the sample and acceleration rate, are there any other potential reasons that might cause this issue?

[mmuckley]

Hello @asaksena98, I can't think of any further reasons than the one you mentioned. If the network has never seen an acceleration other than 5x, you wouldn't expect it to perform well at 4x or 8x.

[tianweiy]

@mmuckley I think OP trained using 4x and doesn't work well on 4x test set? the sample rate 0.2 refers to using a 1/5 subset?

[mmuckley]

Ah in that case I misunderstood it. I'll try to check the model I ran when I get a chance to reproduce this.

[asaksena98]

Ah in that case I misunderstood it. I'll try to check the model I ran when I get a chance to reproduce this.

Sure, thank you!

[mmuckley]

Hello @asaksena98, could you let me know if the items in PR #60 might address your issues?

[asaksena98]

Hello @asaksena98, could you let me know if the items in PR #60 might address your issues?

yup, i'll try running it with those changes

[asaksena98]

Also, for the varnet submission on the public leaderboard, what acceleration rate was used for training? Since you can only submit one reconstructions zip file, I am unsure whether to train strictly on 4x, 8x or pass [4, 8] into the --accelerations argument

[mmuckley]

In that code there is some logic in the evaluation script that detects the acceleration level and runs the corresponding model. So it runs a 4x model for the 4x files and the 8x model for the 8x files.

[asaksena98]

Hello @asaksena98, could you let me know if the items in PR #60 might address your issues?

Hello, so i tried running the model after making the changes to mri_module.py as suggested in PR #60 but I see no discernible differences in the ssim score after each epoch. I ran a small experiment where the sample rate = 0.01, and the ssim score after 3 epochs is around 0.92 both before and after the fix. I suppose what confuses me is that this score seems too high to be true for such a small sample rate given that the training and validation datasets are completely different

[mmuckley]

Okay thanks for this information. I'm going to be putting out a PR on including cs in the experimental folder soon and then I'll look into this.

[asaksena98]

sure, thank you!

[mmuckley]

Hello @asaksena98, I don't know if you fixed this on your end, but I did discover an issue with how masks were handled in the test set that could cause a discrepancy between validation and leaderboard performance:

https://github.com/facebookresearch/fastMRI/pull/67/files#diff-59b11ebf74d2dab3f0129abc1d00bd45L176-L181

This is fixed now in master. If this was your problem, you don't need to retrain; you just need to run with the new test script. Let me know if this helps.

[asaksena98]

Sure, thank you for the update!

[mmuckley]

Hello @asaksena98 any updates on your end? If not we can close this issue.

[asaksena98]

Yup its resolved on my end. Thank you!