Cannot index into a NestedTensor with grad ie nt[0]

[erichan1]

🐛 Bug

Can index into a NestedTensor without grad, but cannot index into a NestedTensor with grad

To Reproduce

Steps to reproduce the behavior:

import torch
c = torch.Tensor([1,2])
d = torch.nested_tensor([c,c])
print(d[0]) # works
d.requires_grad=True
print(d[0]) # fails

Traceback (most recent call last):
  File "/fsx/users/erichan1/work/erichan1_test/random/nt_index_test.py", line 6, in <module>
    print(d[0]) # fails
RuntimeError: Internal error: NestedTensorImpl doesn't support sizes. Please file an issue on https://github.com/pytorch/nestedtensor

Expected behavior

The second print should print tensor([1., 2.]), but fails instead.

Environment

• PyTorch Version (e.g., 1.0): 1.13.0a0+gitfd17ac2
• OS (e.g., Linux): Linux
• How you installed PyTorch (conda, pip, source): Pip
• Build command you used (if compiling from source): pip install -e . (python setup.py develop)
• Python version: 3.9
• CUDA/cuDNN version: n/a
• GPU models and configuration: n/a
• Any other relevant information:

Additional context

n/a