Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update pytorch_data_loader_with_multiple_workers.py #86

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

ankit-amazon
Copy link
Contributor

improved compliant example

Copy link
Contributor

@san2488 san2488 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure this the right fix. I think the issue is that for non-zero num_workers, the dataset should not be stored in a list. So the fix is to not use the list.

Using num_workers as zero is a trivial fix but is likely not what the customer wants.

@linghuiluo
Copy link

Good point @san2488 I agree with it. It is a compliant case when num_workers = 0. I'd suggest to instead of using datasets.ImageFolder, using a user-defined class and it should have a field which uses numpy array to store image path like DataIter
in this issue pytorch/pytorch#13246, but self.data = [] should be changed to numpy array. In this case, num_workers can be any number bigger than 0


class DataIter(Dataset):
def __init__(self):
self.data_np = np.array([x for x in range(24000000)])

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can that data be more realistic rather than just numbers? e.g. path to image files in a folder.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants