-
Notifications
You must be signed in to change notification settings - Fork 1.5k
feat(era): Add EraStream
that downloads all era files
#15613
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
great start, I think most of the functionality is already there.
I have some suggestions for structuring this, I think converting this into an actual stream type will be more flexible later, for example if we need to then route the downloaded files somewhere else we can easily add a channel on top
c767154
to
b707b64
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is looking great, love the tests.
have some (pedantic) nits.
I think we should also consider sprinkling a few trace!
calls here and there, e.g. when we started/finished downloading a file
EraClient
that downloads all era filesEraStream
that downloads all era files
3caf789
to
dfda1d1
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nice, left some suggestions, but otherwise this lgtm
The polls are happening very quickly even when nothing is happening
this could be due to the waker, which, from what I can see, can be removed entirely
Add checksum validation to detect files integrity
When continuing download from an interrupted run, the latest file might have not been downloaded fully
Any other data integrity corruption
all of these additional features we def want but can be added as followups
Previous behavior would end the stream and throw away the error
dfda1d1
to
e12c930
Compare
4e54784
to
cc8358e
Compare
… calls Makes the tests stable, deterministic and faster
9dae6e5
to
12af74a
Compare
Saves 8 MB on test data
7714f35
to
8cbad9a
Compare
Closes #15496
Adds
EraStream
to stream filenames of downloaded files in a givenfolder
.Some rests:
Tests could be rewritten so that they don't depend on real network