Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Async support for atomic-lib #12

Closed
3 tasks
joepio opened this issue Sep 16, 2020 · 2 comments
Closed
3 tasks

Async support for atomic-lib #12

joepio opened this issue Sep 16, 2020 · 2 comments
Labels
performance Speed improvements

Comments

@joepio
Copy link
Member

joepio commented Sep 16, 2020

Since Atomic Data uses a lot of data fetching, some calls might take a while. Sometimes, multiple resources will need to be fetched at once. Instead of executing each request serially, having an parallelized async process for this would minimize execution time. Currently, every call is blocking in atomic_lib, and the HTTP library ureq is sync, too. Perhaps this will cause issues, but we can change ureq for some other HTTP library.

I'm kind of new to rust, and even more so to async programming, so I need to write down my thoughts on this. First, let's identify which processes might have the highest 'blocking' time. E.g. serializing internal AD3 to JSON. This requires all properties to be fetched, and these might not have been loaded yet.

Now, if we'd start implementing async, a lot needs to happen:

  • ureq has to be replaced by some async HTTP(S) library, preferably with rustls support. Maybe reqwest or surf?
  • a lot of the current functions (serializers that depend on store, get_resource) will have to become async, which will also 'infect' all functions that might use these.
  • the Storelike trait (and perhaps more) need to use async_trait
@joepio joepio added the performance Speed improvements label Nov 22, 2020
@joepio
Copy link
Member Author

joepio commented Jan 3, 2022

I'm not entirely sure whether async is the way to go for this library. Sure, some things could made faster using async (fetching multiple resources, for example), but it comes at a price: runtime overhead.

@joepio
Copy link
Member Author

joepio commented Mar 23, 2022

I'm going to close this, as I don't think it's worth the performance cost

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance Speed improvements
Projects
None yet
Development

No branches or pull requests

1 participant