Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Async runtimes / parallelization in atomic-lib #424

Open
joepio opened this issue Jun 3, 2022 · 2 comments
Open

Async runtimes / parallelization in atomic-lib #424

joepio opened this issue Jun 3, 2022 · 2 comments

Comments

@joepio
Copy link
Member

joepio commented Jun 3, 2022

atomic-server does a lot of async stuff, as an Actix project.

However, atomic-lib does not. @AlexMikhalev recently added paralellization to Db::all_resources, which helped speed it up. There are probably other places, too, where it might help to spin up some threads.

I'm not sure where we can expect to gain a lot by paralellization, but I expect that Db::query and Commit::apply_opts as costly functions that do a lot of loopy stuff could speed up quite a bit.

@AlexMikhalev
Copy link
Collaborator

Add benchmarks to measure the performance of specific queries and I will have a go at speeding it up - let's chunk this task into multiple subtasks.

@AlexMikhalev
Copy link
Collaborator

We have to rethink async and parallelisation in connection with the discussion #433 - Sled:IVec doesn't have Sync implemented so can't scale reads via Arc, we can use https://crates.io/crates/dashmap as cache, serialize it to disk using serde and then at some point migrate from sled if or when we need to. I also saw the news that sled is sunsetting, but I haven't seen confirmation.

For parallel we need to make a choice: are we going to add tokio as a dependency for atomic-data-rust or we need to make calls to parallelized functions - like all_resources sync from atomic-data-server. You can't mix both - tokio async and any other threadpool executors.
It's possible to use tokio for CPU-bound tasks, see here and here but it requires dedicated effort.

@joepio joepio mentioned this issue Nov 25, 2022
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants