Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature: Atomic Server can store blobs on S3 compatible storage - like AWS S3, Cloudflare durable objects #543

Open
AlexMikhalev opened this issue Nov 24, 2022 · 3 comments
Open

Feature: Atomic Server can store blobs on S3 compatible storage - like AWS S3, Cloudflare durable objects #543

AlexMikhalev opened this issue Nov 24, 2022 · 3 comments
Assignees
@AlexMikhalev

Comments

@AlexMikhalev
Copy link
Collaborator

To maintain lightweight for Atomic Server give user the opportunity to add S3 credenitals and endpoint with option to synchronise binary data with external S3 compatible servers.
I would check how https://github.com/quickwit-oss/quickwit does it.
@joepio joepio mentioned this issue Nov 24, 2022
Open
@joepio
Copy link
Member

joepio commented Nov 24, 2022

This would be pretty simple to implement, I think. The logic in download.rs and upload.rs could be altered depending on whether some s3 opts (like keys and bucket ID) are being set.

@AlexMikhalev
Copy link
Collaborator Author

Yes, and QuickWit have example of downloading from S3 in multithreaded way.

@AlexMikhalev AlexMikhalev self-assigned this Dec 24, 2022
@AlexMikhalev
Copy link
Collaborator Author

Prefered way to progress to use OpenDAL https://opendal.apache.org/docs/rust/opendal/services/struct.S3.html

@joepio joepio mentioned this issue Aug 4, 2023
Draft
7 tasks
@joepio joepio mentioned this issue Sep 6, 2023
Open
3 tasks
joepio pushed a commit to metame/atomic-server that referenced this issue Jan 18, 2024
@metame @joepio
S3 Support atomicdata-dev#543
d9403f9 
added s3 config vars

s3 uploads working

prefix file-id with store type

cargo format

download building

download working

possibly fix issue with :

refactor files logic into files module

static str lovelies

clean up log

changelog and readme updates

use tokio streams in upload

check bucket exists before unwrap

Store config in FileStore

cargo fmt

consolidate download fs path logic

move file stores to appstate

no unwrap

refactor

finish refactor

tests passing

back to nicer interface

documentation

move test to tests.rs

readme and dep fix
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@AlexMikhalev AlexMikhalev

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@AlexMikhalev @joepio