Skip to content

Enable ingestion via CDP in horizon verify-range docker image #5553

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tamirms opened this issue Dec 11, 2024 · 1 comment
Open

Enable ingestion via CDP in horizon verify-range docker image #5553

tamirms opened this issue Dec 11, 2024 · 1 comment

Comments

@tamirms
Copy link
Contributor

tamirms commented Dec 11, 2024

There is a verify-range docker image which is used to detect ingestion regressions in horizon release candidates. It works by ingesting a range of ledgers and then dumping the contents of the horizon history tables to disk. Then it repeats the same workflow but this time using the horizon release candidate. Finally, the output from both runs are compared. The job succeeds if the output from both runs are identical.

The verify-range job is effective in detecting regressions but it takes very long to run on full history. Also, the verify-range job often fails on tasks because it consumes too much memory. Both of these issues can be solved if we update the verify-range docker image to ingest from a GCS datastore of ledgers instead of using captive core.

@mollykarcher
Copy link
Contributor

Alternative that was discussed at team meeting on 3/19 was to retire verify-range in it's current form (ex. running in separate AWS account via AWS batch), but to instead recreate the same verification behavior using Stellar ETL, as is being done to verify TTP.

@mollykarcher mollykarcher removed this from the platform sprint 57 milestone Mar 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: To Do
Development

No branches or pull requests

2 participants