Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Evolving: FF-Perf Tests #519

Closed
dechdev opened this issue Feb 14, 2022 · 3 comments
Closed

Evolving: FF-Perf Tests #519

dechdev opened this issue Feb 14, 2022 · 3 comments

Comments

@dechdev
Copy link
Contributor

dechdev commented Feb 14, 2022

Soak Tests

Long term tests of FireFly where we want to find FireFly's limits and push performance

ID Date Snapshot Start Time End Time Total Time Blockchain # Broadcast Workers # Private Msg Workers # Token Mint Workers # On-Chain Workers Github Issue Github Commit Comments Versions
1A 2/11/2022 https://github.com/eberger727/images/blob/main/1A.png 2/11/2022 19:40:00 2/14/2022 12:20:00 64:40:00 Ethereum 150 150 0 0 N/A 5cdf975 No errors found in the logs, performance over time does deteriorate, there is a split in performance between broadcasts and private messages halfway through the test. Logs were overwritten for this period of time, so cause can not be determined -
7f9a832b-b55f-41af-a9c8-d076d967553f 03/03/2022 https://github.com/eberger727/images/blob/main/7f9a832b-b55f-41af-a9c8-d076d967553f-1.png and https://github.com/eberger727/images/blob/main/7f9a832b-b55f-41af-a9c8-d076d967553f-2.png 3-3-2022 10:31_PM 3-8-2022 3:30 PM 112 Hrs 59 Min geth 100 100 100 0 N/A a57cc8a This is a soak test for FireFly v0.13.1. Performance is degraded over time. No errors or rejected transactions. ethconnect:8283da4a9408b626c2bbf55ab6d7f44f132129eb2fbf289eb22e3959b0132033 fabconnect:0dee1f38ed41fcdc9f8eba8ac15e108740afe57dd0b11445c29f2ceca06e127a dataexchange-https:8625bb1685738365359174ae3bafecece934e6cc6a60c15e2a364324565c7bf7 tokens-erc1155:169d5bd8c75a44603eb3d99175f3e2121c086d9fd5a039073fce6e09320df09e tokens-erc20-erc721:dbd7b2c81ec80b6158eaa66e84e3dc635d1f5c31e8296270eab66fd5bbc0afc6 build:null ui:null cli:null//
2d8c2fa8-9e98-471b-b853-3f62f3d9e1f1 04_11_2022_12_12_PM Add Snapshot After Test 04_11_2022_12_12_PM Add After Test Add After Test fabric Add Num Broadcasts Add Num Private Add Num Minters Add Num On-Chain Add related Github Issue 8f219f7 Add After Test ethconnect:7336bb15640a01eea1c440195461862b6982a24e1a33b4ea0d89a30acb8bcf7b fabconnect:d6de00e7377ba9a510f8f8caedcce32cdef471070008d28e7d166c4acb7d617f dataexchange-https:c02c7e9cf283ae84f21eca473614b0f88f9766754339ee58ec5e951fcbe180cf tokens-erc1155:7deef15f636a723857adb25993d7e025a7c9f8a1f2e08322ec804c4d1fdbdc35 tokens-erc20-erc721:a71b0dcc1889de74b6ae0493b5e75c9e7c75b16e81864f26d714d26c9992f326 build:null ui:null cli:null//

Sanity Tests

Short-term tests (30 - 60 minutes) to confirm FireFly works as expected before cutting a release or merging new changes

ID Date Snapshot Start Time End Time Total Time Blockchain # Broadcast Workers # Private Msg Workers # Token Mint Workers # On-Chain Workers Github Issue Github Commit Comments Versions
1B 2/11/2022 https://github.com/eberger727/images/blob/main/1B.png 2/14/22 1:20 PM 2/14/2022 14:00:00 00:40:00 Ethereum 50 50 0 0 N/A cc970ff This was a test to verify a release cut for v0.13.0. No errors found that should block the release -
2B 2/16/2022 https://github.com/eberger727/images/blob/main/2B.png 2/16/22 11:45 AM 2/16/2022 1:05:00 PM 01:20:00 Fabric 150 150 0 0 N/A a8cb82e This was a sanity test to verify fabric'c performance for v0.13.0. No major issues found -
c7f2f2dd-9c81-437c-8606-674879a388d4 03/08/2022 nohup ff-perf msg_broadcast msg_private token_mint -l 1h -r "did:firefly:org/org_1" -w 300 &> ff-perf.log & image - stopped early to diagnose mint 10:30_PM Add After Test Add After Test geth 100 100 100 0 Add related Github Issue e46b921 Add After Test ethconnect:c18b15187f8fbcab066e873b13e3d2c23ce24043a73249d08438f9aa2dd5f7f1 fabconnect:0744f9cb01ea1de4e141d83c29aab9aadf1ee8bc4481b7c8025c712da7f79932 dataexchange-https:8625bb1685738365359174ae3bafecece934e6cc6a60c15e2a364324565c7bf7 tokens-erc1155:2cf3711b1a5e3864bfc9c313e270073edb19148303de37e3f8afc5d6b95c390f tokens-erc20-erc721:142dcd72355a73c2ba69f3c1eeab8a7ab5c304a7e01ee19f14a8a95d20b4262d build:null ui:null cli:null//
6f3be6c8-b774-4013-abec-916cb7dfa73f 03_10_2022_10_09_AM nohup ff-perf msg_broadcast msg_private -l 1h -r "did:firefly:org/org_1" -x "0x4a606008a7c535e438c825d074d2d37e59c97284" -w 300 &> ff-perf.log & showed concurrency issue, with 300 goroutines, because private senders were holding a RunAsGroup DB transaction while calling WriteNewMessage that requires a DB TX on the other thread. 03_10_2022_10_09_AM Add After Test Add After Test geth Add Num Broadcasts Add Num Private Add Num Minters Add Num On-Chain Add related Github Issue de1a6a8 Add After Test ethconnect:c18b15187f8fbcab066e873b13e3d2c23ce24043a73249d08438f9aa2dd5f7f1 fabconnect:0744f9cb01ea1de4e141d83c29aab9aadf1ee8bc4481b7c8025c712da7f79932 dataexchange-https:8625bb1685738365359174ae3bafecece934e6cc6a60c15e2a364324565c7bf7 tokens-erc1155:2cf3711b1a5e3864bfc9c313e270073edb19148303de37e3f8afc5d6b95c390f tokens-erc20-erc721:142dcd72355a73c2ba69f3c1eeab8a7ab5c304a7e01ee19f14a8a95d20b4262d build:null ui:null cli:null//
94d3daaa-ba27-45a5-8fae-4c0f4a48e6bc 03_10_2022_10_57_AM nohup ff-perf msg_broadcast msg_private -l 1h -r "did:firefly:org/org_1" -x "0x28672465e3d8808e9c4d0f9337629edb8efafbbc" -w 300 &> ff-perf.log & image 03_10_2022_10_57_AM Add After Test Add After Test geth Add Num Broadcasts Add Num Private Add Num Minters Add Num On-Chain Add related Github Issue 81582de Add After Test ethconnect:c18b15187f8fbcab066e873b13e3d2c23ce24043a73249d08438f9aa2dd5f7f1 fabconnect:0744f9cb01ea1de4e141d83c29aab9aadf1ee8bc4481b7c8025c712da7f79932 dataexchange-https:8625bb1685738365359174ae3bafecece934e6cc6a60c15e2a364324565c7bf7 tokens-erc1155:2cf3711b1a5e3864bfc9c313e270073edb19148303de37e3f8afc5d6b95c390f tokens-erc20-erc721:142dcd72355a73c2ba69f3c1eeab8a7ab5c304a7e01ee19f14a8a95d20b4262d build:null ui:null cli:null//
71bf6f9c-2013-418e-a0a6-82063bb40daa 03_14_2022_12_44_AM `nohup ff-perf msg_broadcast msg_private -l 8h -r "did:firefly:org/org_1" -x "0x09152d14b435c6ef69e4905452825ee038f577e2" -w 900 -a 0xa62b2a6c61107d9396a6b8d1103ded5cba82cbdc &> ff-perf.log &
` Screen Shot 2022-03-14 at 8 00 54 AM Add Snapshot After Test 03_14_2022_12_44_AM Add After Test Add After Test geth Add Num Broadcasts Add Num Private Add Num Minters Add Num On-Chain Add related Github Issue 77fb3fe Add After Test ethconnect:c18b15187f8fbcab066e873b13e3d2c23ce24043a73249d08438f9aa2dd5f7f1 fabconnect:0744f9cb01ea1de4e141d83c29aab9aadf1ee8bc4481b7c8025c712da7f79932 dataexchange-https:8625bb1685738365359174ae3bafecece934e6cc6a60c15e2a364324565c7bf7 tokens-erc1155:2cf3711b1a5e3864bfc9c313e270073edb19148303de37e3f8afc5d6b95c390f tokens-erc20-erc721:142dcd72355a73c2ba69f3c1eeab8a7ab5c304a7e01ee19f14a8a95d20b4262d build:null ui:null cli:null//
@dechdev dechdev pinned this issue Feb 14, 2022
@dechdev
Copy link
Contributor Author

dechdev commented Feb 22, 2022

#541

@jimthematrix
Copy link
Contributor

jimthematrix commented Apr 11, 2022

Long Running Test - Fabric

kicked off on jmeter2 (http://3.135.221.171/) at 2022-04-11T21:47:55.791Z (ask Jim for the ssh private key)

The run stopped at 2022-04-12T02:47:36.672Z due to Delinquent Messages

Grafana: http://3.135.221.171:3000/d/lNnIS4A7z/ff-performance?orgId=1&refresh=5s (ask Jim for credentials)

@dechdev dechdev unpinned this issue Apr 26, 2022
@dechdev
Copy link
Contributor Author

dechdev commented Apr 26, 2022

This is being tracked in Discord on an ongoing basis now

@dechdev dechdev closed this as completed Apr 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants