Ekump/apmsp 1756 add trace exporter integration tests backup #894

ekump · 2025-02-24T18:11:17Z

DO NOT MERGE

…on to send data integration tests

If the decoder sees a key it doesn't recognize for a span we return a Decode error to the caller. The least we can do is tell them the problematic key. The original error message was getting swallowed when mapping errors.

pr-commenter · 2025-02-24T18:21:38Z

Benchmarks

Comparison

Benchmark execution time: 2025-02-24 18:21:12

Comparing candidate commit 3cf2678 in PR branch ekump/APMSP-1756-add-trace-exporter-integration-tests-BACKUP with baseline commit e47f5e9 in branch main.

Found 3 performance improvements and 47 performance regressions! Performance is the same for 2 metrics, 2 unstable metrics.

scenario:benching deserializing traces from msgpack to their internal representation

🟥 execution_time [+3.116ms; +3.219ms] or [+5.690%; +5.879%]

scenario:benching string interning on wordpress profile

🟩 execution_time [-12.045µs; -11.926µs] or [-7.990%; -7.911%]

scenario:concentrator/add_spans_to_concentrator

🟥 execution_time [+314.425µs; +317.850µs] or [+5.213%; +5.270%]

scenario:credit_card/is_card_number/

🟥 execution_time [+373.941ns; +374.917ns] or [+9.555%; +9.580%]
🟥 throughput [-22339753.339op/s; -22281907.327op/s] or [-8.743%; -8.720%]

scenario:credit_card/is_card_number/ 3782-8224-6310-005

🟥 execution_time [+7.879µs; +8.138µs] or [+9.542%; +9.856%]
🟥 throughput [-1088395.272op/s; -1053589.374op/s] or [-8.987%; -8.699%]

scenario:credit_card/is_card_number/ 378282246310005

🟥 execution_time [+3.955µs; +4.138µs] or [+4.975%; +5.204%]
🟥 throughput [-623033.441op/s; -595926.005op/s] or [-4.953%; -4.738%]

scenario:credit_card/is_card_number/37828224631

🟥 execution_time [+371.748ns; +374.191ns] or [+9.496%; +9.559%]
🟥 throughput [-22296460.689op/s; -22146171.218op/s] or [-8.728%; -8.669%]

scenario:credit_card/is_card_number/378282246310005

🟥 execution_time [+4.363µs; +4.502µs] or [+5.723%; +5.905%]
🟥 throughput [-732710.139op/s; -709402.864op/s] or [-5.586%; -5.408%]

scenario:credit_card/is_card_number/37828224631000521389798

🟥 execution_time [+7.226µs; +7.238µs] or [+14.050%; +14.073%]
🟥 throughput [-2398864.366op/s; -2394934.782op/s] or [-12.338%; -12.318%]

scenario:credit_card/is_card_number/x371413321323331

🟥 execution_time [+405.251ns; +407.245ns] or [+6.713%; +6.746%]
🟥 throughput [-10472319.369op/s; -10419400.069op/s] or [-6.322%; -6.290%]

scenario:credit_card/is_card_number_no_luhn/

🟥 execution_time [+374.423ns; +375.433ns] or [+9.567%; +9.593%]
🟥 throughput [-22369439.562op/s; -22310538.731op/s] or [-8.754%; -8.731%]

scenario:credit_card/is_card_number_no_luhn/ 3782-8224-6310-005

🟥 execution_time [+3.735µs; +3.844µs] or [+5.645%; +5.810%]
🟥 throughput [-829724.856op/s; -807143.478op/s] or [-5.490%; -5.341%]

scenario:credit_card/is_card_number_no_luhn/ 378282246310005

🟥 execution_time [+4.880µs; +4.993µs] or [+8.172%; +8.363%]
🟥 throughput [-1292043.517op/s; -1264636.568op/s] or [-7.715%; -7.551%]

scenario:credit_card/is_card_number_no_luhn/37828224631

🟥 execution_time [+373.987ns; +375.068ns] or [+9.557%; +9.584%]
🟥 throughput [-22351358.941op/s; -22287631.016op/s] or [-8.747%; -8.722%]

scenario:credit_card/is_card_number_no_luhn/378282246310005

🟥 execution_time [+5.035µs; +5.084µs] or [+8.854%; +8.940%]
🟥 throughput [-1444291.233op/s; -1429474.886op/s] or [-8.213%; -8.129%]

scenario:credit_card/is_card_number_no_luhn/37828224631000521389798

🟥 execution_time [+7.228µs; +7.243µs] or [+14.054%; +14.082%]
🟥 throughput [-2400106.333op/s; -2395544.978op/s] or [-12.345%; -12.321%]

scenario:credit_card/is_card_number_no_luhn/x371413321323331

🟥 execution_time [+402.943ns; +405.329ns] or [+6.674%; +6.713%]
🟥 throughput [-10422317.613op/s; -10358414.101op/s] or [-6.293%; -6.254%]

scenario:ip_address/quantize_peer_ip_address_benchmark

🟥 execution_time [+377.975ns; +394.371ns] or [+7.444%; +7.767%]

scenario:normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...

🟩 execution_time [-27.263µs; -26.846µs] or [-13.036%; -12.837%]
🟩 throughput [+704584.524op/s; +717081.550op/s] or [+14.735%; +14.997%]

scenario:normalization/normalize_name/normalize_name/bad-name

🟥 execution_time [+2.836µs; +2.888µs] or [+15.459%; +15.741%]
🟥 throughput [-7421300.566op/s; -7291884.190op/s] or [-13.614%; -13.376%]

scenario:normalization/normalize_name/normalize_name/good

🟥 execution_time [+3.532µs; +3.547µs] or [+32.868%; +33.003%]
🟥 throughput [-23098134.422op/s; -23010770.713op/s] or [-24.822%; -24.728%]

scenario:normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...

🟥 execution_time [+195.738µs; +196.034µs] or [+38.701%; +38.759%]
🟥 throughput [-552416.736op/s; -551539.859op/s] or [-27.940%; -27.895%]

scenario:normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて

🟥 execution_time [+20.256µs; +20.385µs] or [+4.466%; +4.495%]
🟥 throughput [-94857.105op/s; -94266.375op/s] or [-4.302%; -4.275%]

scenario:normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters

🟥 execution_time [+14.552µs; +14.644µs] or [+8.238%; +8.290%]
🟥 throughput [-433507.000op/s; -430694.712op/s] or [-7.658%; -7.608%]

scenario:normalization/normalize_service/normalize_service/[empty string]

🟥 execution_time [+8.807µs; +8.831µs] or [+23.400%; +23.463%]
🟥 throughput [-5049980.076op/s; -5037342.457op/s] or [-19.007%; -18.959%]

scenario:normalization/normalize_trace/test_trace

🟥 execution_time [+16.760ns; +21.592ns] or [+6.574%; +8.470%]

scenario:redis/obfuscate_redis_string

🟥 execution_time [+6.736µs; +7.200µs] or [+20.441%; +21.850%]

scenario:sql/obfuscate_sql_string

🟥 execution_time [+2.905µs; +2.987µs] or [+4.343%; +4.466%]

scenario:tags/replace_trace_tags

🟥 execution_time [+279.060ns; +287.455ns] or [+11.397%; +11.739%]

Candidate

Candidate benchmark details

Group 1

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`3cf2678`	1740420328	ekump/APMSP-1756-add-trace-exporter-integration-tests-BACKUP

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
redis/obfuscate_redis_string	execution_time	39.356µs	39.919µs ± 0.882µs	39.515µs ± 0.054µs	39.601µs	41.781µs	41.819µs	43.111µs	9.10%	1.727	1.226	2.20%	0.062µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
redis/obfuscate_redis_string	execution_time	[39.797µs; 40.041µs] or [-0.306%; +0.306%]	None	None	None

Group 2

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`3cf2678`	1740420328	ekump/APMSP-1756-add-trace-exporter-integration-tests-BACKUP

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
normalization/normalize_trace/test_trace	execution_time	263.675ns	274.107ns ± 13.706ns	267.594ns ± 2.394ns	273.990ns	304.047ns	318.248ns	320.224ns	19.67%	1.751	2.023	4.99%	0.969ns	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
normalization/normalize_trace/test_trace	execution_time	[272.208ns; 276.007ns] or [-0.693%; +0.693%]	None	None	None

Group 3

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`3cf2678`	1740420328	ekump/APMSP-1756-add-trace-exporter-integration-tests-BACKUP

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
concentrator/add_spans_to_concentrator	execution_time	6.328ms	6.347ms ± 0.010ms	6.346ms ± 0.005ms	6.352ms	6.358ms	6.390ms	6.426ms	1.26%	3.851	24.453	0.16%	0.001ms	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
concentrator/add_spans_to_concentrator	execution_time	[6.346ms; 6.349ms] or [-0.022%; +0.022%]	None	None	None

Group 4

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`3cf2678`	1740420328	ekump/APMSP-1756-add-trace-exporter-integration-tests-BACKUP

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
sql/obfuscate_sql_string	execution_time	69.603µs	69.836µs ± 0.230µs	69.814µs ± 0.037µs	69.857µs	69.952µs	70.042µs	72.921µs	4.45%	12.146	160.461	0.33%	0.016µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
sql/obfuscate_sql_string	execution_time	[69.804µs; 69.868µs] or [-0.046%; +0.046%]	None	None	None

Group 5

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`3cf2678`	1740420328	ekump/APMSP-1756-add-trace-exporter-integration-tests-BACKUP

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
ip_address/quantize_peer_ip_address_benchmark	execution_time	5.395µs	5.464µs ± 0.038µs	5.460µs ± 0.037µs	5.499µs	5.510µs	5.528µs	5.547µs	1.59%	-0.015	-1.443	0.69%	0.003µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
ip_address/quantize_peer_ip_address_benchmark	execution_time	[5.459µs; 5.469µs] or [-0.096%; +0.096%]	None	None	None

Group 6

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`3cf2678`	1740420328	ekump/APMSP-1756-add-trace-exporter-integration-tests-BACKUP

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
benching string interning on wordpress profile	execution_time	138.156µs	138.768µs ± 0.297µs	138.724µs ± 0.141µs	138.893µs	139.358µs	139.834µs	139.994µs	0.92%	1.321	3.185	0.21%	0.021µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
benching string interning on wordpress profile	execution_time	[138.727µs; 138.809µs] or [-0.030%; +0.030%]	None	None	None

Group 7

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`3cf2678`	1740420328	ekump/APMSP-1756-add-trace-exporter-integration-tests-BACKUP

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...	execution_time	179.500µs	182.081µs ± 1.499µs	181.930µs ± 1.090µs	183.112µs	184.598µs	185.769µs	187.712µs	3.18%	0.587	0.167	0.82%	0.106µs	1	200
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...	throughput	5327312.760op/s	5492431.244op/s ± 45011.008op/s	5496631.322op/s ± 32742.742op/s	5527832.517op/s	5556734.085op/s	5566731.970op/s	5571022.882op/s	1.35%	-0.543	0.058	0.82%	3182.759op/s	1	200
normalization/normalize_name/normalize_name/bad-name	execution_time	20.912µs	21.207µs ± 0.138µs	21.184µs ± 0.104µs	21.315µs	21.421µs	21.488µs	21.757µs	2.70%	0.515	0.053	0.65%	0.010µs	1	200
normalization/normalize_name/normalize_name/bad-name	throughput	45961954.992op/s	47156262.891op/s ± 305090.110op/s	47204920.915op/s ± 231213.297op/s	47402065.367op/s	47571619.606op/s	47655974.923op/s	47819474.652op/s	1.30%	-0.480	-0.044	0.65%	21573.129op/s	1	200
normalization/normalize_name/normalize_name/good	execution_time	14.202µs	14.286µs ± 0.045µs	14.281µs ± 0.031µs	14.314µs	14.362µs	14.408µs	14.413µs	0.93%	0.523	-0.172	0.32%	0.003µs	1	200
normalization/normalize_name/normalize_name/good	throughput	69381728.522op/s	69999663.118op/s ± 221839.642op/s	70025336.440op/s ± 154252.646op/s	70167004.728op/s	70328838.568op/s	70387751.004op/s	70412822.815op/s	0.55%	-0.508	-0.194	0.32%	15686.432op/s	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...	execution_time	[181.873µs; 182.289µs] or [-0.114%; +0.114%]	None	None	None
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...	throughput	[5486193.152op/s; 5498669.337op/s] or [-0.114%; +0.114%]	None	None	None
normalization/normalize_name/normalize_name/bad-name	execution_time	[21.188µs; 21.226µs] or [-0.090%; +0.090%]	None	None	None
normalization/normalize_name/normalize_name/bad-name	throughput	[47113980.336op/s; 47198545.446op/s] or [-0.090%; +0.090%]	None	None	None
normalization/normalize_name/normalize_name/good	execution_time	[14.280µs; 14.292µs] or [-0.044%; +0.044%]	None	None	None
normalization/normalize_name/normalize_name/good	throughput	[69968918.277op/s; 70030407.958op/s] or [-0.044%; +0.044%]	None	None	None

Group 8

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`3cf2678`	1740420328	ekump/APMSP-1756-add-trace-exporter-integration-tests-BACKUP

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...	execution_time	700.261µs	701.660µs ± 0.805µs	701.622µs ± 0.486µs	702.116µs	702.721µs	703.226µs	708.750µs	1.02%	3.505	28.249	0.11%	0.057µs	1	200
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...	throughput	1410934.591op/s	1425192.810op/s ± 1629.556op/s	1425269.658op/s ± 986.710op/s	1426229.004op/s	1427085.456op/s	1427494.398op/s	1428039.915op/s	0.19%	-3.444	27.548	0.11%	115.227op/s	1	200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて	execution_time	472.980µs	473.830µs ± 0.384µs	473.817µs ± 0.260µs	474.077µs	474.483µs	474.814µs	475.013µs	0.25%	0.291	-0.034	0.08%	0.027µs	1	200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて	throughput	2105205.775op/s	2110461.505op/s ± 1711.673op/s	2110519.228op/s ± 1155.841op/s	2111670.117op/s	2113128.292op/s	2113796.881op/s	2114255.949op/s	0.18%	-0.286	-0.039	0.08%	121.034op/s	1	200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters	execution_time	190.340µs	191.247µs ± 0.199µs	191.252µs ± 0.124µs	191.368µs	191.533µs	191.724µs	191.904µs	0.34%	-0.386	2.417	0.10%	0.014µs	1	200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters	throughput	5210947.854op/s	5228846.306op/s ± 5437.753op/s	5228700.687op/s ± 3396.371op/s	5232443.943op/s	5237512.110op/s	5243553.929op/s	5253752.897op/s	0.48%	0.399	2.446	0.10%	384.507op/s	1	200
normalization/normalize_service/normalize_service/[empty string]	execution_time	46.324µs	46.457µs ± 0.074µs	46.455µs ± 0.037µs	46.490µs	46.544µs	46.599µs	47.003µs	1.18%	3.119	20.152	0.16%	0.005µs	1	200
normalization/normalize_service/normalize_service/[empty string]	throughput	21275327.003op/s	21525307.115op/s ± 34099.273op/s	21526172.263op/s ± 16965.747op/s	21543379.872op/s	21570304.005op/s	21582578.370op/s	21587220.122op/s	0.28%	-3.060	19.643	0.16%	2411.183op/s	1	200
normalization/normalize_service/normalize_service/test_ASCII	execution_time	48.970µs	49.121µs ± 0.068µs	49.119µs ± 0.047µs	49.164µs	49.243µs	49.305µs	49.332µs	0.43%	0.473	0.154	0.14%	0.005µs	1	200
normalization/normalize_service/normalize_service/test_ASCII	throughput	20270779.906op/s	20357973.641op/s ± 28322.463op/s	20358696.561op/s ± 19554.964op/s	20379061.202op/s	20398911.628op/s	20415624.658op/s	20420743.296op/s	0.30%	-0.465	0.141	0.14%	2002.701op/s	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...	execution_time	[701.549µs; 701.772µs] or [-0.016%; +0.016%]	None	None	None
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...	throughput	[1424966.969op/s; 1425418.651op/s] or [-0.016%; +0.016%]	None	None	None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて	execution_time	[473.777µs; 473.884µs] or [-0.011%; +0.011%]	None	None	None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて	throughput	[2110224.284op/s; 2110698.726op/s] or [-0.011%; +0.011%]	None	None	None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters	execution_time	[191.219µs; 191.275µs] or [-0.014%; +0.014%]	None	None	None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters	throughput	[5228092.686op/s; 5229599.926op/s] or [-0.014%; +0.014%]	None	None	None
normalization/normalize_service/normalize_service/[empty string]	execution_time	[46.447µs; 46.467µs] or [-0.022%; +0.022%]	None	None	None
normalization/normalize_service/normalize_service/[empty string]	throughput	[21520581.284op/s; 21530032.947op/s] or [-0.022%; +0.022%]	None	None	None
normalization/normalize_service/normalize_service/test_ASCII	execution_time	[49.111µs; 49.130µs] or [-0.019%; +0.019%]	None	None	None
normalization/normalize_service/normalize_service/test_ASCII	throughput	[20354048.420op/s; 20361898.862op/s] or [-0.019%; +0.019%]	None	None	None

Group 9

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`3cf2678`	1740420328	ekump/APMSP-1756-add-trace-exporter-integration-tests-BACKUP

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
two way interface	execution_time	18.887µs	28.450µs ± 13.630µs	19.104µs ± 0.180µs	36.893µs	45.723µs	47.574µs	146.022µs	664.35%	3.815	27.760	47.79%	0.964µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
two way interface	execution_time	[26.561µs; 30.339µs] or [-6.640%; +6.640%]	None	None	None

Group 10

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`3cf2678`	1740420328	ekump/APMSP-1756-add-trace-exporter-integration-tests-BACKUP

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
credit_card/is_card_number/	execution_time	4.270µs	4.288µs ± 0.003µs	4.288µs ± 0.001µs	4.289µs	4.292µs	4.294µs	4.297µs	0.23%	-1.621	14.015	0.06%	0.000µs	1	200
credit_card/is_card_number/	throughput	232694203.910op/s	233202422.061op/s ± 145870.806op/s	233220891.484op/s ± 54148.087op/s	233260654.763op/s	233333481.150op/s	233508019.463op/s	234174242.700op/s	0.41%	1.646	14.153	0.06%	10314.624op/s	1	200
credit_card/is_card_number/ 3782-8224-6310-005	execution_time	89.530µs	90.582µs ± 0.635µs	90.538µs ± 0.421µs	90.965µs	91.498µs	91.888µs	95.076µs	5.01%	1.882	11.050	0.70%	0.045µs	1	200
credit_card/is_card_number/ 3782-8224-6310-005	throughput	10517918.411op/s	11040297.655op/s ± 76424.926op/s	11045040.587op/s ± 51069.737op/s	11095349.385op/s	11145535.921op/s	11165766.624op/s	11169461.766op/s	1.13%	-1.694	9.444	0.69%	5404.058op/s	1	200
credit_card/is_card_number/ 378282246310005	execution_time	83.038µs	83.549µs ± 0.388µs	83.517µs ± 0.125µs	83.658µs	83.858µs	83.993µs	88.263µs	5.68%	8.974	106.798	0.46%	0.027µs	1	200
credit_card/is_card_number/ 378282246310005	throughput	11329744.763op/s	11969270.562op/s ± 53465.721op/s	11973548.648op/s ± 17987.727op/s	11989369.626op/s	12018588.710op/s	12031894.126op/s	12042626.079op/s	0.58%	-8.584	100.602	0.45%	3780.597op/s	1	200
credit_card/is_card_number/37828224631	execution_time	4.269µs	4.288µs ± 0.005µs	4.287µs ± 0.001µs	4.289µs	4.291µs	4.294µs	4.343µs	1.29%	7.905	95.017	0.11%	0.000µs	1	200
credit_card/is_card_number/37828224631	throughput	230270878.699op/s	233228121.303op/s ± 252653.304op/s	233249046.746op/s ± 56035.699op/s	233293164.524op/s	233462388.270op/s	233551663.605op/s	234249906.797op/s	0.43%	-7.792	93.507	0.11%	17865.286op/s	1	200
credit_card/is_card_number/378282246310005	execution_time	79.896µs	80.673µs ± 0.245µs	80.628µs ± 0.103µs	80.735µs	81.160µs	81.597µs	81.693µs	1.32%	1.375	3.602	0.30%	0.017µs	1	200
credit_card/is_card_number/378282246310005	throughput	12240920.667op/s	12395902.516op/s ± 37552.278op/s	12402620.054op/s ± 15880.058op/s	12417065.409op/s	12437205.281op/s	12457202.449op/s	12516344.428op/s	0.92%	-1.341	3.513	0.30%	2655.347op/s	1	200
credit_card/is_card_number/37828224631000521389798	execution_time	58.475µs	58.665µs ± 0.035µs	58.664µs ± 0.019µs	58.686µs	58.722µs	58.733µs	58.735µs	0.12%	-0.924	3.758	0.06%	0.002µs	1	200
credit_card/is_card_number/37828224631000521389798	throughput	17025599.831op/s	17045989.488op/s ± 10078.052op/s	17046100.231op/s ± 5629.570op/s	17051119.131op/s	17062362.391op/s	17069927.877op/s	17101448.493op/s	0.32%	0.933	3.803	0.06%	712.626op/s	1	200
credit_card/is_card_number/x371413321323331	execution_time	6.433µs	6.443µs ± 0.003µs	6.442µs ± 0.001µs	6.444µs	6.448µs	6.452µs	6.460µs	0.28%	1.153	4.755	0.05%	0.000µs	1	200
credit_card/is_card_number/x371413321323331	throughput	154787769.213op/s	155213988.134op/s ± 82037.443op/s	155222546.075op/s ± 34906.764op/s	155253737.200op/s	155331770.307op/s	155395936.873op/s	155453139.893op/s	0.15%	-1.145	4.720	0.05%	5800.923op/s	1	200
credit_card/is_card_number_no_luhn/	execution_time	4.271µs	4.288µs ± 0.003µs	4.288µs ± 0.001µs	4.290µs	4.293µs	4.296µs	4.300µs	0.29%	-0.445	6.863	0.07%	0.000µs	1	200
credit_card/is_card_number_no_luhn/	throughput	232540283.217op/s	233185726.152op/s ± 161752.886op/s	233203393.109op/s ± 57987.126op/s	233247436.698op/s	233439723.925op/s	233525807.968op/s	234148512.208op/s	0.41%	0.463	6.933	0.07%	11437.656op/s	1	200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005	execution_time	69.193µs	69.957µs ± 0.364µs	69.943µs ± 0.250µs	70.197µs	70.607µs	70.861µs	70.879µs	1.34%	0.307	-0.336	0.52%	0.026µs	1	200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005	throughput	14108552.585op/s	14294813.700op/s ± 74336.034op/s	14297418.602op/s ± 51083.722op/s	14347783.140op/s	14413304.077op/s	14434616.806op/s	14452286.635op/s	1.08%	-0.283	-0.356	0.52%	5256.351op/s	1	200
credit_card/is_card_number_no_luhn/ 378282246310005	execution_time	63.852µs	64.646µs ± 0.399µs	64.603µs ± 0.295µs	64.945µs	65.347µs	65.545µs	65.602µs	1.55%	0.336	-0.658	0.62%	0.028µs	1	200
credit_card/is_card_number_no_luhn/ 378282246310005	throughput	15243514.155op/s	15469398.507op/s ± 95350.274op/s	15479236.931op/s ± 70420.901op/s	15544099.006op/s	15600567.813op/s	15639835.836op/s	15661205.486op/s	1.18%	-0.314	-0.675	0.61%	6742.283op/s	1	200
credit_card/is_card_number_no_luhn/37828224631	execution_time	4.272µs	4.288µs ± 0.003µs	4.288µs ± 0.001µs	4.289µs	4.293µs	4.295µs	4.300µs	0.29%	-0.069	4.346	0.07%	0.000µs	1	200
credit_card/is_card_number_no_luhn/37828224631	throughput	232546479.642op/s	233213727.080op/s ± 165754.910op/s	233224388.655op/s ± 72811.779op/s	233276514.891op/s	233465939.419op/s	233576519.799op/s	234062646.321op/s	0.36%	0.082	4.370	0.07%	11720.642op/s	1	200
credit_card/is_card_number_no_luhn/378282246310005	execution_time	61.764µs	61.925µs ± 0.061µs	61.930µs ± 0.032µs	61.958µs	62.017µs	62.096µs	62.171µs	0.39%	0.195	1.359	0.10%	0.004µs	1	200
credit_card/is_card_number_no_luhn/378282246310005	throughput	16084728.791op/s	16148587.071op/s ± 15961.727op/s	16147347.537op/s ± 8446.993op/s	16157955.384op/s	16175242.829op/s	16187771.224op/s	16190536.631op/s	0.27%	-0.185	1.341	0.10%	1128.665op/s	1	200
credit_card/is_card_number_no_luhn/37828224631000521389798	execution_time	58.546µs	58.669µs ± 0.041µs	58.667µs ± 0.020µs	58.689µs	58.735µs	58.773µs	58.929µs	0.45%	1.308	7.592	0.07%	0.003µs	1	200
credit_card/is_card_number_no_luhn/37828224631000521389798	throughput	16969587.027op/s	17044818.875op/s ± 12035.048op/s	17045462.191op/s ± 5804.171op/s	17050696.632op/s	17062726.222op/s	17069325.879op/s	17080463.366op/s	0.21%	-1.291	7.493	0.07%	851.006op/s	1	200
credit_card/is_card_number_no_luhn/x371413321323331	execution_time	6.433µs	6.442µs ± 0.003µs	6.442µs ± 0.001µs	6.444µs	6.447µs	6.451µs	6.455µs	0.19%	0.402	2.049	0.05%	0.000µs	1	200
credit_card/is_card_number_no_luhn/x371413321323331	throughput	154929795.254op/s	155230846.983op/s ± 78247.697op/s	155229502.521op/s ± 35174.777op/s	155266004.509op/s	155374110.344op/s	155408979.303op/s	155447208.506op/s	0.14%	-0.396	2.038	0.05%	5532.948op/s	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
credit_card/is_card_number/	execution_time	[4.288µs; 4.288µs] or [-0.009%; +0.009%]	None	None	None
credit_card/is_card_number/	throughput	[233182205.770op/s; 233222638.351op/s] or [-0.009%; +0.009%]	None	None	None
credit_card/is_card_number/ 3782-8224-6310-005	execution_time	[90.494µs; 90.670µs] or [-0.097%; +0.097%]	None	None	None
credit_card/is_card_number/ 3782-8224-6310-005	throughput	[11029705.895op/s; 11050889.414op/s] or [-0.096%; +0.096%]	None	None	None
credit_card/is_card_number/ 378282246310005	execution_time	[83.495µs; 83.603µs] or [-0.064%; +0.064%]	None	None	None
credit_card/is_card_number/ 378282246310005	throughput	[11961860.728op/s; 11976680.397op/s] or [-0.062%; +0.062%]	None	None	None
credit_card/is_card_number/37828224631	execution_time	[4.287µs; 4.288µs] or [-0.015%; +0.015%]	None	None	None
credit_card/is_card_number/37828224631	throughput	[233193105.985op/s; 233263136.621op/s] or [-0.015%; +0.015%]	None	None	None
credit_card/is_card_number/378282246310005	execution_time	[80.639µs; 80.707µs] or [-0.042%; +0.042%]	None	None	None
credit_card/is_card_number/378282246310005	throughput	[12390698.132op/s; 12401106.901op/s] or [-0.042%; +0.042%]	None	None	None
credit_card/is_card_number/37828224631000521389798	execution_time	[58.660µs; 58.670µs] or [-0.008%; +0.008%]	None	None	None
credit_card/is_card_number/37828224631000521389798	throughput	[17044592.767op/s; 17047386.209op/s] or [-0.008%; +0.008%]	None	None	None
credit_card/is_card_number/x371413321323331	execution_time	[6.442µs; 6.443µs] or [-0.007%; +0.007%]	None	None	None
credit_card/is_card_number/x371413321323331	throughput	[155202618.533op/s; 155225357.734op/s] or [-0.007%; +0.007%]	None	None	None
credit_card/is_card_number_no_luhn/	execution_time	[4.288µs; 4.289µs] or [-0.010%; +0.010%]	None	None	None
credit_card/is_card_number_no_luhn/	throughput	[233163308.758op/s; 233208143.547op/s] or [-0.010%; +0.010%]	None	None	None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005	execution_time	[69.907µs; 70.008µs] or [-0.072%; +0.072%]	None	None	None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005	throughput	[14284511.441op/s; 14305115.960op/s] or [-0.072%; +0.072%]	None	None	None
credit_card/is_card_number_no_luhn/ 378282246310005	execution_time	[64.591µs; 64.702µs] or [-0.086%; +0.086%]	None	None	None
credit_card/is_card_number_no_luhn/ 378282246310005	throughput	[15456183.876op/s; 15482613.138op/s] or [-0.085%; +0.085%]	None	None	None
credit_card/is_card_number_no_luhn/37828224631	execution_time	[4.287µs; 4.288µs] or [-0.010%; +0.010%]	None	None	None
credit_card/is_card_number_no_luhn/37828224631	throughput	[233190755.043op/s; 233236699.116op/s] or [-0.010%; +0.010%]	None	None	None
credit_card/is_card_number_no_luhn/378282246310005	execution_time	[61.916µs; 61.933µs] or [-0.014%; +0.014%]	None	None	None
credit_card/is_card_number_no_luhn/378282246310005	throughput	[16146374.929op/s; 16150799.213op/s] or [-0.014%; +0.014%]	None	None	None
credit_card/is_card_number_no_luhn/37828224631000521389798	execution_time	[58.663µs; 58.675µs] or [-0.010%; +0.010%]	None	None	None
credit_card/is_card_number_no_luhn/37828224631000521389798	throughput	[17043150.933op/s; 17046486.817op/s] or [-0.010%; +0.010%]	None	None	None
credit_card/is_card_number_no_luhn/x371413321323331	execution_time	[6.442µs; 6.442µs] or [-0.007%; +0.007%]	None	None	None
credit_card/is_card_number_no_luhn/x371413321323331	throughput	[155220002.605op/s; 155241691.362op/s] or [-0.007%; +0.007%]	None	None	None

Group 11

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`3cf2678`	1740420328	ekump/APMSP-1756-add-trace-exporter-integration-tests-BACKUP

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
write only interface	execution_time	1.396µs	3.379µs ± 1.490µs	3.179µs ± 0.023µs	3.203µs	3.808µs	14.570µs	15.606µs	390.97%	7.437	56.048	43.98%	0.105µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
write only interface	execution_time	[3.172µs; 3.585µs] or [-6.110%; +6.110%]	None	None	None

Group 12

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`3cf2678`	1740420328	ekump/APMSP-1756-add-trace-exporter-integration-tests-BACKUP

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
benching deserializing traces from msgpack to their internal representation	execution_time	57.689ms	57.931ms ± 0.159ms	57.898ms ± 0.094ms	58.011ms	58.239ms	58.422ms	58.521ms	1.08%	1.189	1.672	0.27%	0.011ms	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
benching deserializing traces from msgpack to their internal representation	execution_time	[57.909ms; 57.953ms] or [-0.038%; +0.038%]	None	None	None

Group 13

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`3cf2678`	1740420328	ekump/APMSP-1756-add-trace-exporter-integration-tests-BACKUP

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
tags/replace_trace_tags	execution_time	2.667µs	2.732µs ± 0.017µs	2.729µs ± 0.008µs	2.738µs	2.768µs	2.775µs	2.777µs	1.75%	0.156	1.900	0.62%	0.001µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
tags/replace_trace_tags	execution_time	[2.730µs; 2.734µs] or [-0.087%; +0.087%]	None	None	None

Baseline

Omitted due to size.

ivoanjo · 2025-03-20T09:33:45Z

Is this PR still relevant?

ekump · 2025-03-21T13:28:51Z

@ivoanjo, nope. I'll close it out.

ivoanjo · 2025-03-21T13:45:34Z

Cool, thanks for confirming! :)

ekump added 7 commits February 5, 2025 16:11

APMSP-1779 add span deserialization and Payload collection constructi…

3f257c0

…on to send data integration tests

return the offending key when decoding v04 msgpack spans

ad0e57f

If the decoder sees a key it doesn't recognize for a span we return a Decode error to the caller. The least we can do is tell them the problematic key. The original error message was getting swallowed when mapping errors.

wip, broken

029fc39

WIP: refactor send_deser_ser

9f2ca5a

wip

bf87e55

support query params for communicating with the test agent

530581c

wip, broken test

3cf2678

ekump closed this Mar 21, 2025

ekump deleted the ekump/APMSP-1756-add-trace-exporter-integration-tests-BACKUP branch March 21, 2025 13:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ekump/apmsp 1756 add trace exporter integration tests backup #894

Ekump/apmsp 1756 add trace exporter integration tests backup #894

ekump commented Feb 24, 2025

pr-commenter bot commented Feb 24, 2025 •

edited

Loading

Group 1

Group 2

Group 3

Group 4

Group 5

Group 6

Group 7

Group 8

Group 9

Group 10

Group 11

Group 12

Group 13

ivoanjo commented Mar 20, 2025

ekump commented Mar 21, 2025

ivoanjo commented Mar 21, 2025

Ekump/apmsp 1756 add trace exporter integration tests backup #894

Ekump/apmsp 1756 add trace exporter integration tests backup #894

Conversation

ekump commented Feb 24, 2025

pr-commenter bot commented Feb 24, 2025 • edited Loading

Benchmarks

Comparison

scenario:benching deserializing traces from msgpack to their internal representation

scenario:benching string interning on wordpress profile

scenario:concentrator/add_spans_to_concentrator

scenario:credit_card/is_card_number/

scenario:credit_card/is_card_number/ 3782-8224-6310-005

scenario:credit_card/is_card_number/ 378282246310005

scenario:credit_card/is_card_number/37828224631

scenario:credit_card/is_card_number/378282246310005

scenario:credit_card/is_card_number/37828224631000521389798

scenario:credit_card/is_card_number/x371413321323331

scenario:credit_card/is_card_number_no_luhn/

scenario:credit_card/is_card_number_no_luhn/ 3782-8224-6310-005

scenario:credit_card/is_card_number_no_luhn/ 378282246310005

scenario:credit_card/is_card_number_no_luhn/37828224631

scenario:credit_card/is_card_number_no_luhn/378282246310005

scenario:credit_card/is_card_number_no_luhn/37828224631000521389798

scenario:credit_card/is_card_number_no_luhn/x371413321323331

scenario:ip_address/quantize_peer_ip_address_benchmark

scenario:normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...

scenario:normalization/normalize_name/normalize_name/bad-name

scenario:normalization/normalize_name/normalize_name/good

scenario:normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...

scenario:normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて

scenario:normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters

scenario:normalization/normalize_service/normalize_service/[empty string]

scenario:normalization/normalize_trace/test_trace

scenario:redis/obfuscate_redis_string

scenario:sql/obfuscate_sql_string

scenario:tags/replace_trace_tags

Candidate

Group 1

Group 2

Group 3

Group 4

Group 5

Group 6

Group 7

Group 8

Group 9

Group 10

Group 11

Group 12

Group 13

Baseline

ivoanjo commented Mar 20, 2025

ekump commented Mar 21, 2025

ivoanjo commented Mar 21, 2025

pr-commenter bot commented Feb 24, 2025 •

edited

Loading