chore: Batch writer benchmarks #1552

disq · 2024-03-05T12:20:22Z

https://docs.google.com/spreadsheets/d/1NxKyOtvckW7yCbwLTL4VOu91XPNyd-Q0dYDMxjT-yH8/edit?usp=sharing

Current state (1000 * N)

GoLand2023.2/tmp/GoLand/___gobench_writers_test_go.test -test.v -test.paniconexit0 -test.bench ^\QBenchmarkWriterMemory\E$ -test.run ^$
goos: darwin
goarch: arm64
pkg: github.com/cloudquery/plugin-sdk/v4/writers
BenchmarkWriterMemory
BenchmarkWriterMemory/BatchWriter_batch10k_bytes100M
BenchmarkWriterMemory/BatchWriter_batch10k_bytes100M-10         	     465	   2556097 ns/op	        35.23 bytes/op
BenchmarkWriterMemory/BatchWriter_defaults
BenchmarkWriterMemory/BatchWriter_defaults-10                   	     469	   2595365 ns/op	        25.22 bytes/op
BenchmarkWriterMemory/BatchWriter_wide_batch10k_bytes100M
BenchmarkWriterMemory/BatchWriter_wide_batch10k_bytes100M-10    	       8	 129711115 ns/op	    178917 bytes/op
BenchmarkWriterMemory/BatchWriter_wide_defaults
BenchmarkWriterMemory/BatchWriter_wide_defaults-10              	       8	 151473391 ns/op	     48066 bytes/op
BenchmarkWriterMemory/MixedBatchWriter_batch10k_bytes100M
BenchmarkWriterMemory/MixedBatchWriter_batch10k_bytes100M-10    	     672	   1719326 ns/op	      1394 bytes/op
BenchmarkWriterMemory/MixedBatchWriter_defaults
BenchmarkWriterMemory/MixedBatchWriter_defaults-10              	     585	   1752100 ns/op	      1394 bytes/op
BenchmarkWriterMemory/MixedBatchWriter_wide_batch10k_bytes100M
BenchmarkWriterMemory/MixedBatchWriter_wide_batch10k_bytes100M-10         	       8	 142323412 ns/op	    185406 bytes/op
BenchmarkWriterMemory/MixedBatchWriter_wide_defaults
BenchmarkWriterMemory/MixedBatchWriter_wide_defaults-10                   	       8	 127524651 ns/op	     96165 bytes/op
BenchmarkWriterMemory/StreamingBatchWriter_bytes100M
BenchmarkWriterMemory/StreamingBatchWriter_bytes100M-10                   	     400	   2812296 ns/op	      1393 bytes/op
BenchmarkWriterMemory/StreamingBatchWriter_defaults
BenchmarkWriterMemory/StreamingBatchWriter_defaults-10                    	     394	   2803509 ns/op	      1393 bytes/op
BenchmarkWriterMemory/StreamingBatchWriter_wide_bytes100M
BenchmarkWriterMemory/StreamingBatchWriter_wide_bytes100M-10              	       9	 120658421 ns/op	     75653 bytes/op
BenchmarkWriterMemory/StreamingBatchWriter_wide_defaults
BenchmarkWriterMemory/StreamingBatchWriter_wide_defaults-10               	       9	 117394778 ns/op	     78724 bytes/op
PASS

Process finished with the exit code 0

github-actions · 2024-03-05T12:34:54Z

⏱️ Benchmark results

Glob-8 ns/op: 92.3
WriterMemory_BatchWriter_batch10k_bytes100M-8 bytes/op: 1,017
WriterMemory_BatchWriter_batch10k_bytes100M-8 ns/op: 2,800
WriterMemory_BatchWriter_defaults-8 bytes/op: 1,212
WriterMemory_BatchWriter_defaults-8 ns/op: 2,694
WriterMemory_BatchWriter_wide_batch10k_bytes100M-8 bytes/op: 197,164
WriterMemory_BatchWriter_wide_batch10k_bytes100M-8 ns/op: 226,966
BenchmarkWriterMemory_BatchWriter_wide_defaults-8_bytes_per_op: 1.845
WriterMemory_BatchWriter_wide_defaults-8 ns/op: 173,995
WriterMemory_MixedBatchWriter_batch10k_bytes100M-8 bytes/op: 1,426
WriterMemory_MixedBatchWriter_batch10k_bytes100M-8 ns/op: 2,617
WriterMemory_MixedBatchWriter_defaults-8 bytes/op: 1,426
WriterMemory_MixedBatchWriter_defaults-8 ns/op: 2,515
WriterMemory_MixedBatchWriter_wide_batch10k_bytes100M-8 bytes/op: 190,174
WriterMemory_MixedBatchWriter_wide_batch10k_bytes100M-8 ns/op: 221,517
WriterMemory_MixedBatchWriter_wide_defaults-8 bytes/op: 12,350
WriterMemory_MixedBatchWriter_wide_defaults-8 ns/op: 173,206
WriterMemory_StreamingBatchWriter_bytes100M-8 bytes/op: 1,394
WriterMemory_StreamingBatchWriter_bytes100M-8 ns/op: 3,139
WriterMemory_StreamingBatchWriter_defaults-8 bytes/op: 1,393
WriterMemory_StreamingBatchWriter_defaults-8 ns/op: 3,036
WriterMemory_StreamingBatchWriter_wide_bytes100M-8 bytes/op: 8,229
WriterMemory_StreamingBatchWriter_wide_bytes100M-8 ns/op: 154,501
WriterMemory_StreamingBatchWriter_wide_defaults-8 bytes/op: 10,784
WriterMemory_StreamingBatchWriter_wide_defaults-8 ns/op: 141,536

disq · 2024-03-05T12:35:27Z

writers/writers_test.go

+	}
+}
+
+func writerMatrix[T writers.Writer, C any, O ~func(T)](prefix string, constructor func(C, ...O) (T, error), client C, recordMaker func() func() arrow.Record, optsMatrix map[string][]O) []bCase {


I think I have a problem...

writers/writers_test.go

hermanschaaf · 2024-03-05T14:10:28Z

writers/writers_test.go

+			runtime.ReadMemStats(&mEnd)
+
+			allocatedBytes := mEnd.Alloc - mStart.Alloc
+			b.ReportMetric(float64(allocatedBytes)/float64(newN), "bytes/op")


Doesn't Go measure memory usage already if we provide -benchmem? What am I missing?

benchmem will report allocs/op (probably the same number as ours but I didn't compare them) and won't report until invoked with -benchmem or toggled from within the test. We also multiply the N to get more stable benchmarks (and hit writer batching spots).

hermanschaaf

Bit complicated logic but since it's a benchmark it's probably okay as long as it helps you find memory-related improvements 😅

Helps with memory. Benched with #1552, old: ``` BenchmarkWriterMemory/MixedBatchWriter_batch10k_bytes100M BenchmarkWriterMemory/MixedBatchWriter_batch10k_bytes100M-10 826 1445595 ns/op 1393 bytes/op BenchmarkWriterMemory/MixedBatchWriter_defaults BenchmarkWriterMemory/MixedBatchWriter_defaults-10 670 1504799 ns/op 1393 bytes/op BenchmarkWriterMemory/MixedBatchWriter_wide_batch10k_bytes100M BenchmarkWriterMemory/MixedBatchWriter_wide_batch10k_bytes100M-10 8 128966609 ns/op 172785 bytes/op BenchmarkWriterMemory/MixedBatchWriter_wide_defaults BenchmarkWriterMemory/MixedBatchWriter_wide_defaults-10 9 118630315 ns/op 60355 bytes/op ``` new: ``` BenchmarkWriterMemory/MixedBatchWriter_batch10k_bytes100M BenchmarkWriterMemory/MixedBatchWriter_batch10k_bytes100M-10 690 1476479 ns/op 1401 bytes/op BenchmarkWriterMemory/MixedBatchWriter_defaults BenchmarkWriterMemory/MixedBatchWriter_defaults-10 687 1539707 ns/op 1401 bytes/op BenchmarkWriterMemory/MixedBatchWriter_wide_batch10k_bytes100M BenchmarkWriterMemory/MixedBatchWriter_wide_batch10k_bytes100M-10 8 129154755 ns/op 173592 bytes/op BenchmarkWriterMemory/MixedBatchWriter_wide_defaults BenchmarkWriterMemory/MixedBatchWriter_wide_defaults-10 9 111416773 ns/op 44588 bytes/op ```

candiduslynx · 2024-03-06T10:34:38Z

writers/writers_test.go

+		for i := 0; i < numWideCols; i++ {
+			bldr.Field(i + 1).(*array.Int64Builder).Append(randVals[i])


Suggested change

for i := 0; i < numWideCols; i++ {

bldr.Field(i + 1).(*array.Int64Builder).Append(randVals[i])

for i := 1; i <= numWideCols; i++ {

bldr.Field(i).(*array.Int64Builder).Append(randVals[i])

you'd need to do randVals[i-1], adds compexity and potential copy/paste errors, just to avoid an + 1

writers/writers_test.go

chore: Batch writer benchmarks

d66fd9c

disq requested review from yevgenypats, hermanschaaf and erezrokah as code owners March 5, 2024 12:20

github-actions bot added ignore-for-release and removed ignore-for-release labels Mar 5, 2024

lint

fe9bbe7

disq commented Mar 5, 2024

View reviewed changes

remove underscore assignments

25d3fb3

disq commented Mar 5, 2024

View reviewed changes

writers/writers_test.go Outdated Show resolved Hide resolved

disq added 2 commits March 5, 2024 12:58

use spaces, prealloc correctly

9539680

use 1000 N multiplier

e188fc7

github-actions bot added ignore-for-release and removed ignore-for-release labels Mar 5, 2024

disq requested a review from candiduslynx March 5, 2024 13:07

disq mentioned this pull request Mar 5, 2024

fix: MixedBatchWriter should nil the slice instead of zeroing #1553

Merged

disq added 3 commits March 5, 2024 13:12

lint

d62fad5

No slash in benchmark delta

a836445

divide by correct/adjusted N

cbefcdb

hermanschaaf reviewed Mar 5, 2024

View reviewed changes

hermanschaaf approved these changes Mar 5, 2024

View reviewed changes

github-actions bot added ignore-for-release and removed ignore-for-release labels Mar 5, 2024

disq added 2 commits March 6, 2024 10:19

generics: infer by casting to interface first

73264ae

readability

6eee11a

candiduslynx approved these changes Mar 6, 2024

View reviewed changes

Clean benchmark: Directly use N and add -test.benchtime

dcaba18

erezrokah approved these changes Mar 6, 2024

View reviewed changes

disq added the automerge label Mar 6, 2024

Merge branch 'main' into chore/batchwriter-benchmarks

ff95763

kodiakhq bot merged commit 3050443 into cloudquery:main Mar 6, 2024
7 checks passed

disq deleted the chore/batchwriter-benchmarks branch March 6, 2024 12:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: Batch writer benchmarks #1552

chore: Batch writer benchmarks #1552

disq commented Mar 5, 2024 •

edited

Loading

github-actions bot commented Mar 5, 2024 •

edited

Loading

disq Mar 5, 2024

hermanschaaf Mar 5, 2024

disq Mar 5, 2024

hermanschaaf left a comment

candiduslynx Mar 6, 2024

disq Mar 6, 2024

		for i := 0; i < numWideCols; i++ {
		bldr.Field(i + 1).(*array.Int64Builder).Append(randVals[i])

chore: Batch writer benchmarks #1552

chore: Batch writer benchmarks #1552

Conversation

disq commented Mar 5, 2024 • edited Loading

github-actions bot commented Mar 5, 2024 • edited Loading

⏱️ Benchmark results

disq Mar 5, 2024

Choose a reason for hiding this comment

hermanschaaf Mar 5, 2024

Choose a reason for hiding this comment

disq Mar 5, 2024

Choose a reason for hiding this comment

hermanschaaf left a comment

Choose a reason for hiding this comment

candiduslynx Mar 6, 2024

Choose a reason for hiding this comment

disq Mar 6, 2024

Choose a reason for hiding this comment

disq commented Mar 5, 2024 •

edited

Loading

github-actions bot commented Mar 5, 2024 •

edited

Loading