increased memory usage after upgrade from 1.11.3 -> 1.11.12

[jkowalski]

I'm running a bunch of e2e tests for Kopia in various configurations, including on 32-bit ARM box (Raspberry PI with 4GB RAM) and they have been pretty stable, until I upgraded compress library to 1.11.12. Since then the tests are seeing a bunch of OOMs like this:

https://github.com/kopia/kopia/runs/2136728758?check_suite_focus=true
https://github.com/kopia/kopia/runs/2136452986?check_suite_focus=true
(there are other flakes due to external infrastructure, but if you look in ARMHF you'll find out of memory) with stack trace like this:

runtime: out of memory: cannot allocate 33554432-byte block (2787573760 in use)
378
fatal error: out of memory
379

380
goroutine 432 [running]:
381
runtime.throw(0x907b30, 0xd)
382
	/usr/local/go/src/runtime/panic.go:1117 +0x5c fp=0x634fa948 sp=0x634fa934 pc=0x496b8
383
runtime.(*mcache).allocLarge(0xb6e522e0, 0x2000000, 0x20101, 0x429a4)
384
	/usr/local/go/src/runtime/mcache.go:226 +0x268 fp=0x634fa978 sp=0x634fa948 pc=0x28058
385
runtime.mallocgc(0x2000000, 0x80d250, 0x1, 0x0)
386
	/usr/local/go/src/runtime/malloc.go:1078 +0x9c8 fp=0x634fa9d4 sp=0x634fa978 pc=0x1e1e8
387
runtime.makeslice(0x80d250, 0x0, 0x2000000, 0xb11847b8)
388
	/usr/local/go/src/runtime/slice.go:98 +0x6c fp=0x634fa9e8 sp=0x634fa9d4 pc=0x628cc
389
github.com/klauspost/compress/zstd.(*fastBase).addBlock(0xba162000, 0x9e642000, 0xc7, 0x10000, 0x1db40)
390
	/root/go/pkg/mod/github.com/klauspost/[email protected]/zstd/enc_base.go:65 +0x70 fp=0x634faa28 sp=0x634fa9e8 pc=0x1e8ce8
391
github.com/klauspost/compress/zstd.(*betterFastEncoder).Encode(0xba162000, 0x9eafaa80, 0x9e642000, 0xc7, 0x10000)
392
	/root/go/pkg/mod/github.com/klauspost/[email protected]/zstd/enc_better.go:95 +0xc4 fp=0x634fab20 sp=0x634faa28 pc=0x1ebd80
393
github.com/klauspost/compress/zstd.(*betterFastEncoder).EncodeNoHist(0xba162000, 0x9eafaa80, 0x9e642000, 0xc7, 0x10000)
394
	/root/go/pkg/mod/github.com/klauspost/[email protected]/zstd/enc_better.go:519 +0x3c fp=0x634fab38 sp=0x634fab20 pc=0x1edb00
395
github.com/klauspost/compress/zstd.(*Encoder).EncodeAll(0x23381e0, 0x9e642000, 0xc7, 0x10000, 0x9e652000, 0x6, 0x10000, 0x0, 0x0, 0x0)
396
	/root/go/pkg/mod/github.com/klauspost/[email protected]/zstd/encoder.go:492 +0xb04 fp=0x634fac24 sp=0x634fab38 pc=0x1f62ac
397
github.com/klauspost/compress/zstd.(*Encoder).nextBlock(0x23381e0, 0x100001, 0xc7, 0x0)
398
	/root/go/pkg/mod/github.com/klauspost/[email protected]/zstd/encoder.go:180 +0x4b4 fp=0x634fac88 sp=0x634fac24 pc=0x1f4e9c
399
github.com/klauspost/compress/zstd.(*Encoder).Close(0x23381e0, 0x1, 0x4e)
400
	/root/go/pkg/mod/github.com/klauspost/[email protected]/zstd/encoder.go:387 +0x30 fp=0x634facbc sp=0x634fac88 pc=0x1f545c
401
github.com/kopia/kopia/repo/compression.(*zstdCompressor).Compress(0x2026780, 0x22e4f60, 0x94532000, 0xc7, 0x100000, 0x0, 0x0)
402
	/tmp/github-runner-kopia/kopia/kopia/repo/compression/compressor_zstd.go:54 +0x154 fp=0x634fad08 sp=0x634facbc pc=0x203a48
403
github.com/kopia/kopia/repo/object.maybeCompressedContentBytes(0xa226ec, 0x2026780, 0x22e4f60, 0x94532000, 0xc7, 0x100000, 0x0, 0x2026f00, 0x8, 0x0, ...)
404

The container that it's running this has ~2.5GB of RAM. The flake is probably a result of too heavy parallelization of tests, so I can try to fix that, but I wanted to make sure that increased memory usage is expected on ARM.

[klauspost]

Though it doesn't match the versions you mention, it likely is because of the increase in default window size: #247

You can tweak that using the WithWindowSize encoding parameter. Down to 512KB will reduce allocations. I will add an option to make it even lower.

[klauspost]

Otherwise, switch to the default compression mode which avoids this allocation altogether.