-
- Downloads
Use parallel gzip instead of gzip for compression
golangs compress/gzip isn't a parallel implementation, so it's quite a bit slower on most modern servers then pgzip. The below benchmark run shows that publishing a debian bullseye mirror snapshot (amd64, arm64, armhf, source) shows a gain of about 35% in publishing time (when skipping bz2 using MR #1081) ``` hyperfine -w 1 -m 3 -L aptly aptly-nobz2,aptly-nobz2-pgzip -p "{aptly} -config aptly.conf publish drop bullseye || true" "{aptly} -config aptly.conf publish snapshot --skip-bz2=true --skip-contents --skip-signing bullseye" Benchmark 1: aptly-nobz2 -config aptly.conf publish snapshot --skip-bz2=true --skip-contents --skip-signing bullseye Time (mean ± σ): 35.548 s ± 0.378 s [User: 39.465 s, System: 10.046 s] Range (min … max): 35.149 s … 35.902 s 3 runs Benchmark 2: aptly-nobz2-pgzip -config aptly.conf publish snapshot --skip-bz2=true --skip-contents --skip-signing bullseye Time (mean ± σ): 26.592 s ± 0.069 s [User: 42.207 s, System: 9.676 s] Range (min … max): 26.521 s … 26.660 s 3 runs Summary 'aptly-nobz2-pgzip -config aptly.conf publish snapshot --skip-bz2=true --skip-contents --skip-signing bullseye' ran 1.34 ± 0.01 times faster than 'aptly-nobz2 -config aptly.conf publish snapshot --skip-bz2=true --skip-contents --skip-signing bullseye' ``` Signed-off-by:Sjoerd Simons <sjoerd@collabora.com> Upstream MR: https://github.com/aptly-dev/aptly/pull/1082
... | @@ -17,6 +17,7 @@ require ( | ... | @@ -17,6 +17,7 @@ require ( |
github.com/jlaffaye/ftp v0.0.0-20180404123514-2403248fa8cc // indirect | github.com/jlaffaye/ftp v0.0.0-20180404123514-2403248fa8cc // indirect | ||
github.com/kjk/lzma v0.0.0-20161016003348-3fd93898850d | github.com/kjk/lzma v0.0.0-20161016003348-3fd93898850d | ||
github.com/klauspost/compress v1.13.6 | github.com/klauspost/compress v1.13.6 | ||
github.com/klauspost/pgzip v1.2.5 | |||
github.com/mattn/go-colorable v0.1.2 // indirect | github.com/mattn/go-colorable v0.1.2 // indirect | ||
github.com/mattn/go-isatty v0.0.14 // indirect | github.com/mattn/go-isatty v0.0.14 // indirect | ||
github.com/mattn/go-runewidth v0.0.2 // indirect | github.com/mattn/go-runewidth v0.0.2 // indirect | ||
... | ... |
Please register or sign in to comment