Revert "recommend using bz compression over gz"

This reverts commit ece1374.

experience with bz prooved less stable than desired

Author: maxlath

.gitignore

@@ -1,5 +1,4 @@
 node_modules
 build
 *ndjson
-*.extract.*
 *.sublime*

README.md

@@ -59,10 +59,10 @@ See [CHANGELOG.md](CHANGELOG.md) for version info
 ## Download dump
 
 ### Wikidata dumps
-Wikidata provides a bunch of [database dumps](https://www.wikidata.org/wiki/Wikidata:Database_download), among which the desired [JSON dump](https://www.wikidata.org/wiki/Wikidata:Database_download#JSON_dumps_.28recommended.29). As a Wikidata dump is a very laaarge file (September 2020: 55GB compressed), it is recommended to download that file first before doing operations on it, so that if anything crashes, you don't have to start the download from zero (the download time being usually the bottleneck).
+Wikidata provides a bunch of [database dumps](https://www.wikidata.org/wiki/Wikidata:Database_download), among which the desired [JSON dump](https://www.wikidata.org/wiki/Wikidata:Database_download#JSON_dumps_.28recommended.29). As a Wikidata dump is a very laaarge file (April 2020: 75GB compressed), it is recommended to download that file first before doing operations on it, so that if anything crashes, you don't have to start the download from zero (the download time being usually the bottleneck).
 ```sh
-wget -C https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.json.bz2
-cat latest-all.json.bz2 | bzcat | wikibase-dump-filter --claim P31:Q5 > humans.ndjson
+wget -C https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.json.gz
+cat latest-all.json.gz | gzip -d | wikibase-dump-filter --claim P31:Q5 > humans.ndjson
 ```
 
 ### Your own Wikibase instance dump

docs/cli.md

@@ -41,7 +41,7 @@ this command filters `entities_dump.json` into a subset where all lines are the
 
 * **directly from a Wikidata dump**
 ```sh
-curl https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.json.bz2 | bzcat | wikibase-dump-filter --claim P31:Q5 > humans.ndjson
+curl https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.json.gz | gzip -d | wikibase-dump-filter --claim P31:Q5 > humans.ndjson
 ```
 this can be quite convinient when you don't have enough space to keep the whole decompressed dump on your disk: here you only write the desired subset.
 

docs/examples.md

@@ -5,6 +5,6 @@
 The [equivalent SPARQL query](https://query.wikidata.org/#SELECT%20%3Fs%20%3FsLabel%20WHERE%20%7B%0A%20%20%20%20%3Farticlea%20schema%3Aabout%20%3Fs%20.%0A%20%20%20%20%3Farticlea%20schema%3AisPartOf%20%3Chttps%3A%2F%2Fzh.wikipedia.org%2F%3E%20.%0A%20%20%20%20%3Farticleb%20schema%3Aabout%20%3Fs%20.%0A%20%20%20%20%3Farticleb%20schema%3AisPartOf%20%3Chttps%3A%2F%2Ffr.wikipedia.org%2F%3E%20.%0A%20%20%20%20%23%20The%20article%20shouldn%27t%20be%20a%20disambiguation%20page%0A%20%20%09FILTER%20NOT%20EXISTS%20%7B%20%3Fs%20wdt%3AP31%20wd%3AQ4167410.%20%7D%0A%7D%0ALIMIT%205) times out
 
 ```sh
-DUMP='https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.json.bz2'
-curl $DUMP | bzcat | wikibase-dump-filter --sitelink 'zhwiki&frwiki' --keep id,labels,sitelinks --languages zh,fr --simplify > subset.ndjson
+DUMP='https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.json.gz'
+curl $DUMP | gzip -d | wikibase-dump-filter --sitelink 'zhwiki&frwiki' --keep id,labels,sitelinks --languages zh,fr --simplify > subset.ndjson
 ```

docs/parallelize.md

@@ -1,19 +1,19 @@
 ## Parallelize
 If your hardware happens to have several cores, we can do better:
-* replace `bzcat` by [`pbzip2`](https://linux.die.net/man/1/pbzip2)
+* replace `gzip` by [`pigz`](https://zlib.net/pigz/)
 * load balance lines over several `wikibase-dump-filter` processes using [`load-balance-lines`](https://github.com/maxlath/load-balance-lines) or something that does the same job
 
 ```sh
 # install those new dependencies
-sudo apt-get install pbzip2
+sudo apt-get install pigz
 npm install --global load-balance-lines
 
 # increase the max RAM available to node processes, to prevent allocation errors
 NODE_OPTIONS=--max_old_space_size=4096
 
-wget --continue https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.json.bz2
-nice -n+19 pbzip2 -cd < latest-all.json.bz2 | nice -n+19 load-balance-lines wikibase-dump-filter --claim P31:Q5 > humans.ndjson
+wget --continue https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.json.gz
+nice -n+19 pigz -d < latest-all.json.gz | nice -n+19 load-balance-lines wikibase-dump-filter --claim P31:Q5 > humans.ndjson
 ```
 
 Using [`nice`](http://man7.org/linux/man-pages/man1/nice.1.html) to tell the system that those processes, while eager to eat all the CPUs, should have the lowest priority.
-If you are not familiar with the `<` operator, it's the equivalent of `cat latest-all.json.bz2 | nice -n+19 pbzip2 -cd` but in a shell built-in way. (See [I/O Redirection doc](http://www.tldp.org/LDP/abs/html/io-redirection.html))
+If you are not familiar with the `<` operator, it's the equivalent of `cat latest-all.json.gz | nice -n+19 pigz -d` but in a shell built-in way. (See [I/O Redirection doc](http://www.tldp.org/LDP/abs/html/io-redirection.html))

scripts/init_fixtures

@@ -3,6 +3,6 @@
 
 mkdir -p test/fixtures
 # get the first 50 lines of the latest dump
-curl -s https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.json.bz2 | bzcat | head -n 50 > ./test/fixtures/entities
+curl -s https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.json.gz |gzip -d |head -n 50 > ./test/fixtures/entities
 
 cat ./test/fixtures/entities |head -n 2 |tail -n 1 |sed 's/,$//' > ./test/fixtures/entity