archive/tar: Go 1.9beta1 source archive is packed in some strange way

[dim13]

What version of Go are you using (go version)?

1.9beta1

What operating system and processor architecture are you using (go env)?

OpenBSD 6.1/amd64

What did you do?

tar zxvf go1.9beta1.src.tar.gz

What did you expect to see?

extracted tar

What did you see instead?

tar: Invalid header, starting valid header search.
tar: Invalid header, starting valid header search.
tar: Invalid header, starting valid header search.
tar: Invalid header, starting valid header search.
tar: Invalid header, starting valid header search.

exit code: 1

[bradfitz]

@dsnet, could you take a look? OpenBSD 6.1's tar apparently doesn't like our tar.

[dsnet]

If I may ask, what is the exact version of the tar utility being used? This works on both BSD and GNU tar:

$ bsdtar --version
bsdtar 3.1.2 - libarchive 3.1.2

$ bsdtar -zxf go1.9beta1.src.tar.gz; echo $?
0

$ gnutar --version
tar (GNU tar) 1.27.1
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by John Gilmore and Jay Fenlason.

$ gnutar -zxf go1.9beta1.src.tar.gz; echo $?
0

[dsnet]

I'm going to guess this is the implementation of the tar utility that you are using?
https://github.com/openbsd/src/tree/master/bin/pax

I don't have an OpenBSD setup, so this may take me some time to repro.

[dim13]

Yes, you are right, it is a standard tar which is pax.

Steps to reproduce:

$ ftp https://storage.googleapis.com/golang/go1.9beta1.src.tar.gz

$ sha256 go1.9beta1.src.tar.gz
SHA256 (go1.9beta1.src.tar.gz) = e42dbd2071aadb28a4d293225b04b6b4215a35a7f04417a0e47ffa38f81d642d

$ tar ztf go1.9beta1.src.tar.gz 1>/dev/null
tar: Invalid header, starting valid header search.
tar: Invalid header, starting valid header search.
tar: Invalid header, starting valid header search.
tar: Invalid header, starting valid header search.
tar: Invalid header, starting valid header search.

$ echo $?
1

$ uname -a
OpenBSD dim13.org 6.1 GENERIC.MP#6 amd64

Note: this is a first time it occurs. Never had any troubles with any previous archives.

[dim13]

Previous release to compare (on same system):

$ ftp https://storage.googleapis.com/golang/go1.8.3.src.tar.gz

$ sha256 go1.8.3.src.tar.gz
SHA256 (go1.8.3.src.tar.gz) = 5f5dea2447e7dcfdc50fa6b94c512e58bfba5673c039259fd843f68829d99fa6

$ tar ztf go1.8.3.src.tar.gz 1>/dev/null

$ echo $?
0

[bradfitz]

@dsnet, any ideas?

[dim13]

After some tweaking I found at least a block in archive which after the things start to break apart: offset into ungzipped tar: 0x01B69600

The block contains a long file name ("go/src/cmd/vendor/github.com/google/pprof/internal/driver/testdata/PaxHeaders.0/pprof.contention.fla") truncated to 100 chars.

Probably reference issue #17630

[dsnet]

When Go1.8 was released, we ran into a similar breakage internally when dpkg failed to work on .deb packages since the Debian package manager only accepted the GNU format and not PAX. For that issue, I filed #18710.

It's possible that the BSD version of tar doesn't support PAX, but my brief glance through their source code seemed to indicate that it does.

The other thing I can think of is that the BSD tar tool doesn't like a non-NULL terminated name.

I'll finally get around to setting up OpenBSD tomorrow and test that theory out.

[dim13]

The null-termination of name doesn't matter. (See 2 attached files.)

The main difference is, how filename is encoded. BSD tar (pax) splits path in prefix and remainder such that length of remainder is less then 100 bytes. (Data Interchange Format - Extended tar header format - POSIX 1003.1-1990)

Go tar-ball is encoded differently. The full name is truncated on mark of 100 bytes. The actual name is stored in PAXRecord format ("%d %s=%s\n" % (size, key, value)).

longname.tar.gz
longname_without_null.tar.gz

[bradfitz]

@dsnet, you can get access to our OpenBSD builders. No need to set up your own.

[gopherbot]

CL https://golang.org/cl/46914 mentions this issue.

[dsnet]

Sent out a fix... I wish the OpenBSD implementation would output more debug information than just "invalid header"; it's really unhelpful. Honestly, I think this is an issue with OpenBSD's tar utility, but I'll make a change to Go's writer to play a little nicer.