GHCup fails to install on armv7

[hasufell]

In GitLab by @j-hui on Aug 12, 2022, 03:54

I'm unable to install the GHC toolchain using ghcup on my armv7 board running Debian 10 (buster).

In particular, running curl --proto '=https' --tlsv1.2 -sSf https://get-ghcup.haskell.org | sh leads to:

... # normal setup process

[ Info  ] verifying digest of: ghc-8.10.7-armv7-deb10-linux.tar.xz
[ Info  ] Unpacking: ghc-8.10.7-armv7-deb10-linux.tar.xz to /home/j-hui/.ghcup/tmp/ghcup-425fcb2a
[ Info  ] Installing GHC (this may take a while)
[ Info  ] Merging file tree from "/home/j-hui/.ghcup/tmp/ghcup-f66ab128/home/j-hui/.ghcup/ghc/8.10.7" to "/home/j-hui/.ghcup/ghc/8.10.7"
[ Error ] Failed to merge file tree from /home/j-hui/.ghcup/tmp/ghcup-f66ab128/home/j-hui/.ghcup/ghc/8.10.7 to /home/j-hui/.ghcup/ghc/8.10.7
[ ...   ] exception was: /home/j-hui/.ghcup/tmp/ghcup-f66ab128/home/j-hui/.ghcup/ghc/8.10.7/bin: install: not a regular file or symlink: illegal operation
[ ...   ] ...you may need to delete /home/j-hui/.ghcup/ghc/8.10.7 manually. Make sure it's gone.
[ Error ] Also check the logs in /home/j-hui/.ghcup/logs
"_eghcup --cache install ghc recommended" failed!

I've already installed all dependencies that the build script suggests; is there something else I'm missing?

[hasufell]

In GitLab by @maerwald on Aug 12, 2022, 10:22

I don't have an armv7 to investigate. It's possible that our new installation logic has problems on some platforms.

You can download the previous release and use that: https://downloads.haskell.org/~ghcup/0.1.17.8/armv7-linux-ghcup-0.1.17.8

[hraftery]

I had the same "Failed to merge file tree" issue on armv7 boards (eg. Raspberry Pi 3 or CM3) running Debian 11. This does indeed seem to be related to new ghcup behaviour, since reverting to ghcup-0.1.17.8 does not present the issue.

This can be reproduced without an armv7/arm32v7 with a fairly minimal example using Docker, using the following Dockerfile:

FROM balenalib/raspberrypi3-debian:bullseye-build

RUN echo `uname -s`
RUN echo `uname -m`
RUN apt update
RUN apt install build-essential curl libffi-dev libffi7 libgmp-dev libgmp10 libncurses-dev libncurses5 libtinfo5
RUN curl --proto '=https' --tlsv1.2 -sSf https://get-ghcup.haskell.org | \
    BOOTSTRAP_HASKELL_NONINTERACTIVE=1 BOOTSTRAP_HASKELL_YAML="https://raw.githubusercontent.com/haskell/ghcup-metadata/develop/ghcup-0.0.6.yaml" sh

The explicit yaml is because 0.0.7, released 10 days ago, results in a "Unable to find a download for the requested version/distro." error instead! Pinning it to 0.0.6 in this (undocumented?) way is enough to reproduce the original issue.

Then run the following command in a directory with just the Dockerfile to reproduce the issue.

docker build --progress=plain .

You can also substitute the generic Debian image by changing the first line of Dockerfile to something like FROM debian:bullseye and specifying the architecture by running the following command instead:

docker build --progress=plain --platform=linux/arm/v7  .

The same error is produced.

---

Having confirmed this is an issue, to workaround it I needed a way to programmatically specify the ghcup version. Unfortunately it seems this is not supported by the bootstrap script, despite some promising functionality, because ghcup upgrade is always called. After some experimentation, the best option that worked for me was to download the version of the script at the revision before 0.1.18.0, and then because we know what we're getting, strip the upgrade command out of it.

The following did the trick for me:

grep -vw 'eghcup upgrade' bootstrap-haskell-60299b6b | \
    BOOTSTRAP_HASKELL_NONINTERACTIVE=1 BOOTSTRAP_HASKELL_MINIMAL=1 bash

where bootstrap-haskell-60299b6b is a local copy of the script at that commit id. Subsequently calling ghcup install ghc succeeds, demonstrating that this workaround is effective.

FWIW, it's seems ironic that the marvellous version handling power of ghcup doesn't extend to ghcup installation itself. I really value being able to pin to certain versions so infrastructure we build doesn't silently break in the future.

[hasufell]

This can be reproduced without an armv7/arm32v7 with a fairly minimal example using Docker, using the following Dockerfile:

This doesn't work on amd64 architecture. You need qemu emulation.

[hasufell]

Unable to find a download for the requested version/distro."

This is because 9.2.5 doesn't provide armv7 anymore: https://downloads.haskell.org/~ghc/9.2.5/

I think GHC devs have stopped supporting this platform.

I don't mind supporting it unofficially, but then other people have to step up and contribute the bindists.

[hasufell]

Unfortunately it seems this is not supported by the bootstrap script

I don't mind supporting these things in the bootstrap script:

1. overwriting of ghcup version
2. skipping of ghcup upgrade

Feel free to provide a PR. The script is here: https://github.com/haskell/ghcup-hs/blob/master/scripts/bootstrap/bootstrap-haskell

[hraftery]

This doesn't work on amd64 architecture.

Oh good to know! It's all still a bit mysterious to me. Seems to work out of the (Docker Desktop) box on x86_64 and arm64, but as you say, not on amd64. That would require some buildx/QEMU trickery.

This is because 9.2.5 doesn't provide armv7

Oh, and I see 8.10.7 (which is the version specified in my use case) is no longer the recommended/default version in the 0.0.7 yaml. I overlooked the default version aspect when I created the example.

contribute the bindists

Yeah I'm not too fussed about supporting the latest version on old architectures. I'm sure there be dragons. But the same version on same architecture continuing to work would be grand. I think in this case, as long as the binaries don't go away, the existing bindists will do?

supporting these things in the bootstrap script

Okay, thanks for the invite. Selfishly, having done the work to determine the root cause and create a workaround, the business case for contributing further is entirely ego-based now. If I clear the plate of obligations, then my ego will come knocking ;-)

[hasufell]

I've done some investigation and here's some odd output:

Prelude System.Posix.RawFilePath.Directory.Traversals> getDirectoryContents "/root/.ghcup/tmp/ghcup-b8f5ebb6/root/.ghcup/ghc/8.10.7/"
[(DirType 72,".."),(DirType 219,"."),(DirType 160,"share"),(DirType 35,"bin"),(DirType 255,"lib")]

Prelude System.Posix.RawFilePath.Directory.Traversals> getDirectoryContents "/"
[(DirType 0,"sys"),(DirType 0,".."),(DirType 0,"root"),(DirType 0,"var"),(DirType 0,"home"),(DirType 0,"."),(DirType 0,"etc"),(DirType 0,"srv"),(DirType 0,"mnt"),(DirType 0,"dev"),(DirType 0,"run"),(DirType 0,"tmp"),(DirType 0,"opt"),(DirType 0,"boot"),(DirType 0,"bin"),(DirType 0,"lib"),(DirType 0,"usr"),(DirType 0,"proc"),(DirType 0,"sbin"),(DirType 0,"media"),(DirType 0,".dockerenv")]

Uhm... DirType of 160 does not make any sense. Nor do all the 0 values.

The offending code is here:

ghcup-hs/lib/GHCup/Prelude/File/Posix/Traversals.hs

Lines 36 to 91 in 75b8911

	----------------------------------------------------------
	-- dodgy stuff
	type CDir = ()
	type CDirent = ()
	-- Posix doesn't export DirStream, so to re-use that type we need to use
	-- unsafeCoerce. It's just a newtype, so this is a legitimate usage.
	-- ugly trick.
	unpackDirStream :: DirStream -> Ptr CDir
	unpackDirStream = unsafeCoerce
	-- the __hscore_* functions are defined in the unix package. We can import them and let
	-- the linker figure it out.
	foreign import ccall unsafe "__hscore_readdir"
	c_readdir :: Ptr CDir -> Ptr (Ptr CDirent) -> IO CInt
	foreign import ccall unsafe "__hscore_free_dirent"
	c_freeDirEnt :: Ptr CDirent -> IO ()
	foreign import ccall unsafe "__hscore_d_name"
	c_name :: Ptr CDirent -> IO CString
	foreign import ccall unsafe "__posixdir_d_type"
	c_type :: Ptr CDirent -> IO DirType
	----------------------------------------------------------
	-- less dodgy but still lower-level
	readDirEnt :: DirStream -> IO (DirType, FilePath)
	readDirEnt (unpackDirStream -> dirp) =
	alloca $ \ptr_dEnt -> loop ptr_dEnt
	where
	loop ptr_dEnt = do
	resetErrno
	r <- c_readdir dirp ptr_dEnt
	if r == 0
	then do
	dEnt <- peek ptr_dEnt
	if dEnt == nullPtr
	then return (dtUnknown, mempty)
	else do
	dName <- c_name dEnt >>= peekFilePath
	dType <- c_type dEnt
	c_freeDirEnt dEnt
	return (dType, dName)
	else do
	errno <- getErrno
	if errno == eINTR
	then loop ptr_dEnt
	else do
	let (Errno eo) = errno
	if eo == 0
	then return (dtUnknown, mempty)
	else throwErrno "readDirEnt"

And here:

ghcup-hs/lib/GHCup/Prelude/File/Posix/Foreign.hsc

Line 49 in 75b8911

#{enum DirType, DirType, DT_BLK, DT_CHR, DT_DIR, DT_FIFO, DT_LNK, DT_REG, DT_SOCK, DT_UNKNOWN}

And we use some FFI functions from unix:

https://github.com/haskell/unix/blob/master/cbits/HsUnix.c

And getting d_type we defined ourselves:

ghcup-hs/cbits/dirutils.c

Lines 1 to 7 in 75b8911

	#include "dirutils.h"
	unsigned int
	__posixdir_d_type(struct dirent* d)
	{
	return(d -> d_type);
	}

---

I tried to compile some C code with open and readdir and access d_type field. It works fine. So it's defined and accessible.

Any ideas @angerman ?

[hasufell]

We tracked down the cause to some weird header shenanigans that causes wrong offsets due to missing __USE_FILE_OFFSET64.

struct dirent
  {
#ifndef __USE_FILE_OFFSET64
    __ino_t d_ino;
    __off_t d_off;
#else
    __ino64_t d_ino;
    __off64_t d_off;
#endif
    unsigned short int d_reclen;
    unsigned char d_type;
    char d_name[256];           /* We must not include limits.h! */
  };

Including ghcconfig.h before dirent.h fixes it.

[hasufell]

This will be fixed in next release.

Until then you can use this binary: https://downloads.haskell.org/ghcup/tmp/armv7-linux-ghcup-0.1.18.1