maxima doesn't build on Linux ppc64 (silius on skynet)

[williamstein]

The maxima spkg in sage-4.7.1 fails to build almost instantly with:

...
;;; Emitting code for UNARY.

Internal or unrecoverable error in:
not a lisp data object
  [2: No such file or directory]

Upstream: Not yet reported upstream; Will do shortly.

CC: @mwhansen

Component: porting

Reviewer: Jeroen Demeyer

Issue created by migration from https://trac.sagemath.org/ticket/11708

[williamstein]

comment:1

NOTE: A naive attempt with maxima-5.25 fails in exactly the same way.

[nexttime]

comment:2

Nice. I didn't know we have any Linux PPC in our build farm. :)

[kcrisman]

comment:3

Replying to @nexttime:

Nice. I didn't know we have any Linux PPC in our build farm. :)

I'll ask about this on the Maxima devel list. Maybe there is some little configuration thing that needs to be set.

[kiwifb]

comment:4

Could we see more of that log? And I'd like to see the ecl build log as well if it was possible.

[williamstein]

comment:5

Here's a huge log file

http://sage.math.washington.edu/home/wstein/days/32/silius/install.log-20110818.txt

It is huge due to building ATLAS, etc. So, just search in it.

It looks like maybe everything ends up building except Maxima (and Tachyon, which is fixed by another ticket).

wstein@silius:~/silius/sage-4.7.1> ./sage
----------------------------------------------------------------------
| Sage Version 4.7.1, Release Date: 2011-08-11                       |
| Type notebook() for the GUI, and license() for information.        |
----------------------------------------------------------------------
sage: time n = factorial(10^6)
Time: CPU 0.64 s, Wall: 0.65 s
sage: time n = factorial(10^7)
Time: CPU 11.81 s, Wall: 11.81 s
sage: !uname -a
Linux silius 2.6.32.43-0.4-ppc64 #1 SMP 2011-07-14 14:47:44 +0200 ppc64 ppc64 ppc64 GNU/Linux

[jhpalmieri]

comment:6

Here are log files for ecl and maxima:

• http://sage.math.washington.edu/home/palmieri/misc/ecl-11.1.1.p1.log
• http://sage.math.washington.edu/home/palmieri/misc/maxima-5.23.2.p0.log

[kiwifb]

comment:7

Hum... nothing jumps out at me, especially if sage compiled without problems including sage/libs/ecl.pyx. Is there a maxima rpm for sles 11? If so it may be worth looking in the spec file.

[williamstein]

comment:8

Yes, sage/libs/ecl.pyx compiles and passes all tests:

wstein@silius:~/silius/sage-4.7.1> ./sage -t devel/sage/sage/libs/ecl.pyx
sage -t  "devel/sage/sage/libs/ecl.pyx"                     
         [5.1 s]
 
----------------------------------------------------------------------
All tests passed!

[nexttime]

comment:9

Hmmm, there are a couple of warnings already, including

make[3]: warning:  Clock skew detected.  Your build may be incomplete.

but comparing the log to what I have doesn't look very different.

How about installing the Maxima spkg with strace -f [...] ./sage -i ..., to see which file ecl apparently doesn't find?

It would also perhaps be better to first try to build on a local filesystem (if that wasn't the case, which I assume).

[kcrisman]

comment:10

Here is a response from one Maxima developer.

Maxima probably isn't supported on this platform because probably no
developer normally works on PPC Linux.  I might be wrong about that.

On the other hand, I have built maxima just fine using a 64-bit version
of ecl on Linux (x86) and Solaris (sparc).  So I would guess that the
problem you are seeing is due to ecl.

If there are any other responses, I'll update here.

[nexttime]

comment:11

Did Dave ever manage to build Sage in 64-bit mode on SPARC?

As far as I know the only big-endians we build / test on are (or just use) 32 bits.

I also noticed that silius doesn't have libffi, but I don't think that's relevant here.

[williamstein]

comment:12

Replying to @nexttime:

How about installing the Maxima spkg with strace -f [...] ./sage -i ..., to see which file ecl apparently doesn't find?

It would also perhaps be better to first try to build on a local filesystem (if that wasn't the case, which I assume).

Do you want an account on the machine? Then you can try all this and maybe fix the problem!

Write me offlist at [email protected] for an account.

[kcrisman]

comment:13

By the way, this may be on skynet, but it's not listed at the wiki for Sage on skynet.

[williamstein]

comment:14

The machine was added to skynet 2 or 3 days ago.

The page http://wiki.sagemath.org/skynet is interesting. Note that it is surely out of date, since the compilers are always being updated, operating systems upgraded, etc.

[williamstein]

comment:15

I've added some info about silius stamped "today" to that wiki.

[kcrisman]

comment:16

The machine was added to skynet 2 or 3 days ago.

Cool.

The page http://wiki.sagemath.org/skynet is interesting. Note that it is surely out of date, since the compilers are always being updated, operating systems upgraded, etc.

John P. reminded me of it on another ticket. Even out-of-date, it seems useful.

[benjaminfjones]

comment:17

Replying to @nexttime:

I've been banging my head on this for a while now at Sage Days 32. I've tried the following so far:

• building ECL 11.1.1 independent of Sage and then building Maxima 5.25.0 (latest) on top of that (failed: same "redefining NIL" error that mhansen reported)
• building latest CVS version of ECL, then Maxima 5.25.0 on top of that (failed: same error as above)
• making a new .spkg for the latest ECL, then using that to try building the maxima .spkg (failed with "not a lisp data object" error)

How about installing the Maxima spkg with strace -f [...] ./sage -i ..., to see which file ecl apparently doesn't find?

Just tried this on silius, here is the part of the log just before the error message is printed:

[pid 13982] write(3, "\tif(!((V1)==(VV[26]))){\n", 24) = 24
[pid 13982] write(3, "\tgoto L21;}\n", 12) = 12
[pid 13982] write(3, "\tT0= LC9next(lex0)              "..., 67) = 67
[pid 13982] write(3, "\tcl_env_copy->nvalues=2;\n", 25) = 25
[pid 13982] write(3, "\tcl_env_copy->values[1]=T0;\n", 28) = 28
[pid 13982] write(3, "\tcl_env_copy->values[0]=V1;\n", 28) = 28
[pid 13982] write(3, "\treturn cl_env_copy->values[0];\n", 32) = 32
[pid 13982] write(3, "L21:;\n", 6)      = 6
[pid 13982] write(3, "\tif(!((V1)==(VV[30]))){\n", 24) = 24
[pid 13982] write(3, "\tgoto L25;}\n", 12) = 12
[pid 13982] write(3, "\tV1= LC9next(lex0)              "..., 67) = 67
[pid 13982] write(2, "\nInternal or unrecoverable error"..., 60) = 60
[pid 12588] <... read resumed> "\nInternal or unrecoverable error"..., 8192) = 60
[pid 12588] write(1, "\nInternal or unrecoverable error"..., 60
Internal or unrecoverable error in:
not a lisp data object
 <unfinished ...>
[pid 13982] write(2, "  [2: No such file or directory]"..., 33 <unfinished ...>

The full log is here: http://sage.math.washington.edu/home/bjones/strace_maxima.log.gz (It is VERY large)

It would also perhaps be better to first try to build on a local filesystem (if that wasn't the case, which I assume).

I've tried building the maxima .spkg (and Sage overall) both on my NFS mounted home folder and on the local /tmp on silius with no difference.

Other things I'm trying:

• running the ECL test-suite and comparing to another architectures (sage.math and skynet/eno)

[kiwifb]

comment:18

I am not convinced that the problem is with ecl at all. I would try installing another lisp (unfortunately as far as I can see there are no lisps at all in the SuSE linux 11.SP1 DVDs) and compile maxima against that. If it still fail that would point towards maxima rather than ecl. If it builds that would point to ecl. clisp or sbcl would be appropriate. Unfortunately I don't have access to sagemath or boxen so I cannot login into silius right now.

[benjaminfjones]

comment:19

I've been trying to build clisp on skynet/silius without success. I've tried with both GCC version 4.6.1 and GCC version 4.3. Both build processes end with:

./lisp.run -B . -N locale -E UTF-8 -Epathname 1:1 -Emisc 1:1 -norc -m 2MW -lp ../src/ -x '(and (load "../src/init.lisp") (sys::%saveinitmem) (ext::exit)) (ext::exit t)'
make: *** [interpreted.mem] Segmentation fault

I'm going to move on and try compiling SBCL..

[benjaminfjones]

comment:20

My SBCL build (using ecl-11.1.1 as the cross compilation host) also failed:

;;; Emitting code for LIST-OF-LENGTH-AT-LEAST-P.

Internal or unrecoverable error in:
not a lisp data object
  [2: No such file or directory]

;;; ECL C Backtrace

Same vague error as I get trying to build maxima.

I've searched through the pre-built package lists for SUSE SLE 11 SP 1 looking for "clisp", "sbcl", "gcl", and "lisp" and didn't find anything.

[kiwifb]

comment:21

I couldn't find any lisp on the SLES DVDs either, I tried to look at RedHat because they also provide ppc64 but couldn't find anything so far. A source rpm of maxima or a lisp on ppc64 would have been helpful from any distros.

[benjaminfjones]

comment:22

I found Clozure CL project distributes binary images for ppc64 linux. I tried this out, but ran into yet more problems. Maybe this is going off on a tangent....

bjones@silius:/tmp/bjones/ccl-1.7> ./ppccl64
remap spjump: Invalid argument

Now I read through the platform notes here and see that there is an issue with 16-bit memory references for functions like NIL (that sounds familiar...) not being accessible by non-root processes. This has to do with a parameter set by the OS: mmap_min_addr.

The parameter in question is called mmap_min_addr; one can cat the file /proc/sys/vm/mmap_min_addr to see what the current setting is.

On skynet/silius,

bjones@silius:/tmp/bjones/ccl-1.7> cat /proc/sys/vm/mmap_min_addr
65536

and according to the platform notes, it should be 4096.

In light of this, it might be worth changing this parameter on silius (which requires a reboot, see the platform notes above) and then try Clozure CL (run test suites, etc) and THEN try building Maxima. Also, it would be worth trying to reproduce Mike Hansen's reported error about redefining NIL here after the mmap_min_addr parameter has been changed.

[williamstein]

comment:23

Quick comment: Even if you could build Maxima with another Lisp, that's not a solution, since Sage uses the C library interface to Maxima, which is only available with ECL.

[kiwifb]

comment:24

@benjaminfjones
I think you are bang on. That is probably what is happening and what Mike Hansen was referring to. We should try the recommendation from the CCL web page and rebuild ecl and then maxima. From their comments doing the work in lisp to avoid the issue is not very appealing.

@williamstein
My suggestion of doing so was to do "an isolation fault" procedure. It happened to have bear fruits in an unexcepted way.

[nexttime]

comment:25

So what's the current status?

Replying to @williamstein:

Do you want an account on the machine? Then you can try all this and maybe fix the problem!

Write me offlist at [email protected] for an account.

William, did you receive my mail from last Friday ("Re: Maxima/ECL on Silius (#11708)")?

I could try debugging / fixing this during the weekend.

[kiwifb]

comment:45

Well, the SLES install we have does not include gfortran so at the moment we have to get it from gcc-4.6.1. I have had some experience building gcc on power (power5 so far power7 any days now, ok so I had a G4 too)and it is a bit dodgy at times. I am not sure if it is dodgier on linux or AIX. On AIX I haven't had a g++ free of problems for example. I suspect that the gcc on silius could use a bit of massaging, the question is where and how. And finally it could very well be an obscure gcc bug that only shows up on power.

[kiwifb]

comment:46

Note that that SLES gcc has been built for power4 to ensure compatibility with a wide range of hardware

gcc -v
Using built-in specs.
Target: powerpc64-suse-linux
Configured with: ../configure --prefix=/usr --infodir=/usr/share/info --mandir=/usr/share/man --libdir=/usr/lib64 --libexecdir=/usr/lib64 --enable-languages=c,c++,objc,fortran,obj-c++,java --enable-checking=release --with-gxx-include-dir=/usr/include/c++/4.3 --enable-ssp --disable-libssp --with-bugurl=http://bugs.opensuse.org/ --with-pkgversion='SUSE Linux' --disable-libgcj --disable-libmudflap --with-slibdir=/lib64 --with-system-zlib --enable-__cxa_atexit --enable-libstdcxx-allocator=new --disable-libstdcxx-pch --enable-version-specific-runtime-libs --program-suffix=-4.3 --enable-linux-futex --without-system-libunwind --with-cpu=power4 --enable-secureplt --with-long-double-128 --build=powerpc64-suse-linux
Thread model: posix
gcc version 4.3.4 [gcc-4_3-branch revision 152973] (SUSE Linux)

Not sure about the localy compiled gcc

/usr/local/gcc-4.6.1/ppc64-Linux-power7-suse/bin/gcc -v
Using built-in specs.
COLLECT_GCC=/usr/local/gcc-4.6.1/ppc64-Linux-power7-suse/bin/gcc
COLLECT_LTO_WRAPPER=/usr/local/gcc-4.6.1/ppc64-Linux-power7-suse/libexec/gcc/powerpc64-unknown-linux-gnu/4.6.1/lto-wrapper
Target: powerpc64-unknown-linux-gnu
Configured with: /usr/local/gcc-4.6.1/src/gcc-4.6.1/configure --enable-languages=c,c++,fortran --with-gnu-as --with-gnu-as=/usr/local/binutils-2.21/ppc64-Linux-power7-suse-gcc-4.3.4-suse/bin/as --with-gnu-ld --with-ld=/usr/local/binutils-2.21/ppc64-Linux-power7-suse-gcc-4.3.4-suse/bin/ld --with-gmp=/usr/local/mpir-2.4.0/ppc64-Linux-power7-suse-gcc-4.3.4-suse --with-mpfr=/usr/local/mpfr-3.0.1/ppc64-Linux-power7-suse-mpir-2.4.0-gcc-4.3.4-suse --with-mpc=/usr/local/mpc-0.9/ppc64-Linux-power7-suse-mpir-2.4.0-mpfr-3.0.1-gcc-4.3.4-suse --prefix=/usr/local/gcc-4.6.1/ppc64-Linux-power7-suse
Thread model: posix
gcc version 4.6.1 (GCC) 

There are a number of elements that could play right there from the config.

[nexttime]

comment:47

Funny that you incidentally compiled ECL with some other GCC version...

Perhaps also try some version from the 4.5 series.

Karl-Dieter, which upstream do you have in mind (now)? ;-)

Btw., the "native" SLES GCC is configured with --enable-languages=...,fortran,..., so in principle gfortran should be available, perhaps packaged separately though. (It's only a wrapper anyway, so should be replaceable by some simple shell script.)

[benjaminfjones]

comment:48

I added CC=gcc-4.3; export CC to the ECL's spkg-install script and after that change the maxima-5.23.2.p0 does indeed build successfully! I'm doing some tests now, will report back.

[kcrisman]

comment:49

Karl-Dieter, which upstream do you have in mind (now)? ;-)

I was wondering that myself, but didn't have anything germane to contribute.

[kiwifb]

comment:50

Replying to @nexttime:

Funny that you incidentally compiled ECL with some other GCC version...

YEs, gave me a false sense of victory for a while. Lucky I figured out what
the recipe for success was. Imagine: we have a working install and no one knows
why or is able to reproduce it!

Perhaps also try some version from the 4.5 series.

Definitely something to try.

Btw., the "native" SLES GCC is configured with --enable-languages=...,fortran,..., so in principle gfortran should be available, perhaps packaged separately though. (It's only a wrapper anyway, so should be replaceable by some simple shell script.)

You would think so. However I cannot find anything on the sles11 dvds (for either x86_64 or ppc64) that mention explicitly fortran. It could be part of some other gcc rpm I need to have a closer look.

[nexttime]

comment:51

Replying to @kiwifb:

Replying to @nexttime:

Funny that you incidentally compiled ECL with some other GCC version...

YEs, gave me a false sense of victory for a while. Lucky I figured out what
the recipe for success was. Imagine: we have a working install and no one knows
why or is able to reproduce it!

I was just wondering nobody (else) tried, intentionally... B)

Unfortunately I now also have an account there, but will first spend my time on the Sage 4.7.2.alpha3 release.

Btw., the "native" SLES GCC is configured with --enable-languages=...,fortran,..., so in principle gfortran should be available, perhaps packaged separately though. (It's only a wrapper anyway, so should be replaceable by some simple shell script.)

You would think so. However I cannot find anything on the sles11 dvds (for either x86_64 or ppc64) that mention explicitly fortran. It could be part of some other gcc rpm I need to have a closer look.

There are at least PPC ones for openSUSE (GCC 4.3), e.g. here.

Other versions (>4.3.x) should IMHO also work, or grab the sources and build it yourself... ;-)

[kiwifb]

comment:52

Replying to @nexttime:

There are at least PPC ones for openSUSE (GCC 4.3), e.g. here.

It is definitely not on the dvd for sles 11. I will try my own 4.6.1.

[nexttime]

comment:53

Replying to @kiwifb:

It is definitely not on the dvd for sles 11. I will try my own 4.6.1.

I can only tell that 4.5.x's works with both GCC 4.3.x and 4.4.x.

[nexttime]

comment:54

Did you play with optimization flags when building the ECL spkg?

By default it uses -O2; if you set SAGE_DEBUG=yes, it will be built with -O0.

spkg-install looks quite ugly [again?] btw., the following for example is definitely wrong:

CPPFLAGS="$CPPFLAGS -I$SAGE_LOCAL/include"
LDFLAGS="$LDFLAGS -L$SAGE_LOCAL/lib"

(The order has to be changed in both cases.)

[kiwifb]

comment:55

Replying to @nexttime:

Did you play with optimization flags when building the ECL spkg?

By default it uses -O2; if you set SAGE_DEBUG=yes, it will be built with -O0.

spkg-install looks quite ugly [again?] btw., the following for example is definitely wrong:

CPPFLAGS="$CPPFLAGS -I$SAGE_LOCAL/include"
LDFLAGS="$LDFLAGS -L$SAGE_LOCAL/lib"

(The order has to be changed in both cases.)

Haven't but that's a good idea. I may have OK-ed those change quickly for .p1 when
a problem appeared with altivec and I caouldn't make a new spkg myself due to time
constraints.

[benjaminfjones]

comment:56

Playing around with the SAGE_DEBUG=yes flag, I've that ECL doesn't even build on skynet/silius using the default system GCC 4.6.1. The ECL build fails with:

;;; Compiling (DEFUN PPRINT-RAW-ARRAY ...).
;;; Compiling (DEFUN PPRINT-LAMBDA-LIST ...).

;;;;;; Stack overflow.
;;; Jumping to the outermost toplevel prompt
;;;

Without SAGE_DEBUG set, it builds fine (but them Maxima fails, of course).

[benjaminfjones]

comment:57

Even though Maxima builds when we force ECL to compile under GCC-4.3 on silius, the resulting Sage installation has lots of doctest errors. See the ptestlong.log file.

At a glance, there are a couple different errors happening:

1. OverflowError: value too large to convert to int (may have to do with the architecture?)
2. A couple of NameErrors in optimize.py and pushout.py

File "/tmp/bjones/sage-4.7.2.alpha2/devel/sage-main/sage/numerical/optimize.py", line 571:
    sage: fit[a], fit[b], fit[c]
Exception raised:
    Traceback (most recent call last):
      File "/tmp/bjones/sage-4.7.2.alpha2/local/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/tmp/bjones/sage-4.7.2.alpha2/local/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/tmp/bjones/sage-4.7.2.alpha2/local/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_7[8]>", line 1, in <module>
        fit[a], fit[b], fit[c]###line 571:
    sage: fit[a], fit[b], fit[c]
    NameError: name 'fit' is not defined

3. mpfr errors: RuntimeError: Aborted
4. many ValueError: Refining interval that does not bound unique root! type errors from the elliptic curves modules

etc..

[kiwifb]

comment:58

I think you should put this in #11705 as at first glance errors are not just from maxima. I am going into that log with a fine comb.

[jhpalmieri]

comment:59

Now I'm confused: I just successfully built Sage 4.7.2.alpha4 on silius, with gcc 4.6.2 (the default on that machine). I had to replace the mpir package with the one from #11964, but that was the only change. Can anyone else verify this? I see that mpir is a dependency of ecl; could the new mpir spkg have fixed the problem with maxima? If so, we can close this ticket.

With the build, I do get lots of test failures, as above. I'll post the log at #11705.

[kiwifb]

comment:60

It could. I have to take some time to do a build but the log from Benjamin indicated there was a lot of trouble potentially with mpir. So it could indeed have been cured by the upgrade, although it is difficult to guess the chain of event leading to that. Unless there is a competition between mpir/gmp used by gcc and the one installed in sage.

[nexttime]

comment:61

Replying to @jhpalmieri:

Now I'm confused: I just successfully built Sage 4.7.2.alpha4 on silius, with gcc 4.6.2 (the default on that machine). I had to replace the mpir package with the one from #11964, but that was the only change. Can anyone else verify this?

I haven't tried GCC 4.6.2 yet, although I downloaded it the day it had been released ;-) (but hadn't had the time to build it on Silius)

Perhaps they've fixed some things, although my impression was that something with ECL is wrong.

Note that I've previously built ECL and Maxima "successfully" (modulo lots of doctest errors) with -mcpu=power4 -mtune=power4 (Maxima uses the CFLAGS from ECL anyway), and Sage with GCC 4.6.1 and GCC 4.4.6 (the other parts with -mcpu=power7 -mtune=power7).

I see that mpir is a dependency of ecl; could the new mpir spkg have fixed the problem with maxima? If so, we can close this ticket.

No, it just triggered the rebuild of ECL with GCC 4.6.2.

(I did build Sage 4.7.2.alpha3 which has [almost] the same MPIR spkg, i.e., the p4; the p7 just fixes a configure regression introduced by the p5/p6.)

I wouldn't close this ticket unless you know that none of the doctest errors is related to / caused by Maxima.

[jhpalmieri]

comment:62

Just to clarify: I logged into silius, sourced /usr/local/skynet_bash_profile, set MAKE='make -j12' and SAGE_PARALLEL_SPKG_BUILD=yes, and typed make. I didn't modify any other settings. gcc-4.6.2 is the default compiler there now.

[nexttime]

comment:63

Replying to @jhpalmieri:

Just to clarify: I logged into silius, sourced /usr/local/skynet_bash_profile, set MAKE='make -j12' and SAGE_PARALLEL_SPKG_BUILD=yes, and typed make. I didn't modify any other settings. gcc-4.6.2 is the default compiler there now.

Which means that upgrading from 4.6.1 to 4.6.2 seems worthwhile, although it's not immediately clear (or I don't know yet) what they changed. It is configured slightly differently, but without --with-cpu=... etc.; it might just default to some other CPU now.

Can anybody look at the GCC diffs? I didn't have and don't have the time right now...

[jdemeyer]

comment:64

Proposing to close this, since building sage-4.8.alpha6 on silius works. There are many test failures (probably unrelated to ecl) but at least it builds.

[nexttime]

comment:65

Replying to @jdemeyer:

Proposing to close this, since building sage-4.8.alpha6 on silius works. There are many test failures (probably unrelated to ecl) but at least it builds.

Unless a couple of doctest failures are caused by ECL and/or Maxima...

Maybe François can tell better.

I don't know right now what Maxima/ECL/GCC version you are referring to, but at least [with] GCC 4.6.x [ECL] seemed to produce buggy or invalid code for POWER7; -mcpu=power4 in contrast "worked", at least better, i.e. did build and produced less test failures. [Note that Maxima uses those CFLAGS specified when configuring ECL.]

[jdemeyer]

comment:66

I'm closing this since maxima does build on silius. If there are doctest failures related to the building of Maxima, that's for a different ticket.

[jdemeyer]

Reviewer: Jeroen Demeyer