Change iconv so it builds on Cygwin and Solaris only

[sagetrac-drkirkby]

Since iconv was added to Sage (#8432) as a standard package (needed due to Karl-Dieter Crisman updating R in #6532 whilst overlooking the warning message from the configure script and without testing on Solaris), there have been some problems on Fedora and OpenSuse which look to be related to problems building gd when there are two copies of iconv to be found on the system. Since there is an option to gd's configure script specify the location of the iconv that seemed the most logical option. However, that did not work, so changing iconv to only build on Cygwin and Solaris seemed the most logical.

Dave

CC: @kcrisman

Component: build

Author: David Kirkby

Reviewer: Jaap Spies, Florent Hivert

Merged: sage-4.3.5

Issue created by migration from https://trac.sagemath.org/ticket/8567

[jaapspies]

comment:1

--with-libiconv-prefix=SAGE_LOCAL

does not work for me on Fedora 12 64 bit.

It was created by GD configure 2.0.35, which was
generated by GNU Autoconf 2.61.  Invocation command line was

  $ ./configure --prefix=/home/jaap/downloads/sage-4.3.4.rc0/local --without-jpeg --with-libiconv-prefix=/home/jaap/downloads/sage-4.3.4.rc0/local --without-x --with-zlib=/home/jaap/downloads/sage-4.3.4.rc0/local --with-freetype=/home/jaap/downloads/sage-4.3.4.rc0/local --without-xpm

gcc -fPIC -g -I/home/jaap/downloads/sage-4.3.4.rc0/local/include -I/home/jaap/downloads/sage-4.3.4.rc0/local/include/freetype2/ -Wl,--rpath -Wl,/home/jaap/downloads/sage-4.3.4.rc0/local/lib -o .libs/annotate annotate.o  -L/home/jaap/downloads/sage-4.3.4.rc0/local/lib ./.libs/libgd.so -lfontconfig /home/jaap/downloads/sage-4.3.4.rc0/local/lib/libfreetype.so /home/jaap/downloads/sage-4.3.4.rc0/local/lib/libpng12.so -lz -lm -Wl,--rpath -Wl,/home/jaap/downloads/sage-4.3.4.rc0/local/lib
./.libs/libgd.so: undefined reference to `libiconv'
./.libs/libgd.so: undefined reference to `libiconv_close'
./.libs/libgd.so: undefined reference to `libiconv_open'
collect2: ld returned 1 exit status

Jaap

[jaapspies]

comment:2

moving /usr/bin/iconv and /usr/include/iconv.h out of the way made it possible to build
gd

So nothing wrong with the libiconv in SAGE_LOCAL

Jaap

[hivert]

Replying to @sagetrac-drkirkby:

Since iconv was added to Sage (#8432) as a standard package (needing due to someone updating R without checking it fully), there have been some problems on Fedora and OpenSuse which look to be related to problems building gd when there are two copies of iconv to be found on the system.

First of all I have the same problem on Gentoo. Ther is also a report of someone having the same problem on Mandriva.

I manage to get it work by the following workaround:
I replaced

$MAKE

by

LIBICONV="$SAGE_LOCAL"/lib/libiconv.so $MAKE

in spkg-install. I've no idea how robust this workaround is. I posted a spkg with the fix.

Florent

[jhpalmieri]

comment:4

By the way, in spkg-install, I think it should say "--with-x=no" rather than "--without-x", and the same for jpeg and xpm.

[hivert]

comment:5

Replying to @hivert:

First of all I have the same problem on Gentoo. Ther is also a report of someone having the same problem on Mandriva.

I manage to get it work by the following workaround:
I replaced

$MAKE

by

LIBICONV="$SAGE_LOCAL"/lib/libiconv.so $MAKE

in spkg-install. I've no idea how robust this workaround is. I posted a spkg with the fix.

Sorry ! This spkg is completely broken... I'm trying to put a working one.

Florent

[jaapspies]

comment:6

Don't put an spkg as attachment! Please give a link to the spkg.

Jaap

[hivert]

comment:7

Replying to @jaapspies:

Don't put an spkg as attachment! Please give a link to the spkg.

Jaap

Oops sorry ! My workaround does work actually. I find out that python distutils have the same problems much later...

[jhpalmieri]

comment:8

This doesn't help on these systems (output from uname -a):

Linux eno 2.6.31.12-174.2.22.fc12.x86_64 #1 SMP Fri Feb 19 18:55:03 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux
Linux lena 2.6.31.12-174.2.19.fc12.x86_64 #1 SMP Thu Feb 11 07:07:16 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux

I also don't know why it's about 50% bigger than the previous spkg.

[jaapspies]

comment:9

In Fedora 12 x86_64 the installation of libiconv is not complete:

[root@vrede jaap]# find /usr/bin -name "*iconv*"
/usr/bin/iconv.bak
/usr/bin/piconv
[root@vrede jaap]# find /usr/include -name "*iconv*"
/usr/include/evolution-data-server-2.28/camel/camel-iconv.h
/usr/include/evolution-data-server-2.28/libedataserver/e-iconv.h
/usr/include/Qt3Support/q3iconview.h
/usr/include/c++/4.4.3/gnu/gcj/convert/Output_iconv.h
/usr/include/c++/4.4.3/gnu/gcj/convert/Input_iconv.h
/usr/include/gtk-2.0/gtk/gtkiconview.h
/usr/include/Qt/q3iconview.h
/usr/include/xulrunner-sdk-1.9.1/system_wrappers_js/iconv.h
/usr/include/xulrunner-sdk-1.9.1/system_wrappers/iconv.h
/usr/include/iconv.h.bak
[root@vrede jaap]# find /usr/lib -name "*iconv*"
/usr/lib/ruby/1.8/xsd/iconvcharset.rb
[root@vrede jaap]# 

Jaap

[hivert]

comment:10

I maybe manage to have something working by a hack. Could someone try the following spkg:
http://sage.math.washington.edu/home/hivert/spkg/gd-2.0.35.p4.spkg

[jhpalmieri]

comment:11

That appears to work on the two systems I mentioned before. I'll know for sure when a full Sage build is completed...

[sagetrac-drkirkby]

comment:12

Replying to @hivert:

Replying to @sagetrac-drkirkby:

Since iconv was added to Sage (#8432) as a standard package (needing due to someone updating R without checking it fully), there have been some problems on Fedora and OpenSuse which look to be related to problems building gd when there are two copies of iconv to be found on the system.

First of all I have the same problem on Gentoo. Ther is also a report of someone having the same problem on Mandriva.

I manage to get it work by the following workaround:
I replaced

$MAKE

by

LIBICONV="$SAGE_LOCAL"/lib/libiconv.so $MAKE

in spkg-install. I've no idea how robust this workaround is. I posted a spkg with the fix.

Florent

I've got no idea if setting LIBICOV shoudl be set, but if it is, the way to do it should be:

LIBICONV="$SAGE_LOCAL/lib/libiconv.so"

and not

LIBICONV="$SAGE_LOCAL"/lib/libiconv.so

Dave

[sagetrac-drkirkby]

comment:13

Replying to @jaapspies:

--with-libiconv-prefix=SAGE_LOCAL

does not work for me on Fedora 12 64 bit.

Jaap

Jaap,

is that a typo in the trac ticket, or was it in the package? There should be a $ sign in front if SAGE_LOCAL.

[sagetrac-drkirkby]

comment:14

If this gets too much of a mess, we could consider creating a ticket to get the R package reverted to the older version. I've cc'ed Karl-Dieter Crisman who updated R, whilst ignoring the warning from the configure script and not testing on Solaris.

Hopefully iconv can be sorted out, but if it can't be done properly, then reverting the R version is always an option, then iconv will not be needed. We could change iconv's spkg-install to simply exit with an exit code of 0. That would make it relatively easy to do, whilst leaving iconv in Sage to allow it to be sorted out properly prior to updating R.

Dave

[jhpalmieri]

comment:15

For me, the Sage build completed, but the documentation won't build because Sage doesn't start successfully:

Traceback (most recent call last):
  File "/home/palmieri/lena/sage-4.3.4.alpha1/local/bin/sphinx-build", line 6, in <module>
    import sage.all
  File "/home/palmieri/lena/sage-4.3.4.alpha1/local/lib/python2.6/site-packages/sage/all.py", line 73, in <module>
    from sage.matrix.all     import *
  File "/home/palmieri/lena/sage-4.3.4.alpha1/local/lib/python2.6/site-packages/sage/matrix/all.py", line 1, in <module>
    from matrix_space import MatrixSpace, is_MatrixSpace
  File "/home/palmieri/lena/sage-4.3.4.alpha1/local/lib/python2.6/site-packages/sage/matrix/matrix_space.py", line 40, in <module>
    import matrix_mod2_dense
ImportError: /home/palmieri/lena/sage-4.3.4.alpha1/local/lib/libgd.so.2: undefined symbol: libiconv

Running

export LIBICONV="$SAGE_LOCAL/lib/libiconv.so"

right before running Sage doesn't help.

[sagetrac-drkirkby]

comment:16

Replying to @jaapspies:

moving /usr/bin/iconv and /usr/include/iconv.h out of the way made it possible to build
gd

So nothing wrong with the libiconv in SAGE_LOCAL

Jaap

I had a discussion with a libtool developer about a similar issue once before. He said it is wrong to have two copies of a library, so the problem may be more fundamental. However, it is strange gd ignores the option to look for iconv in $SAGE_LOCAL.

One disadvantage of reverting R and not installing iconv is that iconv is needed to build gd on Cygwin, and there is effort into porting Sage to Cygwin.

We could consider only installing iconv on Solaris and Cygwin, though I'm tempted to suggest reverting R and getting the problem solved properly at a later date, rather than a hack now.

Dave

[hivert]

comment:17

Replying to @jhpalmieri:

For me, the Sage build completed, but the documentation won't build because Sage doesn't start successfully:

I got the same behavior on openSuSE 11.1 and Gentoo 1.12.9 both x86_64. The build appears to be complete but sage itself is broken.

I don't know what to do and I've no more time to investigate. My vote is to revert R, remove gd and make as soon as possible a fix release until this is sorted out. We can't seriously make a release which is broken on half the distro.

[jaapspies]

comment:18

Replying to @sagetrac-drkirkby:

I had a discussion with a libtool developer about a similar issue once before. He said it is wrong to have two copies of a library, so the problem may be more fundamental. However, it is strange gd ignores the option to look for iconv in $SAGE_LOCAL.

I'm not so sure anymore I didn't make another typo in spkg-install.

I'll try again.

Jaap

[sagetrac-drkirkby]

comment:19

Another way to solve the fallout from updating R, will probably be to add

if [ "x$UNAME" != xSunOS ] && [ "x$UNAME" != xCYGWIN ] then ; 
  exit 0
fi

to the top of the iconv spkg-install file. That's probably the simplest hack. This is untested and I don't have time to do this until around 2100 GMT today, so if someone beats me to it, go ahead and try it. Make sure it is tested on both Solaris and Linux. It should build iconv on Solaris, but not on Linux

Dave

[jaapspies]

comment:20

About iconv(): according to the man pages the function is part of glibc

VERSIONS
       This function is available in glibc since version 2.1.

CONFORMING TO
       POSIX.1-2001.

SEE ALSO
       iconv_close(3), iconv_open(3)

So for most systems there is no need to have a local libiconv!
Let's say only Solaris and Cygwin.

Jaap

[jaapspies]

comment:21

Replying to @sagetrac-drkirkby:

Another way to solve the fallout from updating R, will probably be to add

if [ "x$UNAME" != xSunOS ] && [ "x$UNAME" != xCYGWIN ] then ; 
  exit 0
fi

This probably will cause trouble on Open Solaris.

Jaap

[sagetrac-GeorgSWeber]

comment:22

Replying to @sagetrac-drkirkby:

Another way to solve the fallout from updating R, will probably be to add

if [ "x$UNAME" != xSunOS ] && [ "x$UNAME" != xCYGWIN ] then ; 
  exit 0
fi

to the top of the iconv spkg-install file. That's probably the simplest hack. This is untested and I don't have time to do this until around 2100 GMT today, so if someone beats me to it, go ahead and try it. Make sure it is tested on both Solaris and Linux. It should build iconv on Solaris, but not on Linux

Dave

Yes, I definitely prefer this way, too. The OpenSolaris topic may be solved later (what does $UNAME give there as output?), and it might be nice to remove any leftovers in "$SAGE_ROOT/bin/" from earlier (especially v3.4.3) Sage installs. Which might be tricky in a "Sage upgrade" use case. But never mind, the road to go is clear now!

Cheers,
Georg

[sagetrac-drkirkby]

comment:24

Jaap,

I don't have time to test it now (really am tied up), but if you can test on OpenSolaris, then make a decision whether iconv needs installing or not.

drkirkby@kestrel:~$ uname -r
5.10

on Solaris 10, and

drkirkby@hawk:~$ uname -r
5.11

on OpenSolaris (aka Solaris 11). So just put an additional clause if needed.

The -r option to uname is POSIX, so we can rely on that. Don't use 'uname -p' since

• The i386 processor is used on Solaris 10 too.
• -p is not a POSIX option to uname, so can cause hassles on other systems.

Also worth checking, in preference to this are whether the version of 'gd' in Sage is old and furthermore the 'gd' developers have fixed the bug that prevents '--with-libiconv-prefix' working properly. If so, consider updating gd.

Had R been properly tested, none of this would have been necessary.

When I wrote the iconv package, it clears out debris, like libraries and binaries, so there should be no need to do that again - just make sure the cleaning up is done before exiting.

Since Jaap and I are looking to port Sage to OpenSolaris, we are not keen to introduce more problems for the future. Let's sort it out now.

Dave

[jaapspies]

comment:25

Replying to @sagetrac-drkirkby:

Jaap,

I don't have time to test it now (really am tied up), but if you can test on OpenSolaris, then make a decision whether iconv needs installing or not.

Opensolaris has it's own iconv included in glibc.

on OpenSolaris (aka Solaris 11). So just put an additional clause if needed.

Yes we should put an extra test in

The -r option to uname is POSIX, so we can rely on that. Don't use 'uname -p' since

Ok.

• The i386 processor is used on Solaris 10 too.
• -p is not a POSIX option to uname, so can cause hassles on other systems.

Also worth checking, in preference to this are whether the version of 'gd' in Sage is old and furthermore the 'gd' developers have fixed the bug that prevents '--with-libiconv-prefix' working properly. If so, consider updating gd.

Let's go for the proposed option and leave alone the gd package for the moment.

Had R been properly tested, none of this would have been necessary.

You are repeating your self :-)!

My proposal is you update the iconv spkg and I and evt. others review it.

I think we can wait some more hours.

Jaap

[jhpalmieri]

comment:26

Replying to @sagetrac-drkirkby:

Another way to solve the fallout from updating R, will probably be to add

if [ "x$UNAME" != xSunOS ] && [ "x$UNAME" != xCYGWIN ] then ; 
  exit 0
fi

to the top of the iconv spkg-install file.

Informally (that is, I threw together a quick spkg), this works on at least one of the linux boxes I had troubles with before, and it also works on sage.math. I haven't tested it on Solaris, and I probably won't have time to; I don't have access to a Windows/Cygwin machine. I would suggest that the spkg-install file actually say

if [ "x$UNAME" != xSunOS ] && [ "x$UNAME" != xCYGWIN ] ; then
  echo "Not installing iconv -- on systems other than Solaris and Cygwin, use the system's iconv"
  exit 0
fi

Or something like that -- I'm not sure exactly what the message should say.

[hivert]

comment:27

Replying to @jhpalmieri:

Replying to @sagetrac-drkirkby:

Another way to solve the fallout from updating R, will probably be to add

if [ "x$UNAME" != xSunOS ] && [ "x$UNAME" != xCYGWIN ] then ; 
  exit 0
fi

The ";" should be before "then"..
.

if [ "x$UNAME" != xSunOS ] && [ "x$UNAME" != xCYGWIN ] ; then
  echo "Not installing iconv -- on systems other than Solaris and Cygwin, use the system's iconv"
  exit 0
fi

Or something like that -- I'm not sure exactly what the message should say.

I'm trying this on gentoo and it seems to work... I'm waiting for the build to finish.

Florent

[sagetrac-drkirkby]

comment:28

I've change the title and description and are in the process of creating a package for others to review.

[sagetrac-drkirkby]

Author: David Kirkby

[sagetrac-drkirkby]

comment:29

I've stuck a package here:

http://sage.math.washington.edu/home/kirkby/iconv/iconv-1.13.1.p0.spkg

[jaapspies]

comment:30

Replying to @sagetrac-drkirkby:

I've stuck a package here:

http://sage.math.washington.edu/home/kirkby/iconv/iconv-1.13.1.p0.spkg

I'll try it asap.

Jaap

[sagetrac-drkirkby]

Mercurial patch to ensure iconv builds on Solaris and Cygwin only

[sagetrac-drkirkby]

comment:31

Attachment: iconv.patch.gz

I've not tested this extensively, but it builds on Solaris and hopefully does not on Linux. I don't have time to do much testing, so others can.

I don't have a Cygwin system set up.

[jaapspies]

comment:32

---

Removing old iconv files if they exist
iconv will not be installed, as it is only installed on
Solaris and Cygwin - see:
#8567

real 0m0.078

gcc version 4.4.3 20100127 (Red Hat 4.4.3-4) (GCC) 
****************************************s
user	0m0.008s
sys	0m0.015s
Successfully installed iconv-1.13.1.p0

echo "Removing old iconv files if they exist"
# If iconv is updated, please double-check these are still necessary
# and that there are no extra files added.

rm -f $SAGE_LOCAL/bin/iconv  $SAGE_LOCAL/lib/charset.alias
rm -f $SAGE_LOCAL/lib/*libiconv*  $SAGE_LOCAL/lib/libcharset*
rm -f $SAGE_LOCAL/include/iconv.h  $SAGE_LOCAL/include/libcharset.h
rm -f $SAGE_LOCAL/include/localcharset.h
rm -rf  $SAGE_LOCAL/share/doc/libiconv
rm -f $SAGE_LOCAL/share/man/man1/iconv* $SAGE_LOCAL/share/man/man3/iconv*

This looks ok for me on Fedora (Linux in general?)

The Opensolaris issue is not resolved, but I think this deserves a positive review.

Jaap

[jaapspies]

Reviewer: Jaap Spies

[hivert]

comment:34

With the spkg here, from a fresh Sage source:

• iconv build is successfully skipped,
• gd is successfully build
  on openSuSE 11.1 and Gentoo 1.12.9. I'm waiting the whole build to finish, but it should be ok... I'll then agree with Jaap on positive review.

Thanks

[hivert]

Changed reviewer from Jaap Spies to Jaap Spies, Florent Hivert

[jaapspies]

comment:35

Replying to @hivert:

With the spkg here, from a fresh Sage source:

• iconv build is successfully skipped,
• gd is successfully build
  on openSuSE 11.1 and Gentoo 1.12.9. I'm waiting the whole build to finish, but it should be ok... I'll then agree with Jaap on positive review.

Thanks

My build finished successful, so let's do it! I leave it to you to give the positive review.

Jaap

[hivert]

comment:36

Replying to @jaapspies:

My build finished successful, so let's do it! I leave it to you to give the positive review.

Mine too on openSuSE and gentoo. I'm ready to give positive review but should it be tested on Cygwin ? Unfortunately, I've no access to this kind of machine.

Cheers,

Florent

[jaapspies]

comment:37

Replying to @hivert:

Replying to @jaapspies:

My build finished successful, so let's do it! I leave it to you to give the positive review.

Mine too on openSuSE and gentoo. I'm ready to give positive review but should it be tested on Cygwin ? Unfortunately, I've no access to this kind of machine.

Cheers,

Florent

Please do! There is no change in SunOS and Cygwin. In both cases iconv spkg will be build as before.

Jaap

[williamstein]

comment:39

Merged into sage-4.3.5.

[sagetrac-mvngu]

Merged: sage-4.3.5

[sagetrac-bascorp2]

comment:42

[deleted spam]