"make" should run Sage once #11926

jdemeyer · 2011-10-15T14:16:06Z

In a multi-user environment, the user compiling Sage must run it at least once to run sage-location and generate .pyc files.

The proposed fix is: in the default make rule, run Sage when local/bin/sage-started.txt does not exist and create that file in sage-location. Also run Sage after upgrading.

This patch also changes the error message when a spkg fails a build or test. Example error message:

Error: Configuring PARI with readline and GMP kernel failed.

real    0m0.100s
user    0m0.012s
sys     0m0.012s
************************************************************************
Error installing package pari-2.4.3.alpha.p7
************************************************************************
Please email sage-devel (http://groups.google.com/group/sage-devel)
explaining the problem and including the relevant part of the log file
  /usr/local/src/sage-4.7.2.rc0/spkg/logs/pari-2.4.3.alpha.p7.log
Describe your computer, operating system, etc.
If you want to try to fix the problem yourself, *don't* just cd to
/usr/local/src/sage-4.7.2.rc0/spkg/build/pari-2.4.3.alpha.p7 and type 'make' or whatever is appropriate.
Instead, the following commands setup all environment variables
correctly and load a subshell for you to debug the error:
  (cd '/usr/local/src/sage-4.7.2.rc0/spkg/build/pari-2.4.3.alpha.p7' && '/usr/local/src/sage-4.7.2.rc0/sage' -sh)
When you are done debugging, you can type "exit" to leave the subshell.
************************************************************************
Error: Failed to install package 'pari'.

Apply:

attachment: 11926.patch to SAGE_ROOT
attachment: 11926_sage_starts.patch, attachment: trac_11926-error-msg.patch, attachment: 11926-error-msg-review.patch to SCRIPTS (local/bin)
attachment: 11926_sage.patch and attachment: 11926_doc.patch to the Sage library.

CC: @pcpa

Component: build

Keywords: Makefile build sage-starts

Author: Jeroen Demeyer

Reviewer: John Palmieri, Leif Leonhardy

Merged: sage-4.8.alpha0

Issue created by migration from https://trac.sagemath.org/ticket/11926

The text was updated successfully, but these errors were encountered:

jhpalmieri · 2011-10-15T17:00:53Z

comment:3

I think I suggested this sort of thing once, and it was criticized. I don't remember the critiques. Anyway, I like the idea. In case some system administrator is building Sage and doesn't want a .sage directory created for them, should we set DOTSAGE to something temporary first?

By the way, running Sage as part of the build process should also take care of some of the issues at #5155 (not all of them, unfortunately).

jdemeyer · 2011-10-15T17:47:04Z

comment:4

Replying to @jhpalmieri:

I think I suggested this sort of thing once, and it was criticized. I don't remember the critiques. Anyway, I like the idea.

In case some system administrator is building Sage and doesn't want a .sage directory created for them, should we set DOTSAGE to something temporary first?

Good idea. Make a temporary DOTSAGE, run Sage, remove the temporary DOTSAGE

By the way, running Sage as part of the build process should also take care of some of the issues at #5155 (not all of them, unfortunately).

Thanks for the pointer. Also #11887 is related.

jdemeyer · 2011-10-15T17:47:04Z

Work Issues: temporary DOTSAGE

nexttime · 2011-10-15T18:32:35Z

comment:5

I don't like the idea of running sage after make build since it certainly doesn't belong to that target, but if at all, it shouldn't be run from spkg/install but the Makefile, preferably in the default rule, not build. Or run sage-location from there if make did anything, but we should in principle run it from elsewhere.

Someone who is going to install Sage system-wide should run make ptestlong or similiar anyway (which also runs sage), just like one runs make check before running make install.

Also note that sage (or sage-location) should be run (at least) after it has been moved to its final installation location, which is likely to happen after having built Sage.

jdemeyer · 2011-10-15T19:04:09Z

comment:6

Replying to @nexttime:

I don't like the idea of running sage after make build since it certainly doesn't belong to that target

I think it does. Currently, after make build (or simply make), Sage does not work for a different user. I think make build should result in a proper Sage installation, which means that it should work for every user.

it shouldn't be run from spkg/install but the Makefile

Why? I like spkg/install better because it allows us to run Sage only if something was actually installed. You don't want to run Sage everytime somebody does make foo for various values of "foo". Also, IIRC, we run spkg/install on upgrading but not a top-level make all or make build.

Or run sage-location from there if make did anything, but we should in principle run it from elsewhere.

Running sage-location is certainly not sufficient to generate all files needed to run Sage.

Someone who is going to install Sage system-wide should run make ptestlong

Let the sysadmin decide whether they want to run make ptestlong or not, I want to give him the option of NOT doing it.

Also note that sage (or sage-location) should be run (at least) after it has been moved to its final installation location, which is likely to happen after having built Sage.

I would think most people run Sage from the location where they installed it, or equivalently, install Sage in the location they want to run it from. In any case, this does not invalidate the reasons for this ticket.

jdemeyer · 2011-10-15T22:17:40Z

comment:7

Updated version of the patch using a temporary DOT_SAGE directory. I have not really tested it well.

jdemeyer · 2011-10-15T22:17:40Z

Changed work issues from temporary DOTSAGE to none

jdemeyer · 2011-10-16T15:51:01Z

comment:8

Tested with a complete build from scratch that this does indeed what it should do. A simple make build allows any user to run Sage.

jhpalmieri · 2011-10-16T19:42:28Z

comment:9

I don't see anything in the new patch about DOT_SAGE.

jdemeyer · 2011-10-17T06:40:11Z

comment:10

Replying to @jhpalmieri:

I don't see anything in the new patch about DOT_SAGE.

So true! I seems I forget a hg qrefresh.

jdemeyer · 2011-10-17T06:51:41Z

comment:11

Replying to @jhpalmieri:

In case some system administrator is building Sage and doesn't want a .sage directory created for them, should we set DOTSAGE to something temporary first?

Related question which comes up: should we also use a temporary DOT_SAGE directory for make doc? In #10163, I found out that make doc does run Sage (and does create a DOT_SAGE directory).

jhpalmieri · 2011-10-17T17:11:05Z

comment:13

I'm somewhat tempted to add a new command-line option to Sage:

diff --git a/sage-sage b/sage-sage
--- a/sage-sage
+++ b/sage-sage
@@ -247,6 +247,27 @@ if [ $# -gt 0 ]; then
   fi
 fi
 
+# The following function creates a temporary file and stores its name
+# in $FILE.  (It would be nice to replace this with 'mktemp', but that
+# command has different syntax on OS X compared to linux or Solaris.)
+sagetempfile() {
+    FILE=`mktemp -d 2>/dev/null`
+    if [ $? -ne 0 ]; then
+        # presumably because the "-d" option to mktemp expects an argument,
+        # e.g., on OS X.
+        FILE=`mktemp -d -t dotsage`
+    fi
+}
+
+if [ "$1" = '--norc' -o "$1" = '--nodotsage' ]; then
+    OLD_DOT_SAGE=$DOT_SAGE
+    sagetempfile
+    DOT_SAGE=$FILE && export DOT_SAGE
+    shift
+    sage "$@"
+    rm -rf "$DOT_SAGE"
+    DOT_SAGE=$OLD_DOT_SAGE && export DOT_SAGE
+fi
 
 LOGFILE="$SAGE_ROOT/sage.log"
 LOGOPT=""

We should probably check exit codes for various things here, and of course this would need to be documented. If we had this, then in Makefile, we could replace "sage --docbuild ..." with "sage --norc --docbuild ...". We could possibly do the same with "make test", etc.

jdemeyer · 2011-10-17T18:31:05Z

comment:14

Replying to @jhpalmieri:

I'm somewhat tempted to add a new command-line option to Sage:

Let's discuss this in a different ticket: #11932.

I think I am going to revert the DOT_SAGE handling here and leave that for #11932, because currently it's very hard to avoid having a $HOME/.sage directory created when installing Sage.

jdemeyer · 2011-11-02T19:56:16Z

comment:74

Added an extra patch with the documentation changes proposed by leif, needs_review.

jdemeyer · 2011-11-03T16:14:43Z

Milestone sage-4.7.3 deleted

jdemeyer · 2011-11-03T16:19:49Z

Changed merged from sage-4.7.3.alpha0 to sage-4.8.alpha0

jdemeyer · 2011-11-03T20:42:03Z

comment:78

Attachment: 11926_doc.patch.gz

Can somebody please review the last patch here?

jhpalmieri · 2011-11-03T21:41:01Z

comment:79

Looks okay to me.

nexttime · 2011-11-03T23:55:30Z

comment:80

Replying to @jhpalmieri:

Looks okay to me.

Yes.

But still doc should depend on start, such that one immediately gets a single, meaningful error message, not intermixed by repeated random tracebacks from Sphinx (after each of which the script btw. still unconditionally prints "Build succeeded. The built documents can be found in...").

And it would be far better to have the output of sage-starts also appended to install.log, for the reasons given earlier.

W.r.t. #11021:

Despite its current title, it involves a "major" rewrite of sage-spkg, i.e., makes changes all over the code, half of which were -- though already positively reviewed -- removed from another ticket for IMHO no reason (since at that point, they did apply and didn't break any other tickets).

So that ticket certainly drags (which mainly is my fault; I've returned to it many times, but then again had to work on too many other tickets), but it doesn't make things easier if other tickets do redundant changes to sage-spkg, thereby just causing tedious rebasing.

One can by the way search for other tickets before working on e.g. scripts or spkgs. And there's btw. also a meta-ticket for changes to the Makefile, #11622.

jdemeyer · 2011-11-04T09:18:27Z

comment:81

Replying to @nexttime:

But still doc should depend on start, such that one immediately gets a single, meaningful error message

I don't find it logical that doc would depend on start. I still think the best solution would be to have make build run sage-starts. So here, we have to agree to disagree.

So that ticket certainly drags (which mainly is my fault; I've returned to it many times, but then again had to work on too many other tickets), but it doesn't make things easier if other tickets do redundant changes to sage-spkg, thereby just causing tedious rebasing.

If we are not allowed to make patches just because some other ticket makes changes to the same code, then we might as well stop Sage development. Especially if the ticket gets "dragged" as you say.

jdemeyer · 2011-11-04T09:31:16Z

comment:82

Replying to @nexttime:

And it would be far better to have the output of sage-starts also appended to install.log, for the reasons given earlier.

I don't have strong feelings about this issue. If you want to make this change on a different ticket, go ahead, I will have a look.

nexttime · 2011-11-04T09:37:32Z

comment:83

Replying to @jdemeyer:

Replying to @nexttime:

But still doc should depend on start, such that one immediately gets a single, meaningful error message

I don't find it logical that doc would depend on start. I still think the best solution would be to have make build run sage-starts. So here, we have to agree to disagree.

The builder imports sage.all, hence doc requires that Sage can start.

Perhaps sage --docbuild should check once that it is functional (with try: import sage.all ...).

If we are not allowed to make patches just because some other ticket makes changes to the same code, then we might as well stop Sage development. Especially if the ticket gets "dragged" as you say.

I was talking about unnecessary, unrelated changes. Any developer should at least be aware of other tickets touching the same code.

It's also bad practice to bundle unrelated changes to different files into a single patch, just because they're in the same repo.

jhpalmieri · 2011-11-04T21:07:17Z

comment:84

See #11991 for what I hope is a quick a follow-up: sage-starts should record the time and version number.

kiwifb · 2011-11-24T22:46:40Z

comment:85

I hate to be late to the party. I was bogged down into work. I appreciate the point of view that most people won't build the stuff system wide. Me and Paulo from Mandriva do. I and probably him will object to that sage-started.txt file being stored in $SAGE_LOCAL/lib/. To us it belongs to DOTSAGE. I know it complicates the testing that Jeroen wants to do. But this goes a long way to making a system wide installation troublesome.

I may have missed it but where is the documentation for someone who wants to make a system wide install? Just this ticket?

jhpalmieri · 2011-11-25T04:29:53Z

comment:86

I think that sage_started.txt should be viewed as a file like sage-current-location.txt: a file which is produced during the build/test process, which ordinary users don't need write-access to. This ticket is related to some of the issues at #5155, and should indeed fix some of them. But maybe I don't understand your complaint; can you explain it more?

jdemeyer · 2011-11-25T09:13:14Z

comment:87

Replying to @kiwifb:

I hate to be late to the party. I was bogged down into work. I appreciate the point of view that most people won't build the stuff system wide. Me and Paulo from Mandriva do. I and probably him will object to that sage-started.txt file being stored in $SAGE_LOCAL/lib/. To us it belongs to DOTSAGE. I know it complicates the testing that Jeroen wants to do. But this goes a long way to making a system wide installation troublesome.

Why do you think this makes a system-wide install harder? In fact, this ticket was exactly supposed to make it easier to do a system-wide install.

I may have missed it but where is the documentation for someone who wants to make a system wide install? Just this ticket?

See http://sagemath.org/doc/installation/source.html#installation-in-a-multiuser-environment. Of course this ticket (and probably others to be merged in sage-4.8) will change this documentation, but the essence should remain the same.

kiwifb · 2011-11-25T11:28:44Z

comment:88

OK. I just completed my first ebuild of a sage-4.8 alpha this afternoon. The first I noticed about this is when I ran the doctests as a normal user. I do that straight away as this is the only validation test I have. A truckload of doctests just failed complaining that they couldn't find "sage-started.txt". Actually so many failed that I am almost wondering if there is something wrong with the ones that didn't.

I spent 3 weeks fixing bad fortran code and bad makefiles so I may have had a bit of a temper there but it just made my life difficult.

I am not sure how it helps with #5155 since it adds stuff that actually wants access to SAGE_ROOT.

I'll admit that I skimmed a bit on reading this thread so I won't attribute bad intent but using DOT_SAGE has been raised and apparently shot down as "system wide install" is not a normal install. You don't have to support the kind of stuff I do but it felt like making it more difficult knowingly.

jdemeyer · 2011-11-25T11:35:31Z

comment:89

How did you build sage? i.e. which shell commands did you execute, starting from the Sage source tarball, to get the totally-non-working Sage?

jdemeyer · 2011-11-25T11:40:48Z

comment:90

Also: precisely which alpha version of Sage did you use? There have been various changes to the build process in the 4.8 series.

jhpalmieri · 2011-11-25T18:16:10Z

comment:91

I tried the following:

build Sage ('make'), version 4.8.alpha2
move the Sage installation and run 'sage' once, as per the multi-user installation instructions
chmod a-w -R sage-x.y.z (so I don't have write access anymore)
sage -tp 18 devel/sage

Almost all tests passed; the only failures were the ones reported at #5515.

The following tests failed:

	sage -t --long devel/sage/sage/modular/hecke/submodule.py # 1 doctests failed
	sage -t --long devel/sage/sage/modular/abvar/abvar.py # 1 doctests failed
	sage -t --long devel/sage/sage/lfunctions/sympow.py # 13 doctests failed
	sage -t --long devel/sage/sage/interfaces/qepcad.py # 3 doctests failed
	sage -t --long devel/sage/sage/schemes/elliptic_curves/ell_rational_field.py # 17 doctests failed

So I think the approach on this ticket should work with a multi-user installation, but as I said earlier, I may be misunderstanding the problem.

kiwifb · 2011-11-25T18:34:37Z

comment:92

Sorry not to reply immediately Jeroen but that last message was the last thing I did before getting some sleep.

Which helped clear my mind a little and take on board some comments properly. I, and am pretty sure that Mandriva does too, bypass the normal sage build system in our packages. You may have noticed the word ebuild (gentoo). So I didn't notice that you basically generated a file that was essential to start sage at the very end of the building process and outside of any spkg (that's a very crucial bit if you are packaging sage). I guess now that I have noticed and realized it is supposed to be there I could just add it and adjust the location.

My real mistake was to think it was created each time sage was launched - never mind that there is no process to remove it.

jdemeyer added this to the sage-4.8 milestone Oct 15, 2011

jdemeyer added c: build labels Oct 15, 2011

jdemeyer assigned sagetrac-GeorgSWeber Oct 15, 2011

This comment has been minimized.

Sign in to view

jdemeyer added the s: needs review label Oct 15, 2011

jdemeyer added s: needs work and removed s: needs review labels Oct 15, 2011

jdemeyer added s: needs review and removed s: needs work labels Oct 15, 2011

This comment has been minimized.

Sign in to view

jdemeyer added s: needs work and removed s: needs review labels Oct 17, 2011

This comment has been minimized.

Sign in to view

jdemeyer removed this from the sage-4.8 milestone Nov 3, 2011

jdemeyer added this to the sage-4.8 milestone Nov 3, 2011

jhpalmieri added s: positive review and removed s: needs review labels Nov 3, 2011

jdemeyer removed the s: positive review label Nov 4, 2011

jdemeyer closed this as completed Nov 4, 2011

tornaria mentioned this issue Nov 26, 2011

Properly canonicalize $SAGE_ROOT #5852

Closed

jdemeyer mentioned this issue May 13, 2012

Allow running Sage with a temporary DOT_SAGE directory #11932

Closed

jhpalmieri mentioned this issue Nov 3, 2011

Remove the necessity to set SAGE_PARALLEL_SPKG_BUILD #11959

Closed

jdemeyer mentioned this issue Nov 14, 2011

Clean up top-level Makefile #11969

Closed

jhpalmieri mentioned this issue Nov 7, 2011

record time, version in sage-starts #11991

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"make" should run Sage once #11926

"make" should run Sage once #11926

jdemeyer commented Oct 15, 2011

This comment has been minimized.

jhpalmieri commented Oct 15, 2011

jdemeyer commented Oct 15, 2011

jdemeyer commented Oct 15, 2011

nexttime mannequin commented Oct 15, 2011

jdemeyer commented Oct 15, 2011

jdemeyer commented Oct 15, 2011

jdemeyer commented Oct 15, 2011

jdemeyer commented Oct 16, 2011

jhpalmieri commented Oct 16, 2011

jdemeyer commented Oct 17, 2011

jdemeyer commented Oct 17, 2011

This comment has been minimized.

jhpalmieri commented Oct 17, 2011

jdemeyer commented Oct 17, 2011

jdemeyer commented Nov 2, 2011

This comment has been minimized.

jdemeyer commented Nov 3, 2011

jdemeyer commented Nov 3, 2011

jdemeyer commented Nov 3, 2011

jhpalmieri commented Nov 3, 2011

nexttime mannequin commented Nov 3, 2011

jdemeyer commented Nov 4, 2011

jdemeyer commented Nov 4, 2011

nexttime mannequin commented Nov 4, 2011

jhpalmieri commented Nov 4, 2011

kiwifb commented Nov 24, 2011

jhpalmieri commented Nov 25, 2011

jdemeyer commented Nov 25, 2011

kiwifb commented Nov 25, 2011

jdemeyer commented Nov 25, 2011

jdemeyer commented Nov 25, 2011

jhpalmieri commented Nov 25, 2011

kiwifb commented Nov 25, 2011

"make" should run Sage once #11926

"make" should run Sage once #11926

Comments

jdemeyer commented Oct 15, 2011

This comment has been minimized.

jhpalmieri commented Oct 15, 2011

jdemeyer commented Oct 15, 2011

jdemeyer commented Oct 15, 2011

nexttime mannequin commented Oct 15, 2011

jdemeyer commented Oct 15, 2011

jdemeyer commented Oct 15, 2011

jdemeyer commented Oct 15, 2011

jdemeyer commented Oct 16, 2011

jhpalmieri commented Oct 16, 2011

jdemeyer commented Oct 17, 2011

jdemeyer commented Oct 17, 2011

This comment has been minimized.

jhpalmieri commented Oct 17, 2011

jdemeyer commented Oct 17, 2011

jdemeyer commented Nov 2, 2011

This comment has been minimized.

jdemeyer commented Nov 3, 2011

jdemeyer commented Nov 3, 2011

jdemeyer commented Nov 3, 2011

jhpalmieri commented Nov 3, 2011

nexttime mannequin commented Nov 3, 2011

jdemeyer commented Nov 4, 2011

jdemeyer commented Nov 4, 2011

nexttime mannequin commented Nov 4, 2011

jhpalmieri commented Nov 4, 2011

kiwifb commented Nov 24, 2011

jhpalmieri commented Nov 25, 2011

jdemeyer commented Nov 25, 2011

kiwifb commented Nov 25, 2011

jdemeyer commented Nov 25, 2011

jdemeyer commented Nov 25, 2011

jhpalmieri commented Nov 25, 2011

kiwifb commented Nov 25, 2011