switch sage to the new directory layout #14480

ohanar · 2013-04-24T05:40:41Z

To switch to a unified repository we need to include some information that are currently in the spkg tarballs, which requires modifying the sage's directory structure. Since we are switching workflows at the same time, it is an ideal time to rethink the directory structure as a whole.

The proposed significant changes

spkg -> build
devel/sage -> src
devel/ext -> src/ext
sage_scripts -> src/bin
spkg/bin -> src/bin

devel/sagenb -> /dev/null (is now considered a typical upstream python library)
/dev/null -> build/pkgs/* (these are now the spkg repositories)

Full changes are described at http://wiki.sagemath.org/WorkflowSEP.

The script used to create the git repository is consolidate-repos.sh, which depends on configuration.sh, git-filter-branch, and the fast-export directory (all located at https://github.com/sagemath/sage-workflow). They require bash4, git 1.8.x, and python 2.7 to run (at the very least, possibly more).

Depends on #13015
Depends on #14781

CC: @jdemeyer @saraedum

Component: distribution

Author: R. Andrew Ohana

Branch/Commit: u/ohanar/build_system @ cd237e7

Reviewer: Volker Braun

Issue created by migration from https://trac.sagemath.org/ticket/14480

The text was updated successfully, but these errors were encountered:

ohanar · 2013-04-24T05:44:26Z

comment:1

This can really only be reviewed on the new trac instance. For now, the branch field on http://trac.tangentspace.org/ticket/13015 links to the list of changes (to actually see the changes click the view changes button -- since this includes some massive changes, it might take awhile to load the side-by-side diff).

jdemeyer · 2013-04-25T05:25:15Z

comment:5

Where is the script that you used to automatically create the git repo from the hg repos? That should be reviewed too and kept for reference.

ohanar · 2013-07-13T07:06:03Z

Changed dependencies from #13015 to #13015, #14781

ohanar · 2013-07-18T03:29:30Z

Branch: u/ohanar/build_system

mezzarobba · 2013-07-21T10:34:28Z

comment:9

A few small issues with this branch. I am not sure which of these should actually be fixed here, so I'm marking the ticket as need_info rather that need_work.

There remain references to src/sage/misc/hg.py (which was removed as part of 48ff645) in src/sage/doctest/control.py.

sage -t --long src/sage/doctest/control.py
**********************************************************************
File "src/sage/doctest/control.py", line 434, in sage.doctest.control.DocTestController.add_files
Failed example:
    DC.add_files()
Exception raised:
    Traceback (most recent call last):
      File "/home/marc/co/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 486, in _run
        self.execute(example, compiled, test.globs)
      File "/home/marc/co/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 845, in execute
        exec compiled in globs
      File "<doctest sage.doctest.control.DocTestController.add_files[10]>", line 1, in <module>
        DC.add_files()
      File "/home/marc/co/sage/local/lib/python2.7/site-packages/sage/doctest/control.py", line 458, in add_files
        from sage.misc.hg import hg_sage
    ImportError: No module named hg
**********************************************************************
1 item had failures:
   1 of  16 in sage.doctest.control.DocTestController.add_files
    [162 tests, 1 failure, 1.97 s]

I also get a few more doctest failures such as the one below due to a missing import that seems to be fixed in the current git version of sagenb.

**********************************************************************
File "local/lib/python2.7/site-packages/sagenb-0.10.4-py2.7.egg/sagenb/misc/misc.py", line 44, in sagenb.misc.misc.print_open_msg
Failed example:
    sage.server.misc.print_open_msg('localhost', 8080, True)
Exception raised:
    Traceback (most recent call last):
      File "/home/marc/co/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 486, in _run
        self.execute(example, compiled, test.globs)
      File "/home/marc/co/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 845, in execute
        exec compiled in globs
      File "<doctest sagenb.misc.misc.print_open_msg[0]>", line 1, in <module>
        sage.server.misc.print_open_msg('localhost', Integer(8080), True)
    AttributeError: 'module' object has no attribute 'misc'
**********************************************************************

Finally, trying to run the notebook produces the following error:

sage: notebook()
The notebook files are stored in: sage_notebook.sagenb
**************************************************
*                                                *
* Open your web browser to http://localhost:8080 *
*                                                *
**************************************************
Executing twistd  --pidfile="sage_notebook.sagenb/sagenb.pid" -ny "sage_notebook.sagenb/twistedconf.tac"
Unhandled Error
Traceback (most recent call last):
  File "/home/marc/co/sage/local/lib/python2.7/site-packages/Twisted-12.3.0-py2.7-linux-x86_64.egg/twisted/application/app.py", line 652, in run
    runApp(config)
  File "/home/marc/co/sage/local/lib/python2.7/site-packages/Twisted-12.3.0-py2.7-linux-x86_64.egg/twisted/scripts/twistd.py", line 23, in runApp
    _SomeApplicationRunner(config).run()
  File "/home/marc/co/sage/local/lib/python2.7/site-packages/Twisted-12.3.0-py2.7-linux-x86_64.egg/twisted/application/app.py", line 386, in run
    self.application = self.createOrGetApplication()
  File "/home/marc/co/sage/local/lib/python2.7/site-packages/Twisted-12.3.0-py2.7-linux-x86_64.egg/twisted/application/app.py", line 451, in createOrGetApplication
    application = getApplication(self.config, passphrase)
--- <exception caught here> ---
  File "/home/marc/co/sage/local/lib/python2.7/site-packages/Twisted-12.3.0-py2.7-linux-x86_64.egg/twisted/application/app.py", line 462, in getApplication
    application = service.loadApplication(filename, style, passphrase)
  File "/home/marc/co/sage/local/lib/python2.7/site-packages/Twisted-12.3.0-py2.7-linux-x86_64.egg/twisted/application/service.py", line 405, in loadApplication
    application = sob.loadValueFromFile(filename, 'application', passphrase)
  File "/home/marc/co/sage/local/lib/python2.7/site-packages/Twisted-12.3.0-py2.7-linux-x86_64.egg/twisted/persisted/sob.py", line 210, in loadValueFromFile
    exec fileObj in d, d
  File "sage_notebook.sagenb/twistedconf.tac", line 15, in <module>
    import base as flask_base
exceptions.ImportError: No module named base

Failed to load application: No module named base

---------------------------------------------------------------------------
error                                     Traceback (most recent call last)
<ipython-input-1-3728cb3d7c7d> in <module>()
----> 1 notebook()

/home/marc/co/sage/local/lib/python2.7/site-packages/sagenb-0.10.4-py2.7.egg/sagenb/notebook/notebook_object.pyc in __call__(self, *args, **kwds)
    221     """
    222     def __call__(self, *args, **kwds):
--> 223         return self.notebook(*args, **kwds)
    224 
    225     notebook = run_notebook.notebook_run

/home/marc/co/sage/local/lib/python2.7/site-packages/sagenb-0.10.4-py2.7.egg/sagenb/notebook/run_notebook.pyc in notebook_run(self, directory, port, interface, port_tries, secure, reset, accounts, openid, server_pool, ulimit, timeout, upload, automatic_login, start_path, fork, quiet, server, profile, subnets, require_login, open_viewer, address)
    626     os.chdir(cwd)
    627     if e == 256:
--> 628         raise socket.error
    629 
    630 def get_admin_passwd():

error:

saraedum · 2013-07-23T09:49:09Z

comment:10

Thanks for having a look.

Replying to @mezzarobba:

There remain references to src/sage/misc/hg.py (which was removed as part of 48ff645) in src/sage/doctest/control.py.

This is now #14954.

I also get a few more doctest failures such as the one below due to a missing import that seems to be fixed in the current git version of sagenb.

This is fixed by #14330.

I believe that these issues are not related to consolidate-repos.sh, so I'll put this back to needs_review. These two tickets are part of the metaticket #13015.

tkluck · 2013-07-23T13:48:04Z

comment:11

These are the changes that actually need review for this ticket:

git diff origin/master..origin/u/ohanar/build_system

As a diff (this usually times out on the trac server):

http://trac.sagemath.org/changeset?new=af353bb626561725cee86bb0ca694a39ffb7b5f4&base=c3e08bfa279f8b4be1d5f924aa4668ae1b675b5e&old=c3e08bfa279f8b4be1d5f924aa4668ae1b675b5e

As a list of commits:

http://trac.sagemath.org/log?rev=af353bb626561725cee86bb0ca694a39ffb7b5f4&stop_rev=c3e08bfa279f8b4be1d5f924aa4668ae1b675b5e

tkluck · 2013-07-24T06:46:00Z

comment:12

The following is my review for the changes between master and
u/ohanar/build_system. I'm assuming that there are no other changes
apart from the directory structure between the old hg repositories
and master.

Overall, I'm impressed by the amount of work this has been.
My review contains mostly questions that you can probably just answer
or address with minor patches.

Maybe it makes sense for at least one more person to go through this diff.

diff --git a/Makefile b/Makefile
 doc-clean:
-	@echo "Deleting devel/sage/doc/output..."
-	rm -rf devel/sage/doc/output
+	@echo "Deleting generated docs..."
+	rm -rf src/doc/en/reference/*/sage
+	rm -rf src/doc/en/reference/*/sagenb
+	rm -rf src/doc/en/reference/sage
+	rm -rf src/doc/en/reference/sagenb
+	rm -rf src/doc/output

why do we clean more directories than before?

+
+bdist-clean: clean
+	@echo "Deleting miscellaneous artifacts generated by build system ..."
 	rm -rf logs
-	rm -f spkg/parallel_make.cfg
 	rm -rf dist

What happened to this parallel_make.cfg file?

Do I understand correctly that, and is it okay that the micro_release code now
only strips the binaries and does not remove build directories?

diff --git a/build/deps b/build/deps
new file mode 100644
index 0000000..2d2d19c
--- /dev/null
+++ b/build/deps
@@ -0,0 +1,523 @@
+###############################################################################
+# This file ($SAGE_ROOT/spkg/standard/deps) will be copied into
+# $SAGE_ROOT/spkg/Makefile by $SAGE_ROOT/spkg/install
+###############################################################################

There's still a reference to the spkg directory here.

diff --git a/build/gen_html b/build/gen_html
deleted file mode 100755
index 6dbad06..0000000
--- a/build/gen_html
+++ /dev/null

What did this script do? Why don't we need it anymore?

diff --git a/build/install b/build/install
index 780fe2a..bb14f76 100755
--- a/build/install
+++ b/build/install
@@ -99,11 +95,6 @@ if [ -d "$SAGE_ROOT/data" ] && [ ! -h "$SAGE_ROOT/data" ]; then
     rm -rf "$SAGE_ROOT/data"
 fi
-# Symlink old SAGE_DATA to SAGE_SHARE for optional/experimental
-# packages that haven't yet been updated.
-rm -f "$SAGE_ROOT/data"
-ln -s local/share "$SAGE_ROOT/data"
-

Should we make sure that this does not break any optional/experimental packages?
Personally, I'm fine with any package that installs anything into this data directory,
which I'm guessing has been deprecated for some time now. (how long?)

@@ -352,25 +343,17 @@ newest_version_base() {
 }
 
 # Usage: newest_version $pkg
-# Print version number of latest (according to modification time)
-# standard or optional package $pkg
+# Print version number of latest standard package $pkg
 newest_version() {
 <SNIP>

This reminds me: will we, at some point, test the installation of the
optional packages into the git framework? And, since we're already distributing
them officially, couldn't we just as well put their metadata in the repository, too?
(We don't need to do that right away).

 
+# glpk's build system uses the output of grep
+unset GREP_OPTIONS
+

This change has been made in quite some places. Why wasn't GREP_OPTIONS being set
a problem before?

diff --git a/sage b/sage
index 6597c54..46c69ba 100755
--- a/sage
+++ b/sage
@@ -124,11 +124,10 @@ fi
 export SAGE_ROOT
 
 # Run the actual Sage script
-if [ -x "$SAGE_ROOT/spkg/bin/sage" ]; then
-    "$SAGE_ROOT/spkg/bin/sage" "$@"
-elif [ -x "$SAGE_ROOT/local/bin/sage-sage" ]; then   # Support sage-4.x
-    export CUR=`pwd`
-    "$SAGE_ROOT/local/bin/sage-sage" "$@"
+if [ -x "$SAGE_ROOT/src/bin/sage" ]; then
+    "$SAGE_ROOT/src/bin/sage" "$@"
+elif [ -x "$SAGE_ROOT/local/bin/sage" ]; then # if in a striped binary
+    "$SAGE_ROOT/local/bin/sage" "$@"
 else
     echo >&2 "$0: no Sage installation found in \$SAGE_ROOT=$SAGE_ROOT"
     exit 1

It feels wrong to me to prefer the src/bin version of the sage script to
the local/bin version. Why is this?

diff --git a/src/bin/sage b/src/bin/sage
index 5c886e3..3762d5d 100755
--- a/src/bin/sage
+++ b/src/bin/sage
<SNIP>
@@ -864,12 +825,8 @@ if [ "$1" = '-pkg_nc' -o "$1" = "--pkg_nc" ]; then
 fi
 
 if [ "$1" = '-sdist' -o "$1" = "--sdist" ]; then
-    if [ $# -ne 2 ]; then
-        echo >&2 "** MISSING VERSION NUMBER! **"
-        exit 2
-    fi
-    maybe_sage_location
-    exec sage-sdist $2 "$SAGE_ROOT"
+    shift
+    exec sage-sdist "$@"

Why doesn't this need sage location anymore?

@@ -904,25 +861,7 @@ fi

 if [ "$1" = '-upgrade' -o "$1" = "--upgrade" ]; then
     shift
-    cd "$SAGE_ROOT"
-
-    # People often move the Sage install right before doing the upgrade, so it's
-    # important to fix any path hardcoding issues first, or certain library
-    # links will fail.
-    sage-location || exit $?
-
-    # Run sage-upgrade twice since when installing sage-scripts and a
-    # running script changes, it gets confused and exits with an error.
-    # Running again (with the script replaced) then fixes the problem.
     sage-upgrade "$@"
-    if [ $? -eq 2 ]; then   # this exit code means the user elected not to do the upgrade at all.
-        exit 2
-    fi
-    echo "Double checking that all packages have been installed."
-    sage-upgrade "$@" || exit $?
-
-    # Check that Sage still works
-    sage-starts
     exit $?
 fi

Why don't we call sage-location anymore? and sage-starts?

diff --git a/src/bin/sage-bdist b/src/bin/sage-bdist
index a776900..c765017 100755
--- a/src/bin/sage-bdist
+++ b/src/bin/sage-bdist
<SNIP>
+# Clone Sage repository
+echo "Cloning Sage repository..."
+git clone "$SAGE_ROOT" "$TMP_DIR/$TARGET"
+( cd "$TMP_DIR/$TARGET" && git remote set-url origin git://github.com/sagemath/sage.git )

We need to set this to the trac server. Also, what branch does this clone? It
should probably be the current one, because those are the binaries it copies.
Is it?

diff --git a/src/bin/sage-download-file b/src/bin/sage-download-file
index cf7dce0..a3428df 100755
--- a/src/bin/sage-download-file
+++ b/src/bin/sage-download-file
@@ -33,7 +33,7 @@ if [ -z "$URL_GRABBER" ]; then
     if [ -n "$SAGE_LOCAL" ] && [ -x "$SAGE_LOCAL/bin/python" ]; then
         URL_GRABBER="$SAGE_LOCAL/bin/python"
     elif command -v curl &>/dev/null; then
-        URL_GRABBER="curl"
+        URL_GRABBER="curl --fail"
     elif command -v wget &>/dev/null; then
         URL_GRABBER="wget -nv -O-"
     # Pick Python last because we don't know which version it is,

This looks fine, but why wasn't it an issue before?

diff --git a/src/bin/sage-env b/src/bin/sage-env
index 6c2408a..5e5af1b 100644
--- a/src/bin/sage-env
+++ b/src/bin/sage-env
<SNIP>
 # Make Mercurial not use any user defaults (like enabling the pager extension)
 # This is needed in order to use mercurial in scripts.  When running hg
-# directly through "sage -hg", we will disable HGPLAIN in spkg/bin/sage.
+# directly through "sage -hg", we will disable HGPLAIN in src/bin/sage.
 # See Trac Ticket #12058
 HGPLAIN="yes" && export HGPLAIN

This can probably go.

+if [ -z "$SAGE_REPO" ]; then
+    SAGE_REPO="git://github.com/sagemath/sage.git"
+    export SAGE_REPO
+fi

This should be set to the trac server.

diff --git a/src/bin/sage-spkg b/src/bin/sage-spkg
index 20c56db..86c03c3 100755
--- a/src/bin/sage-spkg
+++ b/src/bin/sage-spkg

This file has had major modifications so it needs updated author credits. Also,
the comment at the start still explains its old functionality.

 
+DOWNLOAD_ONLY=0
+if [ "$1" = '-d' ]; then
+    DOWNLOAD_ONLY=1
+    shift
+fi
+

I think this is an odd way to change the functionality. Maybe download-only
should be in a script of itself.

diff --git a/src/bin/sage-sync-build.py b/src/bin/sage-sync-build.py
index 8e45386..b5d476a 100755
--- a/src/bin/sage-sync-build.py
+++ b/src/bin/sage-sync-build.py
<SNIP>

Why does sage-sync-build need so many changes? Naively, it would need only a few
changes to some directory names.

diff --git a/src/bin/sage-upgrade b/src/bin/sage-upgrade
index 46c1935..55432d1 100755
--- a/src/bin/sage-upgrade
+++ b/src/bin/sage-upgrade
-./install
+if [ "$#" -gt 0 ]; then
+    BRANCH="$1"
+    shift
+else
+    BRANCH=$(git symbolic-ref HEAD) ||
+        BRANCH=$(git describe --exact-match --tags HEAD | \
+                 sed -e 's;^;refs/tags/;')
+    if [ -n "$BRANCH" ]; then
+        case $BRANCH in
+            refs/heads/master) BRANCH=master;;
+            refs/heads/release) BRANCH=release;;
+            refs/heads/beta) BRANCH=beta;;
+            refs/tags/*) BRANCH=release;;
+        esac
+    fi
+fi

Maybe do a git fetch first? Also, should we add pushing to these branches
in the repo to a script somewhere? Maybe to sage-sdist?

diff --git a/src/c_lib/SConstruct b/src/c_lib/SConstruct
index fd4719a..1813de8 100644
--- a/src/c_lib/SConstruct
+++ b/src/c_lib/SConstruct
-#Here we only copy the files over if we are on Cygwin.  Otherwise, the
-#library will be handled by the symlinks created in
-#$SAGE_ROOT/devel/sage/spkg-install
-if os.environ['UNAME'] == 'CYGWIN':
-    env.Alias("install", "$SAGE_LOCAL/lib")
-else:
-    env.Alias("install", [lib])
-
+env.Alias("install", "$SAGE_LOCAL")

Will this still work on Cygwin?

diff --git a/src/sage/misc/sagedoc.py b/src/sage/misc/sagedoc.py
index 9f0bbe3..12e8434 100644
--- a/src/sage/misc/sagedoc.py
+++ b/src/sage/misc/sagedoc.py
@@ -407,9 +407,9 @@ def format(s, embedded=False):
     EXAMPLES::
 
         sage: from sage.misc.sagedoc import format
-        sage: identity_matrix(2).rook_vector.__doc__[115:184]
+        sage: identity_matrix(2).rook_vector.__doc__[107:176]
         'Let `A` be a general `m` by `n`\n        (0,1)-matrix with `m \\le n`. '
-        sage: format(identity_matrix(2).rook_vector.__doc__[115:184])
+        sage: format(identity_matrix(2).rook_vector.__doc__[107:176])
         'Let A be a general m by n\n   (0,1)-matrix with m <= n.\n'
 
     If the first line of the string is 'nodetex', remove 'nodetex' but

why did this change? Is this related to the removal of whitespace? If so:
okay.

diff --git a/src/sage/tests/cmdline.py b/src/sage/tests/cmdline.py
index 342c3ff..7e45f0f 100644
--- a/src/sage/tests/cmdline.py
+++ b/src/sage/tests/cmdline.py
@@ -208,15 +198,9 @@ def test_executable(args, input="", timeout=100.0, **kwds):
     Test ``sage --info [packages]``, unless this is a binary (bdist)
     distribution which doesn't ship spkgs::
 
-        sage: if os.path.isfile(os.path.join(SAGE_ROOT, 'spkg', 'standard', '.from_bdist')):
-        ...     out = "Found package sqlalchemy in spkg/standard/sqlalchemy-...spkg\n= SQLAlchemy =\n...\nSQLAlchemy is the Python SQL toolkit..."
-        ...     err = ''
-        ...     ret = 0
-        ... else:
-        ...     (out, err, ret) = test_executable(["sage", "--info", "sqlalchemy"])
-        ...
+        sage: out, err, ret = test_executable(["sage", "--info", "sqlalchemy"])
         sage: print out
-        Found package sqlalchemy in spkg/standard/sqlalchemy-...spkg
+        Using local scripts to install sqlalchemy-...
         = SQLAlchemy =
         ...
         SQLAlchemy is the Python SQL toolkit...

Do I understand correctly that we install sqlalchemy when running doctests? What is
happening here?

diff --git a/src/spkg-delauto b/src/spkg-delauto
deleted file mode 100755
index 61d071d..0000000
--- a/src/spkg-delauto
+++ /dev/null

Why don't we need this anymore?

ohanar · 2013-07-24T09:31:16Z

comment:13

Replying to @tkluck:

Maybe it makes sense for at least one more person to go through this diff.

Definitely. And hopefully someone who is already familiar with Sage's build system.

why do we clean more directories than before?

In order to make sure distclean still works as expected. Previously cleaning up everything wasn't important for this since distclean would just delete the whole devel directory.

What happened to this parallel_make.cfg file?

There is no other reference to it in the source code.

Do I understand correctly that, and is it okay that the micro_release code now
only strips the binaries and does not remove build directories?

It seems like I didn't revert some cleanup I made to the Makefile in response to #14721.

diff --git a/build/gen_html b/build/gen_html
deleted file mode 100755
index 6dbad06..0000000
--- a/build/gen_html
+++ /dev/null

What did this script do? Why don't we need it anymore?

it is used to generate the webpages http://sagemath.org/packages/{standard,optional,experimental}, however the actual version of the script that is used in generating these patches disagrees from the version that is included in Sage. Since I was overhauling the build system (and this was in the directory related to the build system), I thought it wise to remove it to avoid any confusion.

Should we make sure that this does not break any optional/experimental packages?
Personally, I'm fine with any package that installs anything into this data directory,
which I'm guessing has been deprecated for some time now. (how long?)

This reminds me: will we, at some point, test the installation of the
optional packages into the git framework? And, since we're already distributing
them officially, couldn't we just as well put their metadata in the repository, too?
(We don't need to do that right away).

We should at least make sure that all optional packages work. Basically all changes that might be needed should be fixable within the mercurial distribution. Since we will have to through the packages anyway, it made perfect sense to me to cleanup the whole SAGE_DATA thing. I've opened #14962 for this task.

This change has been made in quite some places. Why wasn't GREP_OPTIONS being set
a problem before?

It was, however I didn't bother fixing it on the mercurial side because either you have to unset it in sage-env or you have to do it in a bunch of packages (although it just came to me that you could unset it in sage-spkg). Unsetting in sage-env is less than ideal because it means that commands like sage -grep don't respect GREP_OPTIONS, and changing a simple thing like this across a bunch of packages is incredibly painful in the current workflow.

It feels wrong to me to prefer the src/bin version of the sage script to
the local/bin version. Why is this?

I've gone back and forth on this a few times. The reason why I ended up with this was because the latest version of the sage script will always be at src/bin, where as the version at local/bin will only be updated if you have rerun make.

diff --git a/src/bin/sage b/src/bin/sage
index 5c886e3..3762d5d 100755
--- a/src/bin/sage
+++ b/src/bin/sage
<SNIP>
@@ -864,12 +825,8 @@ if [ "$1" = '-pkg_nc' -o "$1" = "--pkg_nc" ]; then
 fi
 
 if [ "$1" = '-sdist' -o "$1" = "--sdist" ]; then
-    if [ $# -ne 2 ]; then
-        echo >&2 "** MISSING VERSION NUMBER! **"
-        exit 2
-    fi
-    maybe_sage_location
-    exec sage-sdist $2 "$SAGE_ROOT"
+    shift
+    exec sage-sdist "$@"

Why doesn't this need sage location anymore?

This was a merging error, sage-location should still be here.

Why don't we call sage-location anymore? and sage-starts?

I moved the sage-location call to sage-real-upgrade, and I think that sage-starts should probably be moved to sage-real-upgrade as well (which I think I meant to do, but it seems like I forgot).

We need to set this to the trac server. Also, what branch does this clone? It
should probably be the current one, because those are the binaries it copies.
Is it?

Without an argument git clone will clone HEAD.

diff --git a/src/bin/sage-download-file b/src/bin/sage-download-file
index cf7dce0..a3428df 100755
--- a/src/bin/sage-download-file
+++ b/src/bin/sage-download-file
@@ -33,7 +33,7 @@ if [ -z "$URL_GRABBER" ]; then
     if [ -n "$SAGE_LOCAL" ] && [ -x "$SAGE_LOCAL/bin/python" ]; then
         URL_GRABBER="$SAGE_LOCAL/bin/python"
     elif command -v curl &>/dev/null; then
-        URL_GRABBER="curl"
+        URL_GRABBER="curl --fail"
     elif command -v wget &>/dev/null; then
         URL_GRABBER="wget -nv -O-"
     # Pick Python last because we don't know which version it is,

This looks fine, but why wasn't it an issue before?

It was, but it is more of an issue now because sage-spkg now tries a few urls for each package, which requires sage-download-file to bail if the server returns an error.

diff --git a/src/bin/sage-sync-build.py b/src/bin/sage-sync-build.py
index 8e45386..b5d476a 100755
--- a/src/bin/sage-sync-build.py
+++ b/src/bin/sage-sync-build.py
<SNIP>
Why does sage-sync-build need so many changes? Naively, it would need only a few
changes to some directory names.

Currently we are symlinking stuff into SAGE_LOCAL from the devel directory (this is to make cloning fast), however this is no longer happening in the git repository. This caused some silly breaking with the sage-sync-build, which was in need of some heavy revision anyway. There is now another ticket out there (don't remember which one) dedicated to fixing + updating this script.

diff --git a/src/bin/sage-upgrade b/src/bin/sage-upgrade
index 46c1935..55432d1 100755
--- a/src/bin/sage-upgrade
+++ b/src/bin/sage-upgrade
-./install
+if [ "$#" -gt 0 ]; then
+    BRANCH="$1"
+    shift
+else
+    BRANCH=$(git symbolic-ref HEAD) ||
+        BRANCH=$(git describe --exact-match --tags HEAD | \
+                 sed -e 's;^;refs/tags/;')
+    if [ -n "$BRANCH" ]; then
+        case $BRANCH in
+            refs/heads/master) BRANCH=master;;
+            refs/heads/release) BRANCH=release;;
+            refs/heads/beta) BRANCH=beta;;
+            refs/tags/*) BRANCH=release;;
+        esac
+    fi
+fi

Maybe do a git fetch first?

Good idea.

Also, should we add pushing to these branches
in the repo to a script somewhere? Maybe to sage-sdist?

Pushing should only be done by the release manager. I think the best place for such stuff would be as part of the development scripts, although I think adding the release manager portion to these scripts should be lower priority than the general developer portion of these scripts.

diff --git a/src/c_lib/SConstruct b/src/c_lib/SConstruct
index fd4719a..1813de8 100644
--- a/src/c_lib/SConstruct
+++ b/src/c_lib/SConstruct
-#Here we only copy the files over if we are on Cygwin.  Otherwise, the
-#library will be handled by the symlinks created in
-#$SAGE_ROOT/devel/sage/spkg-install
-if os.environ['UNAME'] == 'CYGWIN':
-    env.Alias("install", "$SAGE_LOCAL/lib")
-else:
-    env.Alias("install", [lib])
-
+env.Alias("install", "$SAGE_LOCAL")

Will this still work on Cygwin?

I think so, although we really should have someone on Cygwin test. As I mentioned, the symlinks that the comment refers to are no more.

diff --git a/src/sage/tests/cmdline.py b/src/sage/tests/cmdline.py
index 342c3ff..7e45f0f 100644
--- a/src/sage/tests/cmdline.py
+++ b/src/sage/tests/cmdline.py
@@ -208,15 +198,9 @@ def test_executable(args, input="", timeout=100.0, **kwds):
     Test ``sage --info [packages]``, unless this is a binary (bdist)
     distribution which doesn't ship spkgs::
 
-        sage: if os.path.isfile(os.path.join(SAGE_ROOT, 'spkg', 'standard', '.from_bdist')):
-        ...     out = "Found package sqlalchemy in spkg/standard/sqlalchemy-...spkg\n= SQLAlchemy =\n...\nSQLAlchemy is the Python SQL toolkit..."
-        ...     err = ''
-        ...     ret = 0
-        ... else:
-        ...     (out, err, ret) = test_executable(["sage", "--info", "sqlalchemy"])
-        ...
+        sage: out, err, ret = test_executable(["sage", "--info", "sqlalchemy"])
         sage: print out
-        Found package sqlalchemy in spkg/standard/sqlalchemy-...spkg
+        Using local scripts to install sqlalchemy-...
         = SQLAlchemy =
         ...
         SQLAlchemy is the Python SQL toolkit...

Do I understand correctly that we install sqlalchemy when running doctests? What is
happening here?

This can be changed in sage-spkg. That message just prints when it realizes it has the package metadata locally.

diff --git a/src/spkg-delauto b/src/spkg-delauto
deleted file mode 100755
index 61d071d..0000000
--- a/src/spkg-delauto
+++ /dev/null

Why don't we need this anymore?

no other reference in the source code (except for MANIFEST.in, which should be updated)

tkluck · 2013-07-24T21:18:45Z

comment:14

Mind if I start making some of the (simpler) changes?

Currently we are symlinking stuff into SAGE_LOCAL from the devel directory (this is to make cloning fast), however this is no longer happening in the git repository. This caused some silly breaking with the sage-sync-build, which was in need of some heavy revision anyway. There is now another ticket out there (don't remember which one) dedicated to fixing + updating this script.

I couldn't find the ticket. Do you mean that your edits basically fix the ticket? Or do you mean that the version of sage-sync-build in your build_system branch is still in need of some heavy revision?

It feels wrong to me to prefer the src/bin version of the sage script to the local/bin version. Why is this?

I've gone back and forth on this a few times. The reason why I ended up with this was because the latest version of the sage script will always be at src/bin, where as the version at local/bin will only be updated if you have rerun make.

I agree that this is the behavior, but I'm not sure whether that's what we want. After switching branches, you could end up in the situation where the sage script from one branch is using SAGE_LOCAL files from another branch. Imagine that a branch adds both a package imagemagick and a command sage -convert and it seems much more consistent to me to only run the new sage script after make has been called.

But I guess that I don't mind much; whoever is in a situation where the difference affects them, is probably doing something with the internal workings of sage anyway, and should be able to deal with it.

sagetrac-felixs · 2013-08-25T08:50:56Z

comment:15

I think it makes sense to move build/pkgs to pkgs. My current work on top of build_system uses pkgs for the package sources and build is used to build packages. Would it be possible to do this move in build_system?

ohanar · 2013-08-31T05:48:36Z

comment:16

Ok, I think I addressed Timo's comments, with the following exceptions:

HG configuration in sage-env: I don't see a good reason for getting rid of these in the short term, we may not be using hg anymore, but it could still be helpful for the transition period while repositories there are still HG repositories for optional packages.

GREP_OPTIONS: I'm not going to fix this now, instead leave it to a later date (unless I encounter merge conflicts, which so far I haven't).

Replying to @sagetrac-felixs:

I think it makes sense to move build/pkgs to pkgs. My current work on top of build_system uses pkgs for the package sources and build is used to build packages. Would it be possible to do this move in build_system?

I would rather not, I don't see a huge need for change at this moment and currently everything works. We can always change this later down the road, but for now I would like to get this ticket finished.

One other thing is that some functionality (especially sdist and bdist) might be broken until I merge the git spkg, which I will do in this ticket. However, there is still plenty of other code that needs review so I'm going to mark this to needs review for the moment.

ohanar · 2013-08-31T06:16:24Z

comment:18

ok, merged in the git spkg, so all should be good to go

sagetrac-git · 2013-09-01T08:37:02Z

Branch pushed to git repo; I updated commit sha1. New commits:

sagetrac-git · 2013-11-04T22:37:11Z

Changed commit from 952f7e3 to 874ba34

sagetrac-git · 2013-11-06T20:10:51Z

Changed commit from 874ba34 to f5796f2

sagetrac-git · 2013-11-06T20:10:51Z

Branch pushed to git repo; I updated commit sha1. New commits:

`[f5796f2](https://github.com/sagemath/sagetrac-mirror/commit/f5796f2)`	`remove reference to sage/misc/hg from docbuild`

vbraun · 2013-11-06T22:50:36Z

comment:47

Looks good to me!

vbraun · 2013-11-06T22:50:36Z

Reviewer: Volker Braun

sagetrac-git · 2013-11-15T20:46:50Z

Changed commit from f5796f2 to 1b1d236

sagetrac-git · 2013-11-15T20:46:50Z

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

vbraun · 2013-11-23T02:20:55Z

comment:49

For the record, this ticket is positively reviewed. Even if Andrew keeps merging the sage-hg branch until sage-5.13 is finished.

sagetrac-git · 2013-11-27T05:20:49Z

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

sagetrac-git · 2013-11-27T05:20:49Z

Changed commit from 1b1d236 to 47cb70d

mezzarobba · 2013-11-27T10:17:41Z

comment:51

Hi,

The merge of 5.13-beta4 broke make build for me, because http://www.sagemath.org/packages/upstream/r/r-3.0.2.tar is missing. See also #14706 comment:119.

sagetrac-git · 2013-12-06T19:00:03Z

Changed commit from 47cb70d to 33dec52

sagetrac-git · 2013-12-06T19:00:03Z

Branch pushed to git repo; I updated commit sha1. New commits:

sagetrac-git · 2013-12-12T05:59:42Z

Branch pushed to git repo; I updated commit sha1. New commits:

sagetrac-git · 2013-12-12T05:59:42Z

Changed commit from 33dec52 to cd237e7

ohanar added this to the sage-5.10 milestone Apr 24, 2013

ohanar added c: distribution labels Apr 24, 2013

ohanar added the s: needs review label Apr 24, 2013

ohanar modified the milestones: sage-5.10, sage-6.0 Apr 24, 2013

ohanar changed the title ~~switch sage to the new layout~~ switch sage to the new directory layout Apr 24, 2013

This comment has been minimized.

Sign in to view

mezzarobba added s: needs info and removed s: needs review labels Jul 21, 2013

saraedum added s: needs review and removed s: needs info labels Jul 23, 2013

tkluck added s: needs info and removed s: needs review labels Jul 24, 2013

ohanar added s: needs review and removed s: needs info labels Aug 31, 2013

vbraun added s: positive review and removed s: needs review labels Nov 6, 2013

sagetrac-git mannequin added s: needs review and removed s: positive review labels Nov 15, 2013

vbraun added s: positive review and removed s: needs review labels Nov 23, 2013

sagetrac-git mannequin added s: needs review and removed s: positive review labels Nov 27, 2013

vbraun added s: positive review and removed s: needs review labels Dec 16, 2013

vbraun closed this as completed Dec 16, 2013

kini mentioned this issue May 8, 2015

Metaticket: revamp workflow / switch to git #13015

Closed

ohanar mentioned this issue Dec 16, 2013

create the new development scripts/library #14482

Closed

tkluck mentioned this issue Dec 17, 2013

remove hg references from git repository #14954

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

switch sage to the new directory layout #14480

switch sage to the new directory layout #14480

ohanar commented Apr 24, 2013

ohanar commented Apr 24, 2013

jdemeyer commented Apr 25, 2013

This comment has been minimized.

ohanar commented Jul 13, 2013

ohanar commented Jul 18, 2013

mezzarobba commented Jul 21, 2013

saraedum commented Jul 23, 2013

tkluck commented Jul 23, 2013

tkluck commented Jul 24, 2013

ohanar commented Jul 24, 2013

tkluck commented Jul 24, 2013

sagetrac-felixs mannequin commented Aug 25, 2013

ohanar commented Aug 31, 2013

ohanar commented Aug 31, 2013

sagetrac-git mannequin commented Sep 1, 2013

sagetrac-git mannequin commented Nov 4, 2013

sagetrac-git mannequin commented Nov 6, 2013

sagetrac-git mannequin commented Nov 6, 2013

vbraun commented Nov 6, 2013

vbraun commented Nov 6, 2013

sagetrac-git mannequin commented Nov 15, 2013

sagetrac-git mannequin commented Nov 15, 2013

vbraun commented Nov 23, 2013

sagetrac-git mannequin commented Nov 27, 2013

sagetrac-git mannequin commented Nov 27, 2013

mezzarobba commented Nov 27, 2013

sagetrac-git mannequin commented Dec 6, 2013

sagetrac-git mannequin commented Dec 6, 2013

sagetrac-git mannequin commented Dec 12, 2013

sagetrac-git mannequin commented Dec 12, 2013

switch sage to the new directory layout #14480

switch sage to the new directory layout #14480

Comments

ohanar commented Apr 24, 2013

ohanar commented Apr 24, 2013

jdemeyer commented Apr 25, 2013

This comment has been minimized.

ohanar commented Jul 13, 2013

ohanar commented Jul 18, 2013

mezzarobba commented Jul 21, 2013

saraedum commented Jul 23, 2013

tkluck commented Jul 23, 2013

tkluck commented Jul 24, 2013

ohanar commented Jul 24, 2013

tkluck commented Jul 24, 2013

sagetrac-felixs mannequin commented Aug 25, 2013

ohanar commented Aug 31, 2013

ohanar commented Aug 31, 2013

sagetrac-git mannequin commented Sep 1, 2013

sagetrac-git mannequin commented Nov 4, 2013

sagetrac-git mannequin commented Nov 6, 2013

sagetrac-git mannequin commented Nov 6, 2013

vbraun commented Nov 6, 2013

vbraun commented Nov 6, 2013

sagetrac-git mannequin commented Nov 15, 2013

sagetrac-git mannequin commented Nov 15, 2013

vbraun commented Nov 23, 2013

sagetrac-git mannequin commented Nov 27, 2013

sagetrac-git mannequin commented Nov 27, 2013

mezzarobba commented Nov 27, 2013

sagetrac-git mannequin commented Dec 6, 2013

sagetrac-git mannequin commented Dec 6, 2013

sagetrac-git mannequin commented Dec 12, 2013

sagetrac-git mannequin commented Dec 12, 2013