cygwin: get_memory_usage isn't implemented

[williamstein]

sage -t  "devel/sage/sage/misc/getusage.py"                 
**********************************************************************
File "/home/wstein/sage-4.4.3/devel/sage/sage/misc/getusage.py", line 30:
    sage: print "ignore this";  top()              # random output
Exception raised:
    Traceback (most recent call last):
      File "/home/wstein/sage-4.4.3/local/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/home/wstein/sage-4.4.3/local/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/home/wstein/sage-4.4.3/local/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_1[2]>", line 1, in <module>
        print "ignore this";  top()              # random output###line 30:
    sage: print "ignore this";  top()              # random output
      File "/home/wstein/sage-4.4.3/local/lib/python/site-packages/sage/misc/getusage.py", line 57, in top
        raise NotImplementedError("top not implemented on platform %s" % U)
    NotImplementedError: top not implemented on platform cygwin_nt-5.1
**********************************************************************
File "/home/wstein/sage-4.4.3/devel/sage/sage/misc/getusage.py", line 92:
    sage: t = get_memory_usage()
Exception raised:
    Traceback (most recent call last):
      File "/home/wstein/sage-4.4.3/local/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/home/wstein/sage-4.4.3/local/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/home/wstein/sage-4.4.3/local/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_2[2]>", line 1, in <module>
        t = get_memory_usage()###line 92:
    sage: t = get_memory_usage()
      File "/home/wstein/sage-4.4.3/local/lib/python/site-packages/sage/misc/getusage.py", line 128, in get_memory_usage
        raise NotImplementedError("memory usage not implemented on platform %s" % U)
    NotImplementedError: memory usage not implemented on platform cygwin_nt-5.1
**********************************************************************
File "/home/wstein/sage-4.4.3/devel/sage/sage/misc/getusage.py", line 93:
    sage: get_memory_usage(t)          # amount of memory more than when we defined t.
Exception raised:
    Traceback (most recent call last):
      File "/home/wstein/sage-4.4.3/local/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/home/wstein/sage-4.4.3/local/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/home/wstein/sage-4.4.3/local/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_2[3]>", line 1, in <module>
        get_memory_usage(t)          # amount of memory more than when we defined t.###line 93:
    sage: get_memory_usage(t)          # amount of memory more than when we defined t.
    NameError: name 't' is not defined
**********************************************************************

Cygwin has top now, so it's straightforward to fix.

apply

• attachment: trac_9170.patch

CC: @jpflori

Component: porting: Cygwin

Author: Dmitrii Pasechnik

Reviewer: Jean-Pierre Flori

Merged: sage-5.7.beta4

Issue created by migration from https://trac.sagemath.org/ticket/9170

[williamstein]

comment:1

Another test failure caused by this:

sage -t  "devel/sage/sage/rings/tests.py"                   
**********************************************************************
File "/home/wstein/sage-4.4.3/devel/sage/sage/rings/tests.py", line 229:
    sage: sage.rings.tests.test_random_elements(trials=2, seed=0)
Expected:

[sagetrac-drkirkby]

comment:2

I assume this means that

sage: top()

will not work either. I know on Solaris, I called prstat rather than top as that is a standard part of Solaris, and top is not. But I thought on Linux there were some sysem calls in Sage for computing memory usage on Linux. Could it be as simple as changing

if [ "x$UNAME" = xLinux ] ; then 
   // Use Linux code
elif [ "x$UNAME" = xSunOS ] ; then 
  // call prstat

to

if [ "x$UNAME" = xLinux ] || [ "x$UNAME" = xCYGWIN ] ; then 
   // Use Linux code
elif [ "x$UNAME" = xSunOS ] ; then 
  // call prstat

Dave

[sagetrac-gbe]

comment:3

Replying to @sagetrac-drkirkby:

I assume this means that

sage: top()

will not work either.

I'm not at home so I can't test on my machine, but

File "/home/wstein/sage-4.4.3/devel/sage/sage/misc/getusage.py", line 30:
    sage: print "ignore this";  top()              # random output

looks like the line that's causing the first error to be throwing. The offending line(s) should be

    if U == 'linux':
        cmd = 'top -b -n 1 -p %s' % pid
    elif U == 'darwin':
        cmd = 'top -l 1 |grep "^ *%s "' % pid
    elif U == 'sunos':
        cmd = '/usr/bin/prstat -n 100000 1 1  | grep "^ *%s "' % pid
    else:
        raise NotImplementedError("top not implemented on platform %s" % U)

I've double checked, and cygwin does ship with a top, so I suspect all that needs to be done is add a few instances along the line of "or U == 'cygwin'" and these issue would be resolved.

I'll try to test this over the weekend and have a patch prepared.

[sagetrac-drkirkby]

comment:4

Irrespective of whether Cygwin comes with top or not, using it can't be the best way to get memory usage. top is not used on Linux exept as a last resort. There are system calls to get the memory usage, which is the most sensible way to do this. If those system calls work on Cygwin, then it would be

• More accurate
• Faster
• Not require 'top' to be installed.

top is not a standard Unix command. It's not defined by POSIX and will not be installed by default on Cygwin. It's better to avoid calling top if at all possible.

Dave

[sagetrac-gbe]

comment:5

Very true. I should have thought about what the code was doing, not just how to fix the breakage. As for top not being POSIX, I wasn't aware of that. I had always assumed it was. Since it's not POSIX it seems fine to let top() calls fail on Windows if top() is not installed, leaving an appropriately worded explanation.

The only time top is called on linux is via a top() call. To get the memory usage under linux, the /proc//status is inspected. While they don't seem to document how complete it is, cygwin does populate a /proc directory. I'll poke around to see if the cygwin /proc system has what is needed. As far as I can see this the closest Python has to a memory usage call without using external libraries.

Geoff

[sagetrac-drkirkby]

comment:6

Replying to @sagetrac-gbe:

Very true. I should have thought about what the code was doing, not just how to fix the breakage. As for top not being POSIX, I wasn't aware of that. I had always assumed it was. Since it's not POSIX it seems fine to let top() calls fail on Windows if top() is not installed, leaving an appropriately worded explanation.

The only time top is called on linux is via a top() call. To get the memory usage under linux, the /proc//status is inspected. While they don't seem to document how complete it is, cygwin does populate a /proc directory. I'll poke around to see if the cygwin /proc system has what is needed. As far as I can see this the closest Python has to a memory usage call without using external libraries.

Geoff

I don't know, but I thought the plan was to make an installer for Cygwin which installed the perquisites, which would include top. So it should not really fail.

As much as I don't like the idea of using 'top', I think in the short term it is a OK. There are more significant issues causing problems on Cygwin. I would have thought this one of the lower priority ones, but that's just my opinion.

Dave

[kcrisman]

comment:7

Unsurprisingly, this still doesn't work, though most stuff now does/can on Cygwin.

[jpflori]

comment:9

I guess we could prereq for the Cygwin procps package but is this really necessary?

[kcrisman]

comment:10

Well, get_memory_usage is pretty useful. Otherwise the real question would be how to change these doctests so they pass on Cygwin, which isn't clear to me.

[jpflori]

comment:11

After fighting with Cygwin to get enough memory to run heegner.py test (did not manage yet... not sure that I changed anything but I can allocate 500000000 bytes, but not 512000000), I'm not really convinced that get_memory_usage is that useful on Cygwin as you won't really be able to allocate as much memory as you want (without fighting with Cygwin...).

[jpflori]

comment:12

Whatsoever, with Cygwin procps package which provides top, the doctests do not pass.
And this is expected as the top function implemented in getusage.py explicitely raises an Error on not explicitely listed systems, which include Cygwin.

[jpflori]

comment:13

So I still think the best solution is not to test this on Cygwin or to expect a different result...

Or if you insist we add procps as a prereq and modify the Sage's top function implem.

Alternatively, we could modify Sage's top so that it raises an Error iff the system top is not available.
Then only the top implementation would vary, depending on the availability of top, not the doctest which would expect random output on Cygwin (0.0 if top is installed, Error otherwise).
But we would still need to deal with doctests producing different output depending on the system (let's say mark it as random Cygwin so that it will (hackingly) deal with both situation (whether procps is installed or not), rather than making it depend on the fact that it is installed which looks impossible; and expect 0.0 on other systems).

[kcrisman]

comment:14

Given that #12828 just uses completely different commands (i.e., ps) to get this information, which should presumably be available (?) on Cygwin, can we just do that? Note that we also do this on Solaris. I.e. we just add another case, perhaps on top (no pun intended) of #12828.

[jpflori]

comment:15

Yes ps is included in the "cygwin" package itself so always available.

[jpflori]

comment:16

But that won't help, I guess we are stuck with top.

[jpflori]

comment:17

Or we could just directly look into /proc/meminfo, not sure why we don't do that on Linuces.

[jpflori]

comment:18

Or rather things in /proc/$PID/

[jpflori]

comment:19

Surely that is not portable across a large range of Linuces.

[dimpase]

comment:20

I do have top on my Cygwin install and its output is very much Linux-like. So I propose just to hack misc/getusage.py to let it work on cygwin, too.

[dimpase]

Attachment: trac_9170.patch.gz

enabling it on Cygwin

[dimpase]

comment:22

please test the patch (on Cygwin you might need to install top). The patch works on my Cygwin install just fine.

[jpflori]

Reviewer: Jean-Pierre Flori

[jpflori]

Author: Dmitrii Pasechnik

[jdemeyer]

Merged: sage-5.7.beta4