OSError 'Not a directory' when creating env on Jython 2.7.0

[pytoxbot]

• Bitbucket: https://bitbucket.org/hpk42/tox/issue/326
• Originally reported by: @anthrotype
• Originally created at: 2016-04-14T10:54:44.259

Hi,

I cannot use tox with Jython. I'm using the latest tox 2.3.1, and Jython 2.7.0 on OS X 10.10.5 with Java 1.7.0_79.

I tried to clone this repository and run the test suite using jython. Below is the output from test_venv.py::test_create that shows my issue.

It seems that the subprocess call to virtualenv command that creates the new tox environment is failing because a listdir call in session._initlogpath raises OSError because the envlogdir (where the output from the subprocess is to be logged) does not exist. If I browse the .tox directory in Finder, I can see there is no {envdir}/log subfolder.

Thank you for your support.

Cosimo

(jython-venv) cosimolupo@Cosimo-MBP ~/Documents/Github/tox
$ py.test -v tests/test_venv.py::test_create
============================= test session starts ==============================
platform java1.7.0_79 -- Python 2.7.0, pytest-2.9.1, py-1.4.31, pluggy-0.3.1 -- /Users/cosimolupo/Documents/Github/fonttools/jython-venv/bin/jython
cachedir: .cache
tox comes from: '/Users/cosimolupo/Documents/Github/tox/tox/__init__$py.class'
rootdir: /Users/cosimolupo/Documents/Github/tox, inifile: tox.ini
collected 28 items 

tests/test_venv.py::test_create FAILED

=================================== FAILURES ===================================
_________________________________ test_create __________________________________

monkeypatch = <_pytest.monkeypatch.monkeypatch instance at 0x69>
mocksession = <tox._pytestplugin.MockSession instance at 0x6a>
newconfig = <function newconfig at 0x6b>

    def test_create(monkeypatch, mocksession, newconfig):
        config = newconfig([], """
            [testenv:py123]
        """)
        envconfig = config.envconfigs['py123']
        venv = VirtualEnv(envconfig, session=mocksession)
        assert venv.path == envconfig.envdir
        assert not venv.path.check()
        action = mocksession.newaction(venv, "getenv")
>       tox_testenv_create(action=action, venv=venv)

tests/test_venv.py:60: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tox/venv.py:395: in tox_testenv_create
    venv._pcall(args, venv=False, action=action, cwd=basepath)
tox/venv.py:361: in _pcall
    return action.popen(args, cwd=cwd, env=env,
tox/session.py:121: in popen
    fout = self._initlogpath(self.id)
tox/session.py:107: in _initlogpath
    l = logdir.listdir("%s-*" % actionid)
tox/session.py:107: in _initlogpath
    l = logdir.listdir("%s-*" % actionid)
../fonttools/jython-venv/Lib/site-packages/py/_path/local.py:388: in listdir
    names = py.error.checked_call(os.listdir, self.strpath)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <py._error.ErrorMaker object at 0x6c>
func = <java function listdir 0x6d>
args = ('/private/var/folders/jb/rjz76yw92w7144mwqg119jnm0000gn/T/pytest-of-cosimolupo/pytest-3/test_create0/.tox/py123/log',)
kwargs = {}, __tracebackhide__ = False, cls = <class 'py.error.ENOTDIR'>
value = OSError(20, 'Not a directory'), tb = <traceback object at 0x6e>
errno = 20

    def checked_call(self, func, *args, **kwargs):
        """ call a function and raise an errno-exception if applicable. """
        __tracebackhide__ = True
        try:
            return func(*args, **kwargs)
        except self.Error:
            raise
        except (OSError, EnvironmentError):
            cls, value, tb = sys.exc_info()
            if not hasattr(value, 'errno'):
                raise
            __tracebackhide__ = False
            errno = value.errno
            try:
                if not isinstance(value, WindowsError):
                    raise NameError
            except NameError:
                # we are not on Windows, or we got a proper OSError
                cls = self._geterrnoclass(errno)
            else:
                try:
                    cls = self._geterrnoclass(_winerrnomap[errno])
                except KeyError:
                    raise value
>           raise cls("%s%r" % (func.__name__, args))
E           ENOTDIR: [Not a directory]: listdir('/private/var/folders/jb/rjz76yw92w7144mwqg119jnm0000gn/T/pytest-of-cosimolupo/pytest-3/test_create0/.tox/py123/log',)

../fonttools/jython-venv/Lib/site-packages/py/_error.py:84: ENOTDIR
=========================== 1 failed in 2.31 seconds ===========================

[peterjc]

I just hit this bug or something very similar while trying to use Jython 2.7.0 under TravisCI/Tox

https://travis-ci.org/peterjc/biopython/jobs/165876306

$ tox -c .travis-tox.ini -e $TOXENV --notest
jython-nocov create: /home/travis/build/peterjc/biopython/.tox/jython-nocov
Traceback (most recent call last):
  File "/home/travis/jython/bin/tox", line 11, in <module>
    sys.exit(cmdline())
  File "/home/travis/jython/Lib/site-packages/tox/session.py", line 39, in main
    retcode = Session(config).runcommand()
  File "/home/travis/jython/Lib/site-packages/tox/session.py", line 39, in main
    retcode = Session(config).runcommand()
  File "/home/travis/jython/Lib/site-packages/tox/session.py", line 375, in runcommand
    return self.subcommand_test()
  File "/home/travis/jython/Lib/site-packages/tox/session.py", line 526, in subcommand_test
    if self.setupenv(venv):
  File "/home/travis/jython/Lib/site-packages/tox/session.py", line 434, in setupenv
    status = venv.update(action=action)
  File "/home/travis/jython/Lib/site-packages/tox/session.py", line 434, in setupenv
    status = venv.update(action=action)
  File "/home/travis/jython/Lib/site-packages/tox/venv.py", line 146, in update
    self.hook.tox_testenv_create(action=action, venv=self)
  File "/home/travis/jython/Lib/site-packages/tox/venv.py", line 146, in update
    self.hook.tox_testenv_create(action=action, venv=self)
  File "/home/travis/jython/Lib/site-packages/pluggy.py", line 724, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "/home/travis/jython/Lib/site-packages/pluggy.py", line 338, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/travis/jython/Lib/site-packages/pluggy.py", line 332, in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: \
  File "/home/travis/jython/Lib/site-packages/pluggy.py", line 596, in execute
    res = hook_impl.function(*args)
  File "/home/travis/jython/Lib/site-packages/tox/venv.py", line 392, in tox_testenv_create
    venv._pcall(args, venv=False, action=action, cwd=basepath)
  File "/home/travis/jython/Lib/site-packages/tox/venv.py", line 361, in _pcall
    return action.popen(args, cwd=cwd, env=env,
  File "/home/travis/jython/Lib/site-packages/tox/session.py", line 121, in popen
    fout = self._initlogpath(self.id)
  File "/home/travis/jython/Lib/site-packages/tox/session.py", line 107, in _initlogpath
    l = logdir.listdir("%s-*" % actionid)
  File "/home/travis/jython/Lib/site-packages/tox/session.py", line 107, in _initlogpath
    l = logdir.listdir("%s-*" % actionid)
  File "/home/travis/jython/Lib/site-packages/py/_path/local.py", line 388, in listdir
    names = py.error.checked_call(os.listdir, self.strpath)
  File "/home/travis/jython/Lib/site-packages/py/_error.py", line 84, in checked_call
    raise cls("%s%r" % (func.__name__, args))
py.error.ENOTDIR: [Not a directory]: listdir('/home/travis/build/peterjc/biopython/.tox/jython-nocov/log',)
The command "tox -c .travis-tox.ini -e $TOXENV --notest" failed and exited with 255 during .

[RonnyPfannschmidt]

tox/tox/session.py

Line 112 in 04bbe3b

except py.error.ENOENT:

it seems this no lo longer accurate :/ - its not clear to me what python version fixes it

[peterjc]

My earlier comment used tox-2.3.1 virtualenv-15.0.3 on TravisCI, https://travis-ci.org/peterjc/biopython/jobs/165876306

Trying to debug this by pre-creating the directory made no difference, now using tox-2.4.0 virtualenv-15.0.3 on TravisCI, e.g. https://travis-ci.org/peterjc/biopython/jobs/167100293 or https://travis-ci.org/peterjc/biopython/jobs/167103643

[RonnyPfannschmidt]

@peterjc i believe a extra except clause is needed for jython, would you like to add it? (see the code path i linked)

[peterjc]

I'd want to be able to reproduce this locally first - debugging things on TravisCI is pretty hard, although you can ask them for temporary access to a VM for this kind of thing.

[peterjc]

Persists with Jython 2.7, tox-2.6.0, virtualenv-15.1.0, py-1.4.32 as tested under TravisCI, e.g.

https://travis-ci.org/peterjc/biopython/jobs/201911071

I've looked at the code here and don't understand what an extra except clause would do under Jython. If os.listdir(...) is called on a non-existent folder, then Jython 2.7 returns a reasonable OSError which the code seems to handle already.

$ jython2.7 -c "import os; os.listdir('/does/not/exist')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
OSError: [Errno 2] No such file or directory: '/does/not/exist'

[LordGaav]

@RonnyPfannschmidt I have prepared #512 with the fix you proposed. It looks good in my local tests, and Travis is not complaining either.

[obestwalter]

fixed in #512

[peterjc]

Lovely. I don't know how/when you update the changelog, but this should be of interest to any Jython + tox users.

[obestwalter]

@peterjc I think we can cut a new release soonish.

[peterjc]

Confirming fixed in tox 2.8.0rc1, which I installed using pip install --pre tox under Jython 2.7.0, e.g.

https://travis-ci.org/peterjc/biopython/jobs/267995703

(The test run timed out, but nevertheless, tox installed and ran under Jython for me)

[obestwalter]

Thanks for testing and for your patience!