Run Cell does not find Jupyter in virtual environment

[mrvkino]

Environment data

• VS Code version: 1.29.1
• Extension version (available under the Extensions sidebar): 2018.11.0
• OS and version: MacOSX 10.14.1
• Python version (& distribution if applicable, e.g. Anaconda): homebrew python 3.7.1
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): pipenv
• Relevant/affected Python packages and their versions: jupyter notebooks

Actual behavior

Notification with the message: Running cells requires Jupyter notebooks to be installed.

I am able to start jupyter notebook in my virtual environment.

Expected behavior

Jupyter Notebook being started.

Steps to reproduce:

1. Create a jupyter python file
2. Click the Run Cell

Logs

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

Microsoft Python Language Server version 0.1.65.0
Initializing for /Users/.../.local/share/virtualenvs/jupyter_vscode-_1tZN8jX/bin/python

Linter 'flake8' is not installed. Please install it or select another linter".
Error: Module 'flake8' not installed.

Output from Console under the Developer Tools panel (toggle Developer Tools on under Help)

In the console of the Developper Tools I see that the plug-in is looking for Jupyter in a bunch of my pyenv Python but not my virtualenv.

[rchiodo]

Looks like pip venv isn't working. We don't find the activate script.

Could you possibly try the insiders build? I added some support for finding the activate script that isn't in the build you're using yet..

[mrvkino]

I Installed the insider build in a clean install of VSCode Insider v1.30.0 (to prevent issue with my day-to-day VSCode).

If I start VSCode after I started my venv it works.

pipenv shell

And then call:

code-insidesr .

The run cell works.

If I start VSCode outside of the venv, it detects the venv from the Pipfile, but is not able to find Jupyter.

[rchiodo]

Okay thanks. That helps. Looks like virtual pipenvs aren't supported yet then.

[mrvkino]

Ha! I thought since the plugin could find the Pipenv and start it automatically when starting a shell it would do the same for the Jupyter Notebook.

Thanks, really happy to be able to work on my notebook in VSCode. Hope the support for Pipenv comes soon.

[rchiodo]

Seems that pyenv-virtualenv also has the same problem. Likely we need to find the activate script for other virtual environments besides conda.

[kvdveer]

This is the workaround I'm using:

Run this as the first cell:

#%% Activate virtualenv

virtualenv_path = "./.venv"
exec(
    open("%s/bin/activate_this.py" % virtualenv_path).read(), 
    dict(__file__="%s/bin/activate_this.py" % virtualenv_path)
)

This will only work correctly if ipython and your venv use the same python binary.

[rchiodo]

Don's work on microsoft/vscode-python#3330 is a precursor to getting this working.

[AlJohri]

@rchiodo Hi, I just ran into this error as well via a pipenv virtual environmnent after clicking "Run Cell". It's not working even if I start code . within the virtualenv.

I'm using PIPENV_VENV_IN_PROJECT=true so the virtualenv is storied in the .venv folder. The python path is set in settings.json.

"python.pythonPath": ".venv/bin/python",

I've verified that jupyter is definitely installed within the virtualenv and up to date as per these instructions.

For some reason the debug console starts iterating through the other installed pythons versions? As you can see in the screenshot below, it starts with the correct python in the .venv folder and then iterates through my other pythons.

[rchiodo]

We need to 'activate' the virtual env prior to running jupyter. Right now we're just using the same python.

The other jupyters running is us searching for one that will work. We don't actually need the same environment to start jupyter, we just need to point it at the correct one (see jupyter kernel specs).

Although now that we know we need to activate it, this will likely change too, as I doubt Jupyter/iPython activates the virtual environment before using it.

[AlJohri]

@rchiodo I see. So selecting a python interpreter won't also activate the virtualenv? I feel as though this worked in the past when I was regular pipenv without setting PIPENV_VENV_IN_PROJECT=true.

Is there some special handling for pipenv that activate the virtualenv that is no longer being run now that I'm using the .venv/bin/python?

[rchiodo]

Selecting a python interpreter doesn't run the pipenv shell command. At least not yet. That's the change we're working on. It never did. We simply run the python within the same directory. (The one that pipenv install creates/copies)

However if you run vscode from within the pipenv shell, it should work. That's the new part that I'm not sure why it isn't working for you.

[rchiodo]

This should be fixed now in our insider's build (well in about half an hour after the build updates).

Please let me know if it isn't. Thanks.

[leonjonathan]

Thanks, @rchiodo. Insider's build worked for me (venv environment via pyenv-virtualenv).

[lonelygo]

The same problem, pyenv can work with vscode pyhton, expecting support pipenv.

[rchiodo]

@lonelygo are you saying the insider's build doesn't work with pipenv? I tested that so it should work.

[ollema]

So I'm either hitting this or a related bug, let me know if I should open a new issue.

I'm using pyenv with pyenv-virtualenvwrapper.

I have two different folders with two different virtual environments, purl and torch-test that both have jupyter installed. I have selected the correct interpreter for both folder.

When I use vscode and I'm running a cell in the torch-tests folder, I get this:

Notice how I get the

Jupyter kernel cannot be started from 'Python 3.6.7 64-bit'. Using closest match Python 3.6.5 64-bit ('torch-tests': venv) instead.

message - even though Python 3.6.7 64-bit is not my selected interpreter. Fortunately, for this environment, it looks like microsoft/vscode-python#3433 allows vscode to find the correct virtual environment anyways.

However, when I use vscode and I try to run a cell in the purl folder, it again doesn't pick up the correct interpreter:

but this time I'm not so lucky and it uses the torch-tests environment, which does not have the correct packages installed.

---

So I found this issue today so I downloaded the insiders build and tried again.

When I use vscode-insiders and I'm running a cell in the torch-tests folder, I get this:

Now it picks up the correct interpreter! Great!

Unfortunately, when I the tried in the purl folder again, I got this:

it uses the torch-tests environment by default, without showing any error messages.

This behavior persists after closing and starting vscode-insiders again, so it's not like it's related to what environment I first try to start the Python Interactive window from.

---

Let me know if you need any more information or if I should start a separate issue.

This is pretty frustrating since the only way I can start up jupyter with the correct venv is by uninstalling jupyter from all other venvs. :(

[rchiodo]

So I think there's a problem in the insider's build with it not showing any of the message popups. That would explain the different behavior. I'm actually looking into that problem on this issue here: #3608

With your 'purl' environment, can you try running these commands:

python -m jupyter notebook --version
python -m jupyter kernelspec --version

Those are what we use to determine whether or not jupyter is installed and we can use it.

Additionally if you go to Help | Toggle Developer Tools, pick the console, right click and save it to disk, that should tell us why your purl environment isn't being picked.

[ollema]

Okay, so here's what I did:

1. Started vscode-insiders by running code-insiders uni.code-workspace. Note that when it opens up, it remembers that I used to view a file in the purl folder, so that it opens up that file again. However, the interpreter in the bottom bar has the wrong virtualenv:
   
2. This can be "fixed" by viewing another file and viewing this file again - then the interpreter is updated in the bottom bar:
   
3. Bring up the integrated terminal: (Note how the source is called! Not sure why this happens, I start vscode-insiders from a fresh, deactivated terminal)
   
4. Run the requested commands:
   
5. Run a cell, the wrong virtualenv is used, run the requested commands again:
   

vscode.log

---

Hopefully this is detailed enough! There are some strange things going on here...

[ollema]

Also worth noting is that after these tests with code-insider, jupyter in code is totally busted. Can't even open ANY interactive window now, says that No module named jupyter and No module named ipykernel, like the posts above. but maybe this is expected? and I was just lucky before?

[rchiodo]

If you open your settings.json for your workspace, what does python.pythonpath say?

That's what we use for the Python Interactive window. It doesn't use the current file, it (for better or worse) always uses the python path for the entire workspace.

[rchiodo]

[rchiodo]

And this

Also worth noting is that after these tests with code-insider, jupyter in code is totally busted. Can't even open ANY interactive window now, says that No module named jupyter and No module named ipykernel, like the posts above. but maybe this is expected? and I was just lucky before?

is not expected unless the workspace python in use actually doesn't have jupyter installed.

[ollema]

I do have the paths set for the individual folders, but not for the workspace:

Are you saying that the workspace setting is used, but folder settings are ignored?

---

For the other issue, it seems like an connection can't be established to the jupyter server.

It does manage to find jupyter (although in the wrong environment, purl) and it starts a server. The connections is lost however.

Not sure why this is not working anymore... it worked a couple of days ago

Also, note that the notebook started is reachable in my browser:

it just seems like for whatever reason, vscode is not able to reach it

[rchiodo]

Yes the workspace setting is used to determine what python to start jupyter with. We don't currently use the folder setting or per file setting. I assume this means you have multiple folders open in your workspace?

The connection reset issue is new. That's entirely in the jupyterlab npm module we use to talk to them. Not sure why it's resetting.

I did just upgrade our jupyterlab bits though. That might help. The insider's build for that will be available in about 30 minutes.

[AlJohri]

@ollema This may be off-base, but I found myself having similar connection reset issues and pinning tornado to version 5 fixed it for me.

[ollema]

@rchiodo okay, I see! I have a workspace that contains all my university projects where each project has it's own folder. And then another workspace for all work stuff, also with folders. Is this not supported?

@AlJohri thanks, I will give that a try, together with trying the latest insider build.

[rchiodo]

@ollema using multiple folders inside a single workspace would behave the way you describe, so it's not really supported well. It always picks the 'global' python to use for the interactive window.

Would it be better if it just started with the python that matched the first cell's file that you ran? (Or maybe just the file that happens to be active when you start it).

Or maybe you'd like to pick which one to run if you have multiple available?

[ollema]

@rchiodo okay, got it!

Would it be better if it just started with the python that matched the first cell's file that you ran? (Or maybe just the file that happens to be active when you start it).
Not sure it I understand the difference - but I would personally prefer it the interactive window used the Python version associated with the file that it was started from, if that makes sense.

For example, if I start up vscode and I do "Run Cell" in a file in the purl folder, I would expect the interactive window to use the Python path set for purl.

---

Picking from multiple would also be nice! But maybe that's more work for you guys?

[rchiodo]

Yeah it's more work but it might make sense if we can't decide which one is the 'correct' one.

We have this issue here that is basically use the current file:
microsoft/vscode-python#3123

Can you upvote it? I'm going to add the further suggestion that if there are multiple possible for a folder, we should ask the user which one to use.

[ollema]

Upvoted! Thanks!