Support for Jupyter Notebook

[GreyElaina]

still missing now.

[DetachHead]

looks like the jupyter extension is separate from pylance and also open source: https://github.com/microsoft/vscode-jupyter

does that cover it? or is there some pylance-specific functionality that we need to add?

[GreyElaina]

looks like the jupyter extension is separate from pylance and also open source: microsoft/vscode-jupyter

does that cover it? or is there some pylance-specific functionality that we need to add?

When pylance is enabled, code inside notebooks have highlights & intelligence but basedpyright no. i think this part was supported by pylance (closed source XD).

[DetachHead]

ah ok, in that case this might be easier than i thought. will look into it when i have time

[Klaaktu]

Is this relevant? microsoft/vscode-python#23897
Looks like it's only in main and not in any of the releases.

EDIT: It's in pre-release and Jedi now works in each individual cell.

[DetachHead]

i think that's only relevant to language servers that use the python.languageServer setting, which basedpyright doesn't use. i tried that approach in #424 but it didn't work out for the reasons i mentioned there

[HeadCase]

Has anyone figured out a workaround to used basedpyright on python files and fallback to pylance on Jupyter notebooks? Perhaps a filetype association?

[johnpmayer]

I took a tour through some of the linked issues to try to piece together your attempts to address this. Is this the next step?

instead we should add support for typechecking notebooks in the CLI and add support for them in the LSP itself, rather than trying to do it microsoft/pylance-release#5207

Sounds like you have a sort of concrete idea the direction you want to take this? Want to add back the help-wanted label?

It does seem like a good independent feature (to answer this question you posed)

(ie. is type checking notebooks in CI a thing?)

Yes, I would! My use case would be tests, authored as notebooks (because that expands the author pool), for a collection of python packages in a mono repo.

[johnpmayer]

I see, basedpyright currently doesn't know how to interpret the JSON

Does using nbQA feel like a nice path forward? Looks like that's how ruff does it.

https://nbqa.readthedocs.io/en/latest/tutorial.html#writing-your-own-tool

edit: this seems to work out of the box:

uv run nbqa basedpyright <mynotebook>.ipynb

[DetachHead]

Sounds like you have a sort of concrete idea the direction you want to take this? Want to add back the help-wanted label?

i've just deleted the "help wanted" label from this project, since i find it quite misleading. it implies that issues without this label are not open for contributors to work on, which isn't true. i welcome contributors on any currently open issue regardless of whether it has the label.

edit: this seems to work out of the box:

uv run nbqa basedpyright <mynotebook>.ipynb

are you sure? it's probably just treating the json conents of the file as python and type checking that instead. for example i tried it on the following notebook:

and received these nonsense errors:

c:\Users\user\project\asdf.ipynb
  c:\Users\user\project\asdf.ipynb:1:1 - warning: Expression value is unused (reportUnusedExpression)
  c:\Users\user\project\asdf.ipynb:5:23 - error: "null" is not defined (reportUndefinedVariable)     
1 error, 1 warning, 0 notes

because it sees the json contents of the .ipynb file and thinks it's a python dict:

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "outputs": [],
   "source": [
    "foo: int = \"\""
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

ideally it should fail on filetypes it doesn't recognize. see #90

[johnpmayer]

are you sure?

Yes!

You're correct, running basedpyright directly on notebook files doesn't work, because the JSON is not valid python.

However, this specific tool - nbQA - appears to be transforming the input stream by concatenating the code cells into a single python "file". And so, I do get type checking output, though the "line numbers" are of course synthetic. Note again the precise command to run:

uv run nbqa basedpyright <mynotebook>.ipynb

Presumably this works because you've setup python -m basedpyright to work (nice!).

It also seems to support formatting tools, so they must have some kind of way (not sure how sophisticated) of taking the transformed python "file" and mapping that back into the code snippets for each notebook cell.

This doesn't get us all the way on its own (though it does, I think, help to run basedpyright as a CLI on notebook files). Conceptually, the LSP now would need to take the notebook data, run it through something like nbQA's stitching routine, then pass that to basedpyright, extract the messages and map that back to the "relativized" line numbers for the code cells (I bet nbQA has some utilities for this).

[johnpmayer]

What I'm not sure is whether, conventionally, IDEs (and vscode is the one I care about) that want to run an LSP on code from a notebook do this handling themselves (the stitching and un-stitching) or pass the whole JSON.

[DetachHead]

oops, i completely missed that you were running it with this separate nbqa tool

i haven't looked into it much but i don't think it supports language servers. i'm thinking we probably need to just add support for it natively instead of relying on external tools. some of pylance's code for notebooks seems to have made its way back to pyright, some of it we may be able to build on top of

[johnpmayer]

Is it perhaps just the vscode extension config?

Spit balling here:

https://github.com/DetachHead/basedpyright/blob/main/packages/vscode-pyright/src/extension.ts#L182

It's setup with

        documentSelector: [
            { scheme: 'file', language: 'python' },
            { scheme: 'untitled', language: 'python' },
        ],

DocumentSelector type comes from https://github.com/microsoft/vscode-languageserver-node/blob/main/protocol/src/common/protocol.ts#L390

which can be a DocumentFilter - https://github.com/microsoft/vscode-languageserver-node/blob/main/protocol/src/common/protocol.ts#L381

and I think it's only doing TextDocumentFilter but not the (apparently also supported) NotebookCellTextDocumentFilter - https://github.com/microsoft/vscode-languageserver-node/blob/main/protocol/src/common/protocol.ts#L342

So I wonder what will happen if we just add { notebook: '*', language: 'python' }

That might not be able to handle the stitching of code across cells, so you'd probably wind up with lots of undeclared symbols. But maybe some extra metadata is passed along.

[johnpmayer]

Ha... yeah that actually works!

As expected, cross-cell references do not work. But even with treating each cell as an independent snippet, I get squigglies, and that's not nothing!

Worth opening an PR for this narrow thing?

[DetachHead]

sure go ahead! let me know if you get stuck on anything and i will try to help out

[BartoszJanuszNA]

That would be a huge step, I love basedpyright LSP, it's so much better that Pylance, but not having LSP in notebooks is a deal breaker for me unfortunately :-(

Is below possible?

Has anyone figured out a workaround to used basedpyright on python files and fallback to pylance on Jupyter notebooks? Perhaps a filetype association?

[s-stefanov]

Same issue here. Did you managed to get it working?

[DetachHead]

turns out most of the code required to support notebooks was already in pyright. i got it working today in #1035

i just need to add support for them in the CLI, which i hope to get done this week. i know this is the most upvoted issue by far so i appreciate everybody's patience

[edwardbickerton]

Is cross-cell referencing supposed to work, or is it planned for the future?

[DetachHead]

that should be working. do you happen to have an older version of basedpyright installed in your environment than the extension version? you'll have to update both for it to work properly

[edwardbickerton]

My bad, working beautifully in jupyter notebooks now