Thonny + py5 imported mode issue

[villares]

Dear friends, forgive me if my brain fog of late is blocking me to see anything obvious.
I'm not sure this is a @tabreturn's thonny-py5mode issue or a py5 issue.

I have installed Thonny 4.0-beta3-py38portable on Windows and installed thonny-py5mode (that means py5, jdk-17) YAY!
Note: There was a red warning message in the console about the plugin being unable to load, and I had to restart Thonny again after the plugin downloaded the JDK, then it disappeared.

Everything works fine in module mode. In imported mode, not so much, in a curious way. A minimal example, starting by something that works! Should I open an issue? Where?

# this works fine 
def settings():
    size(400, 400)
    
def setup():
    rect(10, 10, 20, 20)

# this doesn't work
def setup():
    size(400, 400)
    rect(10, 10, 20, 20)

When not using the PDE, size() can only be used inside settings().
Remove the size() method from setup(), and add the following:
public void settings() {
  size(400, 400);
}
py5 encountered an error in your code:File "PSurfaceNone.java", line 356, in processing.core.PSurfaceNone$AnimationThread.run

File "PSurfaceAWT.java", line 1440, in processing.awt.PSurfaceAWT$9.callDraw

File "PApplet.java", line 2142, in processing.core.PApplet.handleDraw

File "Sketch.java", line 178, in py5.core.Sketch.setup

File "jdk.proxy2.$Proxy7.java", line -1, in jdk.proxy2.$Proxy7.run_method

File "org.jpype.proxy.JPypeProxy.java", line -1, in org.jpype.proxy.JPypeProxy.invoke

File "org.jpype.proxy.JPypeProxy.java", line -2, in org.jpype.proxy.JPypeProxy.hostInvoke

File "PApplet.java", line 1807, in processing.core.PApplet.size

File "PApplet.java", line 848, in processing.core.PApplet.insideSettings

Exception: Java Exception

The above exception was the direct cause of the following exception:

File "C:\Users\sesc\mypy5.py", line 5, in setup
    3    def setup():
    4        size(400, 400)
--> 5        rect(10, 10, 20, 20)

java.lang.IllegalStateException: java.lang.IllegalStateException: size() cannot be used here, see https://processing.org/reference/size_.html

PS: This might be going for a while, in Thonny 3.3.7 too. I remember that during my pymunk course I had to separate settings/setup too!

PPS: I went to check the P2D/P3D issue on an Intel Mac (confirmed it didn't work) and found the same behaviour as reported here.

[hx2A]

I can't reproduce this, but I see a different bug.

When I put this in Thonny and run it:

def setup():
    size(400, 400)
    rect(10, 10, 20, 20)

The window appears for a split second and then exits. If I add this:

def draw():
    pass

Then the window appears. This should be easy to fix and I'll take care of it for the next release.

I don't know why you are getting that error. It should be totally fine putting the call to size() in setup(). Can you confirm you are using the latest version of py5?

Python 3.9.5 (bundled)
>>> import py5
>>> py5.__version__
'0.8.0a2'

[tabreturn]

Missed this because I only tested a static sketch working on Thonny 4 (and it's not an issue in Thonny 3.3.7). But there's a lot of conditional stuff in the plug-in to handle Thonny 4.

It seems things are one line off here --

File "C:\Users\sesc\mypy5.py", line 5, in setup
    3    def setup():
    4        size(400, 400)
--> 5        rect(10, 10, 20, 20)

java.lang.IllegalStateException: java.lang.IllegalStateException: size() cannot ...

The error in this example, however, is correctly identified (points to the line 2) --

py5 encountered an error in your code:
    1    
--> 2    rect(10)
    3    square(10, 10, 10)

TypeError: The parameter types (int) ...

I'll investigate.

UPDATE:

Okay, the line offset is normal. I tried this in Thonny 3 --

def setup():
    pass
def draw():
    size(400, 400)
    rect(10, 10, 20, 20)

... and got

File "/home/nuc/Desktop/a.py", line 5, in draw
    3    def draw():
    4        size(400, 400)
--> 5        rect(10, 10, 20, 20)

Of course, this works fine in Thonny 4 (and 3) --

def settings():
    size(400, 400)
def draw():
    rect(10, 10, 20, 20)

Not sure what's causing the original issue yet ...

[tabreturn]

Okay -- Thonny 4 seems to do things quite differently. It's problematic in many ways. For example:

size(400, 400)
square(10, 10, 10)
print(width)

results in --

py5 encountered an error in your code:
    1    
    2    square(10, 10, 10)
--> 3    print(width)

NameError: name 'width' is not defined

[hx2A]

It seems things are one line off here --

If you see error messages that are off by one line, note that it could be py5. A lot of work that goes into creating authentic error messages and I grappled with off by 1 issues for a while. It's tricky to split the user's setup() into two functions (which happen to be called _py5_faux_settings() and _py5_faux_setup()) in such a way that any errors in either of those functions is on the same line as the corresponding line in the user's setup().

[villares]

This is something I can't understand, why can't, in an ideal world, the size() function call all the settings() preparations itself, and then on returning control to the rest of the body of setup(), "settings is done already".

[tabreturn]

Okay. After banging my head against a wall for hours, I finally found a 'solution'. Change the line in the py5_tools/imported.py that reads --

_imported_mode = False

to:

_imported_mode = True

Obviously, this approach is no good. Any ideas how I might accomplish the same result without editing the file directly?

The thonny-py5mode plug-in works by calling the run_sketch.py file, and for whatever reason, the Thonny %Run works fine with this in Thonny 3 but not Thonny 4.

[hx2A]

The thonny-py5mode plug-in works by calling the run_sketch.py file, and for whatever reason, the Thonny %Run works fine with this in Thonny 3 but not Thonny 4.

You mean the run_sketch command line utility, which is implemented in run_sketch.py? The distinction is important here. I looked at the code and I see that imported mode is dependent on certain assumptions being met, that are met by the py5 jupyter kernel and (I believe) by the run_sketch utility. Somehow the assumptions are not being met in this situation.

Obviously, this approach is no good. Any ideas how I might accomplish the same result without editing the file directly?

To explain what's going on, let's first look at how the py5 kernel works. When the kernel starts it runs this startup code:

_DEFAULT_STARTUP = """
import py5_tools
py5_tools.set_imported_mode(True)
from py5 import *
from py5_tools import sketch_portal
"""

Observe that it calls py5_tools.set_imported_mode(True) before importing py5.

Over in py5/__init__.py, you see this:

_PY5_USE_IMPORTED_MODE = py5_tools.get_imported_mode()

The problem is that if that is set to False, you're stuck with that value. What's causing @villares 's symptom is how the split setup code is called:

    functions = _split_setup.transform(functions, caller_globals, caller_locals, println, mode='imported' if _PY5_USE_IMPORTED_MODE else 'module')

The split setup code needs to know if the code is module mode or imported mode because when it parses the user's code, it needs to know if it is looking for py5.size() or size().

Now let's look at imported.py.

By design, py5_tools can be imported and will not import py5. This is critical because importing py5 starts the jvm and the user might want to change the classpath or do other things before then.

In _run_code(), you see this:

        set_imported_mode(True)
        import py5

That should set the appropriate variable for imported mode and then import py5. Then _PY5_USE_IMPORTED_MODE should be set correctly. But what if somehow py5 has been imported already??? It could be that Thonny 4 is doing something different and importing py5. But the run_sketch command line utility should be running in its own process or subprocess. That part is confusing. That import py5 above should be the first time py5 is imported.

One solution here might be to ditch the _PY5_USE_IMPORTED_MODE variable and always call py5_tools.get_imported_mode() directly. But that might lead to the situation we have here, where the py5 is half running in imported mode and half running in module mode.

Oh, the reason why in @villares ;s example it is even able to find the size() method all is because of this:

        py5_ns = dict()
        py5_ns.update(py5.__dict__)

        exec(sketch_compiled, py5_ns)

That py5 namespace is how it gets access to the py5 methods without having to precede everything with py5.. That's also the half running in imported mode mentioned above. The other half running in module mode is because somehow what is supposed to be the first import of py5 is not actually the first time py5 is being imported.

[villares]

I think Thonny is importing py5 as it loads the thonny-py5mode plugin, for some reason! This explains the warning I saw before it downloaded the JDK the first time.

Would it be possible to pass some argument to the sketch runner telling it to override the imported mode flag?

[hx2A]

I will take a second look at this to see if perhaps there is a way I can make it more robust. I'm confused why this is a problem at all though; if Thonny is calling the run_sketch command line utility, that is running in its own process. I don't understand how it is even possible for Thonny to interfere in a way that would cause these problems.

[tabreturn]

Thanks @hx2A, @villares! Yes -- in the plug-in, I'm importing py5 in a way that's causing problems. It shouldn't be difficult to fix this 🥳

[tabreturn]

I've published a new thonny-py5mode release with the fix. This includes working code-completion for Thonny 4. Thanks again for the help.

[villares]

With setup only and setup/draw working!!!
(the settings split was failing with setup/draw too previously, just my minimal example was setup only)

And on Thonny 4-beta3 portable with Python 3.10 I could install jpype etc... (maybe necessary to install numpy before py5, not sure...)

I'm going to test this portable on a different Windows machine tomorrow 😄

[hx2A]

Great, I'm glad this is now working! Happy to help.

This is an opportunity to improve py5. Since py5 needs to be set to use imported mode before it is imported, it should throw an exception or something to prevent it from ever getting into a half-imported half-module mode state, like what happened here. This should be an easy fix and will prevent anyone else from having to debug this same issue.

[hx2A]

This is an opportunity to improve py5. Since py5 needs to be set to use imported mode before it is imported, it should throw an exception or something to prevent it from ever getting into a half-imported half-module mode state, like what happened here. This should be an easy fix and will prevent anyone else from having to debug this same issue.

This is issue is fixed with #90.