python 3.13 support

[Xmader]

In order to make PythonMonkey support the upcoming (October 1, 2024) Python 3.13 release, the following changes/actions need to be made in the codebase:

• Python 3.13 added a PyLong_AsNativeBytes API, but also changed the function signature of the _PyLong_AsByteArray API
  ( C API: Consider adding public PyLong_AsByteArray() and PyLong_FromByteArray() functions python/cpython#111140)

• _Py_IsFinalizing becomes a stable API in Python 3.13, and is renamed to Py_IsFinalizing

• Python 3.13 moved several undocumented APIs to private and internal-only, i.e. removed from public use.

  • _PyDictView* APIs are removed.
  • _PyArg_CheckPositional API is removed.
  • _PyErr_SetKeyError API is removed. gh-106320: Remove private _PyErr_SetKeyError() python/cpython#108607
  • _PyThreadState_GetDict(tstate) API gets removed.
    • However, there would be no perfect replacement fix.
    • It's currently impossible in Python 3.13 to get the thread dictionary on a specific thread other than the current thread. (the public PyThreadState_GetDict() API can only get from the current thread)

---

Closes #441

[zollqir]

I noticed that only on python 3.13, the eval-test.simple test says

"Exception ignored in: <_io.BufferedReader name='/home/runner/work/PythonMonkey/PythonMonkey/tests/js/resources/eval-test.js'>"

I looked into it briefly, looks to be something to do with the file being closed more than once somehow.

[zollqir]

Overall looks great, just some minor changes

[zollqir]

Suggested change

	# Python 3.13 dramatically changed how the namespace in `exec`/`eval` works
	# See https://docs.python.org/3.13/whatsnew/3.13.html#defined-mutation-semantics-for-locals
	_locals = {} # keep the local variables inside `eval`/`exec` to a dict
	globalThis.python.eval = lambda x: eval(x, None, _locals)
	globalThis.python.exec = lambda x: exec(x, None, _locals)
	globalThis.python.getenv = os.getenv
	# Python 3.13 dramatically changed how the namespace in `exec`/`eval` works
	# See https://docs.python.org/3.13/whatsnew/3.13.html#defined-mutation-semantics-for-locals
	globalThis.python.eval = lambda x: eval(x, None, sys._getframe(1).f_locals)
	globalThis.python.exec = lambda x: exec(x, None, sys._getframe(1).f_locals)

Without the above change, pythonmonkey would lose the ability to mutate parent scope variables with pm.eval. On main the following assert passes, but it currently fails on this branch. We should add a test to ensure this behaviour as well.

import pythonmonkey as pm

a = 42

pm.eval("python.exec('a = 43')")
assert a == 43

While on the subject, the following currently fails on main, but probably shouldn't. The above change fixes that as well.

import pythonmonkey as pm
def foo():
  b = 42
  pm.eval("python.exec('b = 43')")
  assert b == 43
foo()

[Xmader]

https://docs.python.org/3/library/functions.html#exec

In all cases, if the optional parts are omitted, the code is executed in the current scope. If only globals is provided, it must be a dictionary (and not a subclass of dictionary), which will be used for both the global and the local variables. If globals and locals are given, they are used for the global and local variables, respectively. If provided, locals can be any mapping object. Remember that at the module level, globals and locals are the same dictionary.

https://github.com/python/cpython/blob/v3.9.20/Python/bltinmodule.c#L979-L985

if (globals == Py_None) {
        globals = PyEval_GetGlobals();
        if (locals == Py_None) {
            locals = PyEval_GetLocals();
            if (locals == NULL)
                return NULL;
        }

[Xmader]

On main

In your first example, a is defined in the global scope, exec can catch it and re-assign it without any problem.

However, in your second example, b is defined in the local scope of the foo() function. Since the exec is executed inside of require.py, it can only see the variables defined in require.py as its "locals".

[wesgarland]

"Without the above change, pythonmonkey would lose the ability to mutate parent scope variables with pm.eval. "

Question: is there a problem with losing that ability?

From a clean design POV, it's probably best to not let the languages mutate each other's scopes, except when those scoping objects are deliberately made available to this, i.e. in the case of globalThis.

The fact that Python modules have a scoping semantic not understand in JS-land underscores this from my POV.

If we made python.eval work more like indirect eval in JS, it would probably be a saner coding environment for our users, even though it might be a breaking change. And since the only code we really care about at this early stage is our test cases and BiFrost2, we control everything, so we can fix API and code in lock-step if necessary.

[wesgarland]

i.e. to clarify: if we can change PythonMonkey to work with 3.13, and make it work the same in less than 3.13, I think that's probably okay. If a JS developer REALLY needs to mutate a Python scope, that dev can just pass in the scoping object, right?

moved to other issue

[wesgarland]

LGTM