Weirdness in hosted IronPython3

[mickp]

I've been looking at updating a project that uses hosted IronPython2, but I hit a problem in IronPython3 when trying to import importlib.util: namedtuple tries to build and then exec a class declaration, and fails.

It looks like the underlying issue is that my hosted interpreter is unable to exec multiline expressions.

>>> exec('a=1;b=2')  # This is fine.
>>> exec('a=1\nb=2')  # This is not.
Traceback (most recent call last):
IndexError: Index was outside the bounds of the array.
>>> 

That same expression is fine in IronPython3 Console ... so what's the problem with my hosted interpreter?

Some observations:

• The host app uses a stream to communicate with the interpreter. If I change the encoding for that stream from UTF8 to ASCII, the exec expressions above work correctly, but I still see the same problem when I try to import importlib.util.
• In IronPython Console, __builtins__ is a module. In my hosted interpreter, __builtins__ is a dict. The __builtins__ variable appears on the scope during the call to SourceUnit.Execute in Microsoft.Scripting.Hosting.ScriptSource.Execute the first time ScriptEngine.Execute is called.
• I am using the IronPython 3.4.2 and IronPython.StdLib 3.4.2 nuget packages, and I've confirmed that the contents of the Lib folder used by my hosted interpreter are identical to those distributed with IronPython Console (except for a few pip-related packages that are only present in the latter).

Here's the traceback from the importlib.util problem, which I see however I import util, and with both ASCII and UTF8 stream encoding:

from importlib import util
Traceback (most recent call last):
  Line 1565, in load_module
  Line 597, in _load_module_shim
  Line 1208, in load
  Line 1188, in _load_unlocked
  Line 1117, in _exec
  Line 1459, in exec_module
  File "...\bin\Debug\net8.0-windows10.0.17763.0\win-x64\Lib\importlib\util.py", line 12, in <module>
  Line 1565, in load_module
  Line 597, in _load_module_shim
  Line 1208, in load
  Line 1188, in _load_unlocked
  Line 1117, in _exec
  Line 1459, in exec_module
  File "...\bin\Debug\net8.0-windows10.0.17763.0\win-x64\Lib\contextlib.py", line 5, in <module>
  Line 540, in _check_name_wrapper
  Line 1565, in load_module
  Line 597, in _load_module_shim
  Line 1208, in load
  Line 1188, in _load_unlocked
  Line 1117, in _exec
  Line 1459, in exec_module
  File "...\bin\Debug\net8.0-windows10.0.17763.0\win-x64\Lib\functools.py", line 338, in <module>
  File "...\bin\Debug\net8.0-windows10.0.17763.0\win-x64\Lib\collections\__init__.py", line 364, in namedtuple
  Line 2, in <module>
  Line 540, in _check_name_wrapper
  Line 322, in _call_with_frames_removed
  Line 540, in _check_name_wrapper
  Line 322, in _call_with_frames_removed
  Line 322, in _call_with_frames_removed
IndexError: Index was outside the bounds of the array.

[BCSharp]

The host app uses a stream to communicate with the interpreter. If I change the encoding for that stream from UTF8 to ASCII, the exec expressions above work correctly,

Just a thought: have you checked that the UTF-8 encoding you use is configured to work without the preamble? A BOM character can throw things off, although I would expect in such case IndentationError or SyntaxError, not IndexError.

[mickp]

Thanks for the suggestion. We're using System.Text.Encoding.UTF8 everywhere, so I think BOM use/awareness should be consistent.

[BCSharp]

System.Text.Encoding.UTF8 has the preamble set to BOM (which in UTF-8 is 3 bytes). To use the UTF-8 encoding without the preamble you would need to use new System.Text.UTF8Encoding(encoderShouldEmitUTF8Identifier: false).

[mickp]

System.Text.Encoding.UTF8 has the preamble set to BOM (which in UTF-8 is 3 bytes). To use the UTF-8 encoding without the preamble you would need to use new System.Text.UTF8Encoding(encoderShouldEmitUTF8Identifier: false).

I see the same issue when using encoderShouldEmitUTF8Identifier: false.

[slozier]

Weird one indeed. How are you initializing your engine and streaming the data? Do you get the same issue if you try with 3.4.0 or 3.4.1?

[mickp]

The engine and encoding are set up as follows:

                var runtime = Python.CreateRuntime(options);
                runtime.IO.SetInput(stream, CharacterEncoding);
                runtime.IO.SetOutput(stream, CharacterEncoding);
                runtime.IO.SetErrorOutput(stream, CharacterEncoding);
                pyEngine = Python.GetEngine(runtime);

I think the stream is used exclusively for output.

stream is a System.IO.Stream. Interactive input is grabbed from a WPF TextBox when the enter key is pressed , trimmed, executed with PyEngine.CreateScriptSourceFromFile(text).Compile().Execute(scope). That works fine in 2.7 for both ASCII and UTF8 encoding. In 3.4, it only works with ASCII. (I noticed that we're calling CreateScriptSourceFromFile from the class, but I see the same result when I call it from the pyEngine instance).

I see the same issue with 3.4.0, 3.4.1 and 3.4.2.

[Muny]

Out of curiosity, what happens if you try using UTF-16 instead?

[mickp]

Out of curiosity, what happens if you try using UTF-16 instead?

Same error.

[mickp]

I found it! The host app does some custom trace handling. When a line of code contains eval, the line number from a frame inside the eval is used to index the last-stored command, which contains the eval. It looks like someone added a hack to work around this by splitting the source on \n. I guess that works with ASCII encoding, but fails with Unicode because it sees \\n.

Thanks for the suggestions!