Module present in the snapshot cannot be found

[swanux]

What version of pkg are you using?

6.1.0

What version of Node.js are you using?

20

What operating system are you using?

Fedora 41

What CPU architecture are you using?

x86_64

What Node versions, OSs and CPU architectures are you building for?

node20-linux-x64, node20-windows-x64

Describe the Bug

Calling playwright through the binary (node_modules/.bin), the same way as it used to work with the original vercel/pkg, it throws the following error:

Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/snapshot/autotest/node_modules/playwright/lib/transform/esmLoader.js' imported from /snapshot/autotest/node_modules/playwright/lib/common/esmLoaderHost.js
    at finalizeResolution (node:internal/modules/esm/resolve:265:11)
    at moduleResolve (node:internal/modules/esm/resolve:933:10)
    at defaultResolve (node:internal/modules/esm/resolve:1169:11)
    at nextResolve (node:internal/modules/esm/hooks:868:28)
    at Hooks.resolve (node:internal/modules/esm/hooks:306:30)
    at ModuleLoader.resolve (node:internal/modules/esm/loader:503:35)
    at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:239:38)
    at ModuleLoader.import (node:internal/modules/esm/loader:472:34)
    at Hooks.register (node:internal/modules/esm/hooks:167:34)
    at MessagePort.handleMessage (node:internal/modules/esm/worker:196:24) {
  code: 'ERR_MODULE_NOT_FOUND',
  url: 'file:///snapshot/autotest/node_modules/playwright/lib/transform/esmLoader.js'
}

While the following code (from within the compiled program):
console.log(fs.readdirSync("/snapshot/autotest/node_modules/playwright/lib/transform"));

Gives the following result:

[
  'babelBundle.js',
  'babelBundleImpl.js',
  'compilationCache.js',
  'esmLoader.js',
  'esmUtils.js',
  'portTransport.js',
  'transform.js'
]

Indicating that the required file is clearly present in the snapshot.

Expected Behavior

The module import should be successful.

To Reproduce

Have the following config for pkg in your package.json:

"pkg": {
    "assets": [
      "node_modules/@playwright",
      "node_modules/playwright",
      "node_modules/playwright-core",
      "node_modules/.bin/playwright"
    ],
    "targets": [
      "node20-linux-x64",
      "node20-windows-x64"
    ]

Try to call playwright test from within the compiled code, according to this:

const playwright_bin = path.join(__dirname, "../node_modules/.bin/playwright");
...
const node = spawn("node", [playwright_bin, "test"]);

[robertsLando]

Try to check https://github.com/yao-pkg/pkg#troubleshooting and then if nothing works go to last advanced section to create a debug binary and lunch it to see if you can gain some more informations

[swanux]

As per the troubleshooting section, the first 2 seem to be relevant.

1. If I understand correctly, this applies to the case when I want to call the packaged application itself (?) - if so, then it shouldn't be the problem.
2. This one mentions binaries cannot be executed from the virtual filesystem. Does it also apply to binaries for npx (namely node_modules/.bin/playwright)? If so, is this a side-effect of this work (as it hasn't been a problem with the original)? If that is the case, then I guess I'll just supply the relevant binary and its' dependencies separately, but before that, I'd like to clarify (especially, because the binary can be called, only the imports fail).

For the debug section: I verified that the snapshot contains the files that the code is trying to import, at the same path which is mentioned in the error message - thus making it all the more interesting.

UPDATE:
Calling the binary from the outside is even less feasible:

node:internal/modules/cjs/loader:1228
  throw err;
  ^

Error: Cannot find module 'C:\Users\kdaniel\Desktop\csob3\node_modules\.bin\playwright'
    at Module._resolveFilename (node:internal/modules/cjs/loader:1225:15)
    at Function._resolveFilename (pkg/prelude/bootstrap.js:1959:46)
    at Module._load (node:internal/modules/cjs/loader:1051:27)
    at Function.runMain (pkg/prelude/bootstrap.js:1987:12)
    at node:internal/main/run_main_module:28:49 {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}

Node.js v20.18.0

[robertsLando]

@swanux please try to package using --debug and run the exe using DEBUG_PKG=2 ./app. Paste here the result

[swanux]

@robertsLando See attachment: logs.txt

(I ran the linux binary, but they produce the same error)

[robertsLando]

@swanux The issue is in playwright/lib/common/esmLoaderHost.js, it loads esmLoader in a way that is not supported by pkg, try to manually patch this file before packaging and it will work, I sincerly have no clue what this does:

function registerESMLoader() {
  const {
    port1,
    port2
  } = new MessageChannel();
  // register will wait until the loader is initialized.
  require('node:module').register(_url.default.pathToFileURL(require.resolve('../transform/esmLoader')), {
    parentURL: _url.default.pathToFileURL(__filename),
    data: {
      port: port2
    },
    transferList: [port2]
  });
  loaderChannel = createPortTransport(port1);
  exports.esmLoaderRegistered = esmLoaderRegistered = true;
}

[swanux]

@robertsLando yep, I also looked at this part - but I dismissed it as the cause of the issue, because it works with the original vercel pkg. First I thought it's because of the newer node20 I used for build (because there's an if in the code that executes differently when running on node>=20) but I got the same error with node18 as well.

So why does it work with the old pkg, and not the new one, even using the same node versions?

[robertsLando]

Hummm I have no clue maybe in the old pkg version you were using node 16 or a different playwright version? I have a feel this may work with node 16 target as starting from node 18 there have been some changes to esm

[swanux]

No, that's the weird part. I still use the old pkg with node18 and the same playwright version in production (as I am just experimenting with migrating away from the deprecated version). And it works without an issue.

Maybe one of the changes / patches is somehow affecting it?

[robertsLando]

@swanux starting from I don't remember which minor version nodejs 18 started giving the same problem related to esm modules so I think the reason is that, could you try with 16? If it works with 16 the reason is this

[swanux]

@robertsLando I tried it now with node16 - same results...

[robertsLando]

No clue so I'm sorry, I'm out of ideas now... My feel is that the way playwright uses to load that module is strange and it's not well digested by pkg for some reason