PKCS#11 and Python 3.12

[pikusdlp]

Hi mates,

Thanks for all your hard work in this library, very useful.

My application needs to sign PDF using a Crypto Card, with PKCS#11 protocol. The problems is I need to upgrade to Python 3.12 but I cannot compile PKCS11 library. In the same server with Python 3.8 I have no issue. Seems like something is deprecated and that is the reason why it fails while compiling.

As this library depends directly from pyHanko, I am not sure what to do.

I'm using Windows Server, and this is the error message.

`
Building wheels for collected packages: python-pkcs11
Building wheel for python-pkcs11 (pyproject.toml) ... error
error: subprocess-exited-with-error

× Building wheel for python-pkcs11 (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [375 lines of output]
WARNING setuptools_scm.pyproject_reading toml section missing 'pyproject.toml does not contain a tool.setuptools_scm section'
Traceback (most recent call last):
File "C:\Users\pr-3P-cbulweeyjq\AppData\Local\Temp\6\pip-build-env-wevp01kp\normal\Lib\site-packages\setuptools_scm_integration\pyproject_reading.py", line 36, in read_pyproject
section = defn.get("tool", {})[tool_name]
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
KeyError: 'setuptools_scm'
running bdist_wheel
running build
running build_py
creating build
creating build\lib.win-amd64-cpython-312
creating build\lib.win-amd64-cpython-312\pkcs11
copying pkcs11\constants.py -> build\lib.win-amd64-cpython-312\pkcs11
copying pkcs11\defaults.py -> build\lib.win-amd64-cpython-312\pkcs11
copying pkcs11\exceptions.py -> build\lib.win-amd64-cpython-312\pkcs11
copying pkcs11\mechanisms.py -> build\lib.win-amd64-cpython-312\pkcs11
copying pkcs11\types.py -> build\lib.win-amd64-cpython-312\pkcs11
copying pkcs11_init_.py -> build\lib.win-amd64-cpython-312\pkcs11
creating build\lib.win-amd64-cpython-312\pkcs11\util
copying pkcs11\util\dh.py -> build\lib.win-amd64-cpython-312\pkcs11\util
copying pkcs11\util\dsa.py -> build\lib.win-amd64-cpython-312\pkcs11\util
copying pkcs11\util\ec.py -> build\lib.win-amd64-cpython-312\pkcs11\util
copying pkcs11\util\rsa.py -> build\lib.win-amd64-cpython-312\pkcs11\util
copying pkcs11\util\x509.py -> build\lib.win-amd64-cpython-312\pkcs11\util
copying pkcs11\util_init_.py -> build\lib.win-amd64-cpython-312\pkcs11\util
running egg_info
writing python_pkcs11.egg-info\PKG-INFO
writing dependency_links to python_pkcs11.egg-info\dependency_links.txt
writing requirements to python_pkcs11.egg-info\requires.txt
writing top-level names to python_pkcs11.egg-info\top_level.txt
ERROR setuptools_scm._file_finders.git listing git files failed - pretending there aren't any
reading manifest file 'python_pkcs11.egg-info\SOURCES.txt'
writing manifest file 'python_pkcs11.egg-info\SOURCES.txt'
copying pkcs11_errors.pyx -> build\lib.win-amd64-cpython-312\pkcs11
copying pkcs11_mswin.pxd -> build\lib.win-amd64-cpython-312\pkcs11
copying pkcs11_pkcs11.pyx -> build\lib.win-amd64-cpython-312\pkcs11
copying pkcs11_pkcs11_defn.pxd -> build\lib.win-amd64-cpython-312\pkcs11
copying pkcs11_utils.pyx -> build\lib.win-amd64-cpython-312\pkcs11
running build_ext
warning: pkcs11_pkcs11.pyx:17:0: The 'IF' statement is deprecated and will be removed in a future Cython version. Consider using runtime conditions or C macros instead. See cython/cython#4310
warning: pkcs11_errors.pyx:85:44: The keyword 'nogil' should appear at the end of the function signature line. Placing it before 'except' or 'noexcept' will be disallowed in a future version of Cython.
warning: pkcs11_pkcs11.pyx:1366:4: The 'IF' statement is deprecated and will be removed in a future Cython version. Consider using runtime conditions or C macros instead. See cython/cython#4310
warning: pkcs11_pkcs11.pyx:1390:8: The 'IF' statement is deprecated and will be removed in a future Cython version. Consider using runtime conditions or C macros instead. See cython/cython#4310
warning: pkcs11_pkcs11.pyx:1421:8: The 'IF' statement is deprecated and will be removed in a future Cython version. Consider using runtime conditions or C macros instead. See cython/cython#4310
performance hint: pkcs11_errors.pyx:85:6: Exception check on 'assertRV' will always require the GIL to be acquired.
Possible solutions:
1. Declare 'assertRV' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
2. Use an 'int' return type on 'assertRV' to allow an error code to be returned.
warning: pkcs11_pkcs11.pyx:1391:46: Py_UNICODE* has been removed in Python 3.12. This conversion to a Py_UNICODE* will no longer compile in the latest Python versions. Use Python C API functions like PyUnicode_AsWideCharString if you need to obtain a wchar_t* on Windows (and free the string manually after use).
performance hint: pkcs11_pkcs11.pyx:196:20: Exception check after calling 'assertRV' will always require the GIL to be acquired.
Possible solutions:
1. Declare 'assertRV' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
2. Use an 'int' return type on 'assertRV' to allow an error code to be returned.
performance hint: pkcs11_pkcs11.pyx:211:20: Exception check after calling 'assertRV' will always require the GIL to be acquired.
Compiling pkcs11/_pkcs11.pyx because it changed.
[1/1] Cythonizing pkcs11/_pkcs11.pyx
building 'pkcs11._pkcs11' extension
creating build\temp.win-amd64-cpython-312
creating build\temp.win-amd64-cpython-312\Release
creating build\temp.win-amd64-cpython-312\Release\pkcs11
"C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.40.33807\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD "-IC:\Program Files\Python312\include" "-IC:\Program Files\Python312\Include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.40.33807\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\VS\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.20348.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.20348.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.20348.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.20348.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.20348.0\cppwinrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" /Tcpkcs11/_pkcs11.c /Fobuild\temp.win-amd64-cpython-312\Release\pkcs11/_pkcs11.obj
_pkcs11.c
pkcs11/_pkcs11.c(20632): warning C4013: 'PyUnicode_FromUnicode' undefined; assuming extern returning int
pkcs11/_pkcs11.c(20632): warning C4047: '=': 'PyObject *' differs in levels of indirection from 'int'
pkcs11/_pkcs11.c(25705): warning C4244: '=': conversion from 'Py_ssize_t' to 'CK_ULONG', possible loss of data
pkcs11/_pkcs11.c(32118): warning C4244: '=': conversion from 'Py_ssize_t' to 'CK_ULONG', possible loss of data
pkcs11/_pkcs11.c(32727): warning C4244: '=': conversion from 'Py_ssize_t' to 'CK_ULONG', possible loss of data
pkcs11/_pkcs11.c(33602): warning C4244: '=': conversion from 'Py_ssize_t' to 'CK_ULONG', possible loss of data
pkcs11/_pkcs11.c(37672): warning C4244: '=': conversion from 'Py_ssize_t' to 'CK_ULONG', possible loss of data
pkcs11/_pkcs11.c(38659): warning C4244: '=': conversion from 'Py_ssize_t' to 'CK_ULONG', possible loss of data
pkcs11/_pkcs11.c(39384): warning C4244: '=': conversion from 'Py_ssize_t' to 'CK_ULONG', possible loss of data
pkcs11/_pkcs11.c(40371): warning C4244: '=': conversion from 'Py_ssize_t' to 'CK_ULONG', possible loss of data
pkcs11/_pkcs11.c(41095): warning C4244: '=': conversion from 'Py_ssize_t' to 'CK_ULONG', possible loss of data
pkcs11/_pkcs11.c(41955): warning C4244: '=': conversion from 'Py_ssize_t' to 'CK_ULONG', possible loss of data
pkcs11/_pkcs11.c(42627): warning C4244: '=': conversion from 'Py_ssize_t' to 'CK_ULONG', possible loss of data
pkcs11/_pkcs11.c(42647): warning C4244: '=': conversion from 'Py_ssize_t' to 'CK_ULONG', possible loss of data
pkcs11/_pkcs11.c(43179): warning C4244: '=': conversion from 'Py_ssize_t' to 'CK_ULONG', possible loss of data
pkcs11/_pkcs11.c(43393): warning C4244: '=': conversion from 'Py_ssize_t' to 'CK_ULONG', possible loss of data
pkcs11/_pkcs11.c(45145): warning C4244: '=': conversion from 'Py_ssize_t' to 'CK_ULONG', possible loss of data
pkcs11/_pkcs11.c(46469): warning C4013: 'PyUnicode_AsUnicode' undefined; assuming extern returning int
pkcs11/_pkcs11.c(46469): warning C4047: '=': 'LPCWSTR' differs in levels of indirection from 'int'
pkcs11/_pkcs11.c(61060): warning C4244: '=': conversion from 'Py_ssize_t' to 'long', possible loss of data
pkcs11/_pkcs11.c(61243): warning C4244: '=': conversion from 'Py_ssize_t' to 'long', possible loss of data
"C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.40.33807\bin\HostX86\x64\link.exe" /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO "/LIBPATH:C:\Program Files\Python312\libs" "/LIBPATH:C:\Program Files\Python312" "/LIBPATH:C:\Program Files\Python312\PCbuild\amd64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.40.33807\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.20348.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.20348.0\um\x64" user32.lib /EXPORT:PyInit__pkcs11 build\temp.win-amd64-cpython-312\Release\pkcs11/_pkcs11.obj /OUT:build\lib.win-amd64-cpython-312\pkcs11_pkcs11.cp312-win_amd64.pyd /IMPLIB:build\temp.win-amd64-cpython-312\Release\pkcs11_pkcs11.cp312-win_amd64.lib
Creating library build\temp.win-amd64-cpython-312\Release\pkcs11_pkcs11.cp312-win_amd64.lib and object build\temp.win-amd64-cpython-312\Release\pkcs11_pkcs11.cp312-win_amd64.exp
_pkcs11.obj : error LNK2001: unresolved external symbol PyUnicode_AsUnicode
_pkcs11.obj : error LNK2001: unresolved external symbol PyUnicode_FromUnicode
build\lib.win-amd64-cpython-312\pkcs11_pkcs11.cp312-win_amd64.pyd : fatal error LNK1120: 2 unresolved externals
error: command 'C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.40.33807\bin\HostX86\x64\link.exe' failed with exit code 1120
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for python-pkcs11
Failed to build python-pkcs11
ERROR: Could not build wheels for python-pkcs11, which is required to install pyproject.toml-based projects
`

[MatthiasValvekens]

Hi, I haven't built any Cython packages on Windows in almost a decade, so I can't help you troubleshoot this (heck, it's been years since I last even touched a Windows machine for development work). I guess if you ask very nicely to cut a release with prebuilt wheels for Python 3.12 on Windows, you might be able to get help here: https://github.com/pyauth/python-pkcs11.

Failing that, there's not much I can do other than to suggest that you do your dev work in a Linux VM, under WSL, or running Docker in WSL, where you almost certainly won't face this type of issue ;).