Pip install is failing to build due to av/logging.pyx error

[nicholas-presien]

Overview

I'm currently using the stable version of PyAV (v9.0.2) in a docker container, and building in a docker image for a while with no issue. Recently, my docker image failed due to build errors in pyav when installing pyav using pip.

I'm building on a TX2NX dev kit, running Jetpack 4.6 with an arm64 compiled version of FFMPEG 4.3.2(https://launchpad.net/~jonathonf/+archive/ubuntu/ffmpeg-4), on python v3.6.9

Expected behavior

It should build without error

Actual behavior

It's now building with error, when nothing in the environment has changed due to it being in a docker container.

Build report:

python3 setup.py build_ext --inplace                                                                                                                                                                           
Compiling av/logging.pyx because it changed.                                                                                                                                                                                                  
[1/1] Cythonizing av/logging.pyx                                                                                                                                                                                                              
                                                                                                                                                                                                                                              
Error compiling Cython file:                                                                                                                                                                                                                  
------------------------------------------------------------                                                                                                                                                                                  
...                                                                                                                                                                                                                                           
cdef const char *log_context_name(void *ptr) nogil:                                                                                                                                                                                           
    cdef log_context *obj = <log_context*>ptr                                                                                                                                                                                                 
    return obj.name                                                                                                                                                                                                                           
                                                                                                                                                                                                                                              
cdef lib.AVClass log_class                                                                                                                                                                                                                    
log_class.item_name = log_context_name                                                                                                                                                                                                        
                      ^                                                                                                                                                                                                                       
------------------------------------------------------------                                                                                                                                                                                  
                                                                                                                                                                                                                                              
av/logging.pyx:216:22: Cannot assign type 'const char *(void *) except? NULL nogil' to 'const char *(*)(void *) noexcept nogil'                                                                                                               
                                                                                                                                                                                                                                              
Error compiling Cython file:                                                                                                                                                                                                                  
------------------------------------------------------------                                                                                                                                                                                  
...                                                                                                                                                                                                                                           
                                                                                                                                                                                                                                              
# Start the magic!                                                                                                                                                                                                                            
# We allow the user to fully disable the logging system as it will not play                                                                                                                                                                   
# nicely with subinterpreters due to FFmpeg-created threads.                                                                                                                                                                                  
if os.environ.get('PYAV_LOGGING') != 'off':                                                                                                                                                                                                   
    lib.av_log_set_callback(log_callback)                                                                                                                                                                                                     
                            ^                                                                                                                                                                                                                 
------------------------------------------------------------                                                                                                                                                                                  
                                                                                                                                                                                                                                              
av/logging.pyx:351:28: Cannot assign type 'void (void *, int, const char *, va_list) except * nogil' to 'av_log_callback'                                                                                                                     
Traceback (most recent call last):                                                                                                                                                                                                            
  File "setup.py", line 163, in <module>                                                                                                                                                                                                      
    include_path=["include"],                                                                                                                                                                                                                 
  File "/home/presien/.local/lib/python3.6/site-packages/Cython/Build/Dependencies.py", line 1134, in cythonize                                                                                                                               
    cythonize_one(*args)                                                                                                                                                                                                                      
  File "/home/presien/.local/lib/python3.6/site-packages/Cython/Build/Dependencies.py", line 1301, in cythonize_one                                                                                                                           
    raise CompileError(None, pyx_file)                                                                                                                                                                                                        
Cython.Compiler.Errors.CompileError: av/logging.pyx

Investigation

I've tried to run the build natively on the host using the python3 setup.py method, at branch v9.0.2, and I'm getting the same issue

Reproduction

This might be hard to reproduce since its ran on a Nvidia TX2NX Hardware

• Jetpack OS version 4.6.1
• Install ffmpeg 4.3.2 from https://launchpad.net/~jonathonf/+archive/ubuntu/ffmpeg-4, along with all of the dev libs shown below

libavformat-dev 
libavcodec-dev 
libavdevice-dev 
libavutil-dev 
libavfilter-dev 
libswscale-dev 
libswresample-dev

• clone pyav
• checkout tag v9.0.2
• follow installation guide in readme

Versions

• OS: Nvidia Jetpack 4.6.1
• PyAV runtime:

n/a

• PyAV build:

python3 setup.py config --verbose
Compiling av/logging.pyx because it changed.
[1/1] Cythonizing av/logging.pyx

Error compiling Cython file:
------------------------------------------------------------
...
cdef const char *log_context_name(void *ptr) nogil:
    cdef log_context *obj = <log_context*>ptr
    return obj.name

cdef lib.AVClass log_class
log_class.item_name = log_context_name
                      ^
------------------------------------------------------------

av/logging.pyx:216:22: Cannot assign type 'const char *(void *) except? NULL nogil' to 'const char *(*)(void *) noexcept nogil'

Error compiling Cython file:
------------------------------------------------------------
...

# Start the magic!
# We allow the user to fully disable the logging system as it will not play
# nicely with subinterpreters due to FFmpeg-created threads.
if os.environ.get('PYAV_LOGGING') != 'off':
    lib.av_log_set_callback(log_callback)
                            ^
------------------------------------------------------------

av/logging.pyx:351:28: Cannot assign type 'void (void *, int, const char *, va_list) except * nogil' to 'av_log_callback'
Traceback (most recent call last):
  File "setup.py", line 163, in <module>
    include_path=["include"],
  File "/home/presien/.local/lib/python3.6/site-packages/Cython/Build/Dependencies.py", line 1134, in cythonize
    cythonize_one(*args)
  File "/home/presien/.local/lib/python3.6/site-packages/Cython/Build/Dependencies.py", line 1301, in cythonize_one
    raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: av/logging.pyx

• FFmpeg:

ffmpeg -version
ffmpeg version 4.3.2-0york0~18.04 Copyright (c) 2000-2021 the FFmpeg developers
built with gcc 7 (Ubuntu/Linaro 7.5.0-3ubuntu1~18.04)
configuration: --prefix=/usr --extra-version='0york0~18.04' --toolchain=hardened --libdir=/usr/lib/aarch64-linux-gnu --incdir=/usr/include/aarch64-linux-gnu --arch=arm64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libzimg --enable-pocketsphinx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
libavutil      56. 51.100 / 56. 51.100
libavcodec     58. 91.100 / 58. 91.100
libavformat    58. 45.100 / 58. 45.100
libavdevice    58. 10.100 / 58. 10.100
libavfilter     7. 85.100 /  7. 85.100
libavresample   4.  0.  0 /  4.  0.  0
libswscale      5.  7.100 /  5.  7.100
libswresample   3.  7.100 /  3.  7.100
libpostproc    55.  7.100 / 55.  7.100

Research

I have done the following:

• Checked the PyAV documentation
• Searched on Google
• Searched on Stack Overflow
• Looked through old GitHub issues
• Asked on PyAV Gitter
• ... and waited 72 hours for a response.

[christianglodt]

I get the same issue on x86_64 ubuntu 23.04, and in a docker container based on python:3.11-slim (which is based on debian:bookworm-slim). I can't use a binary wheel because the bundled ffmpeg is compiled without Pulseaudio support, which I need.

[samdbmg]

Same here. I'm wondering if this is due to the release of Cython 3.0.0 - my failing CI shows it's being picked up as part of the PyAV build, and it worked yesterday morning 😄

[dhopsonlb]

I can confirm I'm having the same issue. Likewise I would just use the binaries, but there's neither alsa nor pulse support built into the binary distribution. I tried fixing the errors manually and got Cython to build the Cython source to C, and then it explodes in the generated C source like so:

src/av/video/format.c:3307:40: error: assignment of read-only member ‘__pyx_genexpr_arg_0’

[pal-sanich]

Same here on python:alpine3.16, python:alpine3.17, python:alpine3.18,
Tried:
wheel-0.38.4, wheel-0.37.1,
cython-0.29.24, cython-0.29.36

[araczkowski]

I have the same problem:

~/PyAV $ python3 setup.py build_ext --inplace                                                                                                                                                                           
Compiling av/frame.pyx because it changed.
[1/1] Cythonizing av/frame.pyx
Compiling av/buffer.pyx because it changed.
[1/1] Cythonizing av/buffer.pyx
Compiling av/dictionary.pyx because it changed.
[1/1] Cythonizing av/dictionary.pyx
Compiling av/format.pyx because it changed.
[1/1] Cythonizing av/format.pyx
Compiling av/packet.pyx because it changed.
[1/1] Cythonizing av/packet.pyx
Compiling av/descriptor.pyx because it changed.
[1/1] Cythonizing av/descriptor.pyx
Compiling av/stream.pyx because it changed.
[1/1] Cythonizing av/stream.pyx
Compiling av/plane.pyx because it changed.
[1/1] Cythonizing av/plane.pyx
Compiling av/option.pyx because it changed.
[1/1] Cythonizing av/option.pyx
Compiling av/error.pyx because it changed.
[1/1] Cythonizing av/error.pyx
Compiling av/_core.pyx because it changed.
[1/1] Cythonizing av/_core.pyx
Compiling av/logging.pyx because it changed.
[1/1] Cythonizing av/logging.pyx

Error compiling Cython file:
------------------------------------------------------------
...
cdef const char *log_context_name(void *ptr) nogil:
    cdef log_context *obj = <log_context*>ptr
    return obj.name

cdef lib.AVClass log_class
log_class.item_name = log_context_name
                      ^
------------------------------------------------------------

av/logging.pyx:216:22: Cannot assign type 'const char *(void *) except? NULL nogil' to 'const char *(*)(void *) noexcept nogil'

Error compiling Cython file:
------------------------------------------------------------
...

# Start the magic!
# We allow the user to fully disable the logging system as it will not play
# nicely with subinterpreters due to FFmpeg-created threads.
if os.environ.get('PYAV_LOGGING') != 'off':
    lib.av_log_set_callback(log_callback)
                            ^
------------------------------------------------------------

av/logging.pyx:351:28: Cannot assign type 'void (void *, int, const char *, va_list) except * nogil' to 'av_log_callback'
Traceback (most recent call last):
  File "/data/data/com.termux/files/home/PyAV/setup.py", line 157, in <module>
    ext_modules += cythonize(
                   ^^^^^^^^^^
  File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/Cython/Build/Dependencies.py", line 1134, in cythonize
    cythonize_one(*args)
  File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/Cython/Build/Dependencies.py", line 1301, in cythonize_one
    raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: av/logging.pyx

[abhinavsingh]

We tried pinning to Cython==0.29.36 but that doesn't seem to help either. Is there a recommended workaround here?

[dhopsonlb]

Cython seems to misgenerate the C, so even if you edit the .pyx files with errors and change the complaining function prototypes to "noexcept nogil" like I did, that doesn't fully fix the issue. I had to patch the machine-generated C code myself, wait until Cython got past that file in the build process, and quickly copy the patched version in. I ran pip install . from the PyAV git repo.

What's interesting is it seems every Cython version I tried, as far back as 2019, misgenerated the C.

Something unusual is broken.

[cdce8p]

It's an issue with cython==3.0 which was released a few days ago. Since PyAV doesn't use an upper limit in it's build dependencies, it's always used.

PyAV/pyproject.toml

Line 2 in 6982d81

requires = ["setuptools", "wheel", "cython"]

A workaround is to add a constraint. However, flags are not passed when creating the isolated build environments. For that it's necessary to use an environment variable. Something like this works

echo "cython<3.0" >> c.txt
PIP_CONSTRAINT=c.txt pip install av==10.0.0

[nicholas-presien]

It's an issue with cython==3.0 which was released a few days ago. Since PyAV doesn't use an upper limit in it's build dependencies, it's always used.

PyAV/pyproject.toml

Line 2 in 6982d81

requires = ["setuptools", "wheel", "cython"]

A workaround is to add a constraint. However, flags are not passed when creating the isolated build environments. For that it's necessary to use an environment variable. Something like this works

echo "cython<3.0" >> c.txt
PIP_CONSTRAINT=c.txt pip install av==10.0.0

Can confirm this fixes the build issue. Thanks for the help @cdce8p!

[cdce8p]

Can confirm this fixes the build issue. Thanks for the help @cdce8p!

@nicholas-presien Would you mind reopening this again? My solution is only a workaround, this still needs a real / committed fix.

[leidix]

hope this gets officially fixed soon :)

[dreamsavior]

The problem is a lot of dependencies are not precompiled for Python 32 bit.
I switched to Python 64 bit and the usual python.exe -m pip install av run smoothly.

[yubin-tech]

It's an issue with cython==3.0 which was released a few days ago. Since PyAV doesn't use an upper limit in it's build dependencies, it's always used.

PyAV/pyproject.toml

Line 2 in 6982d81

requires = ["setuptools", "wheel", "cython"]

A workaround is to add a constraint. However, flags are not passed when creating the isolated build environments. For that it's necessary to use an environment variable. Something like this works

echo "cython<3.0" >> c.txt
PIP_CONSTRAINT=c.txt 

I am trying to trying to build pyav using the pip install av --no-binary av. The use of the environment variable did not seems to resolve the compile error. Is there something which I am missing?

[ENate]

Is there any fix to this issue? I am facing the same issue and cannot get it to work with the suggestion by @yubin-tech

[abrvkh]

Have there been any updates? I am facing the same issue. After trying the echo "cython<3.0" >> c.txt
PIP_CONSTRAINT=c.txt I am now getting error: command '/usr/bin/gcc' failed with exit code 1

[ENate]

I dropped to python 3.9 (seems to work) but another did come up when building the doc

[dschneider-l5]

I was able to get it to install on Ubuntu 22.04 / python3.10 with the following (in a venv):

echo 'cython<3.0' > constraint.txt
export PIP_CONSTRAINT=constraint.txt
pip install -v av --no-binary av

(Note the use of "export" when setting the environment variable.)

Furthermore, I confirmed that I was able to use the hevc_cuvid codec for decoding an h.265 video stream from python and making the frames available for online image processing (which was not available in the binar ffmpeg installed by av).

[tim-win]

I'm using alpine 3.17 and the PIP_CONSTRAINT=c.txt workaround did the trick.

[ENate]

@dschneider-l5 I tried it before deciding to drop the Python version because it did not work in my case.

[ENate]

I added

[build-system]
requires = ["setuptools", "wheel", "cython<3.0"]

to the pyproject.toml but it doesn't solve the issue.

[ENate]

@dschneider-l5 did you install the ffmpeg libs seperately on Ubuntu 22.04? I will imagine these libs will installed during the build process, right?. Used your solution but got the following error on Ubuntu 22.04:

/home/miniconda3/envs/transformers/include/python3.11/object.h:581:29: note: in expansion of macro ‘_PyObject_CAST’
        581 |         PyObject *_py_tmp = _PyObject_CAST(op); \
            |                             ^~~~~~~~~~~~~~
      src/av/enum.c:15197:9: note: in expansion of macro ‘Py_CLEAR’
      15197 |         Py_CLEAR(f->f_back);
            |         ^~~~~~~~
      In file included from /home/miniconda3/envs/transformers/include/python3.11/Python.h:44,
                       from src/av/enum.c:33:
      src/av/enum.c:15197:19: error: invalid use of incomplete typedef ‘PyFrameObject’ {aka ‘struct _frame’}
      15197 |         Py_CLEAR(f->f_back);
            |                   ^~
      /home/miniconda3/envs/transformers/include/python3.11/object.h:583:14: note: in definition of macro ‘Py_CLEAR’
        583 |             (op) = NULL;                        \
            |              ^~
      error: command '/usr/bin/gcc' failed with exit code 1

 Building editable for transformers (pyproject.toml) ... done
  Created wheel for transformers: filename=transformers-4.34.0.dev0-0.editable-py3-none-any.whl size=39367 sha256=7e2198c71df124de3f59b62cb5ebaf86cdecbbdf5b7b3ffc8f190445b18b8b50
  Stored in directory: /tmp/pip-ephem-wheel-cache-vif9lk97/wheels/d3/9a/6d/b8fb54dc1ab15d2cf5a608abd895e4fa8b861133f942d602d4
Successfully built transformers

Though it seems it built the wheel but I cant see anything in the /tmp location. Any ideas?

[dschneider-l5]

@ENate
On my system, ffmpeg and its libraries (e.g. libavcodec-dev) are installed at the system level via apt and not during the build process.

[yubin-tech]

How I got around the build issue temporarily is to build it directly from the source instead, and copy the binaries into the dist-package folded of the Python installation.

[ENate]

@yubin-tech can you explain how you did it? Thanks

[gregfalx]

Hello everyone, I had the exact same issue as @nicholas-presien . In my case, downgrading my Python version from 3.12 to 3.11.5 resolved the problem.

[aentwist]

I get that this is fixed, but that is in major version 11. Maybe you would consider backporting it to 9 and 10? Projects rightfully specify deps on the major version, e.g. ^9, because that is as loose as you go without going between majors which are supposed to have breaking changes. So now, any project that has av ^9 or ^10 is broken, because there have been no patch versions with this fix... two entire unusable major versions (as constraining cython is not working for me, maybe because I have a requirement to do a Windows-based build)... although hopefully package authors can just update...