Using APScheduler with uWSGI and gevent

[wobfan]

I am trying to run my Flask uWSGI application with APScheduler.
I was using it with the default in-memory jobstore without problems before, but now I need to persist some jobs that are scheduled far in the future in a PostgreSQL database, so it can live longer than the process.

Unfortunately, as soon as I add the jobstore, the uWSGI event loop doesn't start anymore.

The code I am running is:

from apscheduler.schedulers.background import BackgroundScheduler

SCHEDULER_JOBSTORES = {
    "postgres": SQLAlchemyJobStore(url=SQLALCHEMY_DATABASE_URI)
}

def create_app(config: Config):
    scheduler = BackgroundScheduler(jobstores=SCHEDULER_JOBSTORES)
    app = Flask(__name__)

    app.config.from_object(config)

    scheduler.add_job(id="cache_colors", func=cache_colors, trigger="cron", hour='3')
    scheduler.start()

if __name__ == "__main__":
    create_app(Config())

This is the full uWSGI output log:

flask_backend_container | *** Starting uWSGI 2.0.22 (64bit) on [Sat Jul 29 08:47:13 2023] ***
flask_backend_container | compiled with version: 10.2.1 20210110 on 29 July 2023 08:47:04
flask_backend_container | os: Linux-5.4.0-155-generic #172-Ubuntu SMP Fri Jul 7 16:10:02 UTC 2023
flask_backend_container | nodename: 04746b5c713a
flask_backend_container | machine: x86_64
flask_backend_container | clock source: unix
flask_backend_container | pcre jit disabled
flask_backend_container | detected number of CPU cores: 16
flask_backend_container | current working directory: /app
flask_backend_container | detected binary path: /usr/local/bin/uwsgi
flask_backend_container | uWSGI running as root, you can use --uid/--gid/--chroot options
flask_backend_container | *** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
flask_backend_container | your memory page size is 4096 bytes
flask_backend_container | detected max file descriptor number: 1048576
flask_backend_container | - async cores set to 1000 - fd table size: 1048576
flask_backend_container | lock engine: pthread robust mutexes
flask_backend_container | thunder lock: disabled (you can enable it with --thunder-lock)
flask_backend_container | uWSGI http bound on :5555 fd 4
flask_backend_container | uwsgi socket 0 bound to TCP address 127.0.0.1:43057 (port auto-assigned) fd 3
flask_backend_container | uWSGI running as root, you can use --uid/--gid/--chroot options
flask_backend_container | *** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
flask_backend_container | Python version: 3.9.10 (main, Feb  8 2022, 05:06:38)  [GCC 10.2.1 20210110]
flask_backend_container | Python main interpreter initialized at 0x55c674054aa0
flask_backend_container | uWSGI running as root, you can use --uid/--gid/--chroot options
flask_backend_container | *** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
flask_backend_container | python threads support enabled
flask_backend_container | your server socket listen backlog is limited to 100 connections
flask_backend_container | your mercy for graceful operations on workers is 60 seconds
flask_backend_container | mapped 21036928 bytes (20543 KB) for 1000 cores
flask_backend_container | *** Operational MODE: async ***
flask_backend_container | 'FLASK_ENV' is deprecated and will not be used in Flask 2.3. Use 'FLASK_DEBUG' instead.
flask_backend_container | WARNING:root:/app/app
flask_backend_container | INFO:apscheduler.scheduler:Adding job tentatively -- it will be properly scheduled when the scheduler starts
flask_backend_container | 'FLASK_ENV' is deprecated and will not be used in Flask 2.3. Use 'FLASK_DEBUG' instead.
flask_backend_container | INFO:apscheduler.scheduler:Added job "cache_colors" to job store "default"
flask_backend_container | INFO:apscheduler.scheduler:Scheduler started
flask_backend_container | WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55c674054aa0 pid: 1 (default app)
flask_backend_container | uWSGI running as root, you can use --uid/--gid/--chroot options
flask_backend_container | *** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
flask_backend_container | *** uWSGI is running in multiple interpreter mode ***
flask_backend_container | spawned uWSGI master process (pid: 1)
flask_backend_container | spawned uWSGI worker 1 (pid: 28, cores: 1000)
flask_backend_container | spawned uWSGI http 1 (pid: 29)

After that, nothing happens: I cannot do any requests to Flask, as they fail.

When not initializing the PostgreSQL jobstore, the output log is the same but one additional line pops up at the end:

flask_backend_container | *** running gevent loop engine [addr:0x55997f414e00] ***

This starts the backend successfully and I can do requests to it as used to.

I already read about uWSGI doing some stuff with the GIL, which APScheduler relies on. To fix this, I should add --enable-threads to the uWSGI command line, which I did. But it didn't change anything, unfortunately.

Is there anything I missed when setting the jobstores up?

[agronholm]

What is the APScheduler class that you're instantiating?

[agronholm]

Ok, so how will uWSGI get access to your Flask application object?

[wobfan]

Running uWSGI by executing:

uwsgi --http :5555 --gevent 1000 --http-websockets --enable-threads --master --wsgi-file run.py --callable app

That runs run.py, which is:

from app import create_app, logger
from app.config import Config

app, ws = create_app(Config())

if __name__ == '__main__':
    logger.info("Starting backend service\n")
    ws.run(app, debug=True, host="0.0.0.0", port=5555)

[agronholm]

I'll try this later, but meanwhile: Have you tried the GeventScheduler yet?

[wobfan]

Thanks in advance!
Tried replacing the BackgroundScheduler with GeventScheduler now and it indeed seems to be working. But BackgroundScheduler also hat some runs where it appeared to work for some minutes, until it freezed (like 1 of 20 times, the other times it just didn't start, as described above).
Will report back in some hours or tomorrow! Thanks! :)

[wobfan]

Follow-up: Since I've been using GeventScheduler, everything runs fine and just as expected!