Could be interesting for you: karton-memory-watcher

[rakovskij-stanislav]

Hello! I wrote a prehook&posthook addon for karton consumers to automatically exit (to cause restart in docker / systemctl) if process uses too much RAM.

https://github.com/rakovskij-stanislav/karton_memory_watcher

I noticed that some forensic libraries can cause a slow memory leak (in my case ~500 MB for 2-3 month), so the restart is required. To make it softly (for not losing current task) I am making a decicion of restarting in post-hook logic.

Main module

It has 4 configurable rules for restart:

• proceed_tasks: count of tasks to proceed for restart (if you know that some dependency will make mistakes out of nowhere if it proceeded, for example, 500 files;
• elapsed_time: how many seconds should pass till the start of first job (if you know that there is some time-based issues in your dependencies)
• extra_consumed_memory_percent: extra memory used in % (100% means twice of memory compared to point before starting first task). To deal with memory leaks.
• extra_consumed_megabytes: extra memory used in megabytes (e.g. your service at start uses 60MB and you need to kill it if it consumes extra 500+ MB). Also to deal with memory leaks.

Usage is simple:

from karton.core import Consumer
from karton.memory_watcher import implant_watcher, RestartBehavior, RestartRule

...
class YourConsumer(Consumer):
    pass

if __name__ == "__main__":
    foobar = YourConsumer()
    implant_watcher(
        foobar,
        RestartRule(
            extra_consumed_memory_percent=80,
            # call_before_exit=(close_db_connections, )
        )
    )
    foobar.loop()
    

I hope this post-hook will help in long-term runs :)