Fix leaked zombie processes when closing surfaces #4605

goonzoid · 2025-01-05T01:43:58Z

Issue description can be found at #4554, which this fixes.

I'm not thrilled about this solution, but it's the pragmatic option. I reached my wits' end trying to figure out why things weren't quite the same on linux, and we'd need to revisit that if we want to solve this in a different way.

Some notes:

we were failing to call waitpid on subprocesses that exit independently of ghostty (i.e. they terminate, rather than us terminating them by e.g. closing a tab)
std.posix.waitpid panics if waitpid returned ECHILD, which happens if the process has already been reaped when we call waitpid
on linux, something mysterious is causing the subprocesses to be reaped without us calling waitpid, and so adding in a call to waitpid caused the IO thread to panic (this is what @slovdahl was seeing below with an older version of my fix, I believe)

Details of the fix:

to enable us to always be sure to call waitpid, but to avoid panicking on linux, I switched us over to using std.c.waitpid instead of std.posix.waitpid
to make this change easier, I reviewed our use of the Command.wait method, and discovered we could simplify it a bit
this also provided an opportunity to de-duplicate some of the looping logic around non-blocking calls to waitpid that had been introduced

The only other option (other than getting to the bottom of the linux reaping mystery, which may still be worth doing), is to introduce a comptime check around the additional call to waitpid so that it happens on MacOS only. Would be happy to switch to that approach if it's preferred.

I definitely would understand if you don't want to merge this as is and would prefer to really get to the bottom of what's happening on linux. I just don't have more time to dedicate to that right now. I could look more next week, but would probably benefit from a second pair of eyes to ensure I'm not missing something obvious.

slovdahl · 2025-01-05T06:12:08Z

I checked out your branch on my Ubuntu 24.04 and built it (zig build -Doptimize=ReleaseFast). It's not working very well though. When I press ctrl-d in a tab, the ghostty window hangs and I have to press "Force Quit" in the popup that appears after a few seconds.

Log output after pressing ctrl-d:

Destroying EGL context
warning(io_exec): error getting pgid for kill

goonzoid · 2025-01-05T06:16:34Z

Yikes! That’s no good. I think I know what the issue is though. Thanks for checking it out!

I’ll try to get a Linux VM set up soon so that I can fix it correctly for both platforms.

goonzoid · 2025-01-05T06:49:51Z

Okay, thanks to @slovdahl's feedback and a fresh look at the code, I realized there's a much simpler fix available, which I've just pushed.

I think there's a higher than average risk of future bugs here with any slight changes in this area, so I still think it may be worth a rethink of much of subprocess management code. That said, this PR should fix at least some (possibly all) zombie leak cases without introducing any risk of us getting stuck in a loop and hanging.

Note that I still haven't had a chance to test on Linux, so someone should definitely do that before we merge.

slovdahl · 2025-01-05T06:56:06Z

I'm afraid the ghostty process still hangs when pressing ctrl-d on my Ubuntu 24.04 at least. 🤔

goonzoid · 2025-01-05T07:06:37Z

Hmm, that's surprising. Is it possible that you're still building the previous version of my branch? Note that I did a force-push in order to update this PR, so you may need to do a hard reset or similar (a simple pull likely won't do it).

jorismak · 2025-01-05T08:58:57Z

Doing a first pull of your repo/branch and build now, let's see.
But in the issue / discussion, there was talk of the same happening on MacOS. Could be a complete different issue of course, but maybe it's not a Linux specific thing...

jorismak · 2025-01-05T09:05:05Z

info: ghostty version=1.0.2-HEAD+d58b4353
info: ghostty build optimize=ReleaseFast
info: runtime=apprt.Runtime.gtk
info: font_backend=font.main.Backend.fontconfig_freetype
info: dependency harfbuzz=8.4.0
info: dependency fontconfig=21402
info: renderer=renderer.OpenGL
info: libxev backend=main.Backend.io_uring

When exiting the shell with ctrl+d or exit still this and then nothing:

info(io_exec): started subcommand path=/bin/sh pid=46244
info(io_exec): subcommand cgroup=/user.slice/user-1000.slice/user@1000.service/app.slice/app-ghostty-transient-46203.scope/surfaces/6802C70.scope
info(grid): reallocating GPU buffer old=0 new=8
warning(stream): unimplemented OSC callback: terminal.osc.Command{ .end_of_command = terminal.osc.Command.Command__struct_94772{ .exit_code = null } }
warning(stream): unimplemented OSC command: end_of_command
Using integer scale 2 for EGL window (1384 930 => 2768 1860)
Destroying EGL context

Tried again with gtk-single-instance forced to false, but still the same.

And with closing the window through Gnome / window manager:

info(io_exec): started subcommand path=/bin/sh pid=46855
info(io_exec): subcommand cgroup=-
info(grid): reallocating GPU buffer old=0 new=8
warning(stream): unimplemented OSC callback: terminal.osc.Command{ .end_of_command = terminal.osc.Command.Command__struct_94772{ .exit_code = null } }
warning(stream): unimplemented OSC command: end_of_command
Using integer scale 2 for EGL window (1384 930 => 2768 1860)
Destroying EGL context
Destroying EGL context
info(io_exec): read thread got quit signal
info(surface): surface closed addr=2ace9060

During the gtk2 era when I used glib, it was also tricky to get threads to nicely quit. Resulted in 'the thread must always exit itself', you can't force-abort it from the main thread, or it will stay behind.

But, as I said, that was ages ago, and I hope that modern Zig doesn't have the same issues as an ageing glib version :).

slovdahl · 2025-01-05T10:07:49Z

Hmm, that's surprising. Is it possible that you're still building the previous version of my branch? Note that I did a force-push in order to update this PR, so you may need to do a hard reset or similar (a simple pull likely won't do it).

Yeah, I deleted the local branch before fetching your branch again. Re-did it once more and verified I have d58b435 locally as my HEAD.

Noticed something interesting now though. This is ctrl-d with more than one tab open (which still hangs):

But, if I press ctrl-d with only one tab open, the window is closed as expected. The process keeps running in the background though, and nothing else than Destroying EGL context is logged.

jorismak · 2025-01-05T10:16:23Z

Never tested more tabs, so I don't know the 'original' 1.0.0 / 1.0.1 behaviour. But this branch indeed 'hangs' / stalls when opening multiple tabs and closing one with ctrl+d. Funnily, if I 'split right' for example, it seems to close OK with ctrl+d.

I don't know if it's related. But if I have an alacritty terminal running with top inside it, I see very high CPU idle with 0% wait. The moment I start a ghostty shell, the waits reach up to just over 10% (and thus my idle CPU goes down, while I don't have the feeling it's doing anything more).

If I then close the ghostty window with ctrl+d, the waits keep showing. If I type killall ghostty in another alacritty terminal, the waits instantly drop back to 0% again.

slovdahl · 2025-01-05T10:18:01Z

I see very high CPU idle with 0% wait. The moment I start a ghostty shell, the waits reach up to just over 10% (and thus my idle CPU goes down, while I don't have the feeling it's doing anything more).

Likely unrelated: #3224

slovdahl · 2025-01-05T10:24:33Z

Never tested more tabs, so I don't know the 'original' 1.0.0 / 1.0.1 behaviour. But this branch indeed 'hangs' / stalls when opening multiple tabs and closing one with ctrl+d. Funnily, if I 'split right' for example, it seems to close OK with ctrl+d.

FWIW, windows close normally with ctrl-d for me on latest main.

jorismak · 2025-01-05T10:31:36Z

Never tested more tabs, so I don't know the 'original' 1.0.0 / 1.0.1 behaviour. But this branch indeed 'hangs' / stalls when opening multiple tabs and closing one with ctrl+d. Funnily, if I 'split right' for example, it seems to close OK with ctrl+d.

FWIW, windows close normally with ctrl-d for me on latest main.

Yeah, for me as well. The problem was never with windows not closing. The problem is with the ghostty process not exiting when the shell decides to quit.

slovdahl · 2025-01-05T10:47:10Z

Built myself a debug build. All cases with only a single tab.

On main (but using epoll instead of io_uring) when I press ctrl-d and the window is closed but the process remains:

debug(app): mailbox message=redraw_surface
debug(app): mailbox message=redraw_surface
debug(app): mailbox message=redraw_surface
debug(app): mailbox message=redraw_surface
debug(io_thread): mailbox message=termio.message.Message{ .write_small = termio.message.MessageData(u8,38).Small{ .data = { 4, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170 }, .len = 1 } }
debug(io_exec): child process exited status=127 runtime=21997ms
debug(app): mailbox message=surface_message
debug(gtk_surface): gl surface unrealized
debug(opengl): program destroyed id=165
debug(opengl): program destroyed id=162
debug(renderer_thread): mailbox message=renderer.message.Message{ .reset_cursor_blink = void }
Destroying EGL context
debug(app): focus event focused=false
debug(io_exec): termios change mode=pty.Mode{ .canonical = true, .echo = true }
debug(renderer_thread): mailbox message=renderer.message.Message{ .focus = false }
debug(io_thread): mailbox message=termio.message.Message{ .focused = false }
debug(gtk): window destroy
debug(app): mailbox message=redraw_surface
debug(app): mailbox message=redraw_surface
debug(app): mailbox message=redraw_surface
debug(app): mailbox message=redraw_surface

Pressing the X button in the top right corner to close the window:

debug(app): mailbox message=redraw_surface
debug(app): mailbox message=redraw_surface
debug(app): mailbox message=redraw_surface
debug(app): mailbox message=redraw_surface
debug(gtk): window close request
debug(app): focus event focused=false
debug(gtk_surface): gl surface unrealized
debug(opengl): program destroyed id=165
debug(opengl): program destroyed id=162
Destroying EGL context
Destroying EGL context
debug(renderer_thread): mailbox message=renderer.message.Message{ .focus = false }
debug(io_thread): mailbox message=termio.message.Message{ .focused = false }
debug(gtk): tab box destroy
debug(gtk_surface): gl destroy
debug(renderer_thread): starting renderer thread shutdown
debug(renderer_thread): renderer thread exited
debug(io_thread): starting IO thread shutdown
debug(io_exec): process group killed pgid=576934
debug(io_exec): waitpid result=0
debug(io_exec): process group killed pgid=576934
debug(io_exec): waitpid result=576934
info(io_exec): read thread got quit signal
debug(io_thread): IO thread exited
info(surface): surface closed addr=7e5e51a62000
debug(gtk): window destroy
debug(app): mailbox message=redraw_surface
debug(app): mailbox message=redraw_surface

This PR when pressing ctrl-d:

debug(app): mailbox message=redraw_surface
debug(app): mailbox message=redraw_surface
debug(app): mailbox message=redraw_surface
debug(app): mailbox message=redraw_surface
debug(io_thread): mailbox message=termio.message.Message{ .write_small = termio.message.MessageData(u8,38).Small{ .data = { 4, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170 }, .len = 1 } }
debug(io_exec): child process exited status=0 runtime=14748ms
debug(app): mailbox message=surface_message
debug(renderer_thread): mailbox message=renderer.message.Message{ .reset_cursor_blink = void }
debug(gtk_surface): gl surface unrealized
debug(opengl): program destroyed id=165
debug(opengl): program destroyed id=162
debug(io_exec): termios change mode=pty.Mode{ .canonical = true, .echo = true }
Destroying EGL context
debug(app): focus event focused=false
debug(renderer_thread): mailbox message=renderer.message.Message{ .focus = false }
debug(io_thread): mailbox message=termio.message.Message{ .focused = false }
debug(gtk): window destroy
debug(app): mailbox message=redraw_surface
debug(app): mailbox message=redraw_surface
debug(app): mailbox message=redraw_surface
debug(app): mailbox message=redraw_surface

This PR when pressing the X button in the top right corner:

debug(app): mailbox message=redraw_surface
debug(app): mailbox message=redraw_surface
debug(app): mailbox message=redraw_surface
debug(app): mailbox message=redraw_surface
debug(gtk): window close request
debug(app): focus event focused=false
debug(gtk_surface): gl surface unrealized
debug(opengl): program destroyed id=165
debug(opengl): program destroyed id=162
Destroying EGL context
Destroying EGL context
debug(io_thread): mailbox message=termio.message.Message{ .focused = false }
debug(renderer_thread): mailbox message=renderer.message.Message{ .focus = false }
debug(gtk): tab box destroy
debug(gtk_surface): gl destroy
debug(renderer_thread): starting renderer thread shutdown
debug(renderer_thread): renderer thread exited
debug(io_thread): starting IO thread shutdown
debug(io_exec): process group killed pgid=580075
debug(io_exec): waitpid result=0
debug(io_exec): process group killed pgid=580075
debug(io_exec): waitpid result=580075
info(io_exec): read thread got quit signal
debug(io_thread): IO thread exited
info(surface): surface closed addr=73b675df0000
debug(gtk): window destroy
debug(app): mailbox message=redraw_surface
debug(app): mailbox message=redraw_surface

goonzoid · 2025-01-05T16:26:47Z

Thanks for all the info folks. Clearly I need to get a Linux test setup before proceeding much further. This all works well on MacOS (at least for me). I have some theories about what might be causing the stuck process, but don't want to keep guessing when I can't test it out locally.

slovdahl · 2025-01-05T19:57:23Z

I tried to debut the GTK side of this a little bit. First time I'm touching both Zig and GTK, so bear with me.. 😁

I'm not sure if it's the problem, but one thing that I find a bit strange is that App.deleteSurface is never called when the window is closed after pressing ctrl-d with only a single tab open.. The only place the method is called from is errdefer self.app.core_app.deleteSurface(self); in the realize method in apprt/gtk/Surface.zig. AFAICT, this means that quit_timer is never called.

cc'ing @jcollie as the original author of the quit_timer code paths.

goonzoid · 2025-01-08T05:33:35Z

Small progress here: managed to get a linux environment stood up today, and can reproduce the behavior exactly as described above. Ran out of time to debug but should be able to get to that soon.

goonzoid · 2025-01-10T06:33:24Z

I need to boot up my Windows machine to test these changes, since something is obviously amiss there. Will be able to do that tomorrow I hope.

jorismak · 2025-01-10T08:12:53Z

Can I try on Linux already? Or do you have something like 'just wait and sit till I say it is ready' :).

goonzoid · 2025-01-10T18:01:42Z

Can I try on Linux already? Or do you have something like 'just wait and sit till I say it is ready' :).

No harm in trying. I don't think it fixes anything there, but I never dug deep in to the linux specific issues so it's possible that it does.

mitchellh · 2025-01-24T21:27:31Z

Superseded by #5353. Thanks for the initial work @goonzoid ❤️

goonzoid force-pushed the waitpid_fix branch from a9ceb9a to 5335d32 Compare January 5, 2025 06:43

goonzoid mentioned this pull request Jan 5, 2025

Zombie process leaks #4554

Closed

goonzoid force-pushed the waitpid_fix branch 2 times, most recently from 787962f to 37f8741 Compare January 10, 2025 05:59

goonzoid added 3 commits January 9, 2025 21:59

remove unused Command.zig import statements

949a1d1

termio/exec: call wait on subprocess when it exits externally

e8c922b

termio/exec: update comment on externalExit

6c71368

goonzoid force-pushed the waitpid_fix branch from 37f8741 to 4b8f85a Compare January 10, 2025 05:59

goonzoid added 3 commits January 9, 2025 22:09

Command: use std.c.waitpid instead of std.posix.waitpid

5a3a3fb

termio/exec: use Command.wait rather than posix.waitpid

c3e1c4e

Command: add comment about wait blocking on Windows

bb5777c

goonzoid force-pushed the waitpid_fix branch from 4b8f85a to bb5777c Compare January 10, 2025 06:20

mitchellh closed this Jan 24, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix leaked zombie processes when closing surfaces #4605

Fix leaked zombie processes when closing surfaces #4605

goonzoid commented Jan 5, 2025 •

edited

Loading

slovdahl commented Jan 5, 2025 •

edited

Loading

goonzoid commented Jan 5, 2025

goonzoid commented Jan 5, 2025 •

edited

Loading

slovdahl commented Jan 5, 2025

goonzoid commented Jan 5, 2025

jorismak commented Jan 5, 2025

jorismak commented Jan 5, 2025

slovdahl commented Jan 5, 2025

jorismak commented Jan 5, 2025

slovdahl commented Jan 5, 2025

slovdahl commented Jan 5, 2025

jorismak commented Jan 5, 2025

slovdahl commented Jan 5, 2025

goonzoid commented Jan 5, 2025

slovdahl commented Jan 5, 2025

goonzoid commented Jan 8, 2025

goonzoid commented Jan 10, 2025

jorismak commented Jan 10, 2025

goonzoid commented Jan 10, 2025

mitchellh commented Jan 24, 2025

Fix leaked zombie processes when closing surfaces #4605

Fix leaked zombie processes when closing surfaces #4605

Conversation

goonzoid commented Jan 5, 2025 • edited Loading

slovdahl commented Jan 5, 2025 • edited Loading

goonzoid commented Jan 5, 2025

goonzoid commented Jan 5, 2025 • edited Loading

slovdahl commented Jan 5, 2025

goonzoid commented Jan 5, 2025

jorismak commented Jan 5, 2025

jorismak commented Jan 5, 2025

slovdahl commented Jan 5, 2025

jorismak commented Jan 5, 2025

slovdahl commented Jan 5, 2025

slovdahl commented Jan 5, 2025

jorismak commented Jan 5, 2025

slovdahl commented Jan 5, 2025

goonzoid commented Jan 5, 2025

slovdahl commented Jan 5, 2025

goonzoid commented Jan 8, 2025

goonzoid commented Jan 10, 2025

jorismak commented Jan 10, 2025

goonzoid commented Jan 10, 2025

mitchellh commented Jan 24, 2025

goonzoid commented Jan 5, 2025 •

edited

Loading

slovdahl commented Jan 5, 2025 •

edited

Loading

goonzoid commented Jan 5, 2025 •

edited

Loading