AMD RX6700XT causes Gamescope crashes when attempting to launch games

[Xe]

Describe the bug

Launching a game kills gamescope

What did you expect to happen?

The game to launch

Output of rpm-ostree status

State: idle
Deployments:
● ostree-image-signed:docker://ghcr.io/ublue-os/bazzite-deck:testing
                   Digest: sha256:2c2ecb25275361e33a78037c0fe2159126375bd549569f4a6a7c9c47701d9afd
                  Version: 40.20240630.0 (2024-06-30T07:53:52Z)
          LayeredPackages: gettext joystick-support meson python3-distutils-extra
                           python3-matplotlib-gtk3 python3-scipy
                Initramfs: '"-I /etc/crypttab /etc/modprobe.d/amdgpu.conf /etc/modprobe.d/deck-blacklist.conf"' 

  ostree-image-signed:docker://ghcr.io/ublue-os/bazzite-deck:testing
                   Digest: sha256:2c2ecb25275361e33a78037c0fe2159126375bd549569f4a6a7c9c47701d9afd
                  Version: 40.20240630.0 (2024-06-30T07:53:52Z)
          LayeredPackages: gettext joystick-support meson python3-distutils-extra
                           python3-matplotlib-gtk3
                Initramfs: '"-I /etc/crypttab /etc/modprobe.d/amdgpu.conf /etc/modprobe.d/deck-blacklist.conf"'

Hardware

Home theatre PC with 32 GB ram, Ryzen 5 3600, and an AMD RX6700XT

Extra information or context

Stacktraces for gamescope and gldriverquery are attached
gamescope.txt
gldriverquery.txt

[santoshganti]

I have an RX7900XT and this happens on game launches too, actually this happens more often on any arch based distros as well. Key factor I noticed is if Steam Game Recording is enabled, gamescope crashes and recovers and this behavior intermittently happens.

Steam logs show "Driver Deadlock Detected" and hard crashes the session, also in the same logs inside Steam Folder I noticed some thread deadlock on Pipewire's end, Usually noticed when closing a game, entire sessions becomes unresponsive at times.

I can provide more logs if needed, not sure if this is a known bug, but consistently can reproduce

I have the following specs

OS: Bazzite 40 (FROM Fedora Kinoite)
KERNEL: 6.9.12-205.fsync.fc40.x86_64
CPU: AMD Ryzen 9 5900X 12-Core
GPU: AMD Radeon RX 7900 XT (radeonsi, navi31, LLVM 18.1.6, DRM 3.57, 6.9.12-205.fsync.fc40.x86_64)
GPU DRIVER: 4.6 Mesa 24.1.7
RAM: 64 GB

[dosubot]

Hi, @Xe. I'm Dosu, and I'm helping the bazzite team manage their backlog. I'm marking this issue as stale.

Issue Summary:

• You reported Gamescope crashing on a home theatre PC with an AMD RX6700XT.
• User santoshganti noted a similar issue with an RX7900XT on Arch-based distributions.
• Enabling Steam Game Recording seems to trigger the crash.
• Observations include "Driver Deadlock Detected" in Steam logs, possibly related to Pipewire thread deadlocks.
• Santoshganti offered to provide more logs and shared system specifications for context.

Next Steps:

• Please let us know if this issue is still relevant to the latest version of the bazzite repository by commenting here.
• If there is no further activity, this issue will be automatically closed in 14 days.

Thank you for your understanding and contribution!

[santoshganti]

Gamescope Session Crashes with Steam Game Recording – Probable Root Cause Identified

After extensive testing, I have identified a probable root cause for a long-standing issue where Gamescope sessions on Arch-based distributions (including CachyOS, Chimera, and Nobara) would crash upon exiting a game when Steam Game Recording was enabled. This issue was consistently observed on my AMD Radeon 7900XT GPU.

Findings

1. IOMMU & VFIO Parameters Conflict

• Enabling IOMMU-related kernel parameters (iommu=pt, amd_iommu=on, or vfio_* options) appears to cause driver deadlocks in certain Gamescope sessions.
• The problem persisted across multiple distributions but was absent on Bazzite’s default configuration.

2. Mesa or Steam Game Recording May Have Improved

• While this issue was consistently reproducible a few months ago, recent updates to Mesa, Gamescope, or Steam Game Recording may have improved stability.
• However, the correlation with IOMMU settings suggests that certain GPU passthrough or virtualization-related optimizations might interfere with the amdgpu driver in a Gamescope session.

Conclusion & Recommendation

• If users experience crashes when exiting games with Steam Game Recording enabled, check if IOMMU/VFIO parameters are applied and consider removing them if GPU passthrough is not required.
• This issue does not occur with Bazzite’s default setup, suggesting that VFIO-related options should be avoided for standard gaming configurations.
• Further testing across other AMD GPUs may help determine whether this issue affects all RDNA3 cards or specific configurations.
• Users adopting default Bazzite configurations should generally be unaffected by this issue. However, Bazzite allows enabling virtualization, and users on the Deck variant should be cautious when enabling this feature, as it may introduce instability.
• AMD hardware covers a wide range of configurations, and these findings may not apply to all cases of Gamescope crashes.

This may help others encountering similar issues when running AMD GPUs in Gamescope sessions with Steam Game Recording. Let me know if additional details or logs would be helpful.