feat(linux): Add Vulkan video encoder for Linux#4603
Open
neatnoise wants to merge 14 commits intoLizardByte:masterfrom
Open
feat(linux): Add Vulkan video encoder for Linux#4603neatnoise wants to merge 14 commits intoLizardByte:masterfrom
neatnoise wants to merge 14 commits intoLizardByte:masterfrom
Conversation
neatnoise
changed the title
neatnoise
changed the title
ReenigneArcher
requested changes
Jan 19, 2026
Member
There was a problem hiding this comment.
This is just a preliminary review. On top of the comments below, I think these files will probably need updates.
update dependencies:
- https://github.com/LizardByte/Sunshine/blob/master/scripts/linux_build.sh
- https://github.com/LizardByte/Sunshine/blob/master/packaging/sunshine.rb
- https://github.com/LizardByte/Sunshine/blob/master/packaging/linux/Arch/PKGBUILD
- https://github.com/LizardByte/Sunshine/blob/master/packaging/linux/copr/Sunshine.spec
- https://github.com/LizardByte/Sunshine/blob/master/.github/workflows/ci-freebsd.yml
- and maybe the flatpak
Member
|
I will convert this to draft for now, please change it back when you're ready for a full review. Thanks! |
Author
|
Its ready |
Contributor
|
I know this PR is about encoding, but I saw changes to capture as well. Do you know if this could support variable framerate capture, similar to how capture works on Windows? In my limited experience Linux always captures at the stream framerate. The ideal capture process would be capturing only when the image has changed (with duplicated frames subject to minimum_fps_target), as well as capturing using a frame interval timer requested by the client or based on the framerate. For example, an Xbox client works best if frames are captured on a timer with an interval of 59.94fps (60000/1001), a desktop client could just use a 60fps interval. One thing you do not want to do is base anything on the host's vsync interval. |
Member
I left a bunch of review comments. |
Author
Yes, the kms_vblank option already behaves this way with variable frame rate similarly to Windows. It encodes that many frames that are generated by a game. It also respects fps frame rate which are set by a client (if the clients sets 60, it won't generate more than 60 fps with some minimal margin) |
Author
|
@ReenigneArcher I made proposed changes. I also added a tiling fix for RNDA 4 when Gnome DE is used. Would be helpful if somebody with Nvidia and Intel GPUs could test the new encoder |
21 tasks
Author
|
@ReenigneArcher I've tested with vulkan ffmpeg build https://github.com/LizardByte/build-deps/actions/runs/21850415771?pr=606 successfully. Tweaked kms_vblank for the game Cronos The New Dawn meanwhile. Tested in many games with various resolutions with and without vsync in-game and kms_vblank enabled and disabled. Tested with vulkan and vaapi encoders. It works solid. |
This comment was marked as off-topic.
This comment was marked as off-topic.
Member
|
This PR needs to be rebased to pick up the right FFmpeg pre-build, before I can run CI. Edit: actually the build-deps commit that added Vulkan hadn't been merged here yet: #4735 |
Contributor
|
Also, don't forget to verify compatibility with portalgrab when rebasing. This is enough to get started, but you might want to refine/verify: |
Author
|
@XT-Martinez It looks like ffmpeg without built-in vulkan support, you need to build it with ffmpeg vulkan supported binaries (like from link above). @psyke83 will do. I will add vulkan support also for portal grab. Actually I have my portal/pipeware implementation here https://github.com/neatnoise/Sunshine/tree/vulkan%2Bpipewire . So It will be similar. |
Author
The best results on my machine are portal+vulkan (vulkan settings low latency + vbr) and existing portal+vaapi. Portal seems to have a lower latency than KMS. |
Contributor
|
FWIW I found the vblank option quite interesting. (But I don't have much time right now to actually check it out.) |
Contributor
|
With the default settings (vbr, ll), the stream is massively violating the target bitrate (70Mbps target is saturating my wifi connection at 250Mbps+). I didn't notice this with prior testing. |
Contributor
|
This is the issue: Your current PR is assigning VBR as 3 erroneously. When set to 4, I'm not seeing the crazy bitrate overshooting. |
psyke83
reviewed
Feb 20, 2026
Author
Fixed |
21 tasks
Member
|
|
Author
|
@ReenigneArcher rebased with the current master. It works fine: |
ReenigneArcher
requested changes
Feb 24, 2026
Member
ReenigneArcher
left a comment
ReenigneArcher
left a comment
There was a problem hiding this comment.
Thank you, I have a review below.
In addition to my review, I have a question. Will there be issues if there is a mismatch in the vulkan header and vulkan loader versions here versus what is used in the build-deps (FFmpeg) repo?
src_assets/common/assets/web/configs/tabs/encoders/VulkanEncoder.vue
Outdated
Show resolved
Hide resolved
src_assets/common/assets/web/configs/tabs/encoders/VulkanEncoder.vue
Outdated
Show resolved
Hide resolved
src_assets/common/assets/web/configs/tabs/encoders/VulkanEncoder.vue
Outdated
Show resolved
Hide resolved
src_assets/common/assets/web/configs/tabs/encoders/VulkanEncoder.vue
Outdated
Show resolved
Hide resolved
Author
The build-deps repo bundles Vulkan Headers/Loader v1.4.344 (statically linked into FFmpeg), while my Sunshine compiles against the system's v1.4.341 (CachyOS distro). The two |
Author
|
@inkatail please check if the last commit changes anything |
@neatnoise CBR + this commit seems stable enough, but VBR is still pretty unstable, both with and without it. It tends to randomly overshoot the target bitrate by quite a lot. For example, if I set it to 26 Mbps, it can suddenly jump to 48 Mbps for a second. |
Author
|
@inkatail VBR generally is less stable in bitrate compared to CBR. For constrained networks I would recommend to use CBR (like through the internet). |
Author
|
@inkatail I made more changes regarding the bitrate overshoot |
@neatnoise Tested, still has issues at least for me and according to |
Author
|
@inkatail CBR stays as an option if it's important to you |
Yep, that's what I will be using. |
Contributor
|
The PR is impacted by improper pipewire teardown logic that will be fixed by #4768, but there's another issue. It seems that the Vulkan encoder also triggers a crash on mode change even when the teardown logic is fixed. I can reproduce the bug by simply changing the screen mode in KDE's display settings and it will crash with a rough 50% chance. Once #4768 is merged, please double-check this. I believe the below patch may be enough to guard against the issue (I can no longer trigger any crashes), but you may want to verify that other resources in the vulkan-specific code are being deallocated properly (such as DMA-BUF descriptors being closed properly). Edit: the patch is not enough to resolve the issue. I'll look at this again when I have time (if you haven't already resolved it). |
- Support multi-plane DRM modifiers (e.g. AMD DCC) in DMA-BUF import - Average full 2x2 block for chroma subsampling instead of 2 horizontal pixels
…kmsgrab - wlgrab: add Vulkan VRAM encode device support in wlr_vram_t, accept vulkan hwdevice type and route it to the VRAM path - portalgrab: only use Vulkan VRAM when DMA-BUFs are available - kmsgrab: remove dead vk::make_avcodec_encode_device_ram() call from display_ram_t since Vulkan RAM encoding is not implemented
- Remove format qualifiers from compute shader to support both NV12 (8-bit) and P010 (10-bit) with a single SPIR-V binary - Detect P010 from FFmpeg sw_format and use R16/RG16 target views - Rename shader from rgb2nv12 to rgb2yuv to reflect format-agnostic nature
Remove NO_RC_BUF_LIMIT flag from the Vulkan encoder so that rc_buffer_size is set to a single-frame VBV by the generic encoder setup. This constrains the hardware rate controller from producing oversized frames after periods of bitrate undershoot.
When VBR rc_mode is selected, rc_min_rate was pinned to the same value as rc_max_rate and bit_rate. This prevented the encoder from undershooting on simple frames, causing accumulated headroom that led to large bitrate spikes on complex frames. Clear rc_min_rate to 0 in VBR mode so the encoder can freely vary bitrate downward, eliminating the budget buildup that causes overshoot.
Author
|
Rebased to the upstream master to test |
Author
|
It works fine for me when changing display resolutions on the current KDE CachyOS, portalgrab with pipewire 1.4.10 |
|
Now it works for me too but something is off with the framepacing, I get occasional stutter ever 2 second. Might be a general master branch issue though. |
Author
|
@shelterx framepacing issues are generally related to the capturers, not encoders |
Contributor
Thanks for testing - it was only happening with 1.6.0. PipeWire 1.6.1 was released today and I can no longer reproduce any crashes with Vulkan (so far), so I think this PR was never at fault. While troubleshooting, I did identify some descriptor leaks in portalgrab that occur on reinit, so I'll open a new PR with the fixes after verifying everything is OK. Thanks. |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds Vulkan video encoding support for Linux, providing an alternative to VAAPI encoding.
Changes:
h264_vulkan,hevc_vulkan,av1_vulkan) with zero-copy DMA-BUF import and Vulkan compute RGB-to-YUV conversion — no EGL/GL dependency, all GPU work stays in a single Vulkan queuevulkanencoder options indocs/configuration.mdDependencies:
Requires corresponding PR in LizardByte/build-deps (
ffmpeg-vulkanbranch) to enableBUILD_FFMPEG_VULKANoption, which enables Vulkan encoder support in FFmpeg builds.Testing:
Screenshot
Issues Fixed or Closed
Roadmap Issues
Type of Change
Checklist
AI Usage