Hardware acceleration not working on Docker

I have been trying for a long time to get hardware acceleration to work on Jellyfin via Docker and have gotten as far as getting nvidia-smi to successfully run -- which it wasn't previously.

System Specs:

    
Host OS: EndeavourOS
CPU: AMD Ryzen 5 3600
RAM: 32GB
GPU: Nvidia GTX 1660 TI

nvidia-smi

    
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.03              Driver Version: 560.35.03      CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce GTX 1660 Ti     On  |   00000000:26:00.0  On |                  N/A |
|  0%   51C    P3             25W /  140W |     810MiB /   6144MiB |     23%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

Video Info:

    

Video
Title: 4K HEVC HDR
Codec: HEVC
AVC: No
Profile: Main 10
Level: 153
Resolution: 3840x2160
Aspect ratio: 16:9
Anamorphic: No
Interlaced: No
Framerate: 23.976025
Bitrate: 15750 kbps
Bit depth: 10 bit
Video range: HDR
Video range type: DOVIWithHDR10
DV title: DV Profile 8.1 (HDR10)
DV version major: 1
DV version minor: 0
DV profile: 8
DV level: 6
DV rpu preset flag: 1
DV el preset flag: 0
DV bl preset flag: 1
DV bl signal compatibility id: 1
Color space: bt2020nc
Color transfer: smpte2084
Color primaries: bt2020
Pixel format: yuv420p10le
Ref frames: 1

Jellyfin Logs:

    
{"Protocol":0,"Id":"f468f43630f49e2613b07a7c7060c334","Path":"/media/tv/Only Murders in the Building (2021)/Season 4/Only Murders in the Building - S04E06 - Blow-Up 4K.mkv","EncoderPath":null,"EncoderProtocol":null,"Type":0,"Container":"mkv","Size":3973337437,"Name":"Only Murders in the Building - S04E06 - Blow-Up 4K","IsRemote":false,"ETag":"3c4612b2de1ed2d3dc36fdbf3d03cba1","RunTimeTicks":20180800000,"ReadAtNativeFramerate":false,"IgnoreDts":false,"IgnoreIndex":false,"GenPtsInput":false,"SupportsTranscoding":true,"SupportsDirectStream":true,"SupportsDirectPlay":true,"IsInfiniteStream":false,"RequiresOpening":false,"OpenToken":null,"RequiresClosing":false,"LiveStreamId":null,"BufferMs":null,"RequiresLooping":false,"SupportsProbing":true,"VideoType":0,"IsoType":null,"Video3DFormat":null,"MediaStreams":[{"Codec":"hevc","CodecTag":null,"Language":null,"ColorRange":null,"ColorSpace":"bt2020nc","ColorTransfer":"smpte2084","ColorPrimaries":"bt2020","DvVersionMajor":1,"DvVersionMinor":0,"DvProfile":8,"DvLevel":6,"RpuPresentFlag":1,"ElPresentFlag":0,"BlPresentFlag":1,"DvBlSignalCompatibilityId":1,"Comment":null,"TimeBase":"1/1000","CodecTimeBase":null,"Title":null,"VideoRange":2,"VideoRangeType":5,"VideoDoViTitle":"DV Profile 8.1 (HDR10)","AudioSpatialFormat":0,"LocalizedUndefined":null,"LocalizedDefault":null,"LocalizedForced":null,"LocalizedExternal":null,"LocalizedHearingImpaired":null,"DisplayTitle":"4K HEVC HDR","NalLengthSize":null,"IsInterlaced":false,"IsAVC":false,"ChannelLayout":null,"BitRate":15750961,"BitDepth":10,"RefFrames":1,"PacketLength":null,"Channels":null,"SampleRate":null,"IsDefault":true,"IsForced":false,"IsHearingImpaired":false,"Height":2160,"Width":3840,"AverageFrameRate":23.976025,"RealFrameRate":23.976025,"Profile":"Main 10","Type":1,"AspectRatio":"16:9","Index":0,"Score":null,"IsExternal":false,"DeliveryMethod":null,"DeliveryUrl":null,"IsExternalUrl":null,"IsTextSubtitleStream":false,"SupportsExternalStream":false,"Path":null,"PixelFormat":"yuv420p10le","Level":153,"IsAnamorphic":false},{"Codec":"eac3","CodecTag":null,"Language":"eng","ColorRange":null,"ColorSpace":null,"ColorTransfer":null,"ColorPrimaries":null,"DvVersionMajor":null,"DvVersionMinor":null,"DvProfile":null,"DvLevel":null,"RpuPresentFlag":null,"ElPresentFlag":null,"BlPresentFlag":null,"DvBlSignalCompatibilityId":null,"Comment":null,"TimeBase":"1/1000","CodecTimeBase":null,"Title":null,"VideoRange":0,"VideoRangeType":0,"VideoDoViTitle":null,"AudioSpatialFormat":0,"LocalizedUndefined":null,"LocalizedDefault":"Default","LocalizedForced":null,"LocalizedExternal":"External","LocalizedHearingImpaired":null,"DisplayTitle":"English - Dolby Digital\u002B - 5.1 - Default","NalLengthSize":null,"IsInterlaced":false,"IsAVC":false,"ChannelLayout":"5.1","BitRate":256000,"BitDepth":null,"RefFrames":null,"PacketLength":null,"Channels":6,"SampleRate":48000,"IsDefault":true,"IsForced":false,"IsHearingImpaired":false,"Height":null,"Width":null,"AverageFrameRate":null,"RealFrameRate":null,"Profile":null,"Type":0,"AspectRatio":null,"Index":1,"Score":null,"IsExternal":false,"DeliveryMethod":null,"DeliveryUrl":null,"IsExternalUrl":null,"IsTextSubtitleStream":false,"SupportsExternalStream":false,"Path":null,"PixelFormat":null,"Level":0,"IsAnamorphic":null},{"Codec":"subrip","CodecTag":null,"Language":"eng","ColorRange":null,"ColorSpace":null,"ColorTransfer":null,"ColorPrimaries":null,"DvVersionMajor":null,"DvVersionMinor":null,"DvProfile":null,"DvLevel":null,"RpuPresentFlag":null,"ElPresentFlag":null,"BlPresentFlag":null,"DvBlSignalCompatibilityId":null,"Comment":null,"TimeBase":"1/1000","CodecTimeBase":null,"Title":"English [SDH]","VideoRange":0,"VideoRangeType":0,"VideoDoViTitle":null,"AudioSpatialFormat":0,"LocalizedUndefined":"Undefined","LocalizedDefault":"Default","LocalizedForced":"Forced","LocalizedExternal":"External","LocalizedHearingImpaired":"Hearing Impaired","DisplayTitle":"English [SDH] - SUBRIP","NalLengthSize":null,"IsInterlaced":false,"IsAVC":false,"ChannelLayout":null,"BitRate":null,"BitDepth":null,"RefFrames":null,"PacketLength":null,"Channels":null,"SampleRate":null,"IsDefault":false,"IsForced":false,"IsHearingImpaired":false,"Height":0,"Width":0,"AverageFrameRate":null,"RealFrameRate":null,"Profile":null,"Type":2,"AspectRatio":null,"Index":2,"Score":null,"IsExternal":false,"DeliveryMethod":null,"DeliveryUrl":null,"IsExternalUrl":null,"IsTextSubtitleStream":true,"SupportsExternalStream":true,"Path":null,"PixelFormat":null,"Level":0,"IsAnamorphic":null},{"Codec":"subrip","CodecTag":null,"Language":"spa","ColorRange":null,"ColorSpace":null,"ColorTransfer":null,"ColorPrimaries":null,"DvVersionMajor":null,"DvVersionMinor":null,"DvProfile":null,"DvLevel":null,"RpuPresentFlag":null,"ElPresentFlag":null,"BlPresentFlag":null,"DvBlSignalCompatibilityId":null,"Comment":null,"TimeBase":"1/1000","CodecTimeBase":null,"Title":"Spanish [SDH]","VideoRange":0,"VideoRangeType":0,"VideoDoViTitle":null,"AudioSpatialFormat":0,"LocalizedUndefined":"Undefined","LocalizedDefault":"Default","LocalizedForced":"Forced","LocalizedExternal":"External","LocalizedHearingImpaired":"Hearing Impaired","DisplayTitle":"Spanish [SDH] - SUBRIP","NalLengthSize":null,"IsInterlaced":false,"IsAVC":false,"ChannelLayout":null,"BitRate":null,"BitDepth":null,"RefFrames":null,"PacketLength":null,"Channels":null,"SampleRate":null,"IsDefault":false,"IsForced":false,"IsHearingImpaired":false,"Height":0,"Width":0,"AverageFrameRate":null,"RealFrameRate":null,"Profile":null,"Type":2,"AspectRatio":null,"Index":3,"Score":null,"IsExternal":false,"DeliveryMethod":null,"DeliveryUrl":null,"IsExternalUrl":null,"IsTextSubtitleStream":true,"SupportsExternalStream":true,"Path":null,"PixelFormat":null,"Level":0,"IsAnamorphic":null}],"MediaAttachments":[],"Formats":[],"Bitrate":16006961,"Timestamp":null,"RequiredHttpHeaders":{},"TranscodingUrl":null,"TranscodingSubProtocol":0,"TranscodingContainer":null,"AnalyzeDurationMs":null,"DefaultAudioStreamIndex":null,"DefaultSubtitleStreamIndex":null}

/usr/lib/jellyfin-ffmpeg/ffmpeg -analyzeduration 200M -probesize 1G -init_hw_device cuda=cu:0 -filter_hw_device cu -hwaccel cuda -hwaccel_output_format cuda -hwaccel_flags +unsafe_output -threads 1 -noautorotate -i file:"/media/tv/Only Murders in the Building (2021)/Season 4/Only Murders in the Building - S04E06 - Blow-Up 4K.mkv" -noautoscale -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 av1_nvenc -preset p1 -b:v 15750961 -maxrate 15750961 -bufsize 31501922 -g:v:0 72 -keyint_min:v:0 72 -vf "setparams=color_primaries=bt2020:color_trc=smpte2084:colorspace=bt2020nc,tonemap_cuda=format=yuv420p:p=bt709:t=bt709:m=bt709:tonemap=bt2390:peak=100:desat=0" -codec:a:0 libfdk_aac -ac 2 -ab 256000 -af "volume=2" -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type fmp4 -hls_fmp4_init_filename "2a9aeb83dbe04b036badfe08147b7b67-1.mp4" -start_number 0 -hls_segment_filename "/config/cache/transcodes/2a9aeb83dbe04b036badfe08147b7b67%d.mp4" -hls_playlist_type vod -hls_list_size 0 -y "/config/cache/transcodes/2a9aeb83dbe04b036badfe08147b7b67.m3u8"


ffmpeg version 6.0.1-Jellyfin Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 13 (Ubuntu 13.2.0-23ubuntu4)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libxml2 --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      58.  2.100 / 58.  2.100
  libavcodec     60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter     9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample   4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
Input #0, matroska,webm, from 'file:/media/tv/Only Murders in the Building (2021)/Season 4/Only Murders in the Building - S04E06 - Blow-Up 4K.mkv':
  Metadata:
    encoder         : libebml v1.4.5 + libmatroska v1.7.1
  Duration: 00:33:38.08, start: 0.000000, bitrate: 15750 kb/s
  Stream #0:0: Video: hevc (Main 10), yuv420p10le(tv, bt2020nc/bt2020/smpte2084), 3840x2160 [SAR 1:1 DAR 16:9], 23.98 fps, 23.98 tbr, 1k tbn (default)
    Metadata:
      BPS             : 15493265
      DURATION        : 00:33:37.975000000
      NUMBER_OF_FRAMES: 48383
      NUMBER_OF_BYTES : 3908127921
      _STATISTICS_WRITING_APP: mkvmerge v84.0 ('Sleeper') 64-bit
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
    Side data:
      DOVI configuration record: version: 1.0, profile: 8, level: 6, rpu flag: 1, el flag: 0, bl flag: 1, compatibility id: 1
  Stream #0:1(eng): Audio: eac3, 48000 Hz, 5.1(side), fltp, 256 kb/s (default)
    Metadata:
      BPS             : 256000
      DURATION        : 00:33:38.080000000
      NUMBER_OF_FRAMES: 63065
      NUMBER_OF_BYTES : 64578560
      _STATISTICS_WRITING_APP: mkvmerge v84.0 ('Sleeper') 64-bit
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:2(eng): Subtitle: subrip
    Metadata:
      title           : English [SDH]
      BPS             : 122
      DURATION        : 00:33:32.972000000
      NUMBER_OF_FRAMES: 746
      NUMBER_OF_BYTES : 30903
      _STATISTICS_WRITING_APP: mkvmerge v84.0 ('Sleeper') 64-bit
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:3(spa): Subtitle: subrip
    Metadata:
      title           : Spanish [SDH]
      BPS             : 130
      DURATION        : 00:33:32.972000000
      NUMBER_OF_FRAMES: 752
      NUMBER_OF_BYTES : 32900
      _STATISTICS_WRITING_APP: mkvmerge v84.0 ('Sleeper') 64-bit
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> av1 (av1_nvenc))
  Stream #0:1 -> #0:1 (eac3 (native) -> aac (libfdk_aac))
Press [q] to stop, [?] for help
[av1_nvenc @ 0x604c49805800] Codec not supported
[av1_nvenc @ 0x604c49805800] Provided device doesn't support required NVENC features
[vost#0:0/av1_nvenc @ 0x604c49805500] Error initializing output stream: Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
[libfdk_aac @ 0x604c49806600] 2 frames left in the queue on closing
Conversion failed!

I seem to get this error on any video I try to transcode (at least through my browser):

    
[av1_nvenc @ 0x604c49805800] Codec not supported
[av1_nvenc @ 0x604c49805800] Provided device doesn't support required NVENC features
[vost#0:0/av1_nvenc @ 0x604c49805500] Error initializing output stream: Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
[libfdk_aac @ 0x604c49806600] 2 frames left in the queue on closing
Conversion failed!

Does anyone have any idea what might be happening here?

9 comments

Are you using the official docker image? if yes, try linuxserver.io's image. they also have a proper guide to using HWA.
- I’m using the linuxserver.io image
I interpreted the log, that jellyfin tries to transcode to av1 and I don't think the 1660 Ti supports AV1. Can you show me your transcoding config?
- You're right. Here's the NVIDIA GPU Codec Support Matrix: https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new
  and I looked up GTX 1660 Ti, the only codec it doesn't support is av1. That's the issue for sure.
Nvidia-smi from inside the docker or system itself?

Also quick searching brought this thread: https://forum.jellyfin.org/t-gpu-transcoding-on-unraid
- Within the container. I’ll look at the forum post, I don’t recall seeing that particular one.
  
  docker exec -it jellyfin nvidia-smi or just docker exec -it jellyfin /bin/bash to get a basic command line
Modify your docker-compose.yml to contain the following:
version: '3.5'

services: jellyfin: image: jellyfin/jellyfin container_name: jellyfin user: 1000:1000 environment: NVIDIA_DRIVER_CAPABILITIES: all NVIDIA_VISIBLE_DEVICES: all deploy: resources: reservations: devices: - capabilities: [gpu]
That's what worked for me
- I had that previously but will try again later today and report back.