Invalid /=/library/calibre-web path being written to xklb-metadata.db

[deldesir]

Adding '=' leads to prepending an extra /, resulting in an invalid /=/library/calibre-web path being written to xklb-metadata.db database. This causes an error in the download process that is displayed as a 500 INTERNAL ERROR in the task page.

2025-07-07 13:32:04 - [Debug] [https://www.youtube.com/watch?v=lbvpP-CEwhk]: Downloaded to =/library/downloads/calibre-web/Youtube/chevy/Ryce & Chevy - everyday (live band in a van version 🚙✨)_264.76k_[lbvpP-CEwhk].mp4

Originally posted by @deldesir in f3b037c

[holta]

@deldesir does this possibly relate to either (or both of...?)

• "504 Gateway Time-out" when clicking "Watch in Browser - mp4" #319
• "404 Not Found" after clicking "Watch in Browser - mp4" #320

And apparently... !?

• PR Add explicit '=' to some lb args to avoid parse error #312

cc: @ZachLiebl

[deldesir]

This happens in every download. It eventually goes silent (error disappeared) when the download task moves the media file and updates the path, but the invalid directory remains.

@deldesir does this possibly relate to either (or both of...?)

• "504 Gateway Time-out" when clicking "Watch in Browser - mp4" #319
• "404 Not Found" after clicking "Watch in Browser - mp4" #320

cc: @ZachLiebl

This issue is caused by '&' and emojis passed to the header. When I encode them as unicode, nginx cannot find the file. I already tried several workarounds, including use of nginx lua module, perl script... Now I am investigating renaming the path, hence the discovery of the '=' directory being created.

[holta]

Looks like @deldesir might have a solid solution?! Using...

urllib.parse.quote "is a function in Python's urllib.parse module used for URL encoding strings. Its primary purpose is to safely encode special characters within a URL component, typically the path or query parameters, by replacing them with their corresponding percent-encoded (%xx) escape sequences."

• https://docs.python.org/3/library/urllib.parse.html#url-quoting
• https://stackoverflow.com/questions/31827012/python-importing-urllib-quote (beware, this doc is from 2015, re: Python 2.x!)

[holta]

Looks like @deldesir caught at least 2 bugs! As my VM from last night (#319, #320) confirms...

root@box:~# ls /
'='   bin   boot   dev   etc   home   lib   lib64   library   lost+found   media   mnt   opt   proc   root   run   sbin   snap   srv   sys   tmp   usr   var

root@box:~# tree /=
/=
└── library
    └── downloads
        └── calibre-web
            └── Youtube
                ├── Amadou & Mariam
                └── chevy

[avni]

What's perhaps interesting is that the following URL, which also has an "=" works and plays successfully in-browser: https://www.youtube.com/watch?v=tYa6OLQHrEc ?!

This is on Raspberry Pi OS 13 Trixie pre-release.

The behavior for the URLs shared in #319 and #320 is the same as reported in the issues (504 Gateway Time-out, 404 Not Found, respectively) on this OS.

root@box:/library/calibre-web# tree --du -h
[ 34M]  .
├── [ 19M]  Amadou
│   ├── [ 14M]  Amadou & Mariam (feat. Manu Chao) - Sénégal Fast Food (Official Music Video) (1)
│   │   ├── [ 14M]  Amadou & Mariam (feat. Manu Chao) - Séné - Amadou.mp4
│   │   └── [110K]  cover.jpg
│   └── [5.3M]  Amadou & Mariam - Sénégal Fast Food (Official Audio) (2)
│       ├── [5.2M]  Amadou & Mariam - Sénégal Fast Food (O - Amadou.mp4
│       └── [ 76K]  cover.jpg
├── [128K]  config
│   └── [124K]  app.db
├── [348K]  metadata.db
├── [ 13K]  metadata_db_prefs_backup.json
├── [ 14M]  The Old Branch
│   └── [ 14M]  Caterpillar Shoes , Sweet rhyming bedtime story for kids! (3)
│       ├── [ 14M]  Caterpillar Shoes , Sweet rhyming bedtime - The Old Branch.mp4
│       └── [ 44K]  cover.jpg
└── [312K]  xklb-metadata.db

  34M used in 7 directories, 10 files

root@box:/library/calibre-web# cd ~
root@box:~# ls /
'='   bin   boot   dev	 etc   home   lib   library   lost+found   media   mnt	 opt   proc   root   run   sbin   srv   sys   tmp   usr   var
root@box:~# tree /=
/=
└── library
    └── downloads
        └── calibre-web
            └── Youtube
                ├── Amadou & Mariam
                └── The Old Branch

7 directories, 0 files

[holta]

URL, which also has an "=" works and plays successfully in-browser

• "=" sign in URLs likely unrelated!?
• xklb-metadata.db is being badly polluted with corrupted paths — e.g. /=/library/calibre-web/...
• For reasons that @deldesir is investigating (and hopefully @ZachLiebl too!)
• @deldesir suggests the "=" signs that @ZachLiebl added to scripts/lb-wrapper might possibly be related: f3b037c#commitcomment-161610929 (that's PR Add explicit '=' to some lb args to avoid parse error #312)

tree -du

Just FWIW --du (with 2 dashes, not a single dash) is usually what's needed — reporting the cumulative size of each directory — in other words, similar to the du (disk usage) command.

[avni]

"=" sign in URLs likely unrelated!?

ack, OK. I misunderstood this issue, originally. Any URL with a GET parameters will have an "="... so why one works and another doesn't is puzzling.

Just FWIW --du (with 2 dashes, not a single dash) is usually what's needed — reporting the cumulative size of each directory — in other words, similar to the du (disk usage) command.

Thx, updated above. 🙏

[ZachLiebl]

Here is a hack that probably fixes it

• PR remove unneeded equal sign in YT_DLP_OPTIONS #323

[holta]

Here is a hack that probably fixes it

• PR remove unneeded equal sign in YT_DLP_OPTIONS #323

THANKS @ZachLiebl:

• Recap from our call earlier this afternoon, where the consensus from all 4 of us... is that that PR Make path safe for response headers [ampersand (&) in YouTube video titles preventing playback] #322 solves a completely different bug.
• Huge Thanks to @deldesir for identifying both issues back on Monday!

[holta]

@deldesir can you help list a few XKLB (library) commands in sequence — that function as a simpler "reproducer" — to make this problem more clear for @chapmanjacobd ?

[mpimble]

As seen above, I was able to reproduce this directory issue after uploading https://www.youtube.com/watch?v=lbvpP-CEwhk on calibre-web with Ubuntu 24.04, 25.04, 25.10, Debian 12, and Debian 13. None of the IIABs had errors downloading the video from Youtube, though I am unsure if the video would be playable on any of these instances.

[holta]

Huge thanks @mpimble for confirming this affects all 5 OS's / distros!

[holta]

@chapmanjacobd

Does it help that @mpimble reproduced this bug on a clean VM, without IIAB Calibre-Web? By running these 2 commands:

lb-wrapper tubeadd https://www.youtube.com/watch?v=lbvpP-CEwhk
lb-wrapper dl https://www.youtube.com/watch?v=lbvpP-CEwhk

In other words these 2 XKLB (library) commands:

lb tubeadd '/library/calibre-web/xklb-metadata.db' 'https://www.youtube.com/watch?v=lbvpP-CEwhk' --force -vv
lb dl '/library/calibre-web/xklb-metadata.db' --video --search 'https://www.youtube.com/watch?v=lbvpP-CEwhk' --subs --write-thumbnail --live --live-from-start -o='/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s' --format=best --format-sort='tbr~1000' -vv

The result is that he found the erroneous path in xklb-metadata.db (starting with an '=' sign) — and an erroneous directory was created in the current working directory here:

Just FYI these 2 simpler commands also reproduce the problem — on a completely clean VM without IIAB Calibre-Web at all:

lb tubeadd xklb-metadata.db https://www.youtube.com/watch?v=lbvpP-CEwhk -vv
lb dl xklb-metadata.db https://www.youtube.com/watch?v=lbvpP-CEwhk -o='/tmp/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s' -vv

Interim Workaround:

• PR remove unneeded equal sign in YT_DLP_OPTIONS #323

SEPARATELY/ASIDE: Anybody know why shortening the above URL (to https://youtu.be/lbvpP-CEwhk) no longer downloads anything at all? ("Error 2", "No media found")

root@box:~/tmp# lb-wrapper tubeadd https://youtu.be/lbvpP-CEwhk
/usr/local/bin/lb
/usr/local/bin/yt-dlp
2025-07-16 18:37:22 - [Info] Using yt-dlp 2025.06.30
2025-07-16 18:37:22 - [Info] Running xklb command: lb tubeadd '/library/calibre-web/xklb-metadata.db' 'https://youtu.be/lbvpP-CEwhk' --force -vv
2025-07-16 18:37:23 - [Debug] library v3.0.105 :: /root/.local/share/pipx/venvs/library/bin
2025-07-16 18:37:23 - [Debug] ['/usr/local/bin/lb', 'tubeadd', '/library/calibre-web/xklb-metadata.db', 'https://youtu.be/lbvpP-CEwhk', '--force', '-vv']
2025-07-16 18:37:23 - [Debug] {'force': True, 'paths': ['https://youtu.be/lbvpP-CEwhk']}
2025-07-16 18:37:23 - [Debug] {'noprogress': True, 'skip_download': True, 'lazy_playlist': True, 'extract_flat': True, 'no_check_certificate': True, 'skip_playlist_after_errors': 21, 'playlistend': 20000, 'force': True}
2025-07-16 18:37:23 - [Debug] yt-dlp initialized 0.0738
2025-07-16 18:37:23 - [Info] [youtube] Extracting URL: https://youtu.be/lbvpP-CEwhk
2025-07-16 18:37:23 - [Info] [youtube] lbvpP-CEwhk: Downloading webpage
2025-07-16 18:37:24 - [Info] [youtube] lbvpP-CEwhk: Downloading tv client config
2025-07-16 18:37:24 - [Info] [youtube] lbvpP-CEwhk: Downloading tv player API JSON
2025-07-16 18:37:24 - [Info] [youtube] lbvpP-CEwhk: Downloading ios player API JSON
2025-07-16 18:37:26 - [Debug] Importing playlist-less media https://youtu.be/lbvpP-CEwhk
2025-07-16 18:37:26 - [Debug] Extra media data {'media_type': 'video'}
2025-07-16 18:37:26 - [Debug] media.playlist_media_add 3.4158
2025-07-16 18:37:26 - [Debug] ydl.extract_info done 0.0570
2025-07-16 18:37:26 - [Info]
2025-07-16 18:37:27 - [Info] lb-wrapper's xklb command (tubeadd) completed successfully.

root@box:~/tmp# lb-wrapper dl https://youtu.be/lbvpP-CEwhk
/usr/local/bin/lb
/usr/local/bin/yt-dlp
2025-07-16 18:37:34 - [Info] Using yt-dlp 2025.06.30
2025-07-16 18:37:34 - [Info] Running xklb command: lb dl '/library/calibre-web/xklb-metadata.db' --video --search 'https://youtu.be/lbvpP-CEwhk' --subs --write-thumbnail --live --live-from-start -o='/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s' --format=best --format-sort='tbr~1000' -vv
2025-07-16 18:37:34 - [Debug] library v3.0.105 :: /root/.local/share/pipx/venvs/library/bin
2025-07-16 18:37:34 - [Debug] ['/usr/local/bin/lb', 'dl', '/library/calibre-web/xklb-metadata.db', '--video', '--search', 'https://youtu.be/lbvpP-CEwhk', '--subs', '--write-thumbnail', '--live', '--live-from-start', '-o=/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s', '--format=best', '--format-sort=tbr~1000', '-vv']
2025-07-16 18:37:34 - [Debug] {'include': ['https://youtu.be/lbvpP-CEwhk'], 'subs': True, 'live': True, 'paths': []}
2025-07-16 18:37:34 - [Debug] No media found
2025-07-16 18:37:35 - [Error] Error 2 occurred while running lb-wrapper's xklb commands.

[holta]

SEPARATELY/ASIDE: Anybody know why shortening the above URL (to https://youtu.be/lbvpP-CEwhk) no longer downloads anything at all? ("Error 2", "No media found")

CLARIF thanks to @deldesir:

• lb-wrapper dl when run manually needs the canonical YouTube URL (i.e. the .com one).

• Whereas lb itself can use short URLs within both commands (tubeadd and dl), as confirmed here:

root@u2510-321c:~# lb tubeadd xklb-metadata.db https://youtu.be/lbvpP-CEwhk -vv
library v3.0.105 :: /root/.local/share/pipx/venvs/library/bin
['/root/.local/bin/lb', 'tubeadd', 'xklb-metadata.db', 'https://youtu.be/lbvpP-CEwhk', '-vv']
{'paths': ['https://youtu.be/lbvpP-CEwhk']}
{'noprogress': True, 'skip_download': True, 'lazy_playlist': True, 'extract_flat': True, 'no_check_certificate': True, 'skip_playlist_after_errors': 21, 'playlistend': 20000}
yt-dlp initialized 0.0735
[youtube] Extracting URL: https://youtu.be/lbvpP-CEwhk
[youtube] lbvpP-CEwhk: Downloading webpage
[youtube] lbvpP-CEwhk: Downloading tv client config
[youtube] lbvpP-CEwhk: Downloading player 69b31e11-main
[youtube] lbvpP-CEwhk: Downloading tv player API JSON
[youtube] lbvpP-CEwhk: Downloading ios player API JSON
Importing playlist-less media https://youtu.be/lbvpP-CEwhk
Extra media data {'media_type': 'video'}
media.playlist_media_add 3.7997
WARNING: ffmpeg not found. The downloaded format may not be the best available. Installing ffmpeg is strongly recommended: https://github.com/yt-dlp/yt-dlp#dependencies
ydl.extract_info done 0.0216


Optimizing database
Processing table: playlists
Transforming column order: ['id', 'time_created', 'time_modified', 'time_deleted', 'hours_update_delay', 'path', 'extractor_key']
Creating index: id
Creating index: time_created
Creating index: time_modified
Creating index: time_deleted
Creating index: hours_update_delay
Creating index: extractor_config
Creating index: path
Creating fts index: ['path']
Processing table: media
Transforming column order: ['id', 'playlists_id', 'time_created', 'time_modified', 'time_deleted', 'time_uploaded', 'time_downloaded', 'size', 'duration', 'view_count', 'path', 'webpath', 'extractor_id']
Creating index: id
Creating index: playlists_id
Creating index: time_created
Creating index: time_modified
Creating index: time_deleted
Creating index: time_uploaded
Creating index: time_downloaded
Creating index: size
Creating index: duration
Creating index: view_count
Creating index: path
Creating index: uploader
Creating index: live_status
Creating fts index: ['path', 'webpath', 'title']
Processing table: captions
Creating index: media_id
Creating index: time
Creating fts index: ['text']
Running VACUUM
Running ANALYZE

root@u2510-321c:~# lb dl xklb-metadata.db https://youtu.be/lbvpP-CEwhk -o='/tmp/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s' -vv
library v3.0.105 :: /root/.local/share/pipx/venvs/library/bin
['/root/.local/bin/lb', 'dl', 'xklb-metadata.db', 'https://youtu.be/lbvpP-CEwhk', '-o=/tmp/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s', '-vv']
{'paths': ['https://youtu.be/lbvpP-CEwhk']}
None
{'path': 'https://youtu.be/lbvpP-CEwhk'}
{'noprogress': True, 'lazy_playlist': True, 'noplaylist': True, 'no_check_certificate': True, 'skip_playlist_after_errors': 21, 'playlistend': 20000, 'extractor_args': {'youtube': {'skip': ['authcheck']}}, 'check_formats': 'selected', 'logger': <library.createdb.tube_backend.download.<locals>.DictLogger object at 0x7619432adbe0>, 'postprocessors': [{'key': 'FFmpegMetadata'}, {'key': 'FFmpegEmbedSubtitle'}], 'playlist_items': '1', 'extractor_retries': 3, 'retries': 12, 'retry_sleep_functions': {'extractor': <function download.<locals>.<lambda> at 0x7619432f4ae0>, 'http': <function download.<locals>.<lambda> at 0x7619432f4b80>, 'fragment': <function download.<locals>.<lambda> at 0x7619432f4c20>}, 'outtmpl': {'default': '=/tmp/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s'}, 'progress_hooks': [<function download.<locals>.<lambda> at 0x7619432f4cc0>], 'subtitlesformat': 'srt/best'}
[yt-dlp]: Downloading https://youtu.be/lbvpP-CEwhk
downloading   0.0%  Unknown B/s 1024 bytes
downloading   0.0%    2.57MiB/s 3072 bytes
downloading   0.0%    3.21MiB/s 7168 bytes
downloading   0.1%    4.49MiB/s 15360 bytes
downloading   0.2%    7.07MiB/s 31744 bytes
downloading   0.3%   12.23MiB/s 64512 bytes
downloading   0.7%   12.69MiB/s 130048 bytes
downloading   1.3%   17.59MiB/s 261120 bytes
downloading   2.6%   25.15MiB/s 523264 bytes
downloading   5.3%   41.19MiB/s 1047552 bytes
downloading  10.5%   57.44MiB/s 2096128 bytes
downloading  21.0%   64.88MiB/s 4193280 bytes
downloading  42.1%   76.03MiB/s 8387584 bytes
downloading  52.0%   71.14MiB/s 10358855 bytes
downloading  52.0%  Unknown B/s 10359879 bytes
downloading  52.0%  Unknown B/s 10361927 bytes
downloading  52.0%    6.18MiB/s 10366023 bytes
downloading  52.1%    7.77MiB/s 10374215 bytes
downloading  52.1%    7.17MiB/s 10390599 bytes
downloading  52.3%    9.38MiB/s 10423367 bytes
downloading  52.6%    3.78MiB/s 10488903 bytes
downloading  53.3%    7.10MiB/s 10619975 bytes
downloading  54.6%   12.90MiB/s 10882119 bytes
downloading  57.2%   21.75MiB/s 11406407 bytes
downloading  62.5%   36.46MiB/s 12454983 bytes
downloading  73.0%   51.02MiB/s 14552135 bytes
downloading  94.1%   47.89MiB/s 18746439 bytes
downloading 100.0%   52.35MiB/s 19926757 bytes
downloading 100.0% 55.24MiB/s 19926757 bytes
[https://youtu.be/lbvpP-CEwhk]: yt-dlp ERROR: Postprocessing: ffmpeg not found. Please install or provide the path using --ffmpeg-location
[https://youtu.be/lbvpP-CEwhk]: yt-dlp returned no info
[https://youtu.be/lbvpP-CEwhk]: Recoverable error matched (will try again later). ERROR: Postprocessing: ffmpeg not found. Please install or provide the path using --ffmpeg-location;ERROR: Postprocessing: ffmpeg not found. Please install or provide the path using --ffmpeg-location

root@u2510-321c:~# tree
.
├── =
│   └── tmp
│       └── Youtube
│           └── chevy
│               └── Ryce & Chevy - everyday (live band in a van version 🚙✨)_317.19k_[lbvpP-CEwhk].mp4
└── xklb-metadata.db

5 directories, 2 files