Ragged sound at 60 fps lock

[GuvaCode]

Hi. I have built your library for fpc - lazarus. https://github.com/GuvaCode/ray4laz/blob/main/source/extras/raymedia.pas

Everything works like clockwork, Thank you very much !!!!
But when playing and lock at 60 fps the sound is jagged and the picture sometimes freaks out. Maybe it's in the buffer, but I could not find it.

[cloudofoz]

Hi @GuvaCode ,

Thank you for your interest in my project and for building the library for FPC-Lazarus! I really appreciate your contribution and feedback.

Regarding the issue you’re experiencing with audio and frame rate locked at 60 FPS, there could be multiple reasons, including the fact that the implementation is single-threaded. Buffering might indeed be a factor, but I need to investigate further.

Meanwhile, you might try adjusting the buffer as suggested in this reply: GitHub Issue Comment. It might help improve the situation!

At the moment, I’m tied up with some urgent deadlines, but as soon as I have time, I’ll try to look into this problem more thoroughly.

Thanks again for your support and patience!

Best,
C.

[cloudofoz]

Just checking in, have you tried adjusting the buffers as suggested? Did it make any difference?

I'll keep the issue open for now. I hope this isn't caused by the single-threaded nature of the library, as I’d like to keep things simple. If needed, I might look into ways to better visualize the buffers state, maybe this could help to identify a bottleneck. Let me know your thoughts!

C.

[GuvaCode]

Yes, I tried, but there are no changes.

[cloudofoz]

The solution to this problem might be simple, but the fact that it only happens for some users and not others makes it trickier.

Have you tried testing with different videos, codecs, and resolutions?

In my case, videos run almost smoothly, except when I’m in debug mode, where they sometimes slow down.

I’d like to understand if there’s a bottleneck in my code that could be optimized to improve audio/video fluidity.

[GuvaCode]

Yes, I've tried using different codecs. The result is the same everywhere, when fps=60 or vsync is enabled. Shouldn't he be playing music? It immediately crashes with an error.

[cloudofoz]

Thanks for reporting! Could you let me know which video you're using?
That might help me reproduce the crash on my end.

I usually run with vsync on and exactly 60fps by default, so if it’s tied to a specific video, that could give me some useful clues. Otherwise, it’s like finding a needle in a haystack.

[GuvaCode]

Thanks for reporting! Could you let me know which video you're using? That might help me reproduce the crash on my end.
https://download.blender.org/peach/bigbuckbunny_movies/

With any of these files.

Also when I try to play an audio file, no matter what ogg, mp3. I get an error.

ERROR: MEDIA: Trying to enqueue packet to a queue with no allocated packets.
And there's a memory leak

[cloudofoz]

Also when I try to play an audio file, no matter what ogg, mp3. I get an error.

Thanks for sharing this, it helped me track down the bug (hopefully!). The fix is now live:
93efc71

It was caused by an uninitialized streamIdx that could falsely appear valid when no video stream was present. Now all stream indices are explicitly set to -1.

With any of these files.

I’ll test those Blender videos with vsync on and 60 fps, fingers crossed it crashes! 😅 That would actually make debugging a whole lot easier.

[cloudofoz]

I've conducted a quick analysis using two of the videos:

• big_buck_bunny_1080p_h264.mov (H.264, ~700 MB)
• big_buck_bunny_480p_stereo.avi

Tested Builds:

• Debug & Release
• Vsync ON, 60 FPS lock

Video	Build	decode_packet (avg/ms)	audio_frame (avg/ms)	video_frame (avg/ms)	Notes
1080p H.264	DEBUG	12.543	0.0109	27.432	High variance, big spikes (~267 ms)
1080p H.264	RELEASE	0.9723	0.0078	27.841	Lower decode time, video still spikes (~44 ms)
480p AVI	DEBUG	0.45	0.0050	2.00	Mostly smooth, rare spikes (~34 ms)
480p AVI	RELEASE	0.65	0.0059	1.98	Fast, stable playback

---

• Audio processing is consistently fast and stable across all formats.
• Video frame processing is heavier, especially at 1080p, with occasional spikes that could cause visual jitter.
• Even at 480p, rare outliers (e.g. 30ms+) exist, but playback is generally smooth.

---

Conclusion (for now):

• The library is single-threaded by design, using dt (delta time) to sync playback with the main loop — in line with raylib's lightweight and minimal philosophy. Playback timing is therefore tightly coupled to the frame rate.
• On my test machine:
  • 480p videos run smoothly, even with vsync at 60 FPS.
  • 1080p performs decently, though some frame updates occur in small “steps” due to decoding time or occasional CPU spikes.

Since you're experiencing issues with all videos, it may point to something related to your system’s performance.

Professional video players typically rely on multithreading or hardware decoding, while this implementation trades off that complexity in favor of portability and simplicity. On some systems, this can lead to the kind of bottlenecks you're encountering.

[GuvaCode]

Oh Good, now the audio files are playing. But the problem doesn't seem to be single-threading.
Here is an example of a fairly easy format

music file : https://api.modarchive.org/downloads.php?moduleid=70673#oxygene_part_iv.s3m

vsync_on.mp4

vsync_off.mp4

[cloudofoz]

This audio file is very useful for testing!

The good news: I’m seeing the same issue with this file, which helps confirm it’s not codec-specific.
The bad news: the root of the problem is a bit trickier, and I’ll try to explain why.

Raylib manages audio internally through miniaudio, which does run in a separate thread. However, to stream audio from user code (like mine), I have to use these two functions, both of which are called from the main thread:

UpdateAudioStream(AudioStream stream, const void *data, int frameCount); // Push new PCM data
IsAudioStreamProcessed(AudioStream stream); // Check if it's time to push new data

And here's the catch:
I have to guess when and how much data to feed it.
If I give too little, we get gaps. Too much, and it overruns.
But I can only push data when IsAudioStreamProcessed() gives the OK, and I can only check that once per frame due to the single-threaded design.

---

Now here’s the twist:

Raylib also offers a callback-based solution, which is actually ideal:

SetAudioStreamCallback(AudioStream stream, AudioCallback callback);

typedef void (*AudioCallback)(void *bufferData, unsigned int frames);

With this, Raylib asks me for the exact number of frames it needs, which is perfect.
But... it doesn’t provide a way to associate a user-defined pointer or context with the callback.
So when the callback fires, I have no idea which MediaStream it belongs to.
That makes it unmanageable in a C99 library like mine, where multiple audio streams may exist at the same time.

---

• The issue likely comes from needing to feed data more frequently than once per frame.
• The polling model via IsAudioStreamProcessed() doesn’t give fine enough control.
• A callback version with a user pointer would likely solve this problem, as I’d know exactly which stream to update, when, and how much.

I’ve discussed this before with Raylib’s creator, and if I recall correctly, the decision not to include a user pointer was due to safety concerns.
Still, in use cases like this, it’s a real limitation.