Change audio output to use synthetic timecode. #564
Conversation
|
And...having made the commit already, I've just read the commit guidelines, and it's wrong, sorry! |
|
@darthsteven Just amend the commit with the fixes and force push the changes to your branch, and it will apply here. |
|
In general, A/V drift in obs-ndi is caused by OBS' audio sync mechanism. OBS adds a short buffering delay to its internal audio engine if some sources provide late timestamps. This can happen with media sources, where the timestamp provided to OBS by the media source matches the file's playback rate and not necessarily realtime. And so, because of the internal buffer growing to accomodate late sources and keep audio and video timestamps in sync, the raw video and raw audio callbacks don't necessarily output related data (audio being delayed internally by OBS). In a nutshell, with OBS, the audio and video timestamps are actually perfectly in sync, but they don't necessarily get provided in sync to obs-ndi. Which for OBS isn't an issue, because the encoder and/or the end-user's player (not exactly sure which one) uses the provided timestamps to play audio and video in sync. But for NDI, the SDK/runtime supposedly does not do any kind of A/V sync based on timestamps, rather relying on the sending application to provide audio and video already synced (or at the lowest capture latency possible). @darthsteven Your fix and explanation sound promising. I can't wait to test it on my end. Things I don't understand though:
|
|
So the culprit is either the stuff mentioned in the lengthy explanation above, or just bad timestamps math. |
|
So some more general discussion before I answer those specific questions: I added some naff debugging code to simply log the timestamp coming from OBS for video and audio, using the (
I'm not 100% but my assumption is that NDI is trying to reconcile the drift between the timecodes that its given. I'd assume that if one supplied audio and video frames where the timecode was only out by a tiny amount it would match them up internally and provide a clean, synced feed to consumers.
I'm not sure, I've seen that message a few times, but didn't take much notice of when I saw that / vs. when it worked. Sorry! If I'm understanding correctly the implication of OBS adding this buffering, there would be no recovering from that, since then the audio/video feed would forever be out of sync anyway?
I did! I absolutely did. You can turn on the visual audio meter and then use a audio/video sync video to purely see the sync. I also used this in a live production, where I took the NDI output, and used the NDI virtual camera to feed it into Zoom and had no complaints about the lip-syncing. Previously it was awful! |
|
Issue #498 mentioned in the first post is my one. So I decided to perform 10 test recordings of both 4.9.0 "stable" vs 4.9.0 "audio_frame.timecode change", the audio_frame.timecode variable change mentioned here. (Basically I cloned tag 4.9.0, made that one change within obs-ndi-output.cpp, compiled, and replaced the stable obs-ndi.dll with this one.) Using my 'static' image appears to speed up any sync issues that might otherwise take hours to occur, athough it's possible streaming for hours might still be the ultimate test... I'm not sure. I'm not going to 100% suggest the fix here is 'the' fix, but my test results do look promising. [60 FPS Recording tests] v4.9.0 - Before 'flashing' (briefly displaying) my static image, the audio was anywhere from in sync with the video to up to 18 frames out, and up to 23 frames out of sync after displaying the static image. v4.9.0 w/ audio_frame.timecode changed - Before 'flashing' the static image, the audio was anywhere from in sync with the video to up to 4 frames out, and after displaying the static image there was little to no impact...if anything it brought it closer to being in sync with the video. So in conclusion this looks extremely promising and is a huge difference compared to the current 4.9.0 release, although it would be good to see some more tests - or, as mentioned, an option to toggle between the current and proposed option within the plugin itself, as I think this could finally/hopefully be the solution that has annoyed many users of this plugin. |
|
@loopy750 that sounds great, can I ask how you're doing your tests, i.e. what are you using to record the NDI output etc.? |
|
@darthsteven Yes, I'm simply just recording directly with OBS's record feature and saving as mp4 files. The sync wasn't perfect every time, but was usually within 10-20ms, vs +300ms at times with the current release. So I wouldn't say it was perfect, but whether that's because of the way it records in OBS, or the NDI plugin, I'm not exactly sure, but I do know the change has made it much much better, so it's a very good find. |
|
Oh, wait, so you're saying that the change here in the NDI plugin changes how OBS's 'built in' recording plugin is working? |
|
Negative. What I'm suggesting is there still seems to be some slight sync variation with the change, but we're talking only a few milliseconds instead of hundreds of milliseconds. Those are my findings anyway. If anyone else can confirm, that'd be good to see too. |
|
@loopy750 So, if you were to completely remove the NDI plugin from your OBS instance, and re-run your tests (nothing seems to need NDI?) then your recording is totally perfect? Edit: Maybe I'm just confused about what exactly you are testing? I wonder if another test would be to record the NDI stream from NDI Studio Monitor, and compare that. |
|
Oh ok, I'll try and clear it up. The test is recording an mp4 from the "RECEIVE" OBS. One OBS instance is on one PC "TRANSMITTING" to the other PC that is receiving the NDI transmission and recording it. Although you do bring up an interesting question regarding general recording in OBS. And yes, with NDI totally out of the picture and just recording a video file playing in the "Source", there is some sync variation from recording to recording. So that could suggest that your OBS code change is "perfect" and it's OBS that slightly varies. Though, to be more confident in my findings, I'd want to see someone else perform the same test/s. |
|
@loopy750 ah hah! That makes perfect sense now. Sorry, I've only been using one OBS, so yeah, I can see what you're testing now. Great. It would be good to get to the bottom of why the timestamp coming out of OBS for video/audio is sometimes so far apart. The 'fix' I'm suggesting is effectively a workaround for that (albeit a seemingly very effective one.) I'm not going to have time to write it for the next few weeks, but maybe a debug output plugin that simply writes the timestamps it gets to a file would shed a little light on this. They should stay perfectly in sync, but from my limited testing they were seconds out (and more so in the latest unreleased versions of OBS.) |
|
How can we make this change effective without needing an NDI update? |
|
@Mega61 The only way to use this change is to install the version that CI built of this PR. |
|
@Mega61 You can download the plugin from https://dev.azure.com/Palakis/obs-ndi/_build/results?buildId=1415&view=artifacts&type=publishedArtifacts and then pop the built plugin into the right place in your OBS install. |
|
I recently did my first wholly NDI shoot with 6-video sources (BirdDog NDI encoders). Unfortunately, several of the video sources drifted during the 1-hour shoot, in some cases up to 2 seconds. Intent on not letting that happen again I bought an HDMI splitter, so I could send the same signal (ffmpeg test signal with timecode burnt in) to all my NDI encoders. I've been testing syncing with this setup and have replicated the drift I saw during my first NDI live shoot (with the latest obs-ndi release). It looks like this might not be an OBS or NDI problem per se but partially an encoder issue. The BirdDog encoders don't have NTP daemons installed, so their clocks don't sync. "Source Timing" appears to use the attached device's timecode, "Network Timing" appears to use encoder timecode. If the encoders were synced with NTP the Network Timing option should produce really good results. Unfortunately my cameras aren't synced, so can't use source timing and network timing is also out due to lack of NTP. The best option is timestamping the frames as they arrive, which is what this commit appears to do. And it looks like it works really nicely. It's been 5 hours and 40 minutes now and everything is still exactly in sync (30fps), provided sources are set to "Low Latency". Audio is also in sync. I note that if I turn on NDI output (Windows 10 machine) and ingest that output into another OBS instance (Linux), the video is extremely low latency, but the audio (after 5 hours and 40 minutes) is delayed by around 4 seconds. The recorded output on the source machine is in sync. Thanks @darthsteven :) P.S. I took apart the BirdDog firmware (binwalk). They use PetaLinux, which has an off-the-shelf NTPD option available. I've asked them if they can add NTPD and they've apparently "added it to the wishlist". |
|
@darthsteven do you have an updated build for the plugin? Doesn't look like there is anything in the artifacts folder and releases isn't accessible. |
|
He will have to force push the commit again in order to trigger another CI build. For some reason I can't do it myself. |
Video output from OBS is essentially clocked by OBS, and the audio is essentially clocked to that, so instead of using the timestamps of the audio frames which seem unreliable, use the timecode synthesize functionality from the NDI library. This should mean that the audio stays synced with the video.
|
@awrebels they are building now: https://dev.azure.com/Palakis/obs-ndi/_build/results?buildId=1586&view=artifacts&pathAsName=false&type=publishedArtifacts so they should appear there in a few minutes. |
|
I've found audio glitches/distorts in OBS with this change. I've reverted to an older version of OBS-NDI that supports "internal" sync (to deal with sync drift on video) and I'm using Audinate's Dante for audio. I think the problem is that the encoders aren't synced. For comparison, the Dante system uses PTP for syncing. I'm using BirdDog 4K Flex and Minis as encoders. |
|
@aquarat so are you using OBS only for video or are you somehow sending OBS audio to the Dante virtual soundcard, and then to the encoder? |
|
The first fully NDI shoot I did, I ran multiple wireless mics into a recorder/mixer and then out to a dedicated audio only camera. That went into OBS via NDI. After much testing I now seperately run audio from wireless mics into a DAW (Ardour) via Dante AVIO hardware and then route the DAW output to the OBS machine, also using Dante, specifically Dante Via. OBS sees the audio as a sound card (WDM or ASIO). Dante Via and Dante virtual sound card handle the routing and syncing. The system works well, but is a bit costly if you include all the hardware... but it gives me incredible flexibility, professional audio interfaces - and it's fun to experiment. You don't need Dante hardware to make it work; Dante Via will route audio to/from other sound interfaces. Critically, Dante uses Precision Time Protocol to sync devices with very high precision and I suspect that's why its reliable. |
|
@darthsteven is giving me trouble with audio from NDI sources. The VU meters for the NDI sources are constantly flashing red and doing weird stuff :) It also prevents me from recording. When I start a recording I will receive an error after 2 seconds saying "An encoder error occured while recording". When I remove the NDI sources from my scene and then press "Start recording" everything is fine. So I guess this build has indeed introduced other issues that make the plugin unworkable. Is there a way you could fix this new issue? |
|
So, as much as I love open source and want to help I used OBS briefly while trying to get a particular workflow going and issues like this NDI forced our team to pursue alternative approaches, which have since worked out mostly just fine. (I do have time to comment though, obviously!) I would say that I thought that the change I made was purely to do with the way that the NDI plugin exports the main output into the NDI library, and thus I'd be surprised if this change caused issues with the NDI sources, which afaik are handled by entirely different code. However, I will refer you to something I said in my initial post:
Best of luck! |
|
@darthsteven Thank you for your honesty! This fix really got my hopes up but I guess it's back to waiting for a real fix by @Palakis |
|
Just reiterating that I find the audio distorts severely with this change. It distorted with the last major release too, but only every few seconds. This distortion is continuous. |
|
@aquarat is your OBS outputting audio via NDI in your setup via the main OBS output? |
|
@darthsteven Apologies, the distortion was coming from my test video source. Please ignore my last comment. The change seems to be working well. |
|
Yeah, I was going to say, I might be wrong, but the code change here should only affect the main NDI output of the OBS programme feed, not monitors etc. |
|
Yeah, it was a bit confusing because the distortion changed between the last release and the build for this change but I think that was just an unrelated timing issue. |
|
Does this mean that Darthsteven's fix has been added to the plugin? Where can I download this new plugin version? |
|
@bvkrieken It looks like no release with this fix has been done yet. You'll have to build the plugin yourself. |
|
@jlamur Ah ok, thank you for your answer :) |
How do you maintain sync between audio/Dante and multiple camera sources? I'm looking to do the same thing though with different equipment (Joeco Blackbox for audio, Birddog Flex 4Ks for videos). This may have an obvious answer though it's new to me. My initial testing with 5 GH5S/5 Birddog Flex 4K IN is that video sync is unreliable at 2160p/4K in OBS, but fine at 1080p. Has anyone else had issues at 4K using multiple camera sources? Maybe this is a PC/processor issue and not OBS (i9-9900KF), I don't know. The connection between the PC and switch is 10Gbps (Netgear GC728XP). P.S.S. I'm not focusing on audio sync as I'll use an independent source. The distortion is pretty bad so don't think I could use it anyway. |
|
Ah sorry @comemyfanatics, I only just realised this was directed at me. If I run Dante directly from the Dante AVIO ADCs to OBS using Dante Via (that software is insanely useful) I have to set sync for the audio in the advanced audio properties of OBS to -130ms (!). All NDI sources were configured for Network timing and low latency. The sync remained stable for roughly 2 hours. This is for a simple two-mic setup. In other words audio arrived before video by about 130 ms. For video sync I've found that anything other than "low latency" mode is unusable. The sources go out of sync by several seconds over time. I hacked my BirdDog encoders to embed the correct system time in the stream and it doesn't seem to have any effect on this issue. For more complex setups, I run the Dante AVIO ADCs into Ardour DAW, then from Ardour into OBS running on a separate machine. This is all done with Dante Via. In this scenario I have to play with the latency settings in Ardour a tiny bit, but the setup is largely in sync and it doesn't drift over the course of roughly two hours. On my next shoot I'm planning on using a Zoom L-8 for mixing instead of a purely software DAW. This is because the Dante AVIOs expect line-ish level input and my mics are a bit soft, so the Zoom L-8 will provide both preamps, mixing and recording facilities. It also frees up a laptop. I'll only use Dante to get the audio from the mixer into OBS. I've found that jitter becomes dangerous on Dante and NDI when you're running more than 4 HD streams on gigabit (at 100 mbit/sec per stream). I run 5 gigabit and 10 gigabit ethernet for the OBS machines to overcome this. Hope that helps. |
I had audio/video sync issues trying to use the NDI plugin to feed OBS video and audio into NDI and then on to Zoom.
After a lot of fiddling around, I found that the culprit was OBS/this plugin. I could use the NDI monitor to see the video + audio output and although they would start in sync they would gradually drift apart, which was pretty useless!
It seems like lots of people have been experiencing similar things with the video/audio sync of the output:
In the end, I made a single line change and now the sync is near-perfect.
This single line has a very long history, and rightly so!
It's changed a little bit since then, but fundamentally it's the same line of code: use the OBS timestamp for video and audio.
So, history lesson over, back to the single line change.
I added some super simple logging to output the timestamp of the frames we are getting from OBS in the raw_video and raw_audio callbacks, and the timestamps were essentially the same for the first couple but began to drift fairly quickly, they were literally seconds apart after 15 minutes. Maybe there's an OBS bug around the timestamps that are sent along, since the audio data seemed fine, it was simply out of sync.
I'm totally new of OBS so don't understand the internals really at all, but as far as I can tell, the video output is performed by a thread that's waiting for frames from the render, and that it's clocked at the framerate of the video. Then it looks like the audio thread generally tries to keep up with the video rendering, once a video frame goes out the corresponding audio frames are sent out too.
Assuming my understanding is correct then we should simply rely on the OBS timestamp from the video only, and let NDI synthesize the audio timecode, since the NDI library will do its thing and the audio will get the timecode of the previously sent video frame.
In my limited testing, this works just fine, and is a decent workaround for this plugin to use.
Maybe this should be raised as an upstream issue in OBS too, since it looks like the timestamps coming out of OBS audio frames are plain wrong.