REGRESSION(299318@main): [GStreamer][MSE] Audio playback broken on instagram #51805

philn · 2025-10-04T11:14:26Z

`4a3ebdf`

REGRESSION(299318@main): [GStreamer][MSE] Audio playback broken on instagram
https://bugs.webkit.org/show_bug.cgi?id=299252

Reviewed by Alicia Boya Garcia.

299318@main introduced a performance regression and also broke Instagram stories audio playback.
Instead of forcing ADTS framing we now rewrite the codec_data supplied by the demuxer only if the
underlying AAC audio_object_type was set to SBR or PS, which the avdec_aac decoder doesn't handle
very well. When any of those object types is detected, we replace it with an LC (Low Complexity)
identifier and re-use the existing sampling frequency and channels bits from the previous
codec_data. avdec_aac handles LC well and there is no noticeable rendering issue on Kaltura and
Instagram stories anymore.

Thanks to Alicia Boya Garcia <aboya@igalia.com> for her precious help during the investigation.

* Source/WebCore/platform/graphics/gstreamer/mse/AppendPipeline.cpp:
(WebCore::aacGetAudioObjectType):
(WebCore::createOptionalParserForFormat):

Canonical link: https://commits.webkit.org/301209@main

4fb454f

Misc	iOS, visionOS, tvOS & watchOS	macOS	Linux	Windows
❌ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	✅ 🛠 wpe	✅ 🛠 win
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-AS-debug	✅ 🧪 wpe-wk2	~~🧪 win-tests~~
✅ 🧪 webkitperl	✅ 🧪 ios-wk2	✅ 🧪 api-mac	✅ 🧪 api-wpe
	✅ 🧪 ios-wk2-wpt	✅ 🧪 mac-wk1	✅ 🛠 wpe-cairo
	✅ 🧪 api-ios	✅ 🧪 mac-wk2	✅ 🛠 gtk
	✅ 🛠 vision	~~🧪 mac-AS-debug-wk2~~	✅ 🧪 gtk-wk2
	✅ 🛠 vision-sim	✅ 🧪 mac-wk2-stress	✅ 🧪 api-gtk
✅ 🛠 🧪 merge	✅ 🧪 vision-wk2	✅ 🧪 mac-intel-wk2	✅ 🛠 playstation
	✅ 🛠 tv	✅ 🛠 mac-safer-cpp
	✅ 🛠 tv-sim
	✅ 🛠 watch
	✅ 🛠 watch-sim

webkit-early-warning-system · 2025-10-04T11:14:34Z

EWS run on previous version of this PR (hash def6a96)

Details

Misc	iOS, visionOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	✅ 🛠 wpe	⏳ 🛠 win
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-AS-debug	❌ 🧪 wpe-wk2	⏳ 🧪 win-tests
✅ 🧪 webkitperl	✅ 🧪 ios-wk2	✅ 🧪 api-mac	✅ 🧪 api-wpe
	✅ 🧪 ios-wk2-wpt	✅ 🧪 mac-wk1	✅ 🛠 wpe-cairo
	✅ 🧪 api-ios	✅ 🧪 mac-wk2	✅ 🛠 gtk
	✅ 🛠 vision	✅ 🧪 mac-AS-debug-wk2	❌ 🧪 gtk-wk2
	✅ 🛠 vision-sim	✅ 🧪 mac-wk2-stress	✅ 🧪 api-gtk
	✅ 🧪 vision-wk2	✅ 🧪 mac-intel-wk2	✅ 🛠 playstation
	✅ 🛠 tv	✅ 🛠 mac-safer-cpp	⏳ 🛠 jsc-armv7
	✅ 🛠 tv-sim		⏳ 🧪 jsc-armv7-tests
	✅ 🛠 watch
	✅ 🛠 watch-sim

webkit-early-warning-system · 2025-10-04T13:44:12Z

EWS run on previous version of this PR (hash 689f0a6)

Details

Misc	iOS, visionOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	✅ 🛠 wpe	✅ 🛠 win
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-AS-debug	✅ 🧪 wpe-wk2	✅ 🧪 win-tests
✅ 🧪 webkitperl	✅ 🧪 ios-wk2	✅ 🧪 api-mac	✅ 🧪 api-wpe
	✅ 🧪 ios-wk2-wpt	✅ 🧪 mac-wk1	✅ 🛠 wpe-cairo
	✅ 🧪 api-ios	✅ 🧪 mac-wk2	✅ 🛠 gtk
	✅ 🛠 vision	✅ 🧪 mac-AS-debug-wk2	✅ 🧪 gtk-wk2
	✅ 🛠 vision-sim	✅ 🧪 mac-wk2-stress	✅ 🧪 api-gtk
	✅ 🧪 vision-wk2	✅ 🧪 mac-intel-wk2	✅ 🛠 playstation
	✅ 🛠 tv	✅ 🛠 mac-safer-cpp	⏳ 🛠 jsc-armv7
	✅ 🛠 tv-sim		⏳ 🧪 jsc-armv7-tests
	✅ 🛠 watch
	✅ 🛠 watch-sim

webkit-early-warning-system · 2025-10-05T09:45:11Z

EWS run on previous version of this PR (hash 796d2cb)

Details

Misc	iOS, visionOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	✅ 🛠 wpe	✅ 🛠 win
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-AS-debug	✅ 🧪 wpe-wk2	❌ 🧪 win-tests
✅ 🧪 webkitperl	✅ 🧪 ios-wk2	✅ 🧪 api-mac	✅ 🧪 api-wpe
	✅ 🧪 ios-wk2-wpt	✅ 🧪 mac-wk1	✅ 🛠 wpe-cairo
	✅ 🧪 api-ios	✅ 🧪 mac-wk2	✅ 🛠 gtk
	✅ 🛠 vision	✅ 🧪 mac-AS-debug-wk2	❌ 🧪 gtk-wk2
	✅ 🛠 vision-sim	✅ 🧪 mac-wk2-stress	✅ 🧪 api-gtk
	✅ 🧪 vision-wk2	✅ 🧪 mac-intel-wk2	✅ 🛠 playstation
	✅ 🛠 tv	✅ 🛠 mac-safer-cpp
	✅ 🛠 tv-sim
	✅ 🛠 watch
	✅ 🛠 watch-sim

ntrrgc

Aside from what I commented, LGTM.

There is one case of explicit signalling we're not handling here: Backwards-compatible signalling. In that case, AOT=2 appears first, and the SBR and PS extensions appear later in the codec-data. It's a bit trickier to parse because it requires parsing GASpecificInfo as well. For now we can let it be, and if we actually encounter a file where that is a problem, handle it then.

Source/WebCore/platform/graphics/gstreamer/mse/AppendPipeline.cpp

webkit-early-warning-system · 2025-10-08T10:40:55Z

EWS run on previous version of this PR (hash e9a9902)

Details

Misc	iOS, visionOS, tvOS & watchOS	macOS	Linux	Windows
❌ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	✅ 🛠 wpe	⏳ 🛠 win
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-AS-debug	✅ 🧪 wpe-wk2	⏳ 🧪 win-tests
✅ 🧪 webkitperl	✅ 🧪 ios-wk2	✅ 🧪 api-mac	✅ 🧪 api-wpe
	✅ 🧪 ios-wk2-wpt	✅ 🧪 mac-wk1	✅ 🛠 wpe-cairo
	✅ 🧪 api-ios	✅ 🧪 mac-wk2	✅ 🛠 gtk
	✅ 🛠 vision	~~🧪 mac-AS-debug-wk2~~	❌ 🧪 gtk-wk2
	✅ 🛠 vision-sim	✅ 🧪 mac-wk2-stress	✅ 🧪 api-gtk
	✅ 🧪 vision-wk2	✅ 🧪 mac-intel-wk2	✅ 🛠 playstation
	✅ 🛠 tv	✅ 🛠 mac-safer-cpp
	✅ 🛠 tv-sim
	✅ 🛠 watch
	✅ 🛠 watch-sim

ntrrgc

LGTM with nits.

ntrrgc · 2025-10-08T12:27:39Z

Source/WebCore/platform/graphics/gstreamer/mse/AppendPipeline.cpp

+            if (!result)
+                break;
+
+            gst_pad_add_probe(demuxerSrcPad.get(), GST_PAD_PROBE_TYPE_EVENT_DOWNSTREAM, reinterpret_cast<GstPadProbeCallback>(+[]([[maybe_unused]] GstPad* pad, GstPadProbeInfo* info, gpointer) -> GstPadProbeReturn {


nit: this is too big of a lambda and would be better off as a separate static function. You could call it aacSbrForceImplicitSignalling or something of the like.

ntrrgc · 2025-10-08T12:29:52Z

Source/WebCore/platform/graphics/gstreamer/mse/AppendPipeline.cpp

+
+                GST_DEBUG_OBJECT(pad, "AAC SBR or PS object type detected in codec data, replacing it with a LC object type. The decoder should be able to handle this properly");
+
+                auto channels = gst_codec_utils_aac_get_channels(data.data(), data.size());


I was concerned that gst_codec_utils_aac_get_channels() could be too high level and actually parse PS as well, but it doesn't. So as long as it is not changed afterwards, this would work.

Personally I would have just read the bits myself since it's not more difficult than reading the frequency bits.

Well I think we can consider that pbutils API behaviour stable, its doc clearly implies the value return should conform with the spec.

ntrrgc · 2025-10-08T12:37:55Z

Source/WebCore/platform/graphics/gstreamer/mse/AppendPipeline.cpp

+
+                GRefPtr<GstBuffer> codecDataBuffer;
+                gst_structure_get(structure, "codec_data", GST_TYPE_BUFFER, &codecDataBuffer.outPtr(), nullptr);
+                GstMappedBuffer data(codecDataBuffer, GST_MAP_READ);


Also return early with GST_PAD_PROBE_OK and a warning if the length of codec_data < 2, since otherwise the only length checks are asserts, which are not compiled in release.

ntrrgc · 2025-10-08T12:41:22Z

The EWS caught an assertion being broken in media/media-source/media-source-init-segment-swap-track-ids.html:

STDERR: ASSERTION FAILED: mediaType == "audio/mpeg"_s

webkit-early-warning-system · 2025-10-08T13:22:29Z

EWS run on current version of this PR (hash 4fb454f)

Details

Misc	iOS, visionOS, tvOS & watchOS	macOS	Linux	Windows
❌ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	✅ 🛠 wpe	✅ 🛠 win
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-AS-debug	✅ 🧪 wpe-wk2	~~🧪 win-tests~~
✅ 🧪 webkitperl	✅ 🧪 ios-wk2	✅ 🧪 api-mac	✅ 🧪 api-wpe
	✅ 🧪 ios-wk2-wpt	✅ 🧪 mac-wk1	✅ 🛠 wpe-cairo
	✅ 🧪 api-ios	✅ 🧪 mac-wk2	✅ 🛠 gtk
	✅ 🛠 vision	~~🧪 mac-AS-debug-wk2~~	✅ 🧪 gtk-wk2
	✅ 🛠 vision-sim	✅ 🧪 mac-wk2-stress	✅ 🧪 api-gtk
✅ 🛠 🧪 merge	✅ 🧪 vision-wk2	✅ 🧪 mac-intel-wk2	✅ 🛠 playstation
	✅ 🛠 tv	✅ 🛠 mac-safer-cpp
	✅ 🛠 tv-sim
	✅ 🛠 watch
	✅ 🛠 watch-sim

ntrrgc

r+

…stagram https://bugs.webkit.org/show_bug.cgi?id=299252 Reviewed by Alicia Boya Garcia. 299318@main introduced a performance regression and also broke Instagram stories audio playback. Instead of forcing ADTS framing we now rewrite the codec_data supplied by the demuxer only if the underlying AAC audio_object_type was set to SBR or PS, which the avdec_aac decoder doesn't handle very well. When any of those object types is detected, we replace it with an LC (Low Complexity) identifier and re-use the existing sampling frequency and channels bits from the previous codec_data. avdec_aac handles LC well and there is no noticeable rendering issue on Kaltura and Instagram stories anymore. Thanks to Alicia Boya Garcia <aboya@igalia.com> for her precious help during the investigation. * Source/WebCore/platform/graphics/gstreamer/mse/AppendPipeline.cpp: (WebCore::aacGetAudioObjectType): (WebCore::createOptionalParserForFormat): Canonical link: https://commits.webkit.org/301209@main

webkit-commit-queue · 2025-10-08T15:50:56Z

Committed 301209@main (4a3ebdf): https://commits.webkit.org/301209@main

Reviewed commits have been landed. Closing PR #51805 and removing active labels.

aperezdc · 2025-10-08T21:26:16Z

Backported into webkitglib/2.50 as commit 8d06309

philn self-assigned this Oct 4, 2025

philn added the Platform Portability improvements and other general platform improvements not driven directly by site bugs. label Oct 4, 2025

philn requested a review from ntrrgc October 4, 2025 11:14

webkit-ews-buildbot added the merging-blocked Applied to prevent a change from being merged label Oct 4, 2025

philn force-pushed the eng/299252 branch from def6a96 to 689f0a6 Compare October 4, 2025 13:44

philn removed GLib Suggested Backport - 2.46 Suggest this merge request be backported to webkitglib/2.46 branch GLib Suggested Backport - 2.48 Suggest this merge request be backported to webkitglib/2.48 branch merging-blocked Applied to prevent a change from being merged labels Oct 5, 2025

philn force-pushed the eng/299252 branch from 689f0a6 to 796d2cb Compare October 5, 2025 09:45

ntrrgc requested changes Oct 7, 2025

View reviewed changes

philn force-pushed the eng/299252 branch from 796d2cb to e9a9902 Compare October 8, 2025 10:40

webkit-ews-buildbot added the merging-blocked Applied to prevent a change from being merged label Oct 8, 2025

ntrrgc approved these changes Oct 8, 2025

View reviewed changes

philn removed the merging-blocked Applied to prevent a change from being merged label Oct 8, 2025

philn force-pushed the eng/299252 branch from e9a9902 to 4fb454f Compare October 8, 2025 13:22

ntrrgc approved these changes Oct 8, 2025

View reviewed changes

philn added the merge-queue Applied to send a pull request to merge-queue label Oct 8, 2025

webkit-commit-queue force-pushed the eng/299252 branch from 4fb454f to 4a3ebdf Compare October 8, 2025 15:50

webkit-commit-queue merged commit 4a3ebdf into WebKit:main Oct 8, 2025

webkit-commit-queue removed the merge-queue Applied to send a pull request to merge-queue label Oct 8, 2025

philn deleted the eng/299252 branch October 8, 2025 17:21

aperezdc removed the GLib Suggested Backport - 2.50 Suggest this merge request be backported to webkitglib/2.50 branch label Oct 8, 2025


		GST_DEBUG_OBJECT(pad, "AAC SBR or PS object type detected in codec data, replacing it with a LC object type. The decoder should be able to handle this properly");

		auto channels = gst_codec_utils_aac_get_channels(data.data(), data.size());

REGRESSION(299318@main): [GStreamer][MSE] Audio playback broken on instagram #51805

REGRESSION(299318@main): [GStreamer][MSE] Audio playback broken on instagram #51805

Uh oh!

Conversation

philn commented Oct 4, 2025 • edited by webkit-early-warning-system Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

4a3ebdf

Uh oh!

webkit-early-warning-system commented Oct 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

webkit-early-warning-system commented Oct 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

webkit-early-warning-system commented Oct 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ntrrgc left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

webkit-early-warning-system commented Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ntrrgc left a comment

Choose a reason for hiding this comment

Uh oh!

ntrrgc Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

ntrrgc Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

philn Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

ntrrgc Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

ntrrgc commented Oct 8, 2025

Uh oh!

webkit-early-warning-system commented Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ntrrgc left a comment

Choose a reason for hiding this comment

Uh oh!

webkit-commit-queue commented Oct 8, 2025

Uh oh!

aperezdc commented Oct 8, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

philn commented Oct 4, 2025 •

edited by webkit-early-warning-system

Loading

`4a3ebdf`

webkit-early-warning-system commented Oct 4, 2025 •

edited

Loading

webkit-early-warning-system commented Oct 4, 2025 •

edited

Loading

webkit-early-warning-system commented Oct 5, 2025 •

edited

Loading

webkit-early-warning-system commented Oct 8, 2025 •

edited

Loading

webkit-early-warning-system commented Oct 8, 2025 •

edited

Loading