[MSE] Video glitches when switching from lower quality to higher quality content

[filipe-norte-red]

During BBC test suite run (e.g. ACT-1070), video glitches / small temporary freezes can be occasionally observed. The glitches happen when a switch from lower quality content (e.g. bitrate, resolution) to higher quality content occurs, but not the other way around.

The high level sequence of events is the following:

• Playback start is triggered
• Low quality data segments are provided covering a sufficiently long period, e.g. [0, 40] seconds
• After a few seconds, e.g. at 10 seconds media time offset, the high quality data init segment is provided
• After a few seconds, e.g. at 15 seconds media time offset, a few segments of high quality data are provided starting from the time the init segment was provided (e.g. [10, 25] seconds)
• As soon as high quality data is provided for the currently playing position (e.g. 18 seconds media time offset), we switch to the high quality data playback
• video glitch is observed

The attached test case reproduces this behavior with wpe-webkit using custom streams (not bbc original). Same test case works without glitches on Firefox.

bbc-tests-video-glitch-custom-stream.20260407.1.zip

The video glitches can be seen on both wpe-2.46 and wpe-2.38, although the sequence here described and reproduction with the test app was confirmed only with wpe-2.46.

The analysis using original streams shows the following:

Lower quality data pushed with 25 fps, leading to total buffered range from 149.76 to 180.48:

0:02:43.678611755    27    0x4a920 TRACE              webkitmse AppendPipeline.cpp:773:pushNewBuffer:<append-pipeline-video-mp4-0> pushing data buffer buffer: 0xa5b3f690, pts 99:99:99.999999999, dts 99:99:99.999999999, dur 99:99:99.999999999, size 425862, offset none, offset_end none, flags 0x0    
0:02:43.684073705    27    0x4a920 TRACE              webkitmse AppendPipeline.cpp:794:pushNewBuffer:<append-pipeline-video-mp4-0> pushing end-of-append buffer buffer: 0xa5b66aa0, pts 99:99:99.999999999, dts 99:99:99.999999999, dur 99:99:99.999999999, size 0, offset none, offset_end none, flags 0x0

0:02:43.690383611    27    0x66e80 DEBUG                qtdemux qtdemux.c:9331:gst_qtdemux_guess_framerate:<qtdemux0> Calculating framerate, timescale 600 gave fps_n 25 fps_d 1 

0:02:43.787276919    27    0x4a920 INFO       webkitmediaplayer GStreamerCommon.cpp:613:didLogMessage: SourceBuffer::sourceBufferPrivateAppendComplete(86330001) buffered = [{149760000/1000000 = 149.76}..{180480000/1000000 = 180.48}] , totalBufferSize: 3075193

Init segment is pushed with higher quality data and 50 fps:

0:02:44.051806894    27    0x4a920 TRACE              webkitmse AppendPipeline.cpp:773:pushNewBuffer:<append-pipeline-video-mp4-0> pushing data buffer buffer: 0xa5b445e8, pts 99:99:99.999999999, dts 99:99:99.999999999, dur 99:99:99.999999999, size 687, offset none, offset_end none, flags 0x0
0:02:44.052169706    27    0x4a920 TRACE              webkitmse AppendPipeline.cpp:794:pushNewBuffer:<append-pipeline-video-mp4-0> pushing end-of-append buffer buffer: 0xa5faac98, pts 99:99:99.999999999, dts 99:99:99.999999999, dur 99:99:99.999999999, size 0, offset none, offset_end none, flags 0x0

0:02:44.060935478    27    0x4a920 DEBUG       webkitvideotrack VideoTrackPrivateGStreamer.cpp:131:updateConfigurationFromCaps:<webkitappsinkwithworkarounds1:sink> Updating video configuration from video/x-h264, stream-format=(string)avc3, alignment=(string)au, level=(string)3.1, profile=(string)high, codec_data=(buffer)0164001fffe000, width=(int)960, height=(int)540, pixel-aspect-ratio=(fraction)1/1
0:02:44.061535769    27    0x4a920 DEBUG       webkitvideotrack VideoTrackPrivateGStreamer.cpp:131:updateConfigurationFromCaps:<webkitappsinkwithworkarounds1:sink> Updating video configuration from video/x-h264, stream-format=(string)avc3, alignment=(string)au, level=(string)3.1, profile=(string)high, codec_data=(buffer)0164001fffe000, width=(int)960, height=(int)540, pixel-aspect-ratio=(fraction)1/1

0:02:44.067008460    27    0x4a920 INFO       webkitmediaplayer GStreamerCommon.cpp:613:didLogMessage: SourceBuffer::sourceBufferPrivateAppendComplete(86330001) buffered = [{149760000/1000000 = 149.76}..{180480000/1000000 = 180.48}] , totalBufferSize: 3075193  

Actual higher quality data follows around 3 seconds later for range 165.12 to 168.94 sec (= 2min48sec.940ms)!

0:02:47.433335464    27    0x4a920 TRACE              webkitmse AppendPipeline.cpp:773:pushNewBuffer:<append-pipeline-video-mp4-0> pushing data buffer buffer: 0xaa9a35e8, pts 99:99:99.999999999, dts 99:99:99.999999999, dur 99:99:99.999999999, size 1336346, offset none, offset_end none, flags 0x0

0:02:47.449566243    27    0x66e80 DEBUG                qtdemux qtdemux.c:9331:gst_qtdemux_guess_framerate:<qtdemux0> Calculating framerate, timescale 600 gave fps_n 50 fps_d 1

0:02:47.451445745    27    0x4a920 TRACE              webkitmse AppendPipeline.cpp:794:pushNewBuffer:<append-pipeline-video-mp4-0> pushing end-of-append buffer buffer: 0xa5b30738, pts 99:99:99.999999999, dts 99:99:99.999999999, dur 99:99:99.999999999, size 0, offset none, offset_end none, flags 0x0

0:02:47.451654891    27    0x4a920 DEBUG       webkitvideotrack VideoTrackPrivateGStreamer.cpp:131:updateConfigurationFromCaps:<webkitappsinkwithworkarounds1:sink> Updating video configuration from video/x-h264, stream-format=(string)avc3, alignment=(string)au, level=(string)3.1, profile=(string)high, codec_data=(buffer)0164001fffe000, width=(int)960, height=(int)540, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)50/1
0:02:47.451779149    27    0x4a920 DEBUG       webkitvideotrack VideoTrackPrivateGStreamer.cpp:131:updateConfigurationFromCaps:<webkitappsinkwithworkarounds1:sink> Updating video configuration from video/x-h264, stream-format=(string)avc3, alignment=(string)au, level=(string)3.1, profile=(string)high, codec_data=(buffer)0164001fffe000, width=(int)960, height=(int)540, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)50/1

0:02:47.455585744    27    0x4a920 TRACE              webkitmse AppendPipeline.cpp:542:appsinkNewSample:<append-pipeline-video-mp4-0> append: trackId=1 PTS={165120000/1000000 = 165.12} DTS={165120000/1000000 = 165.12} DUR={20000/1000000 = 0.02} presentationSize=960x540

...

0:02:47.586452523    27    0x4a920 TRACE              webkitmse AppendPipeline.cpp:542:appsinkNewSample:<append-pipeline-video-mp4-0> append: trackId=1 PTS={168920000/1000000 = 168.92} DTS={168940000/1000000 = 168.94} DUR={20000/1000000 = 0.02} presentationSize=960x540

0:02:47.605508054    27    0x4a920 DEBUG         webkitmediasrc WebKitMediaSourceGStreamer.cpp:686:webKitMediaSrcStreamFlush:<source:src_1> Sending FLUSH_START downstream.

This is the time where a video glitch / small freeze is seen, which is coincident with the flush being triggered.