At least for H.263 video, the smoothing flag takes effect iff the video has exactly 1 frame. 🤦‍♂️
Why? 😕
Do we want to replicate this? 🤔
Here are a couple of random test files I made to discover this: h263_frame_smoothing_tests.zip

And it looks like it always applies to VP6 videos, regardless of the number of frames it has: vp6_frame_smoothing_tests.zip

For Screen (V1, I think) video, the same logic applies as for H.263: screen_frame_smoothing_tests.zip
And to make matters worse, in both cases, the actual number of frames counts, not the numFrames attribute of the DefineVideoStream tag.

Okay, I have added the weird logic for this flag, and provided a whole bunch of handcrafted files above to verify matching behavior.
Although, the num_frames attribute always matches the number of frames in the file, that could also be perturbed for more complete coverage.
For ScreenV2, I have no idea what happens, as I have no material to test it, and found no way to create any either.

Note that with the smoothing flag off, Flash Player produces major vertical glitching when magnifying videos, but there's no way I'm replicating that, as it appears to clearly be a programming mistake.

Seems like this depends on at least the SWF version: changing nemderulki_h263_2frames_smooth.swf to SWF version 8 allows it to be smoothed.

Looks like this is the way _quality works in pre-FP8. With the default _quality = "HIGH";, bitmaps/videos will only obey the smoothing parameter if they are static -- so I think this explains the case of 1-frame videos being smoothed. (In the case of bitmaps, they'd be unsmoothed if they were moving, say, via a tween).

The bitmaps and videos will be always smoothed if _quality = "BEST"; is added to frame 1.

In FP8+, the default "HIGH" quality was changed to always obey the smoothing settings for bitmaps and videos.

edit: As for what action to take, I'd be okay with defaulting to no-smoothing for video in SWF7 and below, like we currently do for bitmaps, until we properly hook up the logic for BEST.

For completeness, the table above also has a first half in the spec:

And indeed, the codec used wasn't the discriminator, it was the SWF version - which was set by ffmpeg depending on the chosen codec...