There's some mixing of units here, which can be observed by zooming in on the text-decoration-trim-010.html testcase: the amount of trim (relative to the text size) varies with the zoom level (or, equivalently, with the devicePixelRatio of the screen).

I think the issue is that gfxFont::Metrics is in device pixels already, so the multiplication by scale is unnecessary and effectively means that the devicePixelRatio has been doubly applied.

I'd suggest something like max(aMetrics.emHeight * 0.125, scale) which should give a trim of 1/8th of an em, clamped to a minimum of one CSS px (rather than one device pixel, which could be excessively narrow on a high-res screen).

I think this gives the right result but is confusing, because trimLeft and trimRight want to be device pixels (see comment at struct DecorationRectParams), so getting them via ToCSSPixels() feels wrong.

It's actually working out because of the prior application of scale, but would be better done as something like

trimLeft = NSAppUnitsToDoublePixels(length.start.ToAppUnits(), app);

so that it doesn't look like we're putting a CSS px value in there.

These can be nscoord, can't they? AFAICS they're just used for (integer) appUnit values.

Can we skip computing and unioning the rect if this returns false? It seems like that should be OK -- if the decoration is going to be entirely trimmed away, we don't need to consider it part of the ink overflow area. (Or am I missing something?)