Since the implementation of #7170 the editor buttons have no control anymore over the size of the modal popup.
You could do this before in the editors-xtd plugins render method:
$button->options = "{handler: 'iframe', size: {x:window.getSize().x-100, y: window.getSize().y-100}}";
This would get placed in the html rel attribute used by the Squeezebox modal.

Now the option is completely disregarded.

These plugins can also be having different scripts inside the modal to control the popup, like closing it on click.
These all now fail ungracefully

Even the core Article button doesn't work anymore (aside from the fact that it looks bad because of the missing padding in the modal).
It will insert the article, but not close the modal, with js error:

jModalClose is not defined

This is a serious B/C issue and this PR needs to get reverted, IMO.