So far, this requirement:

{
    "require": {
        "ext-DOM": "*",
        "ext-dom": "*",
     }
{

passes both composer validate and composer normalize.

Composer itself normalizes those names (see source), I think we might normalize them the same way and prevent similar nonsense requirements.