When you set a color, it currently requires two clicks to reset it:

This feels like an operation that might be better served by a single-click procedure. Perhaps we could add the reset (-) icon to the color button?

While we're here, it might also be good to reduce the vocabulary. Notice how the popover has a "Clear" action, while the design tools popover talks about "Reset"ing.