It is currently not possible to delete a block without having to select the contents. A delete button has been added to the single instance TinyMCE prototype. It will replace the contents with an empty slot and set the selection there.

https://wordpress.github.io/gutenberg/tinymce-single/