Currently, the block can only be dragged from the drag handle.
It has been suggested that removing the drag handle and allowing the user to drag the block from anywhere might be better user experience, that's also something that some other apps already allow.
This would, however, would introduce a new issue of not being able to "drag" inside the block for other reasons, such as selecting text for copying and pasting or selecting text for marking part of the text italic / bold, etc.
An alternative could be allowing dragging from the borders, in a similar way that Google Slides has, for example.
Related: #2252.