The quick inserter has been designed to accommodate a pattern-only view which has never been turned on yet. Let's try using it when the quick inserter is accessed at the top layer of a template. The top level means it's a direct descendant of the document, between any root template parts or existing blocks. In the following example of a block tree there would be two such places:

As a reminder, the design for the quick pattern insert looks like this:

This would optimize for inserting patterns from the theme when building templates and start bringing patterns into stronger focus. In the future we need to improve the browsing and listing experience of this inserter.