There has been a few PRs recently adding a fair bit of stuff to generate_dir_rst and generate_gallery_rst, making them quite long and difficult to understand.

Prompted by trying to read the code after #1323.

A lot of simply pulling code out into helper functions, removing redundant things. Notable changes:

• generate_dir_rst now only writes index.rst file if src_dir is a subsection file
  • previously we over-wrote it later and had a confusing note in generate_gallery_rst about this
  • the old include_toctree already indicated whether it was a subsection or not, we just weren't fully utilising it
• some re-naming of ambiguous terms like 'has_header', which were all just code for whether the user provided an index.rst file or not
• Refactor of collect_gallery_files - this is opinionated so I am happy to revert or amend
  • Moves this helper to utils/ - to prevent circular import as I wanted to use it in gen_rst.py. This is technically a public function but a quick search on github seems to suggest only one or two cases of its use, most are forks of SG
  • We now use it twice:
    • get all example files and check for spaces and duplicates (original use)
      • we previously recursed through all sub folder depths. I've amended this to check one level of subfolder as we only make galleries one level deep, so we shouldn't care how the user has named deeper files (?)
      • we only checked .py files previously but with example_extensions it makes sense to check all supported exts
    • get example files to iterate through and generate rst for
      • it makes sense that this is done with the same function as the above check