Explain the problem.
#7562's move to babel has forced me to confront its use in the default template
Several issues arise in using babel over polyglossia

• Language loading from babel has some problems that I haven't been able to fully trace. Suffice to say that the wiki advice to pass provide=* does not work well enough to just dump the entire used-language list in it. What works consistently for me is to use \babelprovide[langopts]{lang}. Note that some of these (especially arabic, hebrew) might want to pass per-language options, cf end of the linked page and 1.23 in manual)
• lang: he-IL doesn't work under lualatex, complaining of Right-to-Left Support Error: use TeX--XeT or e-TeX engine.. If the global language setting (in documentclass) is removed, (and main passed to the main language's \babelprovide), this is no longer a problem.
• RTL languages need babel to be passed a bidi parameter, set to default under xelatex and basic under lualatex. pdflatex isn't supported by babel here. Note this also means the bidi package doesn't need to be invoked explicitly.
• Per-language fonts aren't supported. Worse, the header-includes expansion in the default template occurs before babel is imported, so a simple \babelfont invocation cannot be added in header-includes. However, if a single font covers all uses, this can be mitigated by just setting mainfont.

Pandoc version?
Self-built 6742e77 (2.14.2)