Previously, we relied on detecting the language from file paths,
then using various regexes associated with the language to identify
token boundaries. However, the code mirror blob view always provides
a full token range, which can be used directly, instead of attempting
to recompute the token boundaries.

For older URLs, we fallback to simple identifiers, which should
work for the vast majority of languages and identifiers.

We cannot yet remove the language detection here because the file
extensions associated with the language are later used for search-based
code navigation.

This patch also makes the language spec optional for search-based
code intel, as we do not have a solution to #56376 which would
guarantee that we always have a language available. If a language
is not available, search-based code intel falls back to searching
other files with the same extension as a best effort guess.

Locally tested for MATLAB code. The ref panel shows up correctly,
unlike the error earlier.

(cherry-picked from c42cad2)