You can separate out rules and include them, which makes it easy to reuse them in various places. See https://github.com/pygments/pygments/blob/master/doc/docs/lexerdevelopment.rst, the example at "You can include the rules of a state in the definition of another. "

Using that construct to try to get closer to full syntax support, I find there are some inconsistencies with whitespace that I'm not sure how best to address.

Should spaces in the code be Token.Text or Token.Text.Whitespace? And, should contiguous whitespace be a single token ((Token.Text, '\n ')) or one token per character ((Token.Text, '\n'), (Token.Text, ' '), ...)?

Whitespace should be Text.Whitespace, that's what the more modern lexers tend to do anyways. I'd try to merge whitespace if possible but there's no requirement to do so. (I just realized this is also documented in tokens.rst.)

Ping. @drmoose, are you still working on this?

@apjanke Oh whoops. This thread completely slipped my mind. I've got a bunch of uncommitted changes and I was fighting with the inconsistent treatment of whitespace when I got distracted by other things .... aaaand that was apparently four months ago. Linear time is annoying.

Let's see how close I can get this to commit-worthy today...

That'd be great; thanks! It would be nice to have this to build on for my work on #1684.

@apjanke Do you have the code snippet you posted in #1684 available in a form other than a screenshot? I'd like to run through my local branch as a test case. I've made no effort to support the arguments syntax but I'm pretty sure I've got property constraints working.

Here's the test string committed in 53ca36c rendered to my console:

Oh, sure! Sorry for not posting code as text too, originally. I should know better.

classdef SomeClass < SomeOtherClass

  properties
    x (1,1) double = 42
    y
  end

  methods
    function this = SomeClass()
    end
  end

end

function anExampleFunction(foo, bar)
  arguments
    foo (1,1) double
    bar string = "whatever"
  end

  fprintf('Hello, world!\n')
end

It's from https://github.com/janklab/MatlabProjectTemplate/blob/main/doc-src-jekyll/index.md.

Not bad. As expected, arguments could be better, and it's getting confused about the type annotations

Here's a more thorough version:

classdef SomeClass < SomeOtherClass

  properties
    x (1,1) double = 42
    y
  end

  methods
    function this = SomeClass()
    end
  end

end

function anExampleFunction(foo, bar, baz, qux)
  arguments
    foo
    bar (1,1) double
    baz string = "whatever"
    qux string = "foo" {mustBeMember(qux, ["foo" "bar" "baz"])}
  end

  fprintf('Hello, world!\n')
end

Cool. If you want to leave off here, I can pick up on arguments and type annotations.

That's probably a good idea as I'm not terribly familiar with either syntax. Fork me if you want, but you now also have write access on my branch. :)

Hopefully you can just get this PR merged soon, and then I can work off of that. I'd rather not do separate PRs or pushes to your branch/PR – that can get messy, and I'm going to be doing a lot of screwing around and testing with this code.

@apjanke Please go ahead :)