A question based on the way the failing tests are set up; is this behavior actually intended?

A question based on the way the failing tests are set up; is this behavior actually intended?

Reading through the failing test, I believe it is intended for state to persist between calls to ts_parse_with_options:

• An initial parse is done with a callback that ensures it exits parsing while balancing is in progress
• The parser is reset.
• A fresh parse is started with the same callback as the original (intending for parsing to stop during balancing)
• parser.parse_with_options() is called again (no reset this time), with the assumption that the parsing state was preserved during calls.

diff --git a/lib/src/parser.c b/lib/src/parser.c
index 71cf7bdb..49c4d440 100644
--- a/lib/src/parser.c
+++ b/lib/src/parser.c
@@ -2238,9 +2238,8 @@ TSTree *ts_parser_parse_with_options(
   self->parse_options = parse_options;
   self->parse_state.payload = parse_options.payload;
   TSTree *result = ts_parser_parse(self, old_tree, input);
-  // Reset parser state & options before further parse calls.
-  self->parse_options = (TSParseOptions) {0};
-  self->parse_state = (TSParseState) {0};
+  // Reset callback before further parse calls
+  self->parse_options.progress_callback = NULL;
   return result;
 }

Nice! Thanks, that makes a lot of sense. So the patch you gave does work, or it still fails the neovim reproducer? Also, if we clear callbacks, should the patch be changed to the following?

diff --git a/lib/src/parser.c b/lib/src/parser.c
index adcdf8d0..0429ce9a 100644
--- a/lib/src/parser.c
+++ b/lib/src/parser.c
@@ -1540,7 +1540,7 @@ static bool ts_parser__check_progress(TSParser *self, Subtree *lookahead, const
   if (self->operation_count >= OP_COUNT_PER_PARSER_TIMEOUT_CHECK) {
     self->operation_count = 0;
   }
-  if (self->parse_options.progress_callback && position != NULL) {
+  if (position != NULL) {
     self->parse_state.current_byte_offset = *position;
     self->parse_state.has_error = self->has_error;
   }
@@ -2238,6 +2238,7 @@ TSTree *ts_parser_parse_with_options(
   self->parse_options = parse_options;
   self->parse_state.payload = parse_options.payload;
   TSTree *result = ts_parser_parse(self, old_tree, input);
+  self->parse_options = (TSParseOptions) {0};
   return result;
 }

that way the parser state is kept consistent even when parsing without a callback (not sure if that is strictly necessary)

So the patch you gave does work, or it still fails the neovim reproducer?

I believe it still fails the neovim reproducer. I was swapping between testing changes to neovim and tree-sitter locally, and I thought the reproducer passed (running make clean in between neovim builds), but I had a suspicion something was getting cached incorrectly. After re-cloning neovim to ensure a fully clean build with the changes the reproducer failed.

Also, if we clear callbacks, should the patch be changed to the following?

That seems correct, but I'm not 100% sure. I'll do some more testing locally to check it out :)

You need to make distclean (or manually delete the stamp file) to rebuild the dependencies in Neovim.

@ribru17 your latest patch is working for me locally! Mind pushing it up?

Successfully created backport PR for release-0.25:

• fix(lib): reset parser options after use #4361