Skip to content

New count() function (Counter Rework)#5716

Closed
jbirnick wants to merge 1 commit intotypst:mainfrom
jbirnick:count
Closed

New count() function (Counter Rework)#5716
jbirnick wants to merge 1 commit intotypst:mainfrom
jbirnick:count

Conversation

@jbirnick
Copy link
Contributor

@jbirnick jbirnick commented Jan 19, 2025

This adds a new hierarchical count() function which will both enable dependent counters and simplify the Rust code for counters at the same time.

I propose that this mechanism will replace the counter element/type, with the reasons and mechanism explained below.
This would be a breaking change and proceed in ~3 phases:

  1. Add the count function. (this PR)
  2. Replace the default show rules of heading, figure, etc. with count and deprecate counter. (future big PR)
  3. Fully remove counter.
OPEN ME: Motivation and thoughts behind it.

The main issue with current counters is that one cannot define e.g. a theorem counter that inherits the first level of the heading counter.

I thought a lot about how a good counter design should look like, while also being idiomatic/nice/efficient to implement.

I came to the conclusion that the counter type has the following flaw which hinders it from easily supporting dependent counters: Stepping and getting a counter happen through the same object.

A typical math document consists of multiple levels of headings and a counted theorem environment, which however only inherits the first level of the heading counter. So counting-wise it's like a tree: The basis are the headings of level 1, and based on that count both headings of level two (and higher) and theorems. Now if you would want to model this with something like counter, you would have something like counter(heading.where(level: 1), heading.where(level: 2) and counter(heading: where(level: 1), "mytheorem"). This models the "get" side very well, this (rough) syntax describes exactly what the user wants. But the problem is that stepping is tied to a counter, i.e., tied to the "getting". In this case, each heading would need to step both of the counters.

The resolution.
Instead, it is naturally the case that stepping and getting happen on different objects.
What you step are the little units, like headings of level 1, or "Theorem", or "Lemma". When you step this, you don't even care in what context it will be counted at some point, you only want to make clear that you insert a new element of something.
Conversely, when you "get"/count, then you are combining a few different of those little units and want to have them counted in a hierarchical way.
So to reflect this in Typst, we should get rid of one combining counter object, and split it into one "marking" and "counting". Luckily, we don't even need a special element for "marking", because we already have elements/labels/metadata, which are pretty much exactly that.
Then we only need a way to count hierarchically, which is what this PR provides with the count() function.

OPEN ME: The `count` function.

The API of count() is very simple. You just pass a list of targets (selectors), and it counts them hierarchically.
That is, it will first count all the elements of the first selector. Then it will count all the elements of the second selector, but only those which appear after the last element of the first selector. And so on.
So what is currently counter(heading).get() would then be something like:

count(heading.where(level: 1), heading.where(level: 2), heading.where(level: 3))

Now if you want to count your theorems as explained before, just tag them with some label (or in the future they will be elements themselves), and do:

count(heading.where(level: 1), <my_theorem_label>)

The function returns an array of integers, which can then be displayed using numbering(...).

OPEN ME: But how do I do X and Y then?!

Now this also throws away the update function. But this is not a problem! I have seen update used in only two ways:

  1. To reset the counter to zero, e.g. for the headings in the appendix.
    With count() you can achieve the same thing, in a semantically much better way. Just put a metadata/label somewhere to indicate that the appendix begins (e.g. metadata(()) <appendix_start>), and then use count(after: <appendix_start>, ...).
  2. To do some actually complicated update stuff. But well, I think then you cannot call it simply "counting" anymore, and you can simply use state.

Another question is maybe what the default show rule of heading will be, because it somehow has infinitely many levels but count() can only take finitely many arguments. The answer is, the show rule for a heading of level k will roughly incorporate count(heading.where(level: 1), ..., heading.where(level: k)). There is still the issue with ignoring headings that don't want to be numbered, that's discussed below.

Now to the issues / points of discussion / todos.

  1. One problem is, how do you get a selector for all headings/figures/etc. which have numbering != none? You would want a selector count(figure.where(numbering !: none). I guess one could introduce this syntax, but that would be pain on adapting parsing (I think right now it's just parsed as a dict). Or use a syntax like figure.wherenot(numbering: none).
    Another solution would be to introduce a new boolean field counted which is automatically computed from whether numbering is something or none.
  2. Another problem is, how to easily adapt the default show rule of headings, given that it's so complicated then.
    For example, if you want to make an appendix as explained above, you don't want to manually change the show rule of heading with code that lists the selectors up until the level of the heading. One option would be to introduce a new field on heading called count-after which is passed as count(after: this.count-after, ...) in the default show rule. But there are also other solutions, I hope we can find a better one.
  3. I guess the pages selector needs to be handled separately, but should be doable?!
  4. What should the syntax be for counting up until the current location vs some specified location? (like get() vs at() for counter) By default until the current location, and have an optional parameter for at/before?
  5. Should there be syntax to make the before/after inclusive/exclusive?
  6. Must-do before merging:
    • add tests
    • add some way of counting at a different location (e.g. through at/before argument)

Let me know what you think! I know the issues seem annoying, and indeed they are, but thinking about this for so long made it clear to me that there definitely needs to be a separation of objects between stepping and counting!

@jbirnick jbirnick marked this pull request as draft January 19, 2025 13:33
@jbirnick jbirnick marked this pull request as ready for review January 19, 2025 13:35
@jbirnick jbirnick marked this pull request as draft January 19, 2025 13:36
@T0mstone
Copy link
Contributor

Some thoughts:

  • What about infinity? Currently, the heading counter has infinite components and you could use a level-35123 heading if you wanted to (tho the output would be basically unusable). I guess every heading would count up to its own level, but there might be something else where this distinction creates an issue.
  • Isn't this just query? What's the performance impact of changing the system like this? I was under the impression that counters were more efficient.
  • Same thing with the point where you say to use state instead: Isn't that less efficient?
  • In fact, the count function could be implemented in typst code today. (Would probably be even slower tho.) Just query all the argument selectors (chained with or) and iterate through the result.

@jbirnick
Copy link
Contributor Author

jbirnick commented Jan 20, 2025

What about infinity? Currently, the heading counter has infinite components and you could use a level-35123 heading if you wanted to (tho the output would be basically unusable). I guess every heading would count up to its own level, but there might be something else where this distinction creates an issue.

I discussed this in my post above, indeed every heading counts up to its own level. The only other issue I see so far is how to change the default show rule for headings then. You would have to recreate the "count up to my own level". Maybe one could introduce a sub element of heading which has the computed counter value, so users don't have to worry about recreating the counter when they style their headings.
Again, I agree this is an annoyance. But Typst needs native dependent counters at some point, and this is the best design I can think of after thinking about it for really long (and writing packages, see below).
I'm also annoyed by these little issues, that's why I post here to find the best solutions.

In fact, the count function could be implemented in typst code today. (Would probably be even slower tho.) Just query all the argument selectors (chained with or) and iterate through the result.

You are absolutely right! In fact, I am the creator of the only package that offers dependent counters (see here) where I have basically implemented this, just with a different API. However, implementing it with after() vs what's in this PR makes it less efficient due to caching stuff, essentially it's about this.
And making it native is necessary to make it the standard. You can't really use both counter and a package like rich-counters at the same time. Well, you can, many people like me are doing it for their documents. But it fails in edge cases. For example, one cannot query for a counter update like counter(heading).update(0) which resets the page counter. Or any other fancy updates. You can only query for actual headings. Fundamentally, count(heading) and counter(heading) count different things. The former counts actual headings, while the latter counts "counter update" elements, or rather, computes what a sequence of those updates yields.

Isn't this just query?

Yes this is just using query in a clever way. :) Which is great, because it simplifies the code.

What's the performance impact of changing the system like this? I was under the impression that counters were more efficient.

No, that's the great point! This is (essentially) not less efficient than counters. If you look at the implementation of counters, you will in fact find that it uses the counting that I'm doing (though only on one level) as a sub step when it computes the counter. Let's think about counting all figures. It works like this:

In the document there are the counter update elements, which in 99% of the cases will just do a +1. Imagine it as an array like this:

[+1, +1, +1, +1, +1, +1, +1, +1, reset to zero, +1,+1,+1]

Then Typst looks at all those update elements, and produces a sequence of states after every update.

[1,2,3,4,5,6,7,8,0,1,2,3]

Now when you call counter(figure).get(), then Typst will count (!) all figures update elements up until the current location. Let's say these are 4 elements. (The first four +1.) Then it will just look at the second array at index four and output this.

Now count() on the other hand will just do a subset of this work. It doesn't produce/need any arrays. It only directly counts figures (which in the old implementation is done anyway), but outputs the count directly. This forbids any fancy update stuff. But update(0) is still possible by just counting only after a particular element, and any actual fancy updates do not fall into "counting" anymore and you can just use state (which works by the exact same principle). At the same time, this new simplification allows for dependent counters.

Same thing with the point where you say to use state instead: Isn't that less efficient?

No, if I'm not mistaken, counter and state are implemented in the exact same way. That's in some sense my critique, because such a state is tied to a specific sequence like (heading level 1, theorem), which is not the intrinsic semantics of counting and poses problems in implementation for arbitrary dependent counters because you need a separate state/counter for every combination of counters.

@T0mstone
Copy link
Contributor

Alright, that's good to hear. In that case I have no further objections. The issue with insanely deep headings is probably best ignored for now. I don't think it's very likely to come up and if it does, we can think about a solution at that time.

@laurmaedje
Copy link
Member

I need a bit of time to gather my thoughts before leaving a comment here.

@laurmaedje laurmaedje added interface PRs that add to or change Typst's user-facing interface as opposed to internals or docs changes. introspection Related to the introspection category labels Mar 24, 2025
@laurmaedje
Copy link
Member

laurmaedje commented Aug 1, 2025

I think it's a very valid observation that "Stepping and getting a counter happen through the same object." is a flaw when viewed through the lens of dependant counters.

The main issue I see with the proposed design is that it disconnects different usages of the same counter: There is no way anymore to affect all usages of a specific counter. If I say counter(heading).update(0), this will reset the heading numbering everywhere --- in headings, the outline, etc. Unless I'm mistaken, in your metadata-based design, to skip a number, I would have to overwrite all of those show rules. The upside of "Stepping and getting a counter happen through the same object." is that there is a well-defined object that I can use to update the view of every place that observes it.

I'm also a bit concerned by the user-facing complexity problem with deeper headings (conceptually I think your approach makes sense) and the API design questions of the count function (get/at, before/after), but my main concern is what I said above.

What I'm not sure about yet, is whether it's really not possible to extend the existing counter construct with multiple keys (e.g. counter(heading, theorem)) in a way where both getting and stepping makes sense. Getting would essentially work like a mix of your design and the existing counters and stepping would perhaps affect all counters with a shared prefix of keys or something like that. I must admit, I didn't think about this for very long, so if you have some insights on why this won't work, please let me know.

@jbirnick
Copy link
Contributor Author

jbirnick commented Aug 1, 2025

Yeah I can understand your points. I think there is a fundamental dilemma:

  • I guess we both agree that separate objects for "set" and "get" are more idiomatic from a programmers perspective: This is simply the natural way how hierarchical counting works, and it fits better with Typst's functional/pure style. (There is no state needed in this approach.)
  • But this seems to come with little annoyances which might make things a little more complicated for the user.

However, the "separate state for every counting combination" approach also has their annoyances. As you said, you somehow need to maintain a global list of all counter combinations that the user uses (which grows exponentially), have a state for each of them, and every e.g. heading would need to update multiple states in that list. You waste memory and computation by storing things redundantly. It's just ugly from a "theory perspective".

I think your problem regarding "reset headings once to zero at it should affect all counters that depend on the heading counter" can be solved. For example there could be a "headings reset here" element/metadata, which would automatically be considered by the count(...) function. (There could be something like this for every selector.)

But there are still the other little annoyances I mentioned above which would need a nice solution.

Ultimately, I just thought that we should go with the idiomatic approach that I sketched, because I always feel like using the non-idiomatic approach will backfire at some point. But yeah, it will need some ideas regarding a good UX design and necessarily everyone's brain will need to adapt to the new approach.

@laurmaedje
Copy link
Member

However, the "separate state for every counting combination" approach also has their annoyances. As you said, you somehow need to maintain a global list of all counter combinations that the user uses (which grows exponentially), have a state for each of them, and every e.g. heading would need to update multiple states in that list. You waste memory and computation by storing things redundantly. It's just ugly from a "theory perspective".

I think keeping the counter API and extending it to support multiple keys does not necessary imply that each counter track behaves like a fully independent state or that we have a global list. I think counter(heading.where(level: 1), theorem).get() could work very closely to how you envisioned count to work implementation-wise. And counter(heading).update(2) would still be a single element, but it would be taken into consideration by counter(heading, theorem).get().

@jbirnick
Copy link
Contributor Author

jbirnick commented Aug 1, 2025

It seems like now you want to mix the two approaches.

  • You want that counter(heading) has an associated state.
    (It looks up the results of get() in a precomputed array.)
  • You want that counter(heading, theorem) does not have an associated state.
    (It computes the results of get() ad-hoc, which is what I suggest.)

Is that correct? Wouldn't it be better if there was a unified approach?

@laurmaedje
Copy link
Member

I'm not so much thinking in terms of something having an associated state and more just of expanding which counter get functions are affected by which counter update elements.

If we have counter(..keys), then

  • its get function would be affected by elements and explicit counter updates for each k in keys
  • the element returned by its update function:
    • if let [k] = keys (only one key): would affect any other counter get calls that reference k
    • if there are multiple keys, potentially it could affect get calls that have keys as a prefix or subsequence of their own keys. Or maybe it's just forbidden to create multi-key updates. I haven't thought too much about this part.

I'm not yet sure how the implementation specifically would look (ad-hoc vs precomputed array). We'd have to see which makes more sense. But the above seems implementable to me.

@jbirnick
Copy link
Contributor Author

jbirnick commented Aug 1, 2025

Yeah, updates to counters with multiple keys are forbidden. That's exactly the point I'm trying to make:

  • You're "stepping" or "updating" or "placing" things with a single key.
  • You're "querying" or "getting" or "counting" a list of keys.

That's simply the nature of hierarchical counting. These two things should be separated, and that's what I'm "fighting" for.

Now for the "querying" part, I feel like we're on the same page. Let's assume you don't use a precomputed array but do it ad-hoc. At that point, what's the value of the object counter(key1, key2) ? It doesn't have any data associated to it. It only exists to call .get() on it. (Recall that step() or update() is called for single keys, not for this multi-key object.) But then you could just as well write count(key1, key2). That's the whole idea behind the count() function. You only need a function, not an object, for the "querying". I feel like you somewhat agree with that part.

The other thing I'm suggesting, and I feel like this is what you're concerned about, is how we handle the "step/update/place" part. I feel like you still want something like counter(heading).update() or counter(theorem).step(). I was suggesting that we could just use the elements directly for counting. Like literally count heading elements instead of heading-counter-update elements. I guess that's where you disagree. Because you want the ability to place update: 0 elements, which is not possible if we just count actual heading elements.

So I guess we could:

  • Still have special "counter step" elements or "counter update" elements. For single keys. For the "stepping" part of counters.
  • Have a count() function which hierarchically counts these special elements (or even considers more complicated updates than just counting). For multiple keys. For the "querying" part of counters.

@laurmaedje
Copy link
Member

That's the whole idea behind the count() function. You only need a function, not an object, for the "querying". I feel like you somewhat agree with that part.

I agree with your mental model of counters, I think where we disagree is just the concrete API.

As for the API, I see value in keeping the counter object for the following reasons:

  • It sidesteps your get vs at dilemma (there is also final)
  • It keeps counter and state symmetrical which is good for learning
  • It keeps the updating / stepping / getting all in one place, which is good for discoverability
  • It is non-breaking while removing it is extremely breaking. I'm willing to make large breaking changes, but there should be a clear benefit

I was suggesting that we could just use the elements directly for counting.

I mean, we already do that on main. A heading element does not produce a counter update by default. Rather, it is directly considered when resolving counter(heading). The counter.update elements are just an additional way to synthetically affect the counter and I'd like to keep that functionality as I think it's quite useful.

@jbirnick
Copy link
Contributor Author

jbirnick commented Aug 3, 2025

Setting and getting in one place might be good for beginners but I think it could become confusing when you make complicated things, because that is simply not how multi-level counters work "by nature".

But more importantly: How exactly would you make the API? Because it seems like you will always need 2 objects and there is (almost) no way around that. For example, take the case where you want have theorems numbered depending on the first heading. Then you will definitely create counter(heading.where(level: 1), "theorem"), right? But as you already noted, you can't step this thing. (What should stepping do?) So you necessarily need another counter("theorem"), and then there is no more value in having both getting and setting on the same object.

The only solution I see to this is to allow stepping on multi-key counters, and it will always step the last key. I guess that could work, and fits your API preferences?

@laurmaedje
Copy link
Member

The only solution I see to this is to allow stepping on multi-key counters, and it will always step the last key. I guess that could work, and fits your API preferences?

Yeah, that would work I think.

@T0mstone T0mstone mentioned this pull request Aug 5, 2025
@laurmaedje
Copy link
Member

What I still wonder is how all of this would be put into practice when I want to use top-level section numbers in my equations (#2652) without rebuilding equation numbering from scratch.

@jbirnick
Copy link
Contributor Author

jbirnick commented Aug 7, 2025

I don't know how equation numbering works on the inside, but how I would expect/suggest that it works is:

  • it shouldn't be any different from the usual counting, it should just count some special elements, that are embedded in the content, up until the current location
  • for section-dependent counters, allow equation to take a dependent-keys argument where you pass e.g. (heading.where(level: 1), heading.where(level: 2)) and then it will just prepend this to it's own counter

Or what is the problem that I don't see yet?

@jbirnick
Copy link
Contributor Author

jbirnick commented Aug 7, 2025

By the way, just a note for the implementation. When we pass heading.where(level: 1) somewhere, this is a selector which only selects actual heading elements. But it should also capture the respective "update elements" like counter(heading.where(level: 1)).update(0). So how these "update elements" are represented internally must be something like they store a selector, and then when we do the counting we will look for:

  • the key itself
  • any update elements that store the key

And another side note. We can't really do counter(heading).update(...) anymore, or at least it would require very special code.. (ugly). Because note that heading itself has "multiple levels". But this should really be allowed anymore. The individual units should be single-number. Then you can compose multiple such units. I guess we could still support counter(heading).get(...) by just treating it as an "expansion" to counter(heading.where(level: 1), ...), but I think the update function should only take in a single number, never something like (0,3).

@T0mstone
Copy link
Contributor

T0mstone commented Aug 7, 2025

Because note that heading itself has "multiple levels". But this should really be allowed anymore. The individual units should be single-number.

I disagree. In a call like counter(a, b, c), only a and b have to be finite-depth; c can still be infinite and this would also work well with update and step.

@laurmaedje
Copy link
Member

for section-dependent counters, allow equation to take a dependent-keys argument where you pass e.g. (heading.where(level: 1), heading.where(level: 2)) and then it will just prepend this to it's own counter

Or what is the problem that I don't see yet?

I'd kinda like to avoid adding a dependent-keys argument to every function that takes a numbering.

@T0mstone
Copy link
Contributor

T0mstone commented Aug 7, 2025

How about letting the numbering argument itself be either a string or a dictionary (pattern: str, prepend: array<counter>)?

@laurmaedje
Copy link
Member

What if I don't want to prepend though? And also, it would be good to have control over the separators.

@jbirnick
Copy link
Contributor Author

jbirnick commented Aug 7, 2025

What if I don't want to prepend though?

You can support both dictionary and pattern string.

You need to put an argument somewhere, right; at some point the user needs to transfer the information of how they want it to appear. How do you imagine the API?

I'd kinda like to avoid adding a dependent-keys argument to every function that takes a numbering.

Come on, isn't it just equation that would be relevant in 99% of the cases?

@laurmaedje
Copy link
Member

You can support both dictionary and pattern string.

What I meant was "what if I want to append or do some other thing" rather than preprend vs do nothing. I wasn't entirely clear.

You need to put an argument somewhere, right; at some point the user needs to transfer the information of how they want it to appear. How do you imagine the API?

Yeah of course. I just think from a user PoV something like numbering: "{heading}.1.1" (as proposed in #2652) is much easier to grasp and also more flexible than a dependent-keys argument. I'm not a huge fan of this particular design as it kind of puts code in the string in a sort of pseudo-interpolation syntax, which is a bit confusing in itself, but something isomorphic to that goes more into the direction I'm thinking.

Come on, isn't it just equation that would be relevant in 99% of the cases?

Perhaps, but I think it's needlessly restrictive. We can manage to come up with a more general design.

@laurmaedje
Copy link
Member

I'm optimistic that we'll find a good design for dependent counters, but I'm not really convinced that the proposed count function and the removal of the counter type are the right way forward for the various reasons mentioned in the discussion above. Hence, I will close the PR. Thank you still for your effort and insights!

@laurmaedje laurmaedje closed this Oct 3, 2025
@laurmaedje laurmaedje mentioned this pull request Feb 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

interface PRs that add to or change Typst's user-facing interface as opposed to internals or docs changes. introspection Related to the introspection category

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants