So looked a bit more into a design for this:

I would suspect that results could look somewhat like this:

{
  "task_name": "SomeRetrievalTask",
  "date": "2026-03-07T14:23:01.123Z",
  "evaluation_time": 45.2,
  "evaluation_phases": [
    {"name": "data_loading",     "start": 0.3,  "end": 2.3},
    {"name": "dataset_transform", "start": 2.3,  "end": 3.1},
    {"name": "encode_queries",   "start": 3.5,  "end": 15.7}, # notice that we can allow breaks for untracked phases
    {"name": "encode_corpus",    "start": 15.7, "end": 38.4},
    {"name": "scoring",          "start": 38.5, "end": 45.1}
  ],
...
}

Of course we can also track so far down and we probably don't want to overdo it. I would imagine the implementation would look somewhat like this:

timing = TimingStack()

# ...
# when you want to track a step:
with timing("load data"):
  dataset = load_data()
with timing("encode"):
  embeddings = model.encode(dataset)

We could even add some simple plotting functionality:

timing.quick_plot()

data_loading      |██                                                     | 2.3s
dataset_transform |  █                                                    | 0.8s
encode_queries    |        ██████████████                                 | 12.5s
encode_corpus     |                      ██████████████████████████████   | 22.7s
scoring           |                                                    ██ | 0.6s
                   45.2s (untracked: 6.3s

Uh oh!

Add evaluation runtime for indexing and retrieval time #4177

Description

Description of the feature

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions