This seemed useful for me when I was testing the amount of space that fields take up. I'm happy to remove it from this PR if we'd like.

Yeah this will be a surprise to users relying on defaults, I think we should try to keep this default value.

++

This implements using my grok prototype. I'm 10000% sure we aren't going to keep this API, but it is something.

And this is how things used to be.

@nik9000 In order to visualize the queries in https://elasticsearch-benchmarks.elastic.co/#tracks/http-logs/nightly/default/30d we will actually need a second challenge. The query visualizations and the chart generator will not currently visualize more than one operations in the same chart (plus it'd get very noisy given that they already visualize 4 percentiles).

So I suggest we create a new challenge e.g. append-no-conflicts-with-runtime-fields or something like that, where we have all the content of the append-no-conflicts challenge plus the operations in the conditionals here. We can skip the if in the new challenge.

I have this condition that I call copy-and-paste blindness. If I see a bunch of stuff that looks copy and pasted I can never find the small bits that differ. Is there any way I can write this as a function and call it twice to get two named challenges? That way the if statement stay? Otherwise I'll never be able to find the difference.

I hear you. You can leverage the {{ rally.collect(parts="") }} helper to reference common sections.

Example approach:

1. Create a common dir under challenges, e.g. like:

├── challenges
│   ├── common
│   │   └── default-schedule.json
│   └── default.json

2. Move the the schedule from the append-no-conflicts challenge to common/default-schedule.json.

3. Change the append-no-conflicts challenge to be:

     {
       "name": "append-no-conflicts",
       "description": "Indexes the whole document corpus using Elasticsearch default settings. We only adjust the number of replicas as we benchmark a single node cluster and Rally will only start the benchmark if the cluster turns green. Document ids are unique so all index operations are append only. After that a couple of queries are run.",
       "default": true,
       "schedule": [
         {{ rally.collect(parts="common/default-schedule.json") }}
       ]
     },
   

4. Create a second challenge with a different name + description, same content, that relies on use specifying the runtime_script_grok track parameter.

This is just an idea, one could split sections from the schedule of the default challenge to more files and include as required.

Yeah this will be a surprise to users relying on defaults, I think we should try to keep this default value.

Similarly I think we should limit the amount of files that contain conditionals. I suggest we have a separate index-runtimefields.json and apply it conditional in the central track.json where we anyway have some conditionals.

I hadn't realized that was a thing. I'm happy to do that.

Maybe we can add in the description that this relies on setting the track param runtime_script_grok to true.

nit: add newline

With this level of indentation the rendered track by Rally looks like:

  "challenges": [
        {
      "name": "append-no-conflicts",
      "description": "Indexes the whole document corpus using Elasticsearch default settings. We only adjust the number of replicas as we benchmark a single node cluster and Rally will only start the benchmark if the cluster turns green. Document ids are unique so all index operations are append only. After that a couple of queries are run.",
      "default": true,
      "schedule": [
        {
  "operation": "delete-index"
},
{
  "operation": {
    "operation-type": "create-index",
    "settings": {}
  }
},

to make things look proper when rendered internally by Rally (you can check those rendered files in your $TMPDIR, they are useful for debugging) we should indent by 8 characters