What's the purpose of 100 elements if only a small subset is used in the benchmarks?

Hello, thanks for your feedback! We're wanting to introduce a suite of benchmarks that highlight some of the performance issues we've been experience with KeyPaths. What we've seen so far can be largely placed into three categories.

The first category involves a collection of KeyPaths not being inlined if they're spread too far apart in memory. The magic number, after some experimentation, happened to be 14. As soon as you have a 15th KeyPath, the inlining in release mode does not occur, and performance falls off a cliff -- being some 100 times worse or more. This is part of the reason why we chose a struct with 100 elements instead of just 10. (I do agree that asking the optimizer to inline KeyPaths pointing to an arbitrarily-wide range is wishful thinking, but perhaps the value of 14 is smaller than some might expect.) I've added a comment right above run_testKeyPathsInlined() which explains this in a bit more detail.

The second category involves nested structs or classes. Any sort of KeyPath projection operation is O(N) in the depth of the object. The benchmark KeyPathNestedStructs uses this.

The third category involves accessing a constant-sized value within structs of various sizes. Namely, performance decreases as more and more variables are added to the struct. I've added a benchmark called KeyPathsSmallStruct to showcase the difference in performance between a struct with 10 elements and one with 100 (KeyPathReadPerformance).

You can use @inline(never) to ensure you are testing the non-inlined case.

Do you still need the 100 elements when using @inline(never)

I'd like to keep it if possible: I've added the @inline(never) directive to both versions ofgetKeypathToElement() -- for the 10-element struct and also for the 100-element struct -- and compared the times for each iteration. In both Debug and Release mode (i.e., after compiling via swift build and swift build --configuration release), the 100- element struct needs about 1.5X more time per iteration when compared with the 10-element struct.

It would also be very useful to add benchmarks for other keypath variants, like getters+setters.

I'm wondering what is meant by this: does this mean adding benchmarks featuring ReferenceWritableKeyPaths, PartialKeyPaths, and others?
Or, benchmarks for KeyPath syntax like this?
get { ... #keyPath(...) }

By the way, this benchmark is already getting somewhat long (over 1300 lines). Would it be a good candidate for multi-source?

Example for a gettable property:

struct S {
  var x: Int { return 27 }
}

let kp = \S.x

Would it be a good candidate for multi-source?

No, keeping it in a single file is fine.

There are several kind of keypath components:

• stored property (what you are testing)
• gettable property (my previous example)
• settable property (the same, just for writing)
• tuple element
• optional chain
• optional force
• optional wrap

Those are handled differently in the runtime. Therefore it would be nice if the benchmarks would cover all of the keypath component types + combinations of them, e.g.

struct T {
  var y: Int
}

struct S {
  var x: (Int, T?) { return (27, T(y: 42)) }
}

let kp = \S.x.1?.y

Ah yes, I see that they're handled differently, especially inside _projectReadOnly<CurValue, NewValue, LeafValue>(...) in KeyPath.swift.
I'll create a few new benchmarks and verify I get all the cases listed above (and that all the cases are being hit inside that function). These'll be coming soon.

I've added benchmarks that'll test the performance of each type of KeyPathComponent. There is now a MarkDown table near the top of the file that indicates which benchmark stresses which type of KeyPathComponent.
I did find another error in the CI output indicating that the setup time for KeyPathWritePerformance was too high; I've moved its KeyPaths, and the ones for KeyPathReadPerformance, into the singleton.

I've updated the main message associated with the PR to discuss the benchmarks that have been added since then.
What is the convention for resolving conversations (via the Resolve conversation button) during workflows like this?

As you like. Just let me somehow know when the PR is ready for another round of review.

Sound good!

Do you still need the 100 elements when using @inline(never)

Why are the keypaths stored as class members?
If I understand correctly, for the purpose of this benchmark it would be sufficient to create the relevant keypaths in the benchmark functions - wrapped into identity()

I've addressed the 100-element issue in another post -- it boils down to 100 elements needing 1.5X more time per iteration that 10, at least as far as reads are concerned.
I've moved these KeyPaths into the class due to a single CI run where it was complaining about a high setup time in KeyPathWritePerformance. It was this one here:
https://ci.swift.org/job/swift-PR-macos-perf/260/console
The error in question was: ERROR runtime: 'KeyPathWritePerformance' has setup overhead of 68 μs (8.3%).
So the idea here was to move everything but the kitchen sink into the class.

As you like. Just let me somehow know when the PR is ready for another round of review.

You need to add identity outside of the loop, because it adds considerable runtime overhead:

let appendedKeyPath = identity(singleHopKeyPath0.appending(path: singleHopKeyPath1)
     .appending(path: singleHopKeyPath2).appending(path: singleHopKeyPath3)
     .appending(path: singleHopKeyPath4))

Please also correct this in all other functions.

Done!