Add dynamic URL to increase result accuracy

Hi @waghanza ,

When wrk performs the request "GET /user/:id" it always defines "GET /user/0", so frameworks that use cache are bypass to process the route, being that route process is one of the most important points to calc. As we will still maintain calls on two other routes ("POST /user" and "GET /"), the cache will still be evaluated, we will have at least 1/3 of the calls having to be completely processed without taking advantage of the cache.

The recommendation is to change the code in the "client.cr" file where it would perform an additional options["script"] if the method was GET and the uri was "/user/0".

https://github.com/the-benchmarker/web-frameworks/blob/58e650f265371dc42117da16030a622c5957a277/tools/src/client.cr#L70-L72

to something like:

      if method == "POST"
        options["script"] = File.expand_path("../../" + "pipeline_post.lua", __FILE__)
      end
      if method == "GET" && uri == "/user/0"
        options["script"] = File.expand_path("../../" + "pipeline_get_id.lua", __FILE__)
      end

The wrk script to create(pipeline_get_id.lua) the entire uri would look something like:

function init(args)
   requests  = 0
end

request = function()
   requests = requests + 1
   local path = "/user/" .. requests .. ""
   return wrk.format(wrk.method, path, wrk.headers)
end

I hope this helps the project in making the evaluation even more accurate :nerd_face:

if the user ID is randomized at some other point disregard this issue :see_no_evil:

Hi @waghanza ,

I think that caching feature, is one of the core feature of any framework, and then not to be avoided. Most of (I hope all of) apps in production are using caching feature, so why not benchmarking frameworks with the same behaviors ? In the future, when resources consumption will be shown, caching impact will be shown

I agree with you 100%, cache is a very important resource and should be tested, but cache is already tested in 2/3 of requests, on route 1 (GET /) and on route 3 (POST /user), so my proposal is test dynamic routes in 1/3 of requests that would be route 2 (GET /user /:id).

I made a quick experiment : injecting a random number in place of 0 in route number 2, to see if results are significantly changing.

Unfortunately its change was inefficient, as the WRK will execute all calls for n seconds for the same route that is created randomly, for example /user/2343 for all request, to be efficient it is the WRK that must change the URI for each route through LUA script(pipeline_get_id.lua).

As I understand, route 1 (GET:/), route 2 (GET:/user/?) and route 3 (POST:/user) are using cache (because of static URL usage).

Route 2 is however, a parametrized URL. Introducing a random parameter could lead to a benefit for you, right ?

Force wrk to change its route, will enhance results, since the cache feature will be bypassed and the entire routing process will be made by any underlying framework.

What do you thing @the-benchmarker/web-frameworks ?

Caching is (IMO) the biggest problem with benchmarks. Many frameworks handle it differently, and it doesn't really test the framework speed since it should be an equalizer. "Proper" results that test how fast a framework can take a requeat, hand it off to the developer, then take that response back, are much more useful to understand how different frameworks perform. For this reason, I think it should err on the side of avoidance. Does wrk have that ability to dynamically change the request?

At the very least, there should be two sets of benchmarks: with and wihout caching. There is a similar style benchmark for JS frontend frameworks that takes a similar approach.

Thanks for your feedback @ahopkins. I think that the main question is what do we want to reflect ?

1. The performance that a framework could provider (the idea here is to help makers enhance their framework)
2. The performances that a service could deliver (the idea here is to help deciders choosing a framework)

IMHO, option 1 means without caching, option 2 means with caching.

Does wrk have that ability to dynamically change the request?

Yes @ahopkins, via lua

Introducing a random parameter could lead to a benefit for you, right ?

it will be a benefit for the-benchmarker/web-frameworks project. I believe that a framework has to know how to work well with caching, even with dynamic routes, but currently this project does not test this scenario, all requests is static testing like famous "GET /helloworld".

we are far away from false positive idea, then 😛

Thanks for your feedback @ahopkins. I think that the main question is what do we want to reflect ?

The problem that I see is that it encourages makers to gear their frameworks towards a specific benchmarking goal, which may mean making compromises that otherwise should not be made. I think the most likely users for any benchmark are the individual developers trying to decide what to use on their next project. But, I think that a benchmark like this should try and be transparent about the compromises that are and are not being made.

But, I think that a benchmark like this should try and be transparent about the compromises that are and are not being made.

Seems that this vision is aligned with mine. Can you be a bit more explicit about how this goal could be achieved ?

Seems that this vision is aligned with mine. Can you be a bit more explicit about how this goal could be achieved ?

Sure. Probably some sort of a writeup (bullet points, and as brief as can be) that explains something like this. I know this is meant to be mainly focused upon providing raw data, so a sort of in depth explanation that any single project takes would probably be way too cumbersome to maintain.

If you have not seen it, this is the JS benchmarking tool I was talking about: https://krausest.github.io/js-framework-benchmark/current.html

I really like this because:

1. You can see results keyed v. non-keyed with an explanation of what this differentiation means
2. It is filterable to only narrow in on a few particular projects of interest
3. It shows how projects perform in different aspects
4. Color provides context at a glance

As I understand @ahopkins, your ideas concerns 3 subjects :

1. Explaining what this benchmark is / is not
2. Having some sort of filtering
3. Provide multiple scenarios

Let me explain my plan :

• Do the 1. before any publication about results
• Do the 2. is a separated project, an UI project
• Do the 3. is some separated project (this one for the api scenario, one for the graphql api ....)

I agree that the UI portion would probably be another project. Not sure about No. 3 though. Regardless, I think No. 1 as you summarized it is spot on.