[Nginx] Improve generated error message#8909
Conversation
In elastic#8762 error log generation was added. The message was a random message generated. This changes it to have a partially more realistic message generated based on some actual logs. The enum values for the different fields are hardcoded. It would be nice to have for host, ip and paths some partially random generation that could be used as function with cardinality.
| max: 100000 | ||
| - name: request.method | ||
| enum: ["POST", "GET", "PUT", "DELETE"] | ||
| - name: request.path |
There was a problem hiding this comment.
@aspacca Are there any generate functions for host names, ip addresses and paths that could be used?
There was a problem hiding this comment.
I realized it is not documented in the docs of corpus generator, but you can define an example on the fields.yml for a keyword field and the random generated values will follow the example "pattern".
we currently support only the following patterns:
a.dot.separated.string
a-dash-separated-string
an_underline_separated_string
a space separated field
the generated value will match the "separator" in the pattern, and how many "words" there are
There was a problem hiding this comment.
Can you share a full yaml example on how this should work in the scenario here for the request path? Tried with dots but in the outcome didn't have dots.
There was a problem hiding this comment.
fields.yml
- name: request.path
type: keyword
example: a.path
no need to define anything specific in config.yml for request.path, indeed, you cannot have an enum on request.path if you want the example from fields.yml to be used. maybe that was the reason why it didn't work for you
There was a problem hiding this comment.
I have now the following in fields.yml:
- name: request.path
type: keyword
example: my.dotted.path
I expected to get paths with 2 dots inside, what I get is values like forgerkingmercurymask.
@aspacca Can you open a PR against this PR to provide a working example, I think that will be quicker.
There was a problem hiding this comment.
@aspacca Can you open a PR against this PR to provide a working example, I think that will be quicker.
@ruflin I identified the bug and it's related specifically to using . as separator (https://github.com/elastic/elastic-integration-corpus-generator-tool/blob/main/pkg/genlib/generator_interface.go#L404)
If you replace . with _, - or a space in fields.yml is going to work.
I'll create a PR to fix the bug.
🚀 Benchmarks reportPackage
|
| Data stream | Previous EPS | New EPS | Diff (%) | Result |
|---|---|---|---|---|
access |
0 | 1730.1 | 1730.1 ( - %) | 👍 |
error |
0 | 11764.71 | 11764.71 ( - %) | 👍 |
|
/test benchmark fullreport |
💚 Build Succeeded
History
cc @ruflin |
In #8762 error log generation was added. The message was a random message generated. This changes it to have a partially more realistic message generated based on some actual logs. The enum values for the different fields are hardcoded. It would be nice to have for host, ip and paths some partially random generation that could be used as function with cardinality.