New decode_csv_fields processor#11753
Conversation
This patch introduces a new processor, `decode_csv_field` that decoded
rows of CSV-formatted data into a string array, one element per column.
processors:
- truncate_fields:
field: message
target: csv
separator: ,
overwrite_keys: false
ignore_missing: false
trim_leading_space: false
andrewkroh
left a comment
There was a problem hiding this comment.
A while back I had hacked together a lookup table processor that could load data from a CSV file. I found it useful to have a setting that allowed me to directly name specific numbered columns like
- processors:
- decode_csv_field:
field: message
target: "user"
columns:
# Target Field Name -> CSV Column Number
email: 0 # Write column 0 to user.email.
name: 2 # Write column 2 to user.name. (Column 1 is ignored.)
When columns is not specified then I'd have it write an array of strings to the target like you have.
|
@andrewkroh So I thought maybe it was better to have this processor to decode to an array and then add a generic "extract_array" processor or do this inside an ingest pipeline. I'm planning to discuss this on today's sync with Beats team. |
This will help me move the CSV processor out of "actions" and improve overall reusability.
So that it can be included selectively by different Beats.
4e87509 to
9e0a78b
Compare
|
This is looking good. I think it just needs a section added to the asciidocs now. |
|
@andrewkroh I've modified the processor a little bit to help align with the rest. Do you mind reviewing again? Now it has docs. Main change is the rename to *_fields and the ability to process more than one field at the same time (I've copied the conf style from your |
There is currently no mechanism to inject this reference config on selected Beats.
Co-Authored-By: adriansr <adrisr83@gmail.com>
* Missing changelog entry for #11753 * Update csv processor to support `when` clause * Docs fixes for csv processor
This patch introduces a new processor, `decode_csv_fields` that decodes
rows of CSV-formatted data into a string array, one element per column.
processors:
- decode_csv_fields:
fields:
message: csv
separator: ,
overwrite_keys: false
ignore_missing: false
trim_leading_space: false
This patch introduces a new processor,
decode_csv_fielddecode_csv_fieldsthat decodesrows of CSV-formatted data into a string array, one element per column.