enhancement(topology): Update transforms to handle multiple definitions #16793

StephenWakely · 2023-03-14T11:56:42Z

This is a pretty big PR, but most of the changes are fairly mundane.

The biggest changes to look out for are:

`Output::log_schema_definitions`

Transforms can now output multiple schema::Definitions. So Output::log_schema_definitions has changed from Option<Definition> to Vec<Definition>.

vector/lib/vector-core/src/config/mod.rs

Lines 107 to 117 in 1c5a8d2

    
               // NOTE: schema definitions are only implemented/supported for log-type events. There is no 
        
               // inherent blocker to support other types as well, but it'll require additional work to add 
        
               // the relevant schemas, and store them separately in this type. 
        
               /// 
        
               /// *sources* if the output type is `Log` there should be a single definition. `Metric` types should 
        
               /// have no definition defined. 
        
               /// 
        
               /// For a *transforms* if `Datatype` is Log, at least one definition should be output. If the 
        
               /// transform has multiple connected sources, it is possible to have multiple output definitions - 
        
               /// one for each input. 
        
               pub log_schema_definitions: Vec<schema::Definition>,

Since sources still must only output zero or one definition, I have removed Output::default and added a number of functions that each component should use instead which enforces a bit better whether they should output zero, one or many definitions.

A vast number of the changes in this PR are just the components using the appropriate function instead.

`TransformConfig::outputs`

All the transform outputs functions have changed so that they output multiple definitions - generally one for each input definition adjusted to match what that definition will now look like after the transform has run.

The transform that needs the closest review is probably remap. In that transform a merged definition is passed in to the compiler which ensures that VRL will compile for all possible inputs. Then for each input definition the types and meanings from the VRL compiled program are copied into the output definition and the meanings that are still valid (the vrl program may have deleted the field that the meaning points to) are copied over.

It should be noted that this VRL does not seem to change the schema definition of the metadata:

  source: |
           %"stdin.host" = 34

I think that is an issue with VRL and not with this PR.

Topology builder

The most significant changes are in src/topology/builder.rs. In build_pieces it now handles the multiple definition outputs from sources (asserting that they are only creating a single definition).

Instead of merging the definitions of transforms it passes multiple ones to each transform. The input_definitions recurses along the upstream components to fetch their definitions. Each definition is tagged with the source OutputId.

Output

I have split Output into SourceOutput and TransformOutput since each component type has different behaviour.

Transforms

Transforms can output multiple definitions (each will be tagged with a different OutputId signifying the definition that will be output based on what has been input - this will be done in a followup PR).

Sources

Sources can have zero (for metrics and traces) or one (for logs) definitions.

If schema.enabled == false sources will specify a default schema definition with only the meanings defined.

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

src/topology/schema.rs

netlify · 2023-03-14T11:56:47Z

✅ Deploy Preview for vrl-playground ready!

Name	Link
🔨 Latest commit	`31d2bb3`
🔍 Latest deploy log	https://app.netlify.com/sites/vrl-playground/deploys/641d9802bdeb9a00083288d0
😎 Deploy Preview	https://deploy-preview-16793--vrl-playground.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

netlify · 2023-03-14T11:56:47Z

✅ Deploy Preview for vector-project canceled.

Name	Link
🔨 Latest commit	`6fbed87`
🔍 Latest deploy log	https://app.netlify.com/sites/vector-project/deploys/642aa758c5ac65000842ff7d

…nsforms

…output definition Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

…ario Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

neuronull

The codeowner flag for the changes to the sources, all look very like you said mundane.
I have no issues with those, so LGTM from that perspective.

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

datadog-vectordotdev · 2023-03-30T11:24:57Z

Datadog Report

Branch report: stephen/lognamespacing_transforms
Commit report: e2265c2

✅ vector: 0 Failed, 0 New Flaky, 2131 Passed, 0 Skipped, 9m 24.81s Wall Time

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

StephenWakely · 2023-03-30T13:34:47Z

lib/vector-core/src/schema/definition.rs

        Self {
            event_kind,
-            metadata_kind: Kind::object(Collection::empty()),
+            metadata_kind: Kind::object(Collection::any()),


@fuchsnj I have had to change this since if schema_enabled is off we copy over the meanings, but not the kinds. To ensure the meanings that point to metadata are still valid this needs to be any.

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

…nsforms

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

github-actions · 2023-03-31T13:15:04Z

Regression Detector Results

Run ID: 9693a24d-c90d-4443-b121-5bc44efe7240
Baseline: 018637a
Comparison: 20e3505
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%.

Fine details of change detection per experiment.

experiment	goal	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison outlier %	comparison CoV	erratic	declared erratic
file_to_blackhole	egress throughput	1.44MiB/CPU-s	23.47	100.00%	6.14MiB/CPU-s	4.57MiB/CPU-s	122.76KiB/CPU-s	0.743389	7.58MiB/CPU-s	3.93MiB/CPU-s	132.12KiB/CPU-s	1.291712	0.518227	True	True
otlp_grpc_to_blackhole	ingress throughput	32.56KiB/CPU-s	3.14	100.00%	1.01MiB/CPU-s	49.74KiB/CPU-s	626.81B/CPU-s	0.047911	1.05MiB/CPU-s	45.62KiB/CPU-s	574.88B/CPU-s	0.0	0.042605	False	False
otlp_http_to_blackhole	ingress throughput	41.64KiB/CPU-s	2.67	100.00%	1.52MiB/CPU-s	122.03KiB/CPU-s	1.5KiB/CPU-s	0.078317	1.56MiB/CPU-s	105.13KiB/CPU-s	1.29KiB/CPU-s	0.0	0.065716	False	False
syslog_humio_logs	ingress throughput	198.21KiB/CPU-s	2.20	100.00%	8.79MiB/CPU-s	274.07KiB/CPU-s	3.37KiB/CPU-s	0.030454	8.98MiB/CPU-s	317.64KiB/CPU-s	3.91KiB/CPU-s	0.0	0.034534	False	False
datadog_agent_remap_blackhole	ingress throughput	137.82KiB/CPU-s	0.44	100.00%	30.76MiB/CPU-s	1.27MiB/CPU-s	15.96KiB/CPU-s	0.041167	30.9MiB/CPU-s	1008.38KiB/CPU-s	12.41KiB/CPU-s	0.0	0.031869	False	False
http_to_http_json	ingress throughput	50.6KiB/CPU-s	0.36	100.00%	13.57MiB/CPU-s	300.78KiB/CPU-s	3.7KiB/CPU-s	0.021641	13.62MiB/CPU-s	227.59KiB/CPU-s	2.8KiB/CPU-s	0.0	0.016316	False	False
splunk_hec_route_s3	ingress throughput	40.8KiB/CPU-s	0.34	99.99%	11.56MiB/CPU-s	600.31KiB/CPU-s	7.39KiB/CPU-s	0.050711	11.6MiB/CPU-s	622.91KiB/CPU-s	7.66KiB/CPU-s	0.0	0.052439	False	False
syslog_loki	ingress throughput	29.17KiB/CPU-s	0.34	100.00%	8.45MiB/CPU-s	211.98KiB/CPU-s	2.61KiB/CPU-s	0.024509	8.47MiB/CPU-s	201.19KiB/CPU-s	2.47KiB/CPU-s	0.0	0.023184	False	False
http_to_http_noack	ingress throughput	3.75KiB/CPU-s	0.03	43.32%	13.61MiB/CPU-s	393.45KiB/CPU-s	4.84KiB/CPU-s	0.028238	13.61MiB/CPU-s	357.58KiB/CPU-s	4.4KiB/CPU-s	0.0	0.025657	False	False
enterprise_http_to_http	ingress throughput	4.0KiB/CPU-s	0.03	77.48%	13.62MiB/CPU-s	222.37KiB/CPU-s	2.74KiB/CPU-s	0.015942	13.62MiB/CPU-s	150.04KiB/CPU-s	1.85KiB/CPU-s	0.0	0.010754	False	False
datadog_agent_remap_blackhole_acks	ingress throughput	7.8KiB/CPU-s	0.02	36.66%	30.99MiB/CPU-s	1019.57KiB/CPU-s	12.54KiB/CPU-s	0.032129	31.0MiB/CPU-s	853.64KiB/CPU-s	10.51KiB/CPU-s	0.0	0.026893	False	False
fluent_elasticsearch	ingress throughput	-286.04B/CPU-s	-0.00	40.89%	45.41MiB/CPU-s	30.23KiB/CPU-s	376.6B/CPU-s	0.00065	45.41MiB/CPU-s	30.19KiB/CPU-s	376.29B/CPU-s	0.0	0.000649	False	False
splunk_hec_to_splunk_hec_logs_acks	ingress throughput	612.68B/CPU-s	0.00	7.98%	13.61MiB/CPU-s	344.4KiB/CPU-s	4.24KiB/CPU-s	0.024704	13.61MiB/CPU-s	342.53KiB/CPU-s	4.21KiB/CPU-s	0.0	0.024568	False	False
splunk_hec_to_splunk_hec_logs_noack	ingress throughput	-1.77KiB/CPU-s	-0.01	30.15%	13.62MiB/CPU-s	255.33KiB/CPU-s	3.14KiB/CPU-s	0.01831	13.61MiB/CPU-s	269.43KiB/CPU-s	3.31KiB/CPU-s	0.0	0.019324	False	False
splunk_hec_indexer_ack_blackhole	ingress throughput	-2.67KiB/CPU-s	-0.02	44.41%	13.62MiB/CPU-s	254.41KiB/CPU-s	3.13KiB/CPU-s	0.018245	13.61MiB/CPU-s	266.16KiB/CPU-s	3.27KiB/CPU-s	0.0	0.019091	False	False
syslog_regex_logs2metric_ddmetrics	ingress throughput	-670.15B/CPU-s	-0.02	7.22%	3.49MiB/CPU-s	382.32KiB/CPU-s	4.7KiB/CPU-s	0.106909	3.49MiB/CPU-s	444.7KiB/CPU-s	5.47KiB/CPU-s	0.0	0.124375	True	True
datadog_agent_remap_datadog_logs_acks	ingress throughput	-40.88KiB/CPU-s	-0.12	98.65%	32.57MiB/CPU-s	977.5KiB/CPU-s	12.03KiB/CPU-s	0.02931	32.53MiB/CPU-s	922.49KiB/CPU-s	11.35KiB/CPU-s	0.0	0.027694	False	False
http_to_http_acks	ingress throughput	-12.67KiB/CPU-s	-0.23	20.53%	5.27MiB/CPU-s	2.76MiB/CPU-s	34.78KiB/CPU-s	0.523601	5.26MiB/CPU-s	2.7MiB/CPU-s	34.07KiB/CPU-s	0.0	0.514008	True	False
syslog_log2metric_humio_metrics	ingress throughput	-23.46KiB/CPU-s	-0.37	100.00%	6.23MiB/CPU-s	216.41KiB/CPU-s	2.66KiB/CPU-s	0.033944	6.2MiB/CPU-s	187.97KiB/CPU-s	2.31KiB/CPU-s	0.0	0.029591	False	False
datadog_agent_remap_datadog_logs	ingress throughput	-205.0KiB/CPU-s	-0.62	100.00%	32.22MiB/CPU-s	1.18MiB/CPU-s	14.91KiB/CPU-s	0.036717	32.02MiB/CPU-s	1.47MiB/CPU-s	18.52KiB/CPU-s	0.0	0.045896	False	False
socket_to_socket_blackhole	ingress throughput	-87.71KiB/CPU-s	-0.64	100.00%	13.48MiB/CPU-s	263.92KiB/CPU-s	3.25KiB/CPU-s	0.019115	13.4MiB/CPU-s	388.05KiB/CPU-s	4.77KiB/CPU-s	0.0	0.028285	False	False
syslog_splunk_hec_logs	ingress throughput	-65.64KiB/CPU-s	-0.72	100.00%	8.91MiB/CPU-s	224.74KiB/CPU-s	2.76KiB/CPU-s	0.024621	8.85MiB/CPU-s	258.26KiB/CPU-s	3.18KiB/CPU-s	0.0	0.028498	False	False
syslog_log2metric_splunk_hec_metrics	ingress throughput	-162.6KiB/CPU-s	-1.71	100.00%	9.31MiB/CPU-s	167.92KiB/CPU-s	2.07KiB/CPU-s	0.017618	9.15MiB/CPU-s	337.9KiB/CPU-s	4.16KiB/CPU-s	0.0	0.036068	False	False
http_text_to_http_json	ingress throughput	-456.66KiB/CPU-s	-1.77	100.00%	25.26MiB/CPU-s	599.48KiB/CPU-s	7.38KiB/CPU-s	0.023177	24.81MiB/CPU-s	591.33KiB/CPU-s	7.28KiB/CPU-s	0.0	0.023273	False	False

github-actions · 2023-03-31T13:32:24Z

Regression Detector Results

Run ID: b570e942-42e4-49c8-8e56-f79456820bac
Baseline: 018637a
Comparison: b9467e4
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%.

Fine details of change detection per experiment.

experiment	goal	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline outlier %	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
otlp_http_to_blackhole	ingress throughput	62.56KiB/CPU-s	4.04	100.00%	1.51MiB/CPU-s	123.24KiB/CPU-s	1.52KiB/CPU-s	0.0	0.079525	1.57MiB/CPU-s	96.38KiB/CPU-s	1.19KiB/CPU-s	0.059779	False	False
otlp_grpc_to_blackhole	ingress throughput	37.29KiB/CPU-s	3.58	100.00%	1.02MiB/CPU-s	52.15KiB/CPU-s	656.98B/CPU-s	0.0	0.050033	1.05MiB/CPU-s	44.54KiB/CPU-s	561.19B/CPU-s	0.041253	False	False
syslog_log2metric_splunk_hec_metrics	ingress throughput	294.64KiB/CPU-s	3.22	100.00%	8.93MiB/CPU-s	382.32KiB/CPU-s	4.7KiB/CPU-s	0.0	0.041806	9.22MiB/CPU-s	345.57KiB/CPU-s	4.25KiB/CPU-s	0.036607	False	False
datadog_agent_remap_blackhole_acks	ingress throughput	867.7KiB/CPU-s	2.85	100.00%	29.78MiB/CPU-s	1.67MiB/CPU-s	21.05KiB/CPU-s	0.0	0.056097	30.63MiB/CPU-s	865.55KiB/CPU-s	10.65KiB/CPU-s	0.027598	False	False
datadog_agent_remap_blackhole	ingress throughput	504.46KiB/CPU-s	1.62	100.00%	30.5MiB/CPU-s	1.11MiB/CPU-s	14.01KiB/CPU-s	0.0	0.036468	30.99MiB/CPU-s	914.92KiB/CPU-s	11.26KiB/CPU-s	0.028829	False	False
syslog_log2metric_humio_metrics	ingress throughput	86.21KiB/CPU-s	1.38	100.00%	6.08MiB/CPU-s	325.32KiB/CPU-s	4.0KiB/CPU-s	0.0	0.052244	6.16MiB/CPU-s	316.29KiB/CPU-s	3.89KiB/CPU-s	0.0501	False	False
syslog_humio_logs	ingress throughput	100.69KiB/CPU-s	1.10	100.00%	8.92MiB/CPU-s	299.34KiB/CPU-s	3.68KiB/CPU-s	0.0	0.032783	9.01MiB/CPU-s	233.62KiB/CPU-s	2.87KiB/CPU-s	0.025307	False	False
splunk_hec_route_s3	ingress throughput	11.9KiB/CPU-s	0.10	78.88%	11.67MiB/CPU-s	537.49KiB/CPU-s	6.61KiB/CPU-s	0.0	0.044977	11.68MiB/CPU-s	556.12KiB/CPU-s	6.84KiB/CPU-s	0.046489	False	False
enterprise_http_to_http	ingress throughput	7.67KiB/CPU-s	0.06	95.17%	13.62MiB/CPU-s	277.84KiB/CPU-s	3.42KiB/CPU-s	0.0	0.019924	13.62MiB/CPU-s	149.88KiB/CPU-s	1.84KiB/CPU-s	0.010742	False	False
http_to_http_noack	ingress throughput	2.12KiB/CPU-s	0.02	27.34%	13.61MiB/CPU-s	357.59KiB/CPU-s	4.4KiB/CPU-s	0.0	0.025657	13.61MiB/CPU-s	339.52KiB/CPU-s	4.18KiB/CPU-s	0.024357	False	False
http_to_http_json	ingress throughput	-550.2B/CPU-s	-0.00	10.86%	13.62MiB/CPU-s	225.46KiB/CPU-s	2.77KiB/CPU-s	0.0	0.016163	13.62MiB/CPU-s	226.88KiB/CPU-s	2.79KiB/CPU-s	0.016265	False	False
fluent_elasticsearch	ingress throughput	157.2B/CPU-s	0.00	23.30%	45.41MiB/CPU-s	30.17KiB/CPU-s	376.03B/CPU-s	0.0	0.000649	45.41MiB/CPU-s	30.02KiB/CPU-s	374.18B/CPU-s	0.000645	False	False
splunk_hec_indexer_ack_blackhole	ingress throughput	-994.18B/CPU-s	-0.01	17.01%	13.62MiB/CPU-s	256.62KiB/CPU-s	3.16KiB/CPU-s	0.0	0.018404	13.61MiB/CPU-s	262.95KiB/CPU-s	3.23KiB/CPU-s	0.01886	False	False
splunk_hec_to_splunk_hec_logs_noack	ingress throughput	-836.38B/CPU-s	-0.01	14.16%	13.62MiB/CPU-s	260.17KiB/CPU-s	3.2KiB/CPU-s	0.0	0.018659	13.61MiB/CPU-s	266.28KiB/CPU-s	3.28KiB/CPU-s	0.019098	False	False
splunk_hec_to_splunk_hec_logs_acks	ingress throughput	-811.91B/CPU-s	-0.01	10.16%	13.61MiB/CPU-s	366.57KiB/CPU-s	4.51KiB/CPU-s	0.0	0.026292	13.61MiB/CPU-s	346.97KiB/CPU-s	4.27KiB/CPU-s	0.024887	False	False
syslog_loki	ingress throughput	-21.93KiB/CPU-s	-0.25	100.00%	8.43MiB/CPU-s	226.88KiB/CPU-s	2.79KiB/CPU-s	0.0	0.026293	8.4MiB/CPU-s	237.47KiB/CPU-s	2.92KiB/CPU-s	0.02759	False	False
http_to_http_acks	ingress throughput	-27.88KiB/CPU-s	-0.52	43.29%	5.27MiB/CPU-s	2.74MiB/CPU-s	34.58KiB/CPU-s	0.0	0.521078	5.24MiB/CPU-s	2.72MiB/CPU-s	34.31KiB/CPU-s	0.519661	True	False
syslog_splunk_hec_logs	ingress throughput	-57.88KiB/CPU-s	-0.63	100.00%	8.95MiB/CPU-s	236.35KiB/CPU-s	2.91KiB/CPU-s	0.0	0.025786	8.89MiB/CPU-s	208.28KiB/CPU-s	2.56KiB/CPU-s	0.022867	False	False
socket_to_socket_blackhole	ingress throughput	-91.59KiB/CPU-s	-0.66	100.00%	13.56MiB/CPU-s	285.5KiB/CPU-s	3.51KiB/CPU-s	0.0	0.020567	13.47MiB/CPU-s	373.64KiB/CPU-s	4.6KiB/CPU-s	0.027094	False	False
datadog_agent_remap_datadog_logs	ingress throughput	-231.78KiB/CPU-s	-0.69	100.00%	32.8MiB/CPU-s	878.97KiB/CPU-s	10.82KiB/CPU-s	0.0	0.026164	32.58MiB/CPU-s	880.98KiB/CPU-s	10.84KiB/CPU-s	0.026406	False	False
datadog_agent_remap_datadog_logs_acks	ingress throughput	-627.07KiB/CPU-s	-1.91	100.00%	32.11MiB/CPU-s	1.22MiB/CPU-s	15.42KiB/CPU-s	0.0	0.038112	31.5MiB/CPU-s	1.63MiB/CPU-s	20.53KiB/CPU-s	0.051735	False	False
http_text_to_http_json	ingress throughput	-535.33KiB/CPU-s	-2.06	100.00%	25.43MiB/CPU-s	656.56KiB/CPU-s	8.08KiB/CPU-s	0.0	0.025208	24.91MiB/CPU-s	700.55KiB/CPU-s	8.66KiB/CPU-s	0.027462	False	False
syslog_regex_logs2metric_ddmetrics	ingress throughput	-138.65KiB/CPU-s	-3.76	100.00%	3.6MiB/CPU-s	343.33KiB/CPU-s	4.23KiB/CPU-s	0.0	0.093165	3.46MiB/CPU-s	468.68KiB/CPU-s	5.77KiB/CPU-s	0.132151	True	True
file_to_blackhole	egress throughput	-589.09KiB/CPU-s	-8.55	99.89%	6.73MiB/CPU-s	4.53MiB/CPU-s	129.85KiB/CPU-s	0.469851	0.672857	6.16MiB/CPU-s	4.48MiB/CPU-s	124.82KiB/CPU-s	0.726606	True	True

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

github-actions · 2023-03-31T16:11:18Z

Regression Detector Results

Run ID: 029646e1-f459-4adf-98f9-a092422f7aa5
Baseline: b7abb08
Comparison: 24d0581
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%.

Fine details of change detection per experiment.

experiment	goal	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline outlier %	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
otlp_grpc_to_blackhole	ingress throughput	35.0KiB/CPU-s	3.36	100.00%	1.02MiB/CPU-s	49.15KiB/CPU-s	619.27B/CPU-s	0.0	0.047121	1.05MiB/CPU-s	41.77KiB/CPU-s	526.35B/CPU-s	0.038748	False	False
http_to_http_acks	ingress throughput	121.31KiB/CPU-s	2.31	98.73%	5.13MiB/CPU-s	2.72MiB/CPU-s	34.4KiB/CPU-s	0.0	0.529412	5.25MiB/CPU-s	2.73MiB/CPU-s	34.41KiB/CPU-s	0.519896	True	False
syslog_log2metric_splunk_hec_metrics	ingress throughput	209.56KiB/CPU-s	2.20	100.00%	9.3MiB/CPU-s	223.99KiB/CPU-s	2.76KiB/CPU-s	0.0	0.023526	9.5MiB/CPU-s	232.76KiB/CPU-s	2.86KiB/CPU-s	0.023921	False	False
syslog_regex_logs2metric_ddmetrics	ingress throughput	51.9KiB/CPU-s	1.48	100.00%	3.42MiB/CPU-s	375.07KiB/CPU-s	4.61KiB/CPU-s	0.0	0.107061	3.47MiB/CPU-s	363.71KiB/CPU-s	4.48KiB/CPU-s	0.102301	True	True
otlp_http_to_blackhole	ingress throughput	22.7KiB/CPU-s	1.42	100.00%	1.56MiB/CPU-s	110.37KiB/CPU-s	1.36KiB/CPU-s	0.0	0.069273	1.58MiB/CPU-s	94.33KiB/CPU-s	1.16KiB/CPU-s	0.058374	False	False
syslog_humio_logs	ingress throughput	52.8KiB/CPU-s	0.57	100.00%	8.97MiB/CPU-s	256.78KiB/CPU-s	3.16KiB/CPU-s	0.0	0.027957	9.02MiB/CPU-s	244.01KiB/CPU-s	3.0KiB/CPU-s	0.026414	False	False
datadog_agent_remap_blackhole_acks	ingress throughput	24.29KiB/CPU-s	0.08	82.05%	31.15MiB/CPU-s	1.19MiB/CPU-s	15.03KiB/CPU-s	0.0	0.038281	31.18MiB/CPU-s	818.64KiB/CPU-s	10.08KiB/CPU-s	0.025639	False	False
splunk_hec_indexer_ack_blackhole	ingress throughput	2.94KiB/CPU-s	0.02	49.33%	13.62MiB/CPU-s	262.69KiB/CPU-s	3.23KiB/CPU-s	0.0	0.01884	13.62MiB/CPU-s	245.21KiB/CPU-s	3.02KiB/CPU-s	0.017583	False	False
syslog_log2metric_humio_metrics	ingress throughput	1.11KiB/CPU-s	0.02	15.77%	6.08MiB/CPU-s	314.03KiB/CPU-s	3.87KiB/CPU-s	0.0	0.050451	6.08MiB/CPU-s	326.81KiB/CPU-s	4.02KiB/CPU-s	0.052496	False	False
splunk_hec_to_splunk_hec_logs_noack	ingress throughput	1.02KiB/CPU-s	0.01	17.17%	13.61MiB/CPU-s	274.15KiB/CPU-s	3.37KiB/CPU-s	0.0	0.019664	13.61MiB/CPU-s	269.12KiB/CPU-s	3.31KiB/CPU-s	0.019302	False	False
splunk_hec_to_splunk_hec_logs_acks	ingress throughput	296.85B/CPU-s	0.00	3.71%	13.61MiB/CPU-s	363.62KiB/CPU-s	4.47KiB/CPU-s	0.0	0.026082	13.61MiB/CPU-s	352.44KiB/CPU-s	4.33KiB/CPU-s	0.025279	False	False
fluent_elasticsearch	ingress throughput	147.02B/CPU-s	0.00	21.83%	45.41MiB/CPU-s	30.2KiB/CPU-s	376.6B/CPU-s	0.0	0.000649	45.41MiB/CPU-s	29.98KiB/CPU-s	373.6B/CPU-s	0.000645	False	False
enterprise_http_to_http	ingress throughput	-1.85KiB/CPU-s	-0.01	39.15%	13.62MiB/CPU-s	191.22KiB/CPU-s	2.35KiB/CPU-s	0.0	0.013707	13.62MiB/CPU-s	222.94KiB/CPU-s	2.74KiB/CPU-s	0.015983	False	False
http_to_http_noack	ingress throughput	-5.31KiB/CPU-s	-0.04	57.66%	13.61MiB/CPU-s	359.64KiB/CPU-s	4.42KiB/CPU-s	0.0	0.025804	13.6MiB/CPU-s	401.8KiB/CPU-s	4.94KiB/CPU-s	0.028841	False	False
datadog_agent_remap_blackhole	ingress throughput	-49.22KiB/CPU-s	-0.15	98.92%	31.16MiB/CPU-s	1.08MiB/CPU-s	13.67KiB/CPU-s	0.0	0.034809	31.12MiB/CPU-s	1.08MiB/CPU-s	13.65KiB/CPU-s	0.034825	False	False
http_to_http_json	ingress throughput	-25.66KiB/CPU-s	-0.18	100.00%	13.61MiB/CPU-s	246.42KiB/CPU-s	3.03KiB/CPU-s	0.0	0.017681	13.58MiB/CPU-s	278.44KiB/CPU-s	3.43KiB/CPU-s	0.020015	False	False
datadog_agent_remap_datadog_logs_acks	ingress throughput	-96.26KiB/CPU-s	-0.30	100.00%	31.79MiB/CPU-s	1.22MiB/CPU-s	15.43KiB/CPU-s	0.0	0.038526	31.69MiB/CPU-s	1.43MiB/CPU-s	18.01KiB/CPU-s	0.045107	False	False
socket_to_socket_blackhole	ingress throughput	-110.4KiB/CPU-s	-0.79	100.00%	13.7MiB/CPU-s	325.14KiB/CPU-s	4.0KiB/CPU-s	0.0	0.02318	13.59MiB/CPU-s	226.48KiB/CPU-s	2.79KiB/CPU-s	0.016274	False	False
file_to_blackhole	egress throughput	-64.22KiB/CPU-s	-0.97	27.71%	6.46MiB/CPU-s	4.35MiB/CPU-s	125.14KiB/CPU-s	1.026856	0.672372	6.4MiB/CPU-s	4.55MiB/CPU-s	130.89KiB/CPU-s	0.709857	True	True
syslog_splunk_hec_logs	ingress throughput	-103.45KiB/CPU-s	-1.15	100.00%	8.81MiB/CPU-s	284.14KiB/CPU-s	3.49KiB/CPU-s	0.0	0.031496	8.71MiB/CPU-s	260.93KiB/CPU-s	3.21KiB/CPU-s	0.029259	False	False
splunk_hec_route_s3	ingress throughput	-141.15KiB/CPU-s	-1.18	100.00%	11.65MiB/CPU-s	556.38KiB/CPU-s	6.84KiB/CPU-s	0.0	0.04663	11.51MiB/CPU-s	584.06KiB/CPU-s	7.19KiB/CPU-s	0.049536	False	False
syslog_loki	ingress throughput	-132.81KiB/CPU-s	-1.53	100.00%	8.49MiB/CPU-s	237.97KiB/CPU-s	2.93KiB/CPU-s	0.0	0.027382	8.36MiB/CPU-s	332.34KiB/CPU-s	4.09KiB/CPU-s	0.038834	False	False
http_text_to_http_json	ingress throughput	-585.77KiB/CPU-s	-2.27	100.00%	25.19MiB/CPU-s	714.92KiB/CPU-s	8.8KiB/CPU-s	0.0	0.027712	24.62MiB/CPU-s	805.02KiB/CPU-s	9.91KiB/CPU-s	0.031929	False	False
datadog_agent_remap_datadog_logs	ingress throughput	-834.83KiB/CPU-s	-2.47	100.00%	33.0MiB/CPU-s	723.13KiB/CPU-s	8.9KiB/CPU-s	0.0	0.0214	32.18MiB/CPU-s	1.25MiB/CPU-s	15.75KiB/CPU-s	0.038855	False	False

src/topology/schema.rs

src/transforms/remap.rs

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

github-actions · 2023-04-03T10:55:07Z

Regression Detector Results

Run ID: de9ad205-29c9-4050-b145-ba9d657b9bf5
Baseline: 00c0316
Comparison: 6fbed87
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%.

Fine details of change detection per experiment.

experiment	goal	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline outlier %	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
file_to_blackhole	egress throughput	922.85KiB/CPU-s	14.01	100.00%	6.43MiB/CPU-s	4.28MiB/CPU-s	124.51KiB/CPU-s	6.124093	0.665604	7.33MiB/CPU-s	3.81MiB/CPU-s	129.26KiB/CPU-s	0.518653	True	True
http_text_to_http_json	ingress throughput	1.15MiB/CPU-s	4.85	100.00%	23.62MiB/CPU-s	534.25KiB/CPU-s	6.58KiB/CPU-s	0.0	0.022091	24.76MiB/CPU-s	586.81KiB/CPU-s	7.22KiB/CPU-s	0.023142	False	False
socket_to_socket_blackhole	ingress throughput	398.62KiB/CPU-s	2.96	100.00%	13.14MiB/CPU-s	317.38KiB/CPU-s	3.91KiB/CPU-s	0.0	0.023592	13.53MiB/CPU-s	245.09KiB/CPU-s	3.02KiB/CPU-s	0.017694	False	False
otlp_grpc_to_blackhole	ingress throughput	27.61KiB/CPU-s	2.64	100.00%	1.02MiB/CPU-s	49.65KiB/CPU-s	625.56B/CPU-s	0.0	0.047386	1.05MiB/CPU-s	39.7KiB/CPU-s	500.35B/CPU-s	0.036914	False	False
otlp_http_to_blackhole	ingress throughput	36.2KiB/CPU-s	2.29	100.00%	1.54MiB/CPU-s	115.66KiB/CPU-s	1.42KiB/CPU-s	0.0	0.073216	1.58MiB/CPU-s	92.56KiB/CPU-s	1.14KiB/CPU-s	0.057277	False	False
syslog_regex_logs2metric_ddmetrics	ingress throughput	57.07KiB/CPU-s	1.55	100.00%	3.61MiB/CPU-s	395.41KiB/CPU-s	4.87KiB/CPU-s	0.0	0.107089	3.66MiB/CPU-s	398.58KiB/CPU-s	4.91KiB/CPU-s	0.106303	True	True
datadog_agent_remap_datadog_logs_acks	ingress throughput	454.75KiB/CPU-s	1.37	100.00%	32.43MiB/CPU-s	889.19KiB/CPU-s	10.94KiB/CPU-s	0.0	0.026777	32.87MiB/CPU-s	755.06KiB/CPU-s	9.29KiB/CPU-s	0.022431	False	False
syslog_humio_logs	ingress throughput	88.19KiB/CPU-s	0.95	100.00%	9.1MiB/CPU-s	221.6KiB/CPU-s	2.73KiB/CPU-s	0.0	0.023776	9.19MiB/CPU-s	202.15KiB/CPU-s	2.49KiB/CPU-s	0.021486	False	False
datadog_agent_remap_blackhole	ingress throughput	244.52KiB/CPU-s	0.77	100.00%	31.05MiB/CPU-s	1.0MiB/CPU-s	12.62KiB/CPU-s	0.0	0.032252	31.29MiB/CPU-s	955.2KiB/CPU-s	11.75KiB/CPU-s	0.029809	False	False
syslog_log2metric_splunk_hec_metrics	ingress throughput	50.82KiB/CPU-s	0.53	100.00%	9.36MiB/CPU-s	349.75KiB/CPU-s	4.3KiB/CPU-s	0.0	0.036492	9.41MiB/CPU-s	303.77KiB/CPU-s	3.74KiB/CPU-s	0.031527	False	False
syslog_splunk_hec_logs	ingress throughput	32.81KiB/CPU-s	0.35	100.00%	9.05MiB/CPU-s	216.01KiB/CPU-s	2.66KiB/CPU-s	0.0	0.023314	9.08MiB/CPU-s	213.41KiB/CPU-s	2.63KiB/CPU-s	0.022953	False	False
datadog_agent_remap_blackhole_acks	ingress throughput	75.86KiB/CPU-s	0.24	100.00%	31.04MiB/CPU-s	1.02MiB/CPU-s	12.85KiB/CPU-s	0.0	0.032847	31.12MiB/CPU-s	901.69KiB/CPU-s	11.1KiB/CPU-s	0.028297	False	False
http_to_http_json	ingress throughput	26.7KiB/CPU-s	0.19	100.00%	13.59MiB/CPU-s	286.35KiB/CPU-s	3.52KiB/CPU-s	0.0	0.020568	13.62MiB/CPU-s	225.66KiB/CPU-s	2.78KiB/CPU-s	0.016178	False	False
enterprise_http_to_http	ingress throughput	7.38KiB/CPU-s	0.05	94.71%	13.62MiB/CPU-s	271.28KiB/CPU-s	3.34KiB/CPU-s	0.0	0.019454	13.62MiB/CPU-s	149.54KiB/CPU-s	1.84KiB/CPU-s	0.010718	False	False
syslog_log2metric_humio_metrics	ingress throughput	3.01KiB/CPU-s	0.05	66.52%	6.2MiB/CPU-s	191.62KiB/CPU-s	2.36KiB/CPU-s	0.0	0.030177	6.2MiB/CPU-s	166.15KiB/CPU-s	2.04KiB/CPU-s	0.026152	False	False
splunk_hec_indexer_ack_blackhole	ingress throughput	1.12KiB/CPU-s	0.01	20.04%	13.62MiB/CPU-s	257.47KiB/CPU-s	3.17KiB/CPU-s	0.0	0.018465	13.62MiB/CPU-s	251.06KiB/CPU-s	3.09KiB/CPU-s	0.018004	False	False
splunk_hec_to_splunk_hec_logs_acks	ingress throughput	-165.65B/CPU-s	-0.00	2.13%	13.61MiB/CPU-s	353.48KiB/CPU-s	4.35KiB/CPU-s	0.0	0.025354	13.61MiB/CPU-s	343.45KiB/CPU-s	4.22KiB/CPU-s	0.024635	False	False
fluent_elasticsearch	ingress throughput	-373.31B/CPU-s	-0.00	51.78%	45.41MiB/CPU-s	30.21KiB/CPU-s	376.56B/CPU-s	0.0	0.00065	45.41MiB/CPU-s	30.05KiB/CPU-s	374.66B/CPU-s	0.000646	False	False
http_to_http_noack	ingress throughput	-2.06KiB/CPU-s	-0.01	25.40%	13.61MiB/CPU-s	356.05KiB/CPU-s	4.38KiB/CPU-s	0.0	0.025547	13.61MiB/CPU-s	375.8KiB/CPU-s	4.62KiB/CPU-s	0.026968	False	False
splunk_hec_to_splunk_hec_logs_noack	ingress throughput	-2.5KiB/CPU-s	-0.02	41.39%	13.62MiB/CPU-s	254.73KiB/CPU-s	3.13KiB/CPU-s	0.0	0.018267	13.61MiB/CPU-s	273.19KiB/CPU-s	3.36KiB/CPU-s	0.019594	False	False
syslog_loki	ingress throughput	-9.4KiB/CPU-s	-0.11	98.82%	8.47MiB/CPU-s	237.19KiB/CPU-s	2.92KiB/CPU-s	0.0	0.027337	8.46MiB/CPU-s	189.26KiB/CPU-s	2.33KiB/CPU-s	0.021837	False	False
datadog_agent_remap_datadog_logs	ingress throughput	-47.93KiB/CPU-s	-0.14	99.80%	33.11MiB/CPU-s	886.39KiB/CPU-s	10.91KiB/CPU-s	0.0	0.026144	33.06MiB/CPU-s	895.82KiB/CPU-s	11.02KiB/CPU-s	0.026459	False	False
splunk_hec_route_s3	ingress throughput	-81.37KiB/CPU-s	-0.67	100.00%	11.78MiB/CPU-s	525.08KiB/CPU-s	6.46KiB/CPU-s	0.0	0.043525	11.7MiB/CPU-s	559.03KiB/CPU-s	6.88KiB/CPU-s	0.046653	False	False
http_to_http_acks	ingress throughput	-59.07KiB/CPU-s	-1.09	77.34%	5.27MiB/CPU-s	2.77MiB/CPU-s	34.93KiB/CPU-s	0.0	0.526001	5.21MiB/CPU-s	2.71MiB/CPU-s	34.15KiB/CPU-s	0.519964	True	False

spencergilbert

LGTM - though I'd wait for an approval from @fuchsnj before merging.

…efinitions (#16793)" This reverts commit e19f4fc. Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

…ltiple definitions (#16793)"" This reverts commit 5dc20f3.

* Revert "chore(topology): Transform outputs hash table of OutputId -> Definition (#17059)" This reverts commit 1bdb24d. * Revert "chore(topology): split `build_pieces` into smaller functions (#17037)" This reverts commit 6e6f1eb. * Revert "enhancement(topology): Update transforms to handle multiple definitions (#16793)" This reverts commit e19f4fc. Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com> --------- Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* Revert "chore(topology): Transform outputs hash table of OutputId -> Definition (#17059)" This reverts commit 1bdb24d. * Revert "chore(topology): split `build_pieces` into smaller functions (#17037)" This reverts commit 6e6f1eb. * Revert "enhancement(topology): Update transforms to handle multiple definitions (#16793)" This reverts commit e19f4fc. Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

…ltiple definitions (#16793)"" This reverts commit 5dc20f3.

* Revert "Revert "enhancement(topology): Update transforms to handle multiple definitions (#16793)"" This reverts commit 5dc20f3. * Revert "Revert "chore(topology): split `build_pieces` into smaller functions (#17037)"" This reverts commit 0e11bc3. * Revert "Revert "chore(topology): Transform outputs hash table of OutputId -> Definition (#17059)"" This reverts commit 8916ec1. --------- Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Update transforms to handle multiple definitions

186a321

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

github-advanced-security bot found potential problems Mar 14, 2023

View reviewed changes

src/topology/schema.rs Fixed Show fixed Hide fixed

Merge remote-tracking branch 'origin' into stephen/lognamespacing_tra…

d7973a3

…nsforms

fuchsnj self-requested a review March 14, 2023 13:23

StephenWakely added 7 commits March 14, 2023 17:21

Remap transform needs to copy metadata kind and relevant meanings to …

d9bea96

…output definition Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Spelling

b743a23

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Replace Output::default with functions appropriate for the given scen…

1584dd6

…ario Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Clippy

d4bc2b4

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Updated comments

8ea00ad

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Further restrain sources to a single definition

174fa5d

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Remove redundant comments

389fabd

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

StephenWakely added the ci-condition: integration tests enable label Mar 16, 2023

StephenWakely added 7 commits March 16, 2023 17:08

Remove metrics definition from datadog agent

1c5a8d2

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Move vrl compiling outside the definition loop

a3aa6d0

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Dont panic on empty definitions

d7c62d8

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

WIP transform outputs need to be a map of port to a list of definitions

9fac795

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Temporarily fix remap errors

c7ef6ff

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

No inputs should have a Definition::any

db847a3

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Add Output Id to input definition

7024637

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

StephenWakely marked this pull request as ready for review March 21, 2023 11:54

StephenWakely requested review from a team March 21, 2023 11:54

StephenWakely requested a review from neuronull as a code owner March 21, 2023 11:54

neuronull reviewed Mar 29, 2023

View reviewed changes

Responding to feedback

91cd5f4

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

StephenWakely requested review from fuchsnj and spencergilbert March 30, 2023 11:17

Vector schema should have any metadata

2a5456a

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

StephenWakely commented Mar 30, 2023

View reviewed changes

StephenWakely added 4 commits March 30, 2023 15:04

Clippy

12e28b6

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Merge remote-tracking branch 'origin' into stephen/lognamespacing_tra…

45eee4c

…nsforms

Remove into_schema_definition

20e3505

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Add todo for dummy output id

b9467e4

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Fixed test

24d0581

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

spencergilbert reviewed Mar 31, 2023

View reviewed changes

src/topology/schema.rs Outdated Show resolved Hide resolved

src/topology/schema.rs Outdated Show resolved Hide resolved

src/transforms/remap.rs Show resolved Hide resolved

Added text to unreachables

6fbed87

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

spencergilbert approved these changes Apr 3, 2023

View reviewed changes

fuchsnj approved these changes Apr 3, 2023

View reviewed changes

StephenWakely merged commit e19f4fc into master Apr 3, 2023

StephenWakely deleted the stephen/lognamespacing_transforms branch April 3, 2023 13:28

StephenWakely added a commit that referenced this pull request Apr 13, 2023

Revert "enhancement(topology): Update transforms to handle multiple d…

5dc20f3

…efinitions (#16793)" This reverts commit e19f4fc. Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

StephenWakely added a commit that referenced this pull request Apr 14, 2023

Revert "Revert "enhancement(topology): Update transforms to handle mu…

e959c42

…ltiple definitions (#16793)"" This reverts commit 5dc20f3.

StephenWakely added a commit that referenced this pull request Apr 25, 2023

Revert "Revert "enhancement(topology): Update transforms to handle mu…

531009e

…ltiple definitions (#16793)"" This reverts commit 5dc20f3.

	// NOTE: schema definitions are only implemented/supported for log-type events. There is no
	// inherent blocker to support other types as well, but it'll require additional work to add
	// the relevant schemas, and store them separately in this type.
	///
	/// sources if the output type is `Log` there should be a single definition. `Metric` types should
	/// have no definition defined.
	///
	/// For a transforms if `Datatype` is Log, at least one definition should be output. If the
	/// transform has multiple connected sources, it is possible to have multiple output definitions -
	/// one for each input.
	pub log_schema_definitions: Vec<schema::Definition>,

enhancement(topology): Update transforms to handle multiple definitions #16793

enhancement(topology): Update transforms to handle multiple definitions #16793

Uh oh!

Conversation

StephenWakely commented Mar 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Output::log_schema_definitions

TransformConfig::outputs

Topology builder

Output

Transforms

Sources

Uh oh!

Uh oh!

netlify bot commented Mar 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for vrl-playground ready!

Uh oh!

netlify bot commented Mar 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for vector-project canceled.

Uh oh!

neuronull left a comment

Choose a reason for hiding this comment

Uh oh!

datadog-vectordotdev bot commented Mar 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Datadog Report

Uh oh!

StephenWakely Mar 30, 2023

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 31, 2023

Regression Detector Results

Uh oh!

github-actions bot commented Mar 31, 2023

Regression Detector Results

Uh oh!

github-actions bot commented Mar 31, 2023

Regression Detector Results

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Apr 3, 2023

Regression Detector Results

Uh oh!

spencergilbert left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

StephenWakely commented Mar 14, 2023 •

edited

Loading

`Output::log_schema_definitions`

`TransformConfig::outputs`

netlify bot commented Mar 14, 2023 •

edited

Loading

netlify bot commented Mar 14, 2023 •

edited

Loading

datadog-vectordotdev bot commented Mar 30, 2023 •

edited

Loading