Skip to content

feature: improve error messages for package field validation checks #994

Open
Open
feature: improve error messages for package field validation checks#994
Labels
Team:EcosystemLabel for the Packages Ecosystem teamstale
@efd6

Description

@efd6
efd6
opened on Sep 29, 2022

I have recently been doing a large update of packages in integrations and have found in that process that the errors that are printed could be improved.

Two example cases:

  1. Invalid field type 
    1. file "/Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/gcp-2.12.0.zip/data_stream/audit/fields/fields.yml" is invalid: field 0.fields.2.type: 0.fields.2.type must be one of the following: "alias", "histogram", "constant_keyword", "text", "match_only_text", "keyword", "long", "integer", "short", "byte", "double", "float", "half_float", "scaled_float", "date", "date_nanos", "boolean", "binary", "integer_range", "float_range", "long_range", "double_range", "date_range", "ip_range", "group", "geo_point", "object", "ip", "nested", "flattened", "wildcard", "version", "unsigned_long"
  2. Duplicated field definition 
    1. field "cloud.instance.id" is defined multiple times for data stream "storage", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/gcp-2.12.0.zip/data_stream/storage/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/gcp-2.12.0.zip/data_stream/storage/fields/ecs.yml

In both cases there is unnecessary information that reduces the readability of the error, particularly when there are many lines of errors. So I would suggest that the file paths be made relative to the package at least, and probably to the data stream.

In the case of the invalid field, the value that is invalid is not displayed, only the values that it may take. This in conjunction with the fact that the name of the field is not printed, only its position in the AST, make it difficult to find the offending value. In the case above it is not particularly hard since it is early in the file, but some are quite deep. So I would suggest that the offending value be printed, the name of the field (i.e. the value of the name field in the yml) and the file position also be provided (e.g. audit/fields/fields.yml:21:13 in the case above) to allow direct movement to the error in an editor.

The impact of the current output format can be clearly demonstrated with the following detail.

Details 
Error: building package failed: invalid content found in built zip package: found 87 validation errors:
   1. field "host.ip" is defined multiple times for data stream "rfb", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/rfb/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/rfb/fields/ecs.yml
   2. field "container.id" is defined multiple times for data stream "rfb", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/rfb/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/rfb/fields/beats.yml
   3. field "container.id" is defined multiple times for data stream "smb_cmd", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/smb_cmd/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/smb_cmd/fields/beats.yml
   4. field "host.ip" is defined multiple times for data stream "smb_cmd", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/smb_cmd/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/smb_cmd/fields/ecs.yml
   5. field "container.id" is defined multiple times for data stream "smb_files", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/smb_files/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/smb_files/fields/beats.yml
   6. field "host.ip" is defined multiple times for data stream "smb_files", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/smb_files/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/smb_files/fields/ecs.yml
   7. field "host.ip" is defined multiple times for data stream "smb_mapping", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/smb_mapping/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/smb_mapping/fields/ecs.yml
   8. field "container.id" is defined multiple times for data stream "smb_mapping", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/smb_mapping/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/smb_mapping/fields/beats.yml
   9. field "container.id" is defined multiple times for data stream "dnp3", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dnp3/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dnp3/fields/beats.yml
  10. field "host.ip" is defined multiple times for data stream "dnp3", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dnp3/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dnp3/fields/ecs.yml
  11. field "host.ip" is defined multiple times for data stream "ntlm", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ntlm/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ntlm/fields/ecs.yml
  12. field "container.id" is defined multiple times for data stream "ntlm", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ntlm/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ntlm/fields/beats.yml
  13. field "container.id" is defined multiple times for data stream "rdp", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/rdp/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/rdp/fields/beats.yml
  14. field "host.ip" is defined multiple times for data stream "rdp", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/rdp/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/rdp/fields/ecs.yml
  15. field "host.ip" is defined multiple times for data stream "http", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/http/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/http/fields/ecs.yml
  16. field "container.id" is defined multiple times for data stream "http", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/http/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/http/fields/beats.yml
  17. field "host.ip" is defined multiple times for data stream "known_certs", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/known_certs/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/known_certs/fields/ecs.yml
  18. field "container.id" is defined multiple times for data stream "known_certs", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/known_certs/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/known_certs/fields/beats.yml
  19. field "host.ip" is defined multiple times for data stream "modbus", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/modbus/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/modbus/fields/ecs.yml
  20. field "container.id" is defined multiple times for data stream "modbus", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/modbus/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/modbus/fields/beats.yml
  21. field "host.ip" is defined multiple times for data stream "mysql", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/mysql/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/mysql/fields/ecs.yml
  22. field "container.id" is defined multiple times for data stream "mysql", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/mysql/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/mysql/fields/beats.yml
  23. field "host.ip" is defined multiple times for data stream "ntp", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ntp/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ntp/fields/ecs.yml
  24. field "container.id" is defined multiple times for data stream "ntp", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ntp/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ntp/fields/beats.yml
  25. field "container.id" is defined multiple times for data stream "dhcp", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dhcp/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dhcp/fields/beats.yml
  26. field "host.ip" is defined multiple times for data stream "dhcp", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dhcp/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dhcp/fields/ecs.yml
  27. field "host.ip" is defined multiple times for data stream "dns", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dns/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dns/fields/ecs.yml
  28. field "container.id" is defined multiple times for data stream "dns", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dns/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dns/fields/beats.yml
  29. field "host.ip" is defined multiple times for data stream "files", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/files/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/files/fields/ecs.yml
  30. field "container.id" is defined multiple times for data stream "files", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/files/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/files/fields/beats.yml
  31. field "host.ip" is defined multiple times for data stream "socks", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/socks/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/socks/fields/ecs.yml
  32. field "container.id" is defined multiple times for data stream "socks", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/socks/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/socks/fields/beats.yml
  33. field "host.ip" is defined multiple times for data stream "tunnel", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/tunnel/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/tunnel/fields/ecs.yml
  34. field "container.id" is defined multiple times for data stream "tunnel", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/tunnel/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/tunnel/fields/beats.yml
  35. field "host.ip" is defined multiple times for data stream "ftp", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ftp/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ftp/fields/ecs.yml
  36. field "container.id" is defined multiple times for data stream "ftp", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ftp/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ftp/fields/beats.yml
  37. field "host.ip" is defined multiple times for data stream "radius", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/radius/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/radius/fields/ecs.yml
  38. field "container.id" is defined multiple times for data stream "radius", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/radius/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/radius/fields/beats.yml
  39. field "host.ip" is defined multiple times for data stream "smtp", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/smtp/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/smtp/fields/ecs.yml
  40. field "container.id" is defined multiple times for data stream "smtp", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/smtp/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/smtp/fields/beats.yml
  41. field "host.ip" is defined multiple times for data stream "signature", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/signature/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/signature/fields/ecs.yml
  42. field "container.id" is defined multiple times for data stream "signature", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/signature/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/signature/fields/beats.yml
  43. field "container.id" is defined multiple times for data stream "ssl", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ssl/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ssl/fields/beats.yml
  44. field "tls.version_protocol" is defined multiple times for data stream "ssl", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ssl/fields/ecs.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ssl/fields/ecs.yml
  45. field "host.ip" is defined multiple times for data stream "ssl", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ssl/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ssl/fields/ecs.yml
  46. field "container.id" is defined multiple times for data stream "syslog", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/syslog/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/syslog/fields/beats.yml
  47. field "host.ip" is defined multiple times for data stream "syslog", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/syslog/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/syslog/fields/ecs.yml
  48. field "host.ip" is defined multiple times for data stream "traceroute", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/traceroute/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/traceroute/fields/ecs.yml
  49. field "container.id" is defined multiple times for data stream "traceroute", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/traceroute/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/traceroute/fields/beats.yml
  50. field "host.ip" is defined multiple times for data stream "intel", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/intel/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/intel/fields/ecs.yml
  51. field "container.id" is defined multiple times for data stream "intel", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/intel/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/intel/fields/beats.yml
  52. field "container.id" is defined multiple times for data stream "known_hosts", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/known_hosts/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/known_hosts/fields/beats.yml
  53. field "host.ip" is defined multiple times for data stream "known_hosts", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/known_hosts/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/known_hosts/fields/ecs.yml
  54. field "host.ip" is defined multiple times for data stream "ocsp", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ocsp/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ocsp/fields/ecs.yml
  55. field "container.id" is defined multiple times for data stream "ocsp", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ocsp/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ocsp/fields/beats.yml
  56. field "container.id" is defined multiple times for data stream "irc", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/irc/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/irc/fields/beats.yml
  57. field "host.ip" is defined multiple times for data stream "irc", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/irc/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/irc/fields/ecs.yml
  58. field "container.id" is defined multiple times for data stream "kerberos", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/kerberos/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/kerberos/fields/beats.yml
  59. field "host.ip" is defined multiple times for data stream "kerberos", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/kerberos/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/kerberos/fields/ecs.yml
  60. field "container.id" is defined multiple times for data stream "sip", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/sip/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/sip/fields/beats.yml
  61. field "host.ip" is defined multiple times for data stream "sip", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/sip/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/sip/fields/ecs.yml
  62. field "container.id" is defined multiple times for data stream "connection", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/connection/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/connection/fields/beats.yml
  63. field "host.ip" is defined multiple times for data stream "connection", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/connection/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/connection/fields/ecs.yml
  64. field "container.id" is defined multiple times for data stream "dpd", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dpd/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dpd/fields/beats.yml
  65. field "host.ip" is defined multiple times for data stream "dpd", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dpd/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dpd/fields/ecs.yml
  66. field "container.id" is defined multiple times for data stream "known_services", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/known_services/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/known_services/fields/beats.yml
  67. field "host.ip" is defined multiple times for data stream "known_services", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/known_services/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/known_services/fields/ecs.yml
  68. field "host.ip" is defined multiple times for data stream "pe", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/pe/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/pe/fields/ecs.yml
  69. field "container.id" is defined multiple times for data stream "pe", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/pe/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/pe/fields/beats.yml
  70. field "container.id" is defined multiple times for data stream "ssh", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ssh/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ssh/fields/beats.yml
  71. field "host.ip" is defined multiple times for data stream "ssh", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ssh/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ssh/fields/ecs.yml
  72. field "host.ip" is defined multiple times for data stream "snmp", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/snmp/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/snmp/fields/ecs.yml
  73. field "container.id" is defined multiple times for data stream "snmp", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/snmp/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/snmp/fields/beats.yml
  74. field "container.id" is defined multiple times for data stream "software", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/software/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/software/fields/beats.yml
  75. field "host.ip" is defined multiple times for data stream "software", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/software/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/software/fields/ecs.yml
  76. field "container.id" is defined multiple times for data stream "stats", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/stats/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/stats/fields/beats.yml
  77. field "host.ip" is defined multiple times for data stream "stats", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/stats/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/stats/fields/ecs.yml
  78. field "host.ip" is defined multiple times for data stream "weird", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/weird/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/weird/fields/ecs.yml
  79. field "container.id" is defined multiple times for data stream "weird", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/weird/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/weird/fields/beats.yml
  80. field "host.ip" is defined multiple times for data stream "x509", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/x509/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/x509/fields/ecs.yml
  81. field "container.id" is defined multiple times for data stream "x509", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/x509/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/x509/fields/beats.yml
  82. field "container.id" is defined multiple times for data stream "capture_loss", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/capture_loss/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/capture_loss/fields/beats.yml
  83. field "host.ip" is defined multiple times for data stream "capture_loss", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/capture_loss/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/capture_loss/fields/ecs.yml
  84. field "host.ip" is defined multiple times for data stream "dce_rpc", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dce_rpc/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dce_rpc/fields/ecs.yml
  85. field "container.id" is defined multiple times for data stream "dce_rpc", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dce_rpc/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dce_rpc/fields/beats.yml
  86. field "container.id" is defined multiple times for data stream "notice", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/notice/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/notice/fields/beats.yml
  87. field "host.ip" is defined multiple times for data stream "notice", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/notice/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/notice/fields/ecs.yml

This would be made much clearer if the path were made relative to the data stream, file positions were emitted and the outputs were made to be consistent between runs (preferably grouped by file in sections in lexical order of datastream name then file name grouped and then by position in file). A sketch of how this might look is in the following detail.

Details 
Error: building package failed: invalid content found in built zip package: found 87 validation errors:
in data stream "capture_loss"
	"container.id" is defined multiple times
		capture_loss/fields/agent.yml:123:34
		capture_loss/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		capture_loss/fields/agent.yml:123:34
		capture_loss/fields/ecs.yml:123:34
in data stream "connection"
	"container.id" is defined multiple times
		connection/fields/agent.yml:123:34
		connection/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		connection/fields/agent.yml:123:34
		connection/fields/ecs.yml:123:34
in data stream "dce_rpc"
	"host.ip" is defined multiple times
		dce_rpc/fields/agent.yml:123:34
		dce_rpc/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		dce_rpc/fields/agent.yml:123:34
		dce_rpc/fields/beats.yml:123:34
in data stream "dhcp"
	"container.id" is defined multiple times
		dhcp/fields/agent.yml:123:34
		dhcp/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		dhcp/fields/agent.yml:123:34
		dhcp/fields/ecs.yml:123:34
in data stream "dnp3"
	"container.id" is defined multiple times
		dnp3/fields/agent.yml:123:34
		dnp3/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		dnp3/fields/agent.yml:123:34
		dnp3/fields/ecs.yml:123:34
in data stream "dns"
	"host.ip" is defined multiple times
		dns/fields/agent.yml:123:34
		dns/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		dns/fields/agent.yml:123:34
		dns/fields/beats.yml:123:34
in data stream "dpd"
	"container.id" is defined multiple times
		dpd/fields/agent.yml:123:34
		dpd/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		dpd/fields/agent.yml:123:34
		dpd/fields/ecs.yml:123:34
in data stream "files"
	"host.ip" is defined multiple times
		files/fields/agent.yml:123:34
		files/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		files/fields/agent.yml:123:34
		files/fields/beats.yml:123:34
in data stream "ftp"
	"host.ip" is defined multiple times
		ftp/fields/agent.yml:123:34
		ftp/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		ftp/fields/agent.yml:123:34
		ftp/fields/beats.yml:123:34
in data stream "http"
	"host.ip" is defined multiple times
		http/fields/agent.yml:123:34
		http/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		http/fields/agent.yml:123:34
		http/fields/beats.yml:123:34
in data stream "intel"
	"host.ip" is defined multiple times
		intel/fields/agent.yml:123:34
		intel/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		intel/fields/agent.yml:123:34
		intel/fields/beats.yml:123:34
in data stream "irc"
	"container.id" is defined multiple times
		irc/fields/agent.yml:123:34
		irc/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		irc/fields/agent.yml:123:34
		irc/fields/ecs.yml:123:34
in data stream "kerberos"
	"container.id" is defined multiple times
		kerberos/fields/agent.yml:123:34
		kerberos/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		kerberos/fields/agent.yml:123:34
		kerberos/fields/ecs.yml:123:34
in data stream "known_certs"
	"host.ip" is defined multiple times
		known_certs/fields/agent.yml:123:34
		known_certs/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		known_certs/fields/agent.yml:123:34
		known_certs/fields/beats.yml:123:34
in data stream "known_hosts"
	"container.id" is defined multiple times
		known_hosts/fields/agent.yml:123:34
		known_hosts/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		known_hosts/fields/agent.yml:123:34
		known_hosts/fields/ecs.yml:123:34
in data stream "known_services"
	"container.id" is defined multiple times
		known_services/fields/agent.yml:123:34
		known_services/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		known_services/fields/agent.yml:123:34
		known_services/fields/ecs.yml:123:34
in data stream "modbus"
	"host.ip" is defined multiple times
		modbus/fields/agent.yml:123:34
		modbus/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		modbus/fields/agent.yml:123:34
		modbus/fields/beats.yml:123:34
in data stream "mysql"
	"host.ip" is defined multiple times
		mysql/fields/agent.yml:123:34
		mysql/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		mysql/fields/agent.yml:123:34
		mysql/fields/beats.yml:123:34
in data stream "notice"
	"container.id" is defined multiple times
		notice/fields/agent.yml:123:34
		notice/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		notice/fields/agent.yml:123:34
		notice/fields/ecs.yml:123:34
in data stream "ntlm"
	"host.ip" is defined multiple times
		ntlm/fields/agent.yml:123:34
		ntlm/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		ntlm/fields/agent.yml:123:34
		ntlm/fields/beats.yml:123:34
in data stream "ntp"
	"host.ip" is defined multiple times
		ntp/fields/agent.yml:123:34
		ntp/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		ntp/fields/agent.yml:123:34
		ntp/fields/beats.yml:123:34
in data stream "ocsp"
	"host.ip" is defined multiple times
		ocsp/fields/agent.yml:123:34
		ocsp/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		ocsp/fields/agent.yml:123:34
		ocsp/fields/beats.yml:123:34
in data stream "pe"
	"host.ip" is defined multiple times
		pe/fields/agent.yml:123:34
		pe/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		pe/fields/agent.yml:123:34
		pe/fields/beats.yml:123:34
in data stream "radius"
	"host.ip" is defined multiple times
		radius/fields/agent.yml:123:34
		radius/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		radius/fields/agent.yml:123:34
		radius/fields/beats.yml:123:34
in data stream "rdp"
	"container.id" is defined multiple times
		rdp/fields/agent.yml:123:34
		rdp/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		rdp/fields/agent.yml:123:34
		rdp/fields/ecs.yml:123:34
in data stream "rfb"
	"host.ip" is defined multiple times
		rfb/fields/agent.yml:123:34
		rfb/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		rfb/fields/agent.yml:123:34
		rfb/fields/beats.yml:123:34
in data stream "signature"
	"host.ip" is defined multiple times
		signature/fields/agent.yml:123:34
		signature/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		signature/fields/agent.yml:123:34
		signature/fields/beats.yml:123:34
in data stream "sip"
	"container.id" is defined multiple times
		sip/fields/agent.yml:123:34
		sip/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		sip/fields/agent.yml:123:34
		sip/fields/ecs.yml:123:34
in data stream "smb_cmd"
	"container.id" is defined multiple times
		smb_cmd/fields/agent.yml:123:34
		smb_cmd/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		smb_cmd/fields/agent.yml:123:34
		smb_cmd/fields/ecs.yml:123:34
in data stream "smb_files"
	"container.id" is defined multiple times
		smb_files/fields/agent.yml:123:34
		smb_files/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		smb_files/fields/agent.yml:123:34
		smb_files/fields/ecs.yml:123:34
in data stream "smb_mapping"
	"host.ip" is defined multiple times
		smb_mapping/fields/agent.yml:123:34
		smb_mapping/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		smb_mapping/fields/agent.yml:123:34
		smb_mapping/fields/beats.yml:123:34
in data stream "smtp"
	"host.ip" is defined multiple times
		smtp/fields/agent.yml:123:34
		smtp/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		smtp/fields/agent.yml:123:34
		smtp/fields/beats.yml:123:34
in data stream "snmp"
	"host.ip" is defined multiple times
		snmp/fields/agent.yml:123:34
		snmp/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		snmp/fields/agent.yml:123:34
		snmp/fields/beats.yml:123:34
in data stream "socks"
	"host.ip" is defined multiple times
		socks/fields/agent.yml:123:34
		socks/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		socks/fields/agent.yml:123:34
		socks/fields/beats.yml:123:34
in data stream "software"
	"container.id" is defined multiple times
		software/fields/agent.yml:123:34
		software/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		software/fields/agent.yml:123:34
		software/fields/ecs.yml:123:34
in data stream "ssh"
	"container.id" is defined multiple times
		ssh/fields/agent.yml:123:34
		ssh/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		ssh/fields/agent.yml:123:34
		ssh/fields/ecs.yml:123:34
in data stream "ssl"
	"container.id" is defined multiple times
		ssl/fields/agent.yml:123:34
		ssl/fields/beats.yml:123:34
	"tls.version_protocol" is defined multiple times
		ssl/fields/ecs.yml:123:34
		ssl/fields/ecs.yml:123:34
	"host.ip" is defined multiple times
		ssl/fields/agent.yml:123:34
		ssl/fields/ecs.yml:123:34
in data stream "stats"
	"container.id" is defined multiple times
		stats/fields/agent.yml:123:34
		stats/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		stats/fields/agent.yml:123:34
		stats/fields/ecs.yml:123:34
in data stream "syslog"
	"container.id" is defined multiple times
		syslog/fields/agent.yml:123:34
		syslog/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		syslog/fields/agent.yml:123:34
		syslog/fields/ecs.yml:123:34
in data stream "traceroute"
	"host.ip" is defined multiple times
		traceroute/fields/agent.yml:123:34
		traceroute/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		traceroute/fields/agent.yml:123:34
		traceroute/fields/beats.yml:123:34
in data stream "tunnel"
	"host.ip" is defined multiple times
		tunnel/fields/agent.yml:123:34
		tunnel/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		tunnel/fields/agent.yml:123:34
		tunnel/fields/beats.yml:123:34
in data stream "weird"
	"host.ip" is defined multiple times
		weird/fields/agent.yml:123:34
		weird/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		weird/fields/agent.yml:123:34
		weird/fields/beats.yml:123:34
in data stream "x509"
	"host.ip" is defined multiple times
		x509/fields/agent.yml:123:34
		x509/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		x509/fields/agent.yml:123:34
		x509/fields/beats.yml:123:34

It is easier to read, follows the ordering of the files in an editor's file pane and the fields are presented in the order that you would find them in the files as they are being edited (not done in the example here but is straighforward to do mechanically).

Metadata

Metadata

Assignees

No one assigned

    Labels

    Team:EcosystemLabel for the Packages Ecosystem teamstale

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions