Skip to content

Cherry-pick #10760 to 6.8: Adding Cisco support for the Syslog parser#15980

Merged
ph merged 1 commit intoelastic:6.8from
hrak:6.8
Mar 5, 2020
Merged

Cherry-pick #10760 to 6.8: Adding Cisco support for the Syslog parser#15980
ph merged 1 commit intoelastic:6.8from
hrak:6.8

Conversation

@hrak
Copy link
Copy Markdown

@hrak hrak commented Jan 31, 2020 •
edited by andresrc
Loading

Adding Cisco support for the Syslog parser

  • Adding Cisco support for the Syslog parser

Add support for the "sequence" number in the log format send by Cisco switch devices.

Fixes: #10654, #15979

(cherry picked from commit dd92b6f)

What does this PR do?

Apart from adding support for the Cisco sequence number in syslog messages, this PR fixes several issues with the hostname parser (f.e. hostnames with dashes, IPv6 addresses in hostname field)

Why is it important?

The hostname parser in the syslog input in filebeat 6 is currently broken, resulting in everything after the timestamp of a rfc3164 syslog message ending up in the message field (hostname, program name, PID, message)

Checklist

- [ ] My code follows the style guidelines of this project
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] I have made corresponding change to the default configuration files
- [ ] I have added tests that prove my fix is effective or that my feature works

@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jan 31, 2020

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

1 similar comment
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jan 31, 2020

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

@ChrsMark ChrsMark requested a review from ph January 31, 2020 11:27
@andresrc andresrc requested review from exekias and urso February 20, 2020 12:54
@urso
Copy link
Copy Markdown

urso commented Feb 20, 2020 •
edited
Loading

Thank you for contributing.

The backport seems to cause some CI failures in some filebeat modules that require the syslog input. @ph any idea if we are missing some other commits?

For example:

[fail] 27.99% test_xpack_modules.XPackTest.test_fileset_file_0_iptables: 4.1255s

[success] 13.15% test_xpack_modules.XPackTest.test_fileset_file_6_suricata: 1.9384s

[success] 13.07% test_xpack_modules.XPackTest.test_fileset_file_5_suricata: 1.9269s

[fail] 12.58% test_xpack_modules.XPackTest.test_fileset_file_1_iptables: 1.8540s

[fail] 11.93% test_xpack_modules.XPackTest.test_fileset_file_2_iptables: 1.7576s

[fail] 11.58% test_xpack_modules.XPackTest.test_fileset_file_3_iptables: 1.7064s

[fail] 9.69% test_xpack_modules.XPackTest.test_fileset_file_4_iptables: 1.4287s

----------------------------------------------------------------------

@ph
Copy link
Copy Markdown
Contributor

ph commented Feb 21, 2020

sorry @ChrsMark I've missed the ping on this.

This is indeed strange the main PR was indeed green when merged. Looking at the tables failures in the test for iptable the expected object doesn't match anymore but we do have a partial document extracted.

So I think we are missing commit for surricata?

@ph
Copy link
Copy Markdown
Contributor

ph commented Feb 21, 2020

Going to build the 6.8 branch locally and see what is going on.

@ph
Copy link
Copy Markdown
Contributor

ph commented Feb 21, 2020

Ok, plain 6.8 branch fails with the same errors.

@ph ph mentioned this pull request Feb 21, 2020
Merged
5 tasks
ph added a commit that referenced this pull request Feb 24, 2020
@ph
Fix the iptables issues (#16491)
c3db742 
* Fix the iptables issues

The fileset of the iptables test didn't skip the @timestamp field, so
I have included the iptables.log data to the list of dataset to remove.

I also changed the assertion slighly to be relevant on the order of the
expectation and instead of dropping the JSON output in the log we will
display the difference as a diff.

Fix issues with #15980
@ph
Copy link
Copy Markdown
Contributor

ph commented Feb 24, 2020

@hrak do you mind rebasing this PR on 6.8?

@ph @hrak
Cherry-pick elastic#10760 to 6.8:
245b39c 
Adding Cisco support for the Syslog parser

* Adding Cisco support for the Syslog parser

Add support for the "sequence" number in the log format send by Cisco switch devices.

Fixes: elastic#10654, elastic#15979

(cherry picked from commit dd92b6f)
@hrak
Copy link
Copy Markdown
Author

hrak commented Mar 5, 2020

@ph done, looking much better now. One unrelated test failure in metricbeat tests:

ERROR: Service 'jolokia' failed to build: The command '/bin/sh -c wget http://archive.apache.org/dist/tomcat/tomcat-7/v${TOMCAT_VERSION}/bin/${TC}.tar.gz;    tar xzf ${TC}.tar.gz -C /usr;    rm ${TC}.tar.gz;    sed -i -e 's/Connector port="8080"/Connector port="8778"/g' /usr/${TC}/conf/server.xml;    wget http://central.maven.org/maven2/org/jolokia/jolokia-war/${JOLOKIA_VERSION}/jolokia-war-${JOLOKIA_VERSION}.war -O /usr/${TC}/webapps/jolokia.war' returned a non-zero code: 1

@urso
Copy link
Copy Markdown

urso commented Mar 5, 2020

LGTM. Failing test is unrelated.

@ph you want to have a final look?

@urso
Copy link
Copy Markdown

urso commented Mar 5, 2020

Jenkins, test this.

ph
ph approved these changes Mar 5, 2020
View reviewed changes
Copy link
Copy Markdown
Contributor

@ph ph left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ph ph merged commit ecd273d into elastic:6.8 Mar 5, 2020
@ph
Copy link
Copy Markdown
Contributor

ph commented Mar 5, 2020

@hrak thanks it's merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@urso urso Awaiting requested review from urso

@exekias exekias Awaiting requested review from exekias

1 more reviewer

@ph ph ph approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

backport review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@hrak @elasticmachine @urso @ph @ChrsMark