Skip to content

Apply newer service_discovery plugin helper methods#3362

Merged
ashie merged 4 commits into
masterfrom
update-out-forward
May 10, 2021
Merged

Apply newer service_discovery plugin helper methods#3362
ashie merged 4 commits into
masterfrom
update-out-forward

Conversation

@ashie

@ashie ashie commented May 7, 2021

Copy link
Copy Markdown
Member

Which issue(s) this PR fixes:
none

What this PR does / why we need it:
Apply a new helper method added at #3299 to out_forward.
This PR is revised version of #3300 which was closed unexpectedly.

Docs Changes:
Same with #3299

Release Note:
Same with #3299

Signed-off-by: TAGOMORI Satoshi <tagomoris@gmail.com>
@ashie

ashie commented May 7, 2021

Copy link
Copy Markdown
Member Author

I'll merge it after CI is completed.

@ashie

ashie commented May 7, 2021

Copy link
Copy Markdown
Member Author

I'll merge it after CI is completed.

Hmm, 2 tests are failed on all platforms 🤔

2021-05-07T07:25:01.5492123Z ===============================================================================
2021-05-07T07:25:01.5512385Z Failure: test: server is an abbreviation of static type of service_discovery(ForwardOutputTest)
2021-05-07T07:25:01.5516251Z /home/runner/work/fluentd/fluentd/test/plugin/test_out_forward.rb:282:in `block in <class:ForwardOutputTest>'
2021-05-07T07:25:01.5518316Z <1234> expected but was
2021-05-07T07:25:01.5519533Z <1235>
2021-05-07T07:25:01.5520310Z 
2021-05-07T07:25:01.5521282Z diff:
2021-05-07T07:25:01.5522222Z ? 1234
2021-05-07T07:25:01.5523154Z ?    5
2021-05-07T07:25:01.5584243Z ===============================================================================
2021-05-07T07:25:03.8035052Z ......F
2021-05-07T07:25:03.8035472Z ===============================================================================
2021-05-07T07:25:03.8036046Z Failure: test: when out_forward has @id(ForwardOutputTest):
2021-05-07T07:25:03.8041597Z   Exception raised:
2021-05-07T07:25:03.8042557Z   RR::Errors::DoubleNotFoundError(<On subject Fluent::Plugin,
2021-05-07T07:25:03.8043314Z   unexpected method invocation:
2021-05-07T07:25:03.8044075Z     new_sd("static", {:parent=>#<Fluent::Plugin::ForwardOutput:000000000329d8>})
2021-05-07T07:25:03.8044761Z   expected invocations:
2021-05-07T07:25:03.8045902Z   - new_sd(:static, {:parent=>anything})>)
2021-05-07T07:25:03.8047111Z   /home/runner/work/fluentd/fluentd/lib/fluent/plugin_helper/service_discovery/manager.rb:43:in `block in configure'
2021-05-07T07:25:03.8184718Z   /home/runner/work/fluentd/fluentd/lib/fluent/plugin_helper/service_discovery/manager.rb:36:in `each'
2021-05-07T07:25:03.8186338Z   /home/runner/work/fluentd/fluentd/lib/fluent/plugin_helper/service_discovery/manager.rb:36:in `configure'
2021-05-07T07:25:03.8187781Z   /home/runner/work/fluentd/fluentd/lib/fluent/plugin_helper/service_discovery.rb:107:in `service_discovery_create_manager'
2021-05-07T07:25:03.8189483Z   /home/runner/work/fluentd/fluentd/lib/fluent/plugin_helper/service_discovery.rb:78:in `service_discovery_configure'
2021-05-07T07:25:03.8190784Z   /home/runner/work/fluentd/fluentd/lib/fluent/plugin/out_forward.rb:230:in `configure'
2021-05-07T07:25:03.8191917Z   /home/runner/work/fluentd/fluentd/lib/fluent/test/driver/base_owner.rb:57:in `configure'
2021-05-07T07:25:03.8193485Z   /home/runner/work/fluentd/fluentd/test/plugin/test_out_forward.rb:1087:in `block (2 levels) in <class:ForwardOutputTest>'
2021-05-07T07:25:03.8195312Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/assertions.rb:627:in `block in assert_nothing_raised'
2021-05-07T07:25:03.8197012Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/assertions.rb:1633:in `_wrap_assertion'
2021-05-07T07:25:03.8198672Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/assertions.rb:618:in `assert_nothing_raised'
2021-05-07T07:25:03.8200271Z   /home/runner/work/fluentd/fluentd/test/plugin/test_out_forward.rb:1086:in `block in <class:ForwardOutputTest>'
2021-05-07T07:25:03.8201774Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/testcase.rb:832:in `run_test'
2021-05-07T07:25:03.8203340Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/testcase.rb:530:in `block (2 levels) in run'
2021-05-07T07:25:03.8205099Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/fixture.rb:276:in `block in create_fixtures_runner'
2021-05-07T07:25:03.8206675Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/fixture.rb:276:in `block in create_fixtures_runner'
2021-05-07T07:25:03.8208467Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/fixture.rb:276:in `block in create_fixtures_runner'
2021-05-07T07:25:03.8209955Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/fixture.rb:257:in `run_fixture'
2021-05-07T07:25:03.8211374Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/fixture.rb:292:in `run_setup'
2021-05-07T07:25:03.8212785Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/testcase.rb:528:in `block in run'
2021-05-07T07:25:03.8214162Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/testcase.rb:527:in `catch'
2021-05-07T07:25:03.8216699Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/testcase.rb:527:in `run'
2021-05-07T07:25:03.8218133Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/testsuite.rb:124:in `run_test'
2021-05-07T07:25:03.8219514Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/testsuite.rb:53:in `run'
2021-05-07T07:25:03.8220914Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/testsuite.rb:124:in `run_test'
2021-05-07T07:25:03.8222285Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/testsuite.rb:53:in `run'
2021-05-07T07:25:03.8223801Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/ui/testrunnermediator.rb:67:in `run_suite'
2021-05-07T07:25:03.8225603Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/ui/testrunnermediator.rb:45:in `block (2 levels) in run'
2021-05-07T07:25:03.8227437Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/ui/testrunnermediator.rb:102:in `with_listener'
2021-05-07T07:25:03.8229110Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/ui/testrunnermediator.rb:41:in `block in run'
2021-05-07T07:25:03.8230721Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/ui/testrunnermediator.rb:39:in `catch'
2021-05-07T07:25:03.8232424Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/ui/testrunnermediator.rb:39:in `run'
2021-05-07T07:25:03.8233995Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/ui/testrunner.rb:40:in `start_mediator'
2021-05-07T07:25:03.8235450Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/ui/testrunner.rb:25:in `start'
2021-05-07T07:25:03.8248847Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/ui/testrunnerutilities.rb:24:in `run'
2021-05-07T07:25:03.8250792Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/autorunner.rb:446:in `block in run'
2021-05-07T07:25:03.8252447Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/autorunner.rb:502:in `change_work_directory'
2021-05-07T07:25:03.8253941Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/autorunner.rb:445:in `run'
2021-05-07T07:25:03.8255331Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/autorunner.rb:66:in `run'
2021-05-07T07:25:03.8256863Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit.rb:518:in `block (2 levels) in <top (required)>'
2021-05-07T07:25:03.8259268Z /home/runner/work/fluentd/fluentd/test/plugin/test_out_forward.rb:1086:in `block in <class:ForwardOutputTest>'
2021-05-07T07:25:03.8259965Z      1083:       end
2021-05-07T07:25:03.8260277Z      1084:     }
2021-05-07T07:25:03.8260557Z      1085: 
2021-05-07T07:25:03.8260930Z   => 1086:     assert_nothing_raised do
2021-05-07T07:25:03.8261421Z      1087:       output.configure(CONFIG + %[
2021-05-07T07:25:03.8261910Z      1088:         @id unique_out_forward
2021-05-07T07:25:03.8262255Z      1089:       ])
2021-05-07T07:25:03.8262653Z ===============================================================================

ashie added 2 commits May 7, 2021 17:57
Plugin type is a string value, so the test should exepect a string
value while the previous out_forward code pass a symbol value as a
service_discovery plugin type by one-off code.

Signed-off-by: Takuro Ashie <ashie@clear-code.com>
A test should ensure to have only one assertion because trailing
assertions aren't evalueated when an assertions is failed. It makes
debugging hard. Here is an example output of the test:

before:

  Failure: test: server is an abbreviation of static type of service_discovery(ForwardOutputTest)
  /home/aho/Projects/Fluentd/fluentd/test/plugin/test_out_forward.rb:282:in `block in <class:ForwardOutputTest>'
  <1234> expected but was
  <1235>

  diff:
  ? 1234
  ?    5

after:

  Failure: test: server is an abbreviation of static type of service_discovery(ForwardOutputTest)
  /home/aho/Projects/Fluentd/fluentd/test/plugin/test_out_forward.rb:281:in `block in <class:ForwardOutputTest>'
  <[{:host=>"127.0.0.1", :port=>1234}, {:host=>"127.0.0.1", :port=>1235}]> expected but was
  <[{:host=>"127.0.0.1", :port=>1235}, {:host=>"127.0.0.1", :port=>1234}]>

  diff:
  ? [{:host=>"127.0.0.1", :port=>1234}, {:host=>"127.0.0.1", :port=>1235}]
  ?                                 5                                  4

The later one is easy to understand that the array is inverted.

Signed-off-by: Takuro Ashie <ashie@clear-code.com>
@ashie

ashie commented May 7, 2021

Copy link
Copy Markdown
Member Author

BTW although tests in fluentd tend to check arrays by separated assertions like this:

    assert_equal 2, d.instance.discovery_manager.services.size
    assert_equal '127.0.0.1', d.instance.discovery_manager.services[0].host
    assert_equal 1234, d.instance.discovery_manager.services[0].port
    assert_equal '127.0.0.1', d.instance.discovery_manager.services[1].host
    assert_equal 1235, d.instance.discovery_manager.services[1].port

I don't like such tests because it makes hard to debug.
A test should ensure to have only one assertion because trailing assertions aren't evaluated when an assertion is failed.
For above case, it can be replaced by the following code:

    assert_equal(
      [
        { host: '127.0.0.1', port: 1234 },
        { host: '127.0.0.1', port: 1235 },
      ],
      d.instance.discovery_manager.services.collect do |service|
        { host: service.host, port: service.port }
      end
    )

Here is an example output:

before:

Failure: test: server is an abbreviation of static type of service_discovery(ForwardOutputTest)
/home/aho/Projects/Fluentd/fluentd/test/plugin/test_out_forward.rb:282:in `block in <class:ForwardOutputTest>'
<1234> expected but was
<1235>

diff:
? 1234
?    5

after:

Failure: test: server is an abbreviation of static type of service_discovery(ForwardOutputTest)
/home/aho/Projects/Fluentd/fluentd/test/plugin/test_out_forward.rb:281:in `block in <class:ForwardOutputTest>'
<[{:host=>"127.0.0.1", :port=>1234}, {:host=>"127.0.0.1", :port=>1235}]> expected but was
<[{:host=>"127.0.0.1", :port=>1235}, {:host=>"127.0.0.1", :port=>1234}]>

diff:
? [{:host=>"127.0.0.1", :port=>1234}, {:host=>"127.0.0.1", :port=>1235}]
?                                 5                                  4

The later one is easy to understand that the array is inverted.

"server" directive should have higher priority than "service" in
"service_discovery" to keep backward compatibility.

Signed-off-by: Takuro Ashie <ashie@clear-code.com>
@ashie

ashie commented May 10, 2021

Copy link
Copy Markdown
Member Author

tests passed (without a known unstable test).

@tagomoris

Copy link
Copy Markdown
Member

Oops, I missed the notification about the merge of #3299 and your follow-up comments.
@ashie I'm sorry and thank you for creating this change.

@ashie

ashie commented Jun 14, 2021

Copy link
Copy Markdown
Member Author

Don't worry 😃
BTW we also modified the document for it: fluent/fluentd-docs-gitbook#321
Please let me know if you find any problem.

@tagomoris

Copy link
Copy Markdown
Member

That looks perfect!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants