Skip to content

WIP: combine galley start mode and add liveness/readiness probe#3

Closed
ayj wants to merge 1 commit intoayj:masterfrom
irisdingbj:issue7500
Closed

WIP: combine galley start mode and add liveness/readiness probe#3
ayj wants to merge 1 commit intoayj:masterfrom
irisdingbj:issue7500

Conversation

@ayj
Copy link
Copy Markdown
Owner

@ayj ayj commented Aug 15, 2018

No description provided.

versionPath = "/version"
)

func startSelfMonitoring(stop <-chan struct{}, port uint) {
Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should be able to share the promhttp handler between the validation and server paths. Consider moving startSelfMonitoring to galley/pkg/server/monitoring.go?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now there is no metrics available in server, so keep the logics for validation only move it once server have added metrics part?

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll be adding metrics shortly. Since you're already moving this code around we might as well put it the common place now :)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done~ Move to galley/pkg/server/galleymonitor.go


var sa = server.DefaultArgs()
var vc = validation.DefaultWebhooktParams()

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could simplify this a bit by hooking PersistantFlags.StringVar() directly to the server and validation args. e.g.

func GetRootCmd(...) {
    var (
        serverArgs = server.DefaultArgs()
        validationARgs = validadtion.DefaultArgs()
    )

    rootCmd := &cobra.Command{ ... }

    rootCmd.PersistentFlags().BoolVar(&serverArgs.server, "server", "Run galley server mode", server.Args.server)
    // etc
}

This would also keep the server.DefaultArgs() and default flag values in sync.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done~

rootCmd.PersistentFlags().BoolVar(&flags.noServer, "no-server", false,
"Run galley server mode")
rootCmd.PersistentFlags().BoolVar(&flags.noValidation, "no-validation", false,
"Run galley validation mode")
Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we reverse the polarity of no-server and no-validation` to avoid double negatives? I think the command line usage would read a bit better, e.g.

galley --no-server=true --no-validation=false
// vs.
galley --enable-server=false --enable-validation

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done~

//Run liveness and readiness check
probeCheckController := galleyProbe.NewProbeCheckController(flags.livenessProbeOptions,flags.readinessProbeOptions)
probeCheckController.Run()
galleyStop := make(chan struct{})
Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we propagate the galleyStop channel to server.RunServer and validation.RunValidation?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in server.RunServer and validation.RunValidation they will call os.Exit(-1) when they encounter errors. So I think we can use this channel to exit if none of them has errors.

noValidation bool
noServer bool
livenessProbeOptions *probe.Options
readinessProbeOptions *probe.Options
Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would declare livenessProbeOptions and readinessProbeOptions as value types instead of pointers and move it to within the scope of GetRootCmd (line ~56). This would eliminate lines 104-105 and the use of global variables.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

"Maximum size of individual gRPC messages")
rootCmd.PersistentFlags().UintVarP(&sa.MaxConcurrentStreams, "server.maxConcurrentStreams", "", sa.MaxConcurrentStreams,
"Maximum number of outstanding RPCs per connection")
rootCmd.PersistentFlags().BoolVarP(&sa.Insecure, "insecure", "", sa.Insecure,
Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing description for insecure flag

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done~

readinessProbeController.Start()
readinessProbe.SetAvailable(nil)
}

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we align this with how mixer uses the readiness probe package? Specifically, the probe package itself seems to already handle updating the health file. We only need to invoke SetAvailable(<status>) based on server and validation readiness.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mixer have not used this part actually because they have not defined liveness and readiness probe options in their helm/docker file. the security part is relying on this mechanism now: https://github.com/istio/istio/blob/master/security/pkg/probe/controller.go , so we are now under the same way with it.

Copy link
Copy Markdown

@irisdingbj irisdingbj Aug 16, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

currently I have used the same logic to detect the liveness and readiness for galley:

func (c *ProbeCheckController) checkLiveness() error {
	content := []byte(`ok`)
	if err := ioutil.WriteFile(c.livenessProbeOptions.Path, content, 0644); err != nil {
		log.Errorf("Health check update of %q failed: %v", c.livenessProbeOptions.Path, err)
		return err

	}
	return nil
}


func (c *ProbeCheckController) checkReadiness() error {
	content := []byte(`ok`)
	if err := ioutil.WriteFile(c.readinessProbeOptions.Path, content, 0644); err != nil {
		log.Errorf("Health check update of %q failed: %v", c.readinessProbeOptions.Path, err)
		return err

	}
	return nil
}

So in case there is error, the probe package itself will get the error and remove the heath file.

So is above logic valid to check the galley liveness and readiness? (Just copy the logic from validation code)?

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See istio#7986 (release-1.0 branch). This change should also be merged into master. This decouples the config reconciliation loop from the health check updates. To do this, I used the istio.io/pkg/probe package instead of writing the file directly. We should use the same mechanism for the galley service health checks.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done~~ Thanks for the clue 👍

// limitations under the License.
package validation

import (
Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI, you'll need to run make format and make lint to help fix the formatting and linter issues before the PR can be merged.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure, Thanks~

@ayj
Copy link
Copy Markdown
Owner Author

ayj commented Aug 15, 2018

fyi @irisdingbj

@ayj ayj closed this Aug 16, 2018
@irisdingbj irisdingbj deleted the issue7500 branch August 29, 2018 01:53
ayj pushed a commit that referenced this pull request Dec 21, 2018
ayj pushed a commit that referenced this pull request Jul 23, 2019
* Fixing iptabes ranges

Signed-off-by: Serguei Bezverkhi <sbezverk@cisco.com>

* fix shellcheck errors

Signed-off-by: Serguei Bezverkhi <sbezverk@cisco.com>

* fixing ci failures #1

Signed-off-by: Serguei Bezverkhi <sbezverk@cisco.com>

* fixing ci failures #2

Signed-off-by: Serguei Bezverkhi <sbezverk@cisco.com>

* fixing ci failures #3

Signed-off-by: Serguei Bezverkhi <sbezverk@cisco.com>

* Addressing comments

Signed-off-by: Serguei Bezverkhi <sbezverk@cisco.com>
ayj pushed a commit that referenced this pull request Jan 13, 2020
Fix domainname of galley in envoy configmap istio-control/istio-discovery
ayj pushed a commit that referenced this pull request Jan 13, 2020
* Installer CRD proto definition

* Review comments

* Review comments

* Add docs and examples

* Review comments

* Review comments

* Review comments

* License text

* Change IstioInstall to IstioControlPlane

* Add components struct layer
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants