fix(controller): ensure log level is respected #1135

onematchfox · 2025-11-26T08:22:22Z

This PR ensures that the log level specified when installing kagent is respected by the controller. Currently this is not respected because when Development: true is set, controller-runtime's zap logger enables verbosity level (V(1)) logs by default, which get logged as log.V(1).Info(...) but displayed as DEBUG in the output. -zap-log-level controls the minimum log level (info, error, etc.), but does NOT control the verbosity threshold for V(n) logs.

Note:

Switching away from development logging by default also switches the default log format from ConsoleEncoder to JsonEncoder (see examples below)¹.

Pre this PR:

2025-11-26T07:07:20Z	INFO	Starting KAgent Controller	{"version": "v0.7.5-1-g9438c9c0", "git_commit": "9438c9c0", "build_date": "2025-11-26"}
2025-11-26T07:07:20Z	INFO	Config	{"config": {"Metrics":{"Addr":"0","CertPath":"","CertName":"tls.crt","CertKey":"tls.key"},"Streaming":{"MaxBufSize":"1Mi","InitialBufSize":"4Ki","Timeout":600000000000},"LeaderElection":false,"ProbeAddr":":8082","SecureMetrics":true,"EnableHTTP2":false,"DefaultModelConfig":{"Namespace":"kagent","Name":"default-model-config"},"HttpServerAddr":":8083","WatchNamespaces":"","A2ABaseUrl":"http://127.0.0.1:8083","Database":{"Type":"sqlite","Path":"/sqlite-volume/kagent.db","Url":"postgres://postgres:kagent@db.kagent.svc.cluster.local:5432/crud"}}}
2025-11-26T07:07:20Z	INFO	Updating GOMAXPROCS=2: determined from CPU quota
2025-11-26T07:07:20Z	INFO	setup	Starting KAgent Controller	{"version": "v0.7.5-1-g9438c9c0", "git_commit": "9438c9c0", "build_date": "2025-11-26"}
2025-11-26T07:07:20Z	INFO	setup	Watching all namespaces (no valid namespaces specified)
2025-11-26T07:07:20Z	INFO	MCPServer CRD not found - controller will not be started	{"controller": "mcpserver"}
2025-11-26T07:07:20Z	INFO	Could not find CRD for tool server - API integration will be disabled	{"toolServerType": "MCPServer.kagent.dev"}
2025-11-26T07:07:20Z	INFO	setup	starting manager
2025-11-26T07:07:20Z	INFO	starting server	{"name": "health probe", "addr": "[::]:8082"}
2025-11-26T07:07:20Z	INFO	http-server	Starting HTTP server	{"address": ":8083"}
2025-11-26T07:07:20Z	INFO	Starting EventSource	{"controller": "service", "controllerGroup": "", "controllerKind": "Service", "source": "kind source: *v1.Service"}
2025-11-26T07:07:20Z	INFO	Starting Controller	{"controller": "service", "controllerGroup": "", "controllerKind": "Service"}
2025-11-26T07:07:20Z	INFO	Starting workers	{"controller": "service", "controllerGroup": "", "controllerKind": "Service", "worker count": 1}
2025-11-26T07:07:20Z	INFO	Starting EventSource	{"controller": "modelconfig", "controllerGroup": "kagent.dev", "controllerKind": "ModelConfig", "source": "kind source: *v1.Secret"}
2025-11-26T07:07:20Z	INFO	Starting EventSource	{"controller": "agent", "controllerGroup": "kagent.dev", "controllerKind": "Agent", "source": "kind source: *v1.Service"}
2025-11-26T07:07:20Z	INFO	setup	starting pprof server
2025-11-26T07:07:20Z	INFO	setup	pprof server started	{"address": ":6060"}
...
2025-11-26T07:07:20Z	INFO	Starting Controller	{"controller": "agent", "controllerGroup": "kagent.dev", "controllerKind": "Agent"}
2025-11-26T07:07:20Z	INFO	Starting workers	{"controller": "agent", "controllerGroup": "kagent.dev", "controllerKind": "Agent", "worker count": 1}
2025-11-26T07:07:20Z	INFO	Starting Controller	{"controller": "remotemcpserver", "controllerGroup": "kagent.dev", "controllerKind": "RemoteMCPServer"}
2025-11-26T07:07:20Z	INFO	Starting workers	{"controller": "remotemcpserver", "controllerGroup": "kagent.dev", "controllerKind": "RemoteMCPServer", "worker count": 1}
2025-11-26T07:07:48Z	DEBUG	http	Request started	{"method": "GET", "path": "/health", "remote_addr": "10.244.0.1:34006"}
2025-11-26T07:07:48Z	DEBUG	http.health-handler	Handling health check request	{"method": "GET", "path": "/health", "remote_addr": "10.244.0.1:34006"}
2025-11-26T07:07:48Z	INFO	http	Request completed	{"method": "GET", "path": "/health", "remote_addr": "10.244.0.1:34006", "status": 200, "duration": "117.034µs"}
2025-11-26T07:07:49Z	DEBUG	http	Request started	{"method": "GET", "path": "/health", "remote_addr": "10.244.0.1:47646"}
2025-11-26T07:07:49Z	DEBUG	http.health-handler	Handling health check request	{"method": "GET", "path": "/health", "remote_addr": "10.244.0.1:47646"}
2025-11-26T07:07:49Z	INFO	http	Request completed	{"method": "GET", "path": "/health", "remote_addr": "10.244.0.1:47646", "status": 200, "duration": "115.967µs"}

Post this PR (without log level being overriden as we do when running locally - note the missing "debug" logs):

{"level":"info","ts":"2025-11-26T07:32:28Z","logger":"setup","msg":"Starting KAgent Controller","version":"v0.7.5-3-g44e8c6d3","git_commit":"44e8c6d3","build_date":"2025-11-26","config":{"Metrics":{"Addr":"0","CertPath":"","CertName":"tls.crt","CertKey":"tls.key"},"Streaming":{"MaxBufSize":"1Mi","InitialBufSize":"4Ki","Timeout":600000000000},"LeaderElection":false,"ProbeAddr":":8082","SecureMetrics":true,"EnableHTTP2":false,"DefaultModelConfig":{"Namespace":"kagent","Name":"default-model-config"},"HttpServerAddr":":8083","WatchNamespaces":"","A2ABaseUrl":"http://127.0.0.1:8083","Database":{"Type":"sqlite","Path":"/sqlite-volume/kagent.db","Url":"postgres://postgres:kagent@db.kagent.svc.cluster.local:5432/crud"}}}
{"level":"info","ts":"2025-11-26T07:32:28Z","msg":"Updating GOMAXPROCS=2: determined from CPU quota"}
{"level":"info","ts":"2025-11-26T07:32:28Z","logger":"setup","msg":"Watching all namespaces (no valid namespaces specified)"}
{"level":"info","ts":"2025-11-26T07:32:28Z","msg":"MCPServer CRD not found - controller will not be started","controller":"mcpserver"}
{"level":"info","ts":"2025-11-26T07:32:28Z","msg":"Could not find CRD for tool server - API integration will be disabled","toolServerType":"MCPServer.kagent.dev"}
{"level":"info","ts":"2025-11-26T07:32:28Z","logger":"setup","msg":"starting manager"}
{"level":"info","ts":"2025-11-26T07:32:28Z","msg":"starting server","name":"health probe","addr":"[::]:8082"}
{"level":"info","ts":"2025-11-26T07:32:28Z","logger":"http-server","msg":"Starting HTTP server","address":":8083"}
{"level":"info","ts":"2025-11-26T07:32:28Z","msg":"Starting EventSource","controller":"service","controllerGroup":"","controllerKind":"Service","source":"kind source: *v1.Service"}
{"level":"info","ts":"2025-11-26T07:32:28Z","msg":"Starting Controller","controller":"service","controllerGroup":"","controllerKind":"Service"}
{"level":"info","ts":"2025-11-26T07:32:28Z","msg":"Starting workers","controller":"service","controllerGroup":"","controllerKind":"Service","worker count":1}
{"level":"info","ts":"2025-11-26T07:32:28Z","msg":"Starting EventSource","controller":"modelconfig","controllerGroup":"kagent.dev","controllerKind":"ModelConfig","source":"kind source: *v1.Secret"}
{"level":"info","ts":"2025-11-26T07:32:28Z","msg":"Starting EventSource","controller":"agent","controllerGroup":"kagent.dev","controllerKind":"Agent","source":"kind source: *v1.Service"}
{"level":"info","ts":"2025-11-26T07:32:28Z","logger":"setup","msg":"starting pprof server"}
{"level":"info","ts":"2025-11-26T07:32:28Z","msg":"Starting EventSource","controller":"agent","controllerGroup":"kagent.dev","controllerKind":"Agent","source":"kind source: *v1.Deployment"}
{"level":"info","ts":"2025-11-26T07:32:28Z","logger":"setup","msg":"pprof server started","address":":6060"}
...
{"level":"info","ts":"2025-11-26T07:32:28Z","msg":"Starting Controller","controller":"modelconfig","controllerGroup":"kagent.dev","controllerKind":"ModelConfig"}
{"level":"info","ts":"2025-11-26T07:32:28Z","msg":"Starting workers","controller":"modelconfig","controllerGroup":"kagent.dev","controllerKind":"ModelConfig","worker count":1}
{"level":"info","ts":"2025-11-26T07:32:28Z","msg":"Starting Controller","controller":"agent","controllerGroup":"kagent.dev","controllerKind":"Agent"}
{"level":"info","ts":"2025-11-26T07:32:28Z","msg":"Starting workers","controller":"agent","controllerGroup":"kagent.dev","controllerKind":"Agent","worker count":1}
{"level":"info","ts":"2025-11-26T07:32:55Z","logger":"http","msg":"Request completed","method":"GET","path":"/health","remote_addr":"10.244.0.1:34516","status":200,"duration":0.000057265}
{"level":"info","ts":"2025-11-26T07:32:55Z","logger":"http","msg":"Request completed","method":"GET","path":"/health","remote_addr":"10.244.0.1:34524","status":200,"duration":0.000050881}

If anyone is opposed to JSON logs locally then we can reconfigure these locally but this will require updates to the Helm chart that I'm trying to avoid pending the switch to using env vars instead of args in feat(controller): support HA deployment #1133. Specifically, I don't want to add controller.args to the values now when that could subsequently be removed (breaking change) in the linked PR. ↩

Signed-off-by: Brian Fox <878612+onematchfox@users.noreply.github.com>

onematchfox · 2025-11-26T08:52:50Z

go/pkg/app/app.go

 	utilruntime.Must(v1alpha2.AddToScheme(scheme))
 	// +kubebuilder:scaffold:scheme
-
-	ctrl.SetLogger(zap.New(zap.UseDevMode(true)))


This was introduced in 2cb72c3 - although I can't see any reason why it's needed.

Copilot

Pull request overview

This PR fixes the controller's log level configuration to properly respect the -zap-log-level flag by removing the hardcoded Development: true setting from zap.Options. Previously, development mode enabled V(1) verbosity logs by default regardless of the specified log level, causing DEBUG-level logs to always appear in production.

Key changes:

Removed unused LOG_LEVEL environment variable from the Helm deployment template
Removed Development: true from zap logger configuration to respect the -zap-log-level flag
Consolidated duplicate startup logging statements into a single log entry
Added explicit --set controller.loglevel=debug to local development Makefile target

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
`helm/kagent/templates/controller-deployment.yaml`	Removed unused LOG_LEVEL environment variable that was never consumed by the application
`go/pkg/app/app.go`	Removed Development mode default, eliminated duplicate logger initialization in init(), and consolidated startup logging
`Makefile`	Added explicit debug log level configuration for local development environment

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

onematchfox added 4 commits November 26, 2025 08:30

fix(controller): remove forced development logging

ae1866d

Signed-off-by: Brian Fox <878612+onematchfox@users.noreply.github.com>

refactor(controller): remove duplicate startup logs

b82617c

Signed-off-by: Brian Fox <878612+onematchfox@users.noreply.github.com>

build: set controller loglevel to debug when running locally

4241a4f

Signed-off-by: Brian Fox <878612+onematchfox@users.noreply.github.com>

fix(helm): remove unused env var

e640f15

Signed-off-by: Brian Fox <878612+onematchfox@users.noreply.github.com>

onematchfox commented Nov 26, 2025

View reviewed changes

onematchfox marked this pull request as ready for review November 26, 2025 08:53

onematchfox requested review from EItanya, ilackarms, peterj and yuval-k as code owners November 26, 2025 08:53

Copilot AI review requested due to automatic review settings November 26, 2025 08:53

Copilot started reviewing on behalf of onematchfox November 26, 2025 08:53 View session

Copilot finished reviewing on behalf of onematchfox November 26, 2025 08:55

Copilot AI reviewed Nov 26, 2025

View reviewed changes

EItanya approved these changes Dec 1, 2025

View reviewed changes

EItanya merged commit 95659bd into kagent-dev:main Dec 1, 2025
26 checks passed

onematchfox deleted the fix-controller-logging branch December 2, 2025 07:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(controller): ensure log level is respected #1135

fix(controller): ensure log level is respected #1135

Uh oh!

onematchfox commented Nov 26, 2025 •

edited

Loading

Uh oh!

onematchfox Nov 26, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

fix(controller): ensure log level is respected #1135

fix(controller): ensure log level is respected #1135

Uh oh!

Conversation

onematchfox commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Footnotes

Uh oh!

onematchfox Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

onematchfox commented Nov 26, 2025 •

edited

Loading