chore(): add telemetry event for auto mode routing fallback#3780
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds telemetry tracking when the auto mode router fails to return a valid model and the system falls back to using the reserved model. This helps monitor router reliability and understand when fallback behavior occurs.
Changes:
- Added a new telemetry event
automode.routerDecisionFallbackwith GDPR annotations to track router failures - The event captures available models, preferred models, and the chosen fallback model when routing fails
Comments suppressed due to low confidence (1)
src/platform/endpoint/node/automodeService.ts:273
- The control flow logic has a critical bug. The code after the telemetry block (lines 262-272) will only execute when routing fails (inside the
if (!selectedModel)block). However, when routing succeeds,selectedModelwill be set and the function will skip all the remaining endpoint creation logic, causing it to reach line 273 without returning a value. This will result in a compilation error since the function is declared to returnPromise<IChatEndpoint>but doesn't return anything in the success path. The code after line 261 (including the_applyVisionFallbackcall and endpoint creation) should be moved outside of theif (!selectedModel)block so it executes in both the fallback path and the success path.
if (selectedModel && shouldRoute) {
// If routing was attempted but failed, emit event that we are falling back to the reserved model
{
/* __GDPR__
"automode.routerDecisionFallback" : {
"owner": "tyleonha",
"comment": "Reports a fallback event when the router fails to return a valid model and we have to fall back to the reserved model.",
"availableModels": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "Comma-separated list of available models for this request" },
"preferredModels": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "Comma-separated list of preferred models for this request, ordered by preference with the reserved model first" }
"chosenModel": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "The model selected by the router" },
}
*/
this._telemetryService.sendMSFTTelemetryEvent("automode.routerDecisionFallback", {
"availableModels": availableModels.join(','),
"preferredModels": preferredModels.join(','),
"chosenModel": selectedModel.model,
})
}
}
selectedModel = this._applyVisionFallback(chatRequest, selectedModel, reserveToken.available_models, knownEndpoints);
// If the session token changed, invalidate all cached endpoints so they get recreated with the new token
const existingEndpoints = (entry && entry.lastSessionToken === reserveToken.session_token) ? entry.endpoints : [];
let autoEndpoint = existingEndpoints.find(e => e.model === selectedModel.model);
if (!autoEndpoint) {
autoEndpoint = this._instantiationService.createInstance(AutoChatEndpoint, selectedModel, reserveToken.session_token, reserveToken.discounted_costs?.[selectedModel.model] || 0, this._calculateDiscountRange(reserveToken.discounted_costs));
existingEndpoints.push(autoEndpoint);
}
this._autoModelCache.set(conversationId, { endpoints: existingEndpoints, tokenBank: reserveTokenBank, lastSessionToken: reserveToken.session_token, lastRoutedPrompt: prompt });
return autoEndpoint;
}
| if (selectedModel && shouldRoute) { | ||
| // If routing was attempted but failed, emit event that we are falling back to the reserved model | ||
| { | ||
| /* __GDPR__ | ||
| "automode.routerDecisionFallback" : { | ||
| "owner": "tyleonha", | ||
| "comment": "Reports a fallback event when the router fails to return a valid model and we have to fall back to the reserved model.", | ||
| "availableModels": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "Comma-separated list of available models for this request" }, | ||
| "preferredModels": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "Comma-separated list of preferred models for this request, ordered by preference with the reserved model first" } | ||
| "chosenModel": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "The model selected by the router" }, | ||
| } | ||
| */ | ||
| this._telemetryService.sendMSFTTelemetryEvent("automode.routerDecisionFallback", { | ||
| "availableModels": availableModels.join(','), | ||
| "preferredModels": preferredModels.join(','), | ||
| "chosenModel": selectedModel.model, | ||
| }) | ||
| } |
There was a problem hiding this comment.
The new telemetry for router fallback lacks test coverage. Consider adding a test case in automodeService.spec.ts that verifies the telemetry event is emitted when the router call fails (e.g., throws an error or returns an invalid model) and the system falls back to the reserved model. This would ensure the fallback telemetry is properly captured and that the code path works correctly.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
|
@Sid200026 aren't you more interested in error telemetry in this case? |
Added telemetry when the auto mode router errors out and fallback model is used.