Implements two new built-in scorers for evaluating knowledge retention
in conversational AI applications:

- LastTurnKnowledgeRetention: Evaluates whether the last AI response
  correctly retains information provided by users in earlier turns
- KnowledgeRetention: Orchestrates per-turn evaluation across entire
  conversations using composition pattern

Key features:
- Binary yes/no feedback with detailed per-turn rationale
- Worst-case aggregation (any turn failure = overall failure)
- Session slicing to provide correct conversation history to each turn
- Customizable via single_turn_scorer parameter
- Comprehensive test coverage (6 tests including multi-turn scenarios)

Implementation details:
- Uses InstructionsJudge directly (not make_judge) per conventions
- Follows BuiltInSessionLevelScorer pattern with lazy imports
- Clear prompt instructions distinguishing relevant vs irrelevant info
- Edge case handling for empty sessions and single-turn conversations

Signed-off-by: Alkis Polyzotis <alkis.polyzotis@databricks.com>

Remove explanatory comments from KnowledgeRetention implementation and
tests as they add unnecessary noise without providing value.

Changes in builtin_scorers.py:
- Removed inline comments explaining obvious operations
- Code is self-documenting through clear variable names and structure

Changes in test_builtin_scorers.py:
- Removed "Turn 0/Turn 1" comments from test setup
- Removed single-line assertion explanations
- Tests remain clear through descriptive test names and assertions

All 5 knowledge retention tests pass with no functionality changes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Signed-off-by: Alkis Polyzotis <alkis.polyzotis@databricks.com>

Update expected scorer count after adding KnowledgeRetention to
get_all_scorers():
- 12 scorers for Databricks (was 11)
- 10 scorers for OSS (was 9)

This fixes the failing test that was checking for the wrong count.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Signed-off-by: Alkis Polyzotis <alkis.polyzotis@databricks.com>

Creates a new SessionLevelScorer base class and makes LastTurnKnowledgeRetention
internal (_LastTurnKnowledgeRetention) to ensure BuiltInScorer and
BuiltInSessionLevelScorer are reserved for truly public scorers.

## Changes

### New SessionLevelScorer Base Class (builtin_scorers.py)
- Created SessionLevelScorer base class inheriting from Judge
- Provides common session-level functionality: judge caching, __call__ method,
  get_input_fields, is_session_level_scorer property
- Designed for use by both public and internal session-level scorers

### Updated BuiltInSessionLevelScorer (builtin_scorers.py)
- Now inherits from both BuiltInScorer and SessionLevelScorer using multiple inheritance
- BuiltInScorer provides special serialization for public API
- SessionLevelScorer provides session-level evaluation logic
- Reserved explicitly for public API scorers only per docstring

### Renamed LastTurnKnowledgeRetention (builtin_scorers.py)
- Renamed to _LastTurnKnowledgeRetention with underscore prefix
- Changed base class from BuiltInSessionLevelScorer to SessionLevelScorer
- Removed @experimental and @format_docstring decorators (internal class)
- Updated docstring to emphasize internal status and removed example
- KnowledgeRetention updated to use _LastTurnKnowledgeRetention as default

### Removed from Public API (__init__.py)
- Removed LastTurnKnowledgeRetention from _LAZY_IMPORTS set
- Removed from TYPE_CHECKING imports
- Removed from __all__ list
- No longer accessible via `from mlflow.genai.scorers import LastTurnKnowledgeRetention`

### Updated Tests (test_builtin_scorers.py)
- Removed LastTurnKnowledgeRetention from imports
- Renamed test_last_turn_knowledge_retention_success to
  test_knowledge_retention_uses_default_single_turn_scorer
- Refactored test to use KnowledgeRetention instead of direct instantiation
- Test now verifies integration with default _LastTurnKnowledgeRetention scorer
- Validates judge is called for each turn (2 turns = 2 calls)

### Documentation (api_inventory.txt)
- Removed all references to LastTurnKnowledgeRetention
- Cleaned up 3 entries from API inventory

## Architectural Benefits

1. **Clear Separation**: BuiltInScorer/BuiltInSessionLevelScorer explicitly for public API
2. **Reusable Base**: SessionLevelScorer available for future internal scorers
3. **No Duplication**: Common session-level logic extracted to one place
4. **Type Safety**: Multiple inheritance preserves both serialization and session logic
5. **Clean API**: Internal classes don't pollute public API exports

## Testing

- All existing session-level scorer tests pass (UserFrustration, ConversationCompleteness, etc.)
- Knowledge retention tests pass with new integration test approach
- Verified LastTurnKnowledgeRetention no longer accessible from public API
- Verified KnowledgeRetention uses _LastTurnKnowledgeRetention by default

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Signed-off-by: Alkis Polyzotis <alkis.polyzotis@databricks.com>

Signed-off-by: Alkis Polyzotis <alkis.polyzotis@databricks.com>

Implements 6 of 9 review comments from smoorjani:

**API Simplification (Comment #1):**
- Remove last_turn_scorer parameter from public API
- Simplify constructor to use default Pydantic initialization
- Parameter kept as internal field for composition pattern

**Code Quality (Comments #3, mlflow#6, mlflow#8):**
- Remove duplicated kwargs validation (already in base class)
- Remove unnecessary one-line docstring from _evaluate_turn
- Use CategoricalRating enum instead of string literals ("yes"/"no")

**Logic Improvements (Comments mlflow#7, mlflow#9):**
- Move empty results check from _compute_aggregate to __call__
- Add per-turn details for success case (symmetric with failure case)

**Code Reuse Analysis (Comments mlflow#4, mlflow#5):**
- Researched _validate_session: serves different purpose (session ID validation)
- Researched resolve_conversation_from_session: returns different type (list[dict])
- Keeping inline implementations as they serve different purposes

**Deferred (Comment #2):**
- _create_judge() architecture issue deferred per reviewer (non-blocking)

All 5 KnowledgeRetention tests passing.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Signed-off-by: Alkis Polyzotis <alkis.polyzotis@databricks.com>

Implements 5 ALKIS review comments:

**Code Reusability:**
- Refactor _validate_session from InstructionsJudge to trace_utils.py
- Add validate_session() utility function for session validation
- Update InstructionsJudge to call refactored utility
- Add validate_session() call in KnowledgeRetention.__call__

**Code Simplification:**
- Remove redundant empty per_turn_results check (session already validated)
- Refactor per-turn rationale assembly into _format_per_turn_rationale()
- Eliminate duplicate code between success/failure branches

**User Experience:**
- Remove truncation from turn summaries (show full rationale)
- Improves readability of evaluation feedback

All 5 KnowledgeRetention tests passing.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Signed-off-by: Alkis Polyzotis <alkis.polyzotis@databricks.com>

Implementation improvements per review feedback:

1. Move imports to top-level per MLflow conventions
   - Add validate_session to top-level imports in all files
   - Add INVALID_PARAMETER_VALUE to top-level imports
   - Remove inline imports from functions
   - Remove TraceMetadataKey from __all__ exports

2. Replace empty session feedback with exception
   - Raise MlflowException for empty session instead of returning feedback
   - Consistent with InstructionsJudge error handling pattern

3. Refactor to use list[Feedback] directly
   - Update _compute_aggregate() to accept list[Feedback]
   - Update _format_per_turn_rationale() to accept list[Feedback]
   - Remove intermediate dict creation in __call__()
   - Cleaner type signatures and more maintainable code

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Signed-off-by: Alkis Polyzotis <alkis.polyzotis@databricks.com>

Test improvements per review feedback:

1. Extract test helper function
   - Add _create_test_trace_with_session() helper
   - Reduces code duplication across 4 tests
   - Makes tests more readable and maintainable

2. Update test for exception handling
   - Add MlflowException import at top-level
   - Update test_knowledge_retention_empty_session to expect exception
   - Consistent with new empty session error handling

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Signed-off-by: Alkis Polyzotis <alkis.polyzotis@databricks.com>