Specifically, this is FeedbackValueType or Literal[<list of values in PbValueType>], right? Can we change the type hint to FeedbackValueType | Literal[<list of values in PbValueType>]?

For variable naming, I'd like us to be a bit more explicit too. feedback_value_type?

cc @alkispoly-db @AveshCSingh

The argument value here is "type" itself, so the more accurate type hint is type[FeedbackValueType]. Yes, we can support a list of PbValueType and a dict of PbValueType to be consistent with FeedbackValueType. Do we want to support Literal in this case? I think Literal is useful for categorical responses (e.g., yes, no), but it violates the type hint type[FeedbackValueType] and there's nothing like type[Literal] in Python iirc, resulting in Any type hint which causes the type accuracy concern.

Yeah, sounds good! We should also support literal

@dbczumar Supported a list and dict of PbValueType, can you take another look?

Let's test this on Databricks to make sure it works with GPT / GPT OSS, Anthropic, Llama. We should introduce some retry logic if response schema is not supported

Yes, I tested FMAPI models and structured output worked well!

Could you elaborate what is missing if we simply serialize the ResponseFormat pydantic model using .model_json_schema()? We still need to implement deserialization but it allow us bypassing serialization and maintaining our own format.

There's no native way to convert json schema back to Pydantic base class. We can simplify the stored artifact and deserialization logic if we have our own simple format rather than using JSON schema. But I agree with the value of using a standard schema, so I’m fine with either way.

Btw, to clarify, we just need to store the feedback_value_type rather than the entire response_format for LLM which is dynamically constructed (renamed the serialize method to clarify). Then the feedback value type itself is not Pydantic base model. To use JSON schema, we need to store the entire response_format, which contains some duplicated information.

Yea no strong preference here. Just thought using Pydantic serialization reduces complexity and spec ambiguity, even if we need to implement deserialization ourselves. Not a blocker.

Do you think it is too aggressive if we default to boolean or mandate this field? The current default setting (free-form text) is usable; users will get inconsistent results and broken aggregation. For example, when I tried it first, I got three different words representing "false". This happens to high-end LLMs as well because there is no mechanism to enforce consistency by default.

It is not great that the default experience gives impression of "make_judge is broken". I would rather harness it or adding extra step, such that users can get meaningful results from the default journey.

str is for backward compatibility. It's not a good experience if the users' existing judge changes the response type when upgrading MLflow version. Assuming the low adoption of make_judge, I'm fine with making boolean or Literal["yes", "on"] our default. What do you think? cc: @dbczumar @alkispoly-db

We're definitely going to break existing judges if we do this; let's keep a string default but update documentation to explain how to use the structured outputs and use it in all of our examples.

It is not great that the default experience gives impression of "make_judge is broken"

I haven't experienced this issue if I tell the judge to "return one of the following values exactly: ..."

How many "existing judge" users do we have? It is much better to break earlier than later if we believe it is makes a product better. The API is marked experimental too.

I haven't experienced this issue if I tell the judge to "return one of the following values exactly: ..."

Do competitors ask similar thing to users? Also structured outputs is a theoretically guaranteed method to enforce outputs, so is more stable than prompting.

As a data point, the oss usage is less than 5 sessions per day in average. Isn't it a good trade-off to make default experience better for hundreds of future users who will be using the API?

@dbczumar can you take a look at Yuki's comment above?

Can pydantic help with this validation, or is it too difficult to express in pydantic because we have to support v1 and v2?

Ifaik, Pydantic helps validate an object on types, but this validation is for validating a type on types. So I'm not sure if we can simplify this logic using Pydantic.

@TomeHirata Anything we can do to make the is_trace_based == True case work with Databricks gpt-oss, which doesn't support structured outputs and tool calling together?

It seems like we could accept the response format as an argument to the _build_system_message method and use it to adjust the evaluation_rating_fields?

    def _build_system_message(self, is_trace_based: bool) -> str:
        """Build the system message based on whether this is trace-based or field-based."""
        output_fields = self.get_output_fields()

        if is_trace_based:
            evaluation_rating_fields = "\n".join(
                [f"- {field.name}: {field.description}" for field in output_fields]
            )
            return INSTRUCTIONS_JUDGE_TRACE_PROMPT_TEMPLATE.format(
                evaluation_rating_fields=evaluation_rating_fields,
                instructions=self._instructions,
            )
        else:
            base_prompt = format_prompt(
                INSTRUCTIONS_JUDGE_SYSTEM_PROMPT, instructions=self._instructions
            )
            return add_output_format_instructions(base_prompt, output_fields=output_fields)

Agree to include the type information in instruction when is_trace_based == True. Added the type information to the evaluation rating field.