Skip to content

Fix failure threshold math #200

@3mei

Description

@3mei

Priority Level

Critical (Total blocker)

Describe the bug

The entire workflow occasionally gets killed when we hit generation failures, despite using a failure threshold of 1.0. In the example below, this happened when working with the LLMCodeColumnConfig column.

Steps/Code to reproduce bug

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/workspace/.venv/lib/python3.11/site-packages/data_designer/interface/data_designer.py", line 168, in create
    builder.build(num_records=num_records, buffer_size=self._buffer_size)
  File "/workspace/.venv/lib/python3.11/site-packages/data_designer/engine/dataset_builders/column_wise_builder.py", line 107, in build
    self._run_batch(generators, batch_mode="batch", group_id=group_id)
  File "/workspace/.venv/lib/python3.11/site-packages/data_designer/engine/dataset_builders/column_wise_builder.py", line 181, in _run_batch
    raise DatasetGenerationError(f"🛑 Failed to process {column_error_str}:\n{e}")
data_designer.engine.dataset_builders.errors.DatasetGenerationError: 🛑 Failed to process column 'sql':
  |-- Data generation was terminated early due to error rate exceeding threshold.
  |-- The summary of encountered errors is: 
{
    "failure_threshold": 1.0,
    "completed_count": 256,
    "success_count": 230,
    "early_shutdown": true,
    "error_count": 26,
    "task_errors": {
        "ModelContextWindowExceededError": 19,
        "ModelGenerationValidationFailureError": 7
    }
}

Expected behavior

Math should check out. Should not terminate the workflow with threshold set to 1.0

Additional context

No response

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions