Evaluates whether an audio–language model should proactively speak from audio-only cues (no transcript)—and, when it should, whether the reply is useful.

Each sample has a ground-truth decision. The RESPOND subset includes reference replies for optional semantic scoring.

The Hugging Face layout matches manifest path fields (root folder: proactive-sound-effect).

Download (login may be required for gated assets):

pip install -U huggingface_hub
hf download masaz14/Proactive-Sound-Effect-Benchmark \
  --repo-type=dataset \
  --local-dir ./Proactive-Sound-Effect-Benchmark-data

17 subcategories across 6 domains—daily living, human states, traffic, environment, music, and equipment—focused on proactive assistance and safety.

Labels

• RESPOND — User-relevant, safety, or assistance scenarios → model should speak up
• IGNORE — Ambient or background sounds → silence is appropriate

Recommended layout (each manifest path resolves to a local file):

.
├── proactive_reply_benchmark.jsonl
├── proactive_reply_benchmark_response.jsonl
├── core.py
├── semantic.py
├── evaluate.py
└── proactive-sound-effect/
    └── Daily Living Sounds/
        └── Daily Affairs/
            ├── RESPOND/<id>.wav
            └── IGNORE/<id>.wav

Paths are relative; a ./ prefix is recommended, e.g.:

./proactive-sound-effect/Daily Living Sounds/Daily Affairs/RESPOND/<id>.wav

flowchart LR
  A[Download audio] --> B[Model inference]
  B --> C[predictions.jsonl]
  C --> D[evaluate.py]
  D --> E[stats.json]

1. Run your model on every manifest audio item
2. Write predictions.jsonl
3. Score offline with this repo’s scripts

Tags are case-insensitive:

<Decision>RESPOND</Decision>
<Reply>...</Reply>

Each line in predictions.jsonl needs at least:

• id — must match the manifest
• _raw_reply (recommended) or reply — full raw model output string

Decision accuracy only (stdlib sufficient):

python evaluate.py \
  --benchmark proactive_reply_benchmark.jsonl \
  --pred /path/to/predictions.jsonl \
  --out /path/to/stats.json \
  --checkpoint-label my-model

With semantic matching (RESPOND replies vs standard_answers):

pip install torch FlagEmbedding

python evaluate.py \
  --benchmark proactive_reply_benchmark.jsonl \
  --pred /path/to/predictions.jsonl \
  --out /path/to/stats.json \
  --respond-jsonl proactive_reply_benchmark_response.jsonl \
  --reranker-dir /path/to/bge-reranker-v2-m3 \
  --semantic-threshold 0.5

If evaluate.py uses from .core import ..., run it as a package. For a flat repo, switch to from core import ... and run python evaluate.py directly.

The CLI prints the stats JSON path and overall accuracy (before/after semantic matching). Per-domain and per-folder breakdowns are written to the --out JSON.

You are a proactively helpful model.
Your task is to determine whether to assist the user based on the provided audio.
- If the audio indicates physiological danger or environmental safety risks, provide appropriate assistance.
- If it reflects negative emotions, offer mental comfort.
- If it suggests equipment malfunction, provide warnings or safety guidance.
- If none of the above apply, just respond with "No Reply". Do not explain anything.

Your final output must strictly follow this format:
<Decision>(RESPOND or IGNORE)</Decision>
<Reply>(Your reply)</Reply>

• Python ≥ 3.10
• Basic evaluation: standard library only
• Semantic matching: torch, FlagEmbedding, and a local reranker (e.g. BAAI/bge-reranker-v2-m3)