[ Leaderboard ] [ Meta Seal ]

OmniSeal Bench is a framework that provides a comprehensive benchmark for evaluating the performance of neural watermarking techniques on image, video and audio contents. The benchmark includes a wide variety of datasets, attacks, evaluation metrics, and an analysis tool to compare different SOTA watermarking models.

# Running an Omniseal Bench involves creating a task for watermarking generation or detection on a given modality,
# then load a model. All attacks, metrics computation are provided and can be customized

from omnisealbench import task, get_model

img_watermarking = task(
    "default",   # Default is end-to-end benchmarking. Other options: "generation", "detection"
    modality="image",
    dataset_dir="my_dataset",
    attacks="all",
)

model = get_model("trustmark", device="cuda")
avg_metrics, results = img_watermarking(model)

summary = img_watermarking.print_scores(results)

For more examples, see notebooks/. For detailed information about the models and metrics used in OmniSeal Bench, please refer to the documentation. This section contains markdown files that describe each model (and how to download it) and metric in detail.

OmniSeal Bench can be installed from GitHub by cloning the repository and installing it. We recomend a fresh Conda/Python environment. Omniseal Bench was tested with Python>=3.10+ and PyTorch>=2.4. It is advisable to check your CUDA version and install a matching PyTorch version for it to have optimal performance.

conda create -n "omnisealbench" python=3.10 pip
conda activate omnisealbench

pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 \
    --index-url https://download.pytorch.org/whl/cu128

# Make sure to install FFMPEG for audio and video
# Alternatively, you can also 
# apt-get install ffmpeg
conda install -c conda-forge ffmpeg

# Clone the OmniSeal Bench repository and install it.
git clone https://github.com/facebookresearch/omnisealbench.git
cd omnisealbench

pip install omnisealbench[all] -e .

You can install only a subset of OmniSeal Bench with less dependencies required.

• Audio watermarking only:

  pip install omnisealbench[audio] -e .

• Image watermarking only:

  pip install omnisealbench[image] -e .

• Video watermarking only:

  pip install omnisealbench[video] -e .

• Results analysis via a leaderboard:

  pip install omnisealbench[leaderboard] -e .

Before running any evaluation, if you want to use the registered models and the default model cards you need to download the checkpoints for the models that require that and dont provide automatic weights download. For this you can run one of the following commands depending on the modality you are using. Make sure to install gdown before running the scripts:

• audio:

bash scripts/download_audio_models.sh

• image:

bash scripts/download_image_models.sh

• The InvisMark repository provides a public OneDrive link for its checkpoint. This file must be downloaded manually, as automatic download is not supported.

By default, models are stored in the OMNISEAL_CHECKPOINTS location: $USER/.cache/omnisealbench/checkpoints. To use a different location, set the environment variable:

export OMNISEAL_CHECKPOINTS=/path/to/your/checkpoints/location

To run the evaluation end-to-end, try the following quick examples. Ensure to download the models and set OMNISEAL_CHECKPOINTS beforehand.

[image]

omniseal eval image
    --dataset_dir=./examples/img
    --result_dir=./results/images
    --ids_to_save=all
    --attacks=all
    --metrics=all
    --model="wam,trustmark"

[audio]

omniseal eval audio
    --dataset_dir=./examples/audios
    --result_dir=./results/audios
    --ids_to_save=all
    --attacks=all
    --metrics=all
    --model="audioseal,timbre,wavmark_fast"

[video]

export FFMPEG_BINARY=... # Path to ffmpeg binary in the system, this will be required for VMAF metric computation
omniseal eval image
    --dataset_dir=./examples/videos
    --result_dir=./results/videos
    --ids_to_save=all
    --attacks=all
    --metrics=all
    --model=videoseal_1.0

Notes:

• You can omit the parameter "--model" if you want to evaluate on all the registered models. See "src/omnisealbench/configs/
• ids_to_save=all will save all examples to the result dir. Other ways is to select a specific examples by ids, e.g. ids_to_save=0,1,2,4, or by range ids_to_save=0-4
• To limit the number of samples e.g. to one example, use --num_samples=1
• By default, Omniseal Bench will skip if the result directory exists with certain structure (see below). To force the rerun, specify --overwrite=True

The result directory will look similar to something below (examples for audio):

📁 no-attack/
    📄 attack_results.jsonl
    📄 metrics.json
📁 updownresample/
    📄 attack_results.jsonl
    📄 metrics.json
📁 bandpass_filter/
    📄 attack_results.jsonl
    📄 metrics.json
📁 speed_julius__speed_1.25/
    📄 attack_results.jsonl
    📄 metrics.json
📁 pink_noise__noise_std_1.0/
    📄 attack_results.jsonl
    📄 metrics.json
📁 lowpass_filter__cutoff_freq_5000/
    📄 attack_results.jsonl
    📄 metrics.json
📁 mp3_compression__bitrate_256k/
    📄 attack_results.jsonl
    📄 metrics.json
📄 metrics.json
📄 report.csv

• The file "metrics.json" consists of average metrics for the whole dataset. For each metrics, it will report three values (in this order): average score, number of total examples evaluated over all attacks, and mean of square of the values.
• "report.csv" stores the raw results of each example. In additiona to the metrics, each row also contains the attack name, attack category, and the parameters used in the attack (attack variant).
• For each attack, the results are stored in a sub-folder. There is a special attack - "identity" - that corresponds to "no attack", i.e. running the detection directly on the original and watermarked contents without transformation. Inside each of these sub-folder, "metrics.json" reports the average scores for the corresponding attack, while "attack_results.jsonl" is the raw results in JSONL format.

Model cards:

The model cards can be found in "src/omnisealbench/cards". IF you want to point to other model cards outside the code repo (e.g. in your own cluster), use the environment variable "OMNISEAL_MODEL_DIR" to store the model card directory.

The Omniseal Bench CLI also provides an individual way to watermark content using the registered models, detect the embedded watermarks, and report various metrics. There are two main commands:

1. generate_watermark watermarks content and evaluates the quality of it:

   omniseal eval [image|audio|video] generate_watermark [OPTIONS]

2. detect_watermark detects watermarks in, potentially altered, watermarked content and evaluates the robustness of the watermarking methods:

   omniseal eval [image|audio|video] detect_watermarks [OPTIONS]

Depending on the command, the [OPTIONS] vary. To learn which options are available, use omniseal eval [image|audio|video] --help. More about in advanced_usage.

• OMNISEAL_HOME: The home directory to store results of the omniseal runs. By default, this is set to ~/.cache/omnisealbench.
• OMNISEAL_DEBUG: By default, the errors in CLI are shortened to avoid verbosity. To print the full stacktrace of the error in CLI mode, pass OMNISEAL_DEBUG=1 when running Omniseal
• OMNISEAL_MESSGE: The messages are randomly generated by default. To reproduce the attacks completely, pass OMNISEAL_MESSAGE='test' when running Omniseal. This will ensure the message follows a strict pattern that can be reproducible for every run.
• OMNISEAL_MODEL_DIR: Refer to the folder where all YAML cards are stored. This is handy for adding external model cards without checking out source code.
• OMNISEAL_CHECKPOINTS: When downloading baselines using the scripts in "scripts", by defaults, all checkpoints are saved to "~/.cache/omnisealbench/checkpoints". Set this environment to change the target location of downloaded baselines.

Full details: Audio , Image , Video

To scale up the evaluation on SLURM with bigger dataset, simply add --slurm to the CLI commands. To specify the runtime configuration for the SLURM job:

omniseal eval audio
    --dataset_type=hf   # Evaluate on a HuggingFace dataset
    --dataset_name=facebook/voxpopuli
    --dataset_hf_subset=en
    --dataset_split=test
    --result_dir=/tmp/tuantran/voxpopuli_audioseal
    --padding=fixed
    --max_length=160000
    --sample_rate=16000
    --message_size=16
    --metrics=all
    --overwrite
    --model=audioseal
    --model_card_or_path=audioseal_wm_16bits
    --num_samples 1000
    --slurm
    --slurm.[OPTONS]=[VALUE]

The following table describes the available options for configuring SLURM jobs. (See omnisealbench.commands.utils.SlurmOptions):

Once we run an evaluation we can load the raw results, aggregate them and visualise examples into omnisealbench_space by pointing to the evaluation results directory. Omniseal Bench leaderboard is hosted in HuggingFace space, but you can also run it locally by following the steps below. This is handy if you want to inspect your own data without uploading to public storages.

Leaderboard is provided as a submodule in Omniseal Bench, run:

git submodule update --recursive

to get its source. Next, you can either setup a Docker image to run it on your own, or install pip install [image,leaderboard] -e . to run the leaderboard in the same environment. The following steps assume you run directly in your (Conda) environment. For Docker build, please see see README in src/leaderboard_space.

cd src/leaderboard_space
gunicorn --chdir backend -b 0.0.0.0:7860 app:app --reload

Then open browser and navigate to "http://localhost:7860", you should see something like this:

Run:

omniseal leaderboard export \
    --modality=image
    --source=results/images \
    --samples=all \
    --dataset_label=my_dataset

This will save all examples in the HF-leaderboard compatible format to a subfolder name "my_dataset" inside "src/leaderboard_space/backend/data".

Now refresh the browser and go to "Image" tab, you should see "my_dataset" listed there. CLick to it and switch to "Image Examples". The leaderboard will reload the results with your evaluation and dataset. Have fun analysing :) !

The common code of Omniseal Bench is licensed under an MIT license. For different watermarking models, the LICENSE can vary, and we keep the LICENSE files under each model code accordingly (models/[MODEL]_src)

See contributing and the code of conduct.