By Pontus Ebelin and Tomas Akenine-Möller, with Jim Nilsson, Magnus Oskarsson, Kalle Åström, Mark D. Fairchild, and Peter Shirley.

This repository implements the LDR-FLIP and HDR-FLIP image error metrics in Python, using the C++ implementation through nanobind. Similarly, it implements the FLIP tool, presented in Ray Tracing Gems II.

Copyright © 2020-2024, NVIDIA Corporation & Affiliates. All rights reserved.

This work is made available under a BSD 3-Clause License.

The repository distributes code for tinyexr, which is subject to a BSD 3-Clause License,
and stb_image, which is subject to an MIT License.

For individual contributions to the project, please confer the Individual Contributor License Agreement.

For business inquiries, please visit our website and submit the form: NVIDIA Research Licensing.

• Setup (with pip):

  pip install flip-evaluator
  

• Usage (API): See example in the script src/python/api_example.py.

• Usage (tool): flip --reference reference.{exr|png} --test test.{exr|png} [--options], where the list of options can be seen by flip -h.

• Tested with pip 24.0, Python 3.11.8, pybind11 2.11.1, and C++20.

• FLIP runs on Windows, Linux (tested on Ubuntu 24.04), and OS X ($\ge$ 10.15), though its output might differ slightly between the different operative systems. The references used for src/tests/test.py are made for Windows. While the mean tests (means compared up to six decimal points) pass for each mentioned operative system, not all error map pixels are identical.

• The code that implements FLIP metrics and the FLIP tool is available in src/cpp/FLIP.h and src/cpp/tool, respectively. The relevant functions are called by the Python API using nanobind (see src/nanobindFLIP.cpp). The Python API is provided in src/flip_evaluator/flip_python_api.py. src/tests/test.py contains simple tests used to test whether code updates alter results. Notice that those scripts require numpy and matplotlib, both of which can be installed using pip.

• Weighted histograms are output as Python scripts. Running the script will create a PDF version of the histogram. Like the test scripts, these scripts require numpy and matplotlib, both of which can be installed using pip.

• The naming convention used for the FLIP tool's output is as follows (where ppd is the assumed number of pixels per degree, tm is the tone mapper assumed by HDR-FLIP, cstart and cstop are the shortest and longest exposures, respectively, assumed by HDR-FLIP, with p indicating a positive value and m indicating a negative value, N is the number of exposures used in the HDR-FLIP calculation, nnn is a counter used to sort the intermediate results, and exp is the exposure used for the intermediate LDR image / FLIP map):

  Default:

  Low dynamic range images:
  

  LDR-FLIP: flip.<reference>.<test>.<ppd>ppd.ldr.png
  Weighted histogram: weighted_histogram.reference>.<test>.<ppd>ppd.ldr.py
  Overlapping weighted histogram: overlapping_weighted_histogram.<reference>.<test1>.<test2>.<ppd>ppd.ldr.py
  Text file: pooled_values.<reference>.<test>.<ppd>ppd.ldr.txt
  

  High dynamic range images:
  

  HDR-FLIP: flip.<reference>.<test>.<ppd>ppd.hdr.<tm>.<cstart>_to_<cstop>.<N>.png
  Exposure map: exposure_map.<reference>.<test>.<ppd>ppd.hdr.<tm>.<cstart>_to_<cstop>.<N>.png
  Intermediate LDR-FLIP maps: flip.<reference>.<test>.<ppd>ppd.ldr.<tm>.<nnn>.<exp>.png
  Intermediate LDR images: <reference|test>.<tm>.<nnn>.<exp>.png
  Weighted histogram: weighted_histogram.<reference>.<test>.<ppd>ppd.hdr.<tm>.<cstart>_to_<cstop>.<N>.py
  Overlapping weighted histogram: overlapping_weighted_histogram.<reference>.<test1>.<test2>.<ppd>ppd.hdr.<tm>.<cstart>_to_<cstop>.<N>.py
  Text file: pooled_values.<reference>.<test>.<ppd>ppd.hdr.<tm>.<cstart>_to_<cstop>.<N>.txt
  

  With --basename <name> (note: not applicable if more than one test image is evaluated):

  Low dynamic range images:
  

  LDR-FLIP: <name>.png
  Weighted histogram: <name>.py
  Overlapping weighted histogram: N/A
  Text file: <name>.txt
  

  High dynamic range images:
  

  HDR-FLIP: <name>.png
  Exposure map: <name>.exposure_map.png
  Intermediate LDR-FLIP maps: <name>.<nnn>.png
  Intermediate LDR images: <name>.reference|test.<nnn>.png
  Weighted histogram: <name>.py
  Overlapping weighted histogram: N/A
  Text file: <name>.txt
  

Example usage: To test the API, please inspect the src/python/api_example.py script. This shows how the available API commands may be used. Please note that not all capabilities of the tool is available through the Python API. For example, the exposure map is not output when running HDR-FLIP. For that, use the tool or the C++ API in FLIP.h.

To test the tool, start a shell, navigate to images/ and try:

flip -r reference.exr -t test.exr

The result should be:

Invoking HDR-FLIP
      Pixels per degree: 67
      Assumed tone mapper: ACES
      Start exposure: -12.5423
      Stop exposure: 0.9427
      Number of exposures: 14

FLIP between reference image <reference.exr> and test image <test.exr>:
      Mean: 0.283478
      Weighted median: 0.339430
      1st weighted quartile: 0.251122
      3rd weighted quartile: 0.434673
      Min: 0.003123
      Max: 0.962022
      Evaluation time: <t> seconds
      FLIP error map location: <path/to/workingDirectory/flip.reference.test.67ppd.hdr.aces.m12.5423_to_p0.9427.14.png>
      FLIP exposure map location: <path/to/workingDirectory/exposure_map.reference.test.67ppd.hdr.aces.m12.5423_to_p0.9427.14.png>

where <t> is the time it took to evaluate HDR-FLIP. In addition, you will now find the files flip.reference.test.67ppd.hdr.aces.m12.5423_to_p0.9427.14.png and exposure_map.reference.test.67ppd.hdr.aces.m12.5423_to_p0.9427.14.png in the working directory, and we urge you to inspect those, which will reveal where the errors in the test image are located.