Clone this repo.

git clone https://github.com/jose13579/variable-hyperparameter-image-impainting.git
cd variable-hyperparameter-image-impainting/

We build our project based on Pytorch and Python libraries. To train and test our project, we suggest create a Conda environment from the provided YAML, e.g.

conda env create -f environment.yml 
conda activate vhii

Or use the Dockerfile to install conda and prerequisites, e.g.

docker build -t vhii-mage .

If you are having issues to install the environment, we suggest you follow the next steps:

• Remove the cupy from the environment.yml
• Create the conda env
• Manually install the cupy with the correct version:

conda install -c conda-force cupy==7.7.0

We prepare the real and masks datasets.

Preparing Places365 Dataset. The dataset can be downloaded from here. The training set has approximately 1.8 million images from 365 scene categories, where there are at most 5,000 images per category. We adopt the places365-Standard datasets (small images 256 * 256 version) to train and validate our proposed method. The dataset should be arranged in the same directory structure as

datasets
    ｜- places365
        |- data_256
            |- a
                |- airport_terminal
                    |- <image_id>.jpg
                    |- <image_id>.jpg
                |- airplane_cabin
                    |- <image_id>.jpg
                    |- <image_id>.jpg
                |- ...
            |- ...
        |- val_256
            |- <image_id>.jpg
            |- <image_id>.jpg

Preparing CelebA Dataset. The dataset can be downloaded from here. This dataset contains more than 200,000 large-scale facial celebrity images. We adopt 162,770 for training and 19,961 for test. The dataset should be arranged in the same directory structure as

datasets
    ｜- celeba_dataset
        |- train
           |- <image_id>.jpg
           |- <image_id>.jpg
        |- val
           |- <image_id>.jpg
           |- <image_id>.jpg
        |- test
           |- <image_id>.jpg
           |- <image_id>.jpg

Preparing Paris Street View (PSV) Dataset. The dataset can be downloaded from here. The training and test set includes 14,900 and 100 images, respectively. This dataset was collected from the street views of Paris, taking a large number of buildings, and structure information, such as windows and doors. The dataset should be arranged in the same directory structure as

datasets
    ｜- psv_dataset
        |- train
           |- <image_id>.jpg
           |- <image_id>.jpg
        |- test
           |- <image_id>.jpg
           |- <image_id>.jpg

Preparing Mask Dataset. The dataset can be downloaded from here. This mask dataset contains 12,000 irregular masks grouped into six intervals according to the mask area on the total image size, where each interval has 2,000 masks. We employed three intervals (20-30%, 30-40%, and 40-50%) for test. The dataset should be arranged in the same directory structure as

datasets
    ｜- test_mask
        |- mask
            |- testing_mask_dataset
                |- 10-20
                    |- <mask_id>.png
                    |- <mask_id>.png                   
                |- 20-30
                    |- <mask_id>.png
                    |- <mask_id>.png
                |- ...

Or you can use the dataset grouped into directories per interval here.

The trained model can be download: places

The trained model can be download: celeba.

The trained model can be download: psv.

The trained models can be download:

• VHII efficient 256-128-64-32.
• VHII efficient 128-64-32-16.
• VHII efficient 64-32-16-8.

Once the dataset is prepared, new models can be trained with the following commands:

bash run_train.sh --train_config_file

For example:

bash run_train.sh configs/psv_proposal_efficient_128_64_32_16_channels.json

To test the models

1. Download the trained models, and save they in trained_models/.

2. Run the test bash file to evaluate/test the trained model.

bash run_test_dataset.sh --model_name --model_path --seed --gt_dataset_path --mask_dataset_path --output_dataset_path 

For example:

bash run_test_dataset.sh "VHII_efficient" "trained_models/celeba/celeba_VHII_efficient/gen_00050.pth" 0 "/data/celeba/celeba_dataset/test/" "/data/pconv/test_mask/20-30/" "test_output_datasets/trained_celeba_VHII_efficient_seed_0/output_images"

The outputs inpainted images are saved at test_output_datasets/.

To measure the quantitative results:

cd metrics
bash run_metrics.sh --gt_dataset_path --output_dataset_path

For example:

bash run_metrics.sh "/data/celeba/celeba_dataset/test/" "/config/variable-hyperparameter-image-impainting/test_output_datasets/trained_celeba_VHII_efficient_seed_0"

To inference a single image like this:

bash run_test_image.sh --model_name --model_path --input_path --mask_path --output_path --output_name

For example:

bash run_test_image.sh "VHII_efficient" trained_models/celeba/celeba_VHII_efficient/gen_00050.pth "examples/img/100_000100_gt.png" "examples/mask/100_000100_mask.png" "examples/output" "100_000100_output"

If you are having problems to test our model (e.g celeba), I suggest you follow the next steps:

• Use the images from examples/ directory, there you can find the examples that you can use to test the proposed model
• Download the celeba model here and put it inside the directory "trained_models/celeba/celeba_VHII_efficient"
• Create the docker image

docker build -t vhii-mage .

• Create a repository

nvidia-docker run --userns=host -it --rm --name vhii-repository -v /work/data/:/data -v /work/code/:/code vhii-mage bash

• Inside the docker repository, you can run the test command

bash run_test_image.sh "VHII_efficient" trained_models/celeba/celeba_VHII_efficient/best_model_celeba.pth "examples/img/100_000100_gt.png" "examples/mask/100_000100_mask.png" "examples/output" "100_000100_output"

@article{Campana2023_Inpainting,
  author=(J.L.F. Campana and L.G.L. Decker and M.R. Souza and H.A. Maia and H. Pedrini}
  title={Variable-Hyperparameter Visual Transformer for Efficient Image Inpainting},
  journal={Computers \& Graphics},
  year={2023}
}

If you have any questions or suggestions about this paper, feel free to contact me (j209820@dac.unicamp.br).