Overview

This PR adds mmcv/runner/hooks/eval.py file and EvalHook and DistEvalHook to MMCV together with related unittest parts. Since those two hooks have been widely used in many MM repos, this PR implements and unifies common functions used in the evaluation part.

Design

Model Evaluation performed in OpenMMLab uses EvalHook and DistEvalHook during training.

In detail, the those hooks are commonly registered in apis/train.py by runner.register_hook(eval_hook(val_dataloader, **eval_cfg)). Users can initialize their models with following steps:

1. Define validation dataset by assigning cfg.data.val, building val_datasetloader
2. Register EvalHook using runner.register_hook.

Once registering it, runner will periodically call the evaluate function to perform evaluation in a fixed mode (by epoch or by iteration, etc.). The high-level workflow of EvalHook in OpenMMLab is:

register EvalHook -> define evaluation setting (start, interval, by_epoch, save_best, comparison rules, etc.) -> perform evaluation after_train_iter or after_train_epoch -> (save the best checkpoint) -> loop back to after_train_xxx

APIs

EvalHook is the base module, and DistEvalHook is a child class to it. They both inherit Hook. Here is the APIs explanation for EvalHook.

Initialization

• dataloader: The PyTorch dataloader for evaluation. It can be built with build_dataloader with provided dataloader setting.

• start: Evaluation starting epoch. It enables evaluation before the training starts if start <= the resuming epoch. If set to None, whether to evaluate is merely decided by interval.

• interval: Evaluation interval.

• by_epoch: Determine whether to perform evaluation by epoch or by iteration. If set to True, it will perform by epoch. Otherwise, by iteration.

• save_best: If a metric is specified, it would measure the best checkpoint during evaluation. The information about best checkpoint would be save in runner.meta['hook_msgs']. Options are the evaluation metrics to the validation dataset. e.g., bbox_mAP, segm_mAP for bbox detection and instance segmentation. AR@100 for proposal recall. If save_best is auto, the first key of the returned OrderedDict result will be used. The interval of CheckpointHook should be divisible by that of EvalHook.

• rule : Comparison rule for best score. If set to None, it will infer a reasonable rule. Keys such as acc, top, etc. will be inferred by greater rule. Keys contain loss will be inferred by less rule. Options are greater, less, None.

  But Note: Since the rule for downstream repos are different, it may need to overwrite the self.greater_keys and self.less_keys

• eval_kwargs: Key word arguments for dataset evaluate function (def evaluate), which will be fed into the evaluate function of the dataset.

• Other hardcoded inner variables: rule_map, init_value_map, greater_keys and less_keys.

  Note that: Since the rule for downstream repos are different, it may need to overwrite these variable due to the specific task.

  • rule_map: A dict containing comparison function, default as {'greater': lambda x, y: x > y, 'less': lambda x, y: x < y}.
  • init_value_map: The initialized value for comparison, default as {'greater': -inf, 'less': inf}.
  • greater_keys : A list containing some rule keys applied for greater function, which means in these rules EvalHook regards greater number as the better one. Note that: If a string is one of the rules' substring, it will be applied to greater function. e.g., for acc: top1_acc, top5_acc, mean_acc are all applied to greater rule.
  • less_keys: Similar to greater_keys but it is for less rules. e.g., for loss: bce_loss, bmn_loss are all applied to less rule.

before_run

This part is to initialize runner.meta.hook_msgs, if users determine to save the best checkpoint.

before_train_xxx

For before_train_iter and before_train_epoch part. It is mainly to determine whether it is the right time to perform evaluation by examining:

• Whether it is the correct time to perform evaluation according to by_epoch
• Whether it achieves the start to perform evaluation

And use self.initial_flag to indicate whether the EvalHook gets into the normal evaluation loop. After getting into the normal evaluation loop, before_train_xxx will be skipped.

after_train_xxx

For after_train_iter and after_train_epoch part. It is mainly to inference the model and do evaluation, as well as save the best checkpoint if save_best is specified. In detail, it will call _do_evaluate() and _save_best().

• _do_evaluate(): inference the model by calling single_gpu_test(), do evaluation and call _save_best()
• _save_best(): compare the score according to the rule, write info into runner.meta['hook_msgs'] and save the best checkpoint in the work_dir

For DistEvalHook, besides the variable and function mentioned above, it inferences model by calling multi_gpu_test() and assigns tmpdir, gpu_collect for multi_gpu_test()

Usages

Users adopt EvalHook by typing in --validate for training command, it will call EvalHook in training.

Config

To use EvalHook with a specific setting, users need to modify the evaluation variable in config files. like this:

evaluation = dict(
    interval=1, metrics=['top_k_accuracy', 'mean_class_accuracy'])

the key-value pairs in the dict are corresponding to EvalHook API.

Migration

Since the key for determine greater an less is related to the downstream task, downstream repos may need to overwrite the self.greater_keys and self.less_keys:

from mmcv.runner import EvalHook as BasicEvalHook

class EvalHook(BasicEvalHook):

    greater_keys = ['a', 'b']
    less_keys = ['c']

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)