HTM-Align is a manually annotated 80-video subset of HowTo100M (HTM) dataset, to evaluate the alignment performance. It is a test set randomly sampled from Food & Entertaining category of HTM. These videos are not used for any training in our project.

• HTM-Align(616KB json) [from Oxford server]

import json
htm_align = json.load(open('htm_align.json'))

print(len(htm_align))  # 80

print(htm_align['-3CEg4y7mQM'][1])
# [1, 10.739, 17.535, 'add extra virgin olive oil and garlic']
# format: [alignability (1/0), start(second), end(second), text]

*: since the model does not have a binary classifier for alignability, for each sentence, we first compute the sentence-visual similarity scores, then take its maximum score over time as the alignability measurement to compute ROC-AUC.

**: in the paper, we only reported CLIP and MIL-NCE results with the 'global' time window setting, since CLIP and MIL-NCE do not use long-range temporal context. Here we also show their results with the 'moving window' setting for a fair comparison.

Note: After fixing a bug (I did not divide in this line) in ROC-AUC metric, the reproduced ROC-AUC scores are different with the numbers originally reported in the paper Table 1. Please consider comparing with the new results here. We will update our arXiv paper for this correction.

If you find this dataset useful for your project, please consider citing our paper:

@InProceedings{Han2022TAN,
    author       = "Tengda Han and Weidi Xie and Andrew Zisserman",
    title        = "Temporal Alignment Networks for Long-term Video",
    booktitle    = "CVPR",
    year         = "2022",
}