This repository contains the question-answering StreamingQA datasets, a list of deduplicated WMT document IDs, and a script to process and filter the WMT documents to be used in conjunction with the paper: StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models (Liška, Kočiský, Gribovskaya, Terzi et al., 2021).

If you use this dataset in your research please cite

@article{streamingqa2022,
  title={StreamingQA: A Benchmark
    for Adaptation
    to New Knowledge over Time
    in Question Answering Models},
  author={Adam Li{\v{s}}ka and
    Tom{\'a}{\v{s}} Ko{\v{c}}isk{\'y} and
    Elena Gribovskaya and
    Tayfun Terzi and
    Eren Sezener and
    Devang Agrawal and
    Cyprien de Masson d'Autume and
    Tim Scholtes and
    Manzil Zaheer and
    Susannah Young and
    Ellen Gilsenan-McMahon
    Sophia Austin and
    Phil Blunsom and
    Angeliki Lazaridou},
  journal={arXiv preprint arXiv:2205.11388},
  year={2022}
}

The paper specific data can be downloaded using the links provided below. These are files stored in Google Cloud Storage in gzipped form.

We downloaded document-split versions of the English WMT News Crawl dataset. As the dataset does not provide document IDs, we used SHA256 hashes of the Base64 encoded unsplit texts of articles as part of "sorting key IDs" (see below).

For the paper, we use a deduplicated subset of the WMT data. To reproduce the subset, please find a list of WMT sorting key IDs, which in conjunction with the extraction script can be used to filter out duplicate documents. The list is stored as newline delimited sorting keys.

The StreamingQA questions and answers (including metadata) are stored in JSONL files. We provide subsets for train, valid, and eval separately.

Each QA entry has attributes:

For detailed definitions of the toxicity classifiers please refer to Perspective API website.

The following table is necessary for this dataset to be indexed by search engines such as Google Dataset Search.

This dataset is based on news articles from various sources and contains a small number of questions or answers, both human written and automatically generated, that are toxic and may be triggering or worrisome for researchers, or cause models to generate such content. We aimed to create a balanced process that identifies most of the toxic content while decreasing the risk of removing false positives. We estimate that 0.5% items in the dataset are toxic after our toxicity filtering. Secondly, questions and answers reflect information from the news articles, and in particular, may not always be factually correct. Furthermore, this dataset is intended to evaluate adaptation of models to new information in news over time, and therefore, it may not be applicable to settings where the assumptions we made don't apply. We provide further toxicity discussion and details of our filtering in the paper.

For installation please run ./run.sh to setup a Python environment and install the necessary Python packages (listed in requirements.txt). The script completes with the output of the test.

After installation, and in the activated Python virtual environment (here streamingqa_env) you may start an interactive Python session and use the following code (if you have downloaded files to the same directory as the code from the links provided above):

We provide a Python script extraction.py that extracts the downloaded WMT data, pre-processes the text, assigns the sorting key IDs, and finally filters out the duplicate documents. The main entry point get_deduplicated_wmt_docs yields WMTDoc objects with attributes being the assigned sorting key ID (sorting_key), the document publication date as UTC timestamp in seconds (publication_ts), and the pre-processed document text (text).

import extraction

_archive_file_names = [
  'news-docs.2007.en.filtered.gz',
  'news-docs.2008.en.filtered.gz',
  'news-docs.2009.en.filtered.gz',
  'news-docs.2010.en.filtered.gz',
  'news-docs.2011.en.filtered.gz',
  'news-docs.2012.en.filtered.gz',
  'news-docs.2013.en.filtered.gz',
  'news-docs.2014.en.filtered.gz',
  'news-docs.2015.en.filtered.gz',
  'news-docs.2016.en.filtered.gz',
  'news-docs.2017.en.filtered.gz',
  'news-docs.2018.en.filtered.gz',
  'news-docs.2019.en.filtered.gz',
  'news-docs.2020.en.filtered.gz',
  'news-docs.2021.en.filtered.gz',
]

wmt_docs = extraction.get_deduplicated_wmt_docs(
  wmt_archive_file_paths_or_objects=_archive_file_names,
  deduplicated_sorting_keys_file_path_or_object='wmt_sorting_key_ids.txt.gz',
)

Furthermore, we also provide a function to reproduce our splits of articles into sentence chunks. These passages can be used as the search space for the retrieval architecture as is discussed in more detail in the paper mentioned above.

wmt_passages = extraction.get_wmt_passages_from_docs(
  wmt_docs=wmt_docs,
  preprend_date=True,
)

import gzip
import json

_file_name_by_streamingqa_subset = {
  'train': 'streaminqa_train.jsonl.gz',
  'valid': 'streaminqa_valid.jsonl.gz',
  'eval': 'streaminqa_eval.jsonl.gz',
}

streamingqa = {}
for subset_name, file_name in _file_name_by_streamingqa_subset.items():
  with open('streamingqa_train.jsonl.gz'), 'rb') as input_file:
    with gzip.open(input_file) as ungzipped_file:
      streamingqa[subset_name] = [
          json.loads(line.decode()) for line in ungzipped_file
      ]

Copyright 2022 DeepMind Technologies Limited

All software is licensed under the Apache License, Version 2.0 (Apache 2.0); you may not use this file except in compliance with the Apache 2.0 license. You may obtain a copy of the Apache 2.0 license at: https://www.apache.org/licenses/LICENSE-2.0

All other materials are licensed under the Creative Commons Attribution 4.0 International License (CC-BY). You may obtain a copy of the CC-BY license at: https://creativecommons.org/licenses/by/4.0/legalcode

Unless required by applicable law or agreed to in writing, all software and materials distributed here under the Apache 2.0 or CC-BY licenses are distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the licenses for the specific language governing permissions and limitations under those licenses.

This is not an official Google product.