Question answering & distractors

A multiple-choice question needs wrong answers that are plausible but different from the correct one. This pipeline creates them automatically from a reading passage: it answers the question, then turns that answer into three distractors. The generated questions can be used to train and evaluate question-answering systems.

Three stages: DistilBERT answers SQuAD questions with 76.52% exact match and 84.77% F1; spaCy entities in the answer are replaced with same-type entities to make distractors, falling back to a predefined list when fewer than three are found; a T5 model answers the resulting multiple-choice questions with 90.53% exact match. — 1. A fine-tuned DistilBERT finds the answer span in the passage. 2. spaCy tags the answer's named entity, which is swapped for other entities of the same type from the passage, or from a predefined list when the passage has fewer than three. 3. A separate T5 model answers the finished multiple-choice question to check it.

Muhanad Tuameh · Metin Usta

Model-based evaluation does not establish that every distractor is incorrect, natural, or useful to a learner.

Models and implementation

DistilBERT is fine-tuned on SQuAD. A pretrained spaCy model supplies named entities. The evaluator is a pretrained T5 model fine-tuned on QASC. Built with Hugging Face Transformers, spaCy, and PyTorch in Jupyter notebooks.

Project overview · Three notebooks