Instructions for Fill-in-the-blank Evaluation Task

Your goal is to analyze how good an artificial intelligence is at generating text that makes sense with respect to the text before and after it. You will be shown the start of a passage of text where the AI's continuation has been highlighted in yellow. You will then be asked:

Does the highlighted text make sense?

Answer "not at all" if:

Answer "completely" if:


You must adjust every slider to be able to submit the HIT. The questions you've already worked on will be marked with a ✅.



Question 1/7



${left} ${pred_0} [...]

Does the highlighted continuation make sense with respect to the text before it?



Now, suppose the text is continued in the following way.

${left} ${pred_0} ${right}

Does the highlighted section make sense given this continuation?




Question 2/7



${left} ${pred_1} [...]

Does the highlighted continuation make sense with respect to the text before it?



Now, suppose the text is continued in the following way.

${left} ${pred_1} ${right}

Does the highlighted section make sense given this continuation?




Question 3/7



${left} ${pred_2} [...]

Does the highlighted continuation make sense with respect to the text before it?



Now, suppose the text is continued in the following way.

${left} ${pred_2} ${right}

Does the highlighted section make sense given this continuation?




Question 4/7



${left} ${pred_3} [...]

Does the highlighted continuation make sense with respect to the text before it?



Now, suppose the text is continued in the following way.

${left} ${pred_3} ${right}

Does the highlighted section make sense given this continuation?




Question 5/7



${left} ${pred_4} [...]

Does the highlighted continuation make sense with respect to the text before it?



Now, suppose the text is continued in the following way.

${left} ${pred_4} ${right}

Does the highlighted section make sense given this continuation?




Question 6/7



${left} ${pred_5} [...]

Does the highlighted continuation make sense with respect to the text before it?



Now, suppose the text is continued in the following way.

${left} ${pred_5} ${right}

Does the highlighted section make sense given this continuation?




Question 7/7



${left} ${pred_6} [...]

Does the highlighted continuation make sense with respect to the text before it?



Now, suppose the text is continued in the following way.

${left} ${pred_6} ${right}

Does the highlighted section make sense given this continuation?