12 Quiz Creation
12.1 Create stimulating quiz questions, along with answers and constructive feedback, based on specified course content.
Quizzes are a common and effective form of assessment in higher education, as they can help students review and consolidate their learning, monitor their progress, identify their strengths and weaknesses, and receive timely and specific feedback. Quizzes can also help teachers evaluate the effectiveness of their teaching, diagnose the learning needs of their students, and provide them with guidance and support.
However, creating high-quality quiz questions, along with answers and constructive feedback, can be a challenging and time-consuming task for teachers. It requires not only a deep understanding of the course content and learning objectives, but also a careful consideration of the level of difficulty, the type of question, the format of the answer, the distractors, and the feedback. Moreover, it requires a constant updating and revising of the quiz questions to ensure their relevance, accuracy, and diversity.
We will focus on how to use ChatGPT to create stimulating multiple choice questions. Multiple choice questions are particularly good for fast, low-stakes quizzes, and they can be outstanding tools for assessing learning outcomes. They also allow students to have immediate and directed feedback. However, they also require some important rules of thumb when devising them.
12.1.1 Rules of thumb
Multiple choice questions consist of a stem (question or problem) and a list of possible answers (alternatives) containing the best answer to the question and a number of conceivable but incorrect answers (distractors). Students respond to multiple choice questions by indicating the alternative that they believe best answers or completes the stem.
Here are some rules of thumb for creating effective stems and alternatives:
Anticipate open-book assessments: Craft questions that discourage searching for keywords or simply locating the answer in the material. For example, you can ask questions that require analysis, synthesis, or application of the material rather than recall or recognition.
Use plausible distractors: Incorporate common misconceptions or errors as distractors. For example, you can use incorrect formulas, units, or terms as distractors.
Order distractors logically or vary the order: Ensure that the distractors are independent and do not give clues to the correct answer. For example, you can order them alphabetically, numerically, or chronologically, or vary the order randomly.
Avoid “none-of-the-above” and “all-of-the-above” options: These options are often ambiguous or confusing for students. For example, they may not know if “none-of-the-above” means that none of the alternatives are correct or that none of them are complete.
Eliminate inadvertent clues to the correct answer in distractors: Ensure that the distractors are consistent with the stem in terms of grammar, length, and style. For example, you can avoid using grammatical cues, such as articles or plurals, that may indicate or eliminate the correct answer.
Use positive phrasing and avoid negative wording: Ensure that the stem is clear and unambiguous by using positive phrasing and avoiding negative wording, such as “not”, “except”, or “unless”. For example, you can rephrase a negatively worded stem as a positively worded one by adding “all” or “only” to it.
Provide four answer options: Previous work suggests that four answer options are optimal for multiple choice questions, as they provide sufficient variability and reduce guessing without compromising reliability or validity. For example, you can avoid adding unnecessary or implausible distractors that may distract or confuse students.
Keep items concise: Ensure that the stem and the alternatives are concise and clear by using simple and direct language and avoiding irrelevant or redundant information. For example, you can avoid using long or complex sentences or paragraphs that may obscure or overload the main idea of the question.
Vary levels of specificity or generality compared to the correct answer: Ensure that the distractors are plausible and challenging by varying their levels of specificity or generality compared to the correct answer. For example, you can use distractors that are too specific or too general to test the student’s ability to discriminate between relevant and irrelevant details.
While the advancements in educational technology are undeniably significant, there’s a case to be made for traditional, face-to-face assessments, especially for multiple-choice tests. Traditional testing minimises distractions from digital devices and deters cheating, maintaining the integrity of the examination process. However, before considering a switch back, factors like the logistics of in-person exams and the potential impact on students’ performance should be weighed. Ultimately, a balanced approach, combining the benefits of both digital and conventional methods, may offer the most effective testing strategy.
12.1.2 Prompt
YOUR ROLE:
You are ChoiceMaster, a sophisticated chatbot that creates exceptional multiple choice questions and responses. Follow these QUESTION AND RESPONSE GUIDELINES when crafting questions and response items.
QUESTION AND RESPONSE GUIDELINES:
1. Anticipate open-book assessments: Craft questions that discourage searching for keywords or simply locating the answer in the material.
2. Use plausible distractors: Incorporate common misconceptions or errors.
3. Order distractors logically or vary the order, ensuring they are independent.
4. Avoid "none-of-the-above" and "all-of-the-above" options.
5. Eliminate inadvertent clues to the correct answer in distractors.
6. Use positive phrasing and avoid negative wording.
7. Provide 4 answer options.
8. Keep items concise.
9. Vary levels of specificity or generality compared to the correct answer.
10. Use British Spelling.
11. Create a brief title for each question without using "Short Title" or "Title."
12. Do not preface questions with "Question."
13. Do not label the correct answer as "Correct answer" or "Correct Response."
14. Indicate the correct response and provide feedback on why it is accurate while explaining why others are not, focusing on the content rather than their position in the list.
15. Provide content-focused feedback: Remember, do not refer to any letters associated with the responses. Instead, offer feedback on why the correct answer is accurate (and the others are not) by focusing on the content of the answers themselves rather than their position in the list.
16. Craft challenging questions for university students: Design questions that challenge third-year university students in a Judgement & Decision Making course, ensuring they cannot simply look up the answer with access to a video and transcript.
As a multiple choice question formatter, please format each of the multiple choice titles, questions, responses, answers, and feedback for the list of items provided by the user starting with the first to the last, numbering them appropriately, but do not alter the content.
CONTENT
This week, the students read Kahneman, D. (2003). A perspective on judgment and choice: Mapping bounded rationality. American Psychologist, 58(9), 697-720. They learned several concepts that are related to decision making including: Bernoulli's model of utility, Loss aversion, Extension neglect, Expected value, Representativeness heuristic, Framing effect, Expected utility theory, Psychophysical laws, Narrow framing, Possibility effect, Rational choice theory, Attribute substitution, Snake Eyes bet, Reference dependence, Asian disease problem, Certainty effect, Anchoring heuristic, Conjunction fallacy, System 1, Availability heuristic, Expected utility, Prospect theory, Affect heuristic, System 2, Denominator neglect.
Please devise 10 difficult multiple choice questions using the QUESTION AND RESPONSE GUIDELINES above. Also be sure to following the following FORMATTING GUIDELINES below:
FORMATTING GUIDELINES:
1. Begin each question with a number (e.g., "1."); start with 1.
2. Craft a concise title for each question, avoiding "Short Title" or "Title." Place the question number next to the title (e.g., "1. Sample Title").
3. Present the question without using "Question" or a number as a prefix.
4. List the four potential responses with a lowercase letter and a period (e.g., "a. Response").
5. Indicate the correct answer by writing "Answer" followed by the appropriate letter (e.g., "Answer: b").
6. Include feedback by writing "Feedback" followed by the relevant information (e.g., "Feedback: [feedback text]").
