Some key takeaways on designing assessment questions:
Instructors may choose to employ a variety of different question types when designing assessments. Three of the most common types of questions are multiple choice, short answer, and essay questions, though others, such as true/false, matching, and fill-in-the-blank are used as well. This page will provide recommendations and examples to guide you through the exam design process.
Align questions with course learning objectives
Effective questions tie back to class activities and learning goals, so that students can apply their knowledge and see connections throughout their learning. Instructors can use backward design to: first, write course learning outcomes; second, develop their assessments that evaluate student learning of those outcomes; and third, design the in-class and out-of-class activities that will help students achieve these learning outcomes and be successful on the assessments.
Use Bloom’s Taxonomy.
Bloom’s Taxonomy, revised in 2001, categorizes six levels of learning from lower- to higher-level thinking skills: Remember, Understand, Apply, Analyze, Evaluate, and Create. Ideally, learning outcomes, class activities, and assessments map to these ascending levels throughout a class or course. Instructors can develop or select a variety of question types that assess different levels of cognition.
Provide opportunities to practice before exams.
Instructors can introduce low-stake questions in class to have students to practice material as they prepare for a test, and these questions can be approached individually, through group work, class discussion, case study, or a variety of other means; moreover, instructors and students can use these practice outcomes to modify instruction and study habits. Additionally, instructors can encourage students to take practice multiple choice questions aligned with course material outside of class while studying for tests.
Utilize Canvas.
Yale’s learning management system is Canvas, which features a variety of tools for constructing different types of assessment questions, grading anonymously, and reviewing statistics for outcomes.
Instructors should be aware of students with accessibility needs and provide accommodations to support students who have a documented disability and may need more time on tests, materials in multiple formats, and other accommodations.
Multiple choice questions are typically composed of (1) a question stem and (2) several choices, including distractors and one correct option. Prior research suggests including a minimum of 3 answer choices to ensure reliable assessment (Haladyna et al. 2002).
Designing the question stem:
Good question stems introduce the central idea clearly and briefly, and are constructed in positive ways that avoid words such as “not” or “except” (so, “In the Dred Scott decision, the Supreme Court enabled slave owners to …”, instead of “In the Dred Scott decision, the Supreme Court’s decision did not …”).
Including distractors:
Good distractors target major student misconceptions, remain clear by avoiding overlapping content and awkward grammar, and avoid catch-all answers like “all of the above” or “none of the above.” Additionally, effective distractors should not provide clues to rule them out and should shuffle order with the right answer from question to question.
Removing low-quality questions:
Many institutions have statistical software programs that provide reports on the quality of multiple-choice questions. Two common reliability tests are item difficulty and item discrimination. Item difficulty is typically reported as the percentage of those taking the test who chose the correct answer for an item. In classical testing theory (McCowan and McCowan, 1999), the optimal difficulty guideline for a 4-option multiple choice test is 63%. Item discrimination is the measure of how an item detects differences between higher and lower scores on a test. Typically, this is measured by the point-biserial, which is a correlation coefficient. Items that discriminate well are answered correctly more often by the higher scoring students and have a higher positive correlation. Good discrimination indexes are 0.4 and above, poor are 0.2 and below. Instructors can use this data from the test to remove or revise lower quality multiple choice questions.
Promoting academic integrity:
Canvas Quizzes allows randomization of quiz question order and randomizing responses within questions (ABCD, ACBD, DBAC). For mathematical questions, the numbers used in an equation can be automatically altered between versions while the formula and process remain the same.
Short Answer and Essay Questions provide instructors with the opportunity to test students’ ability to apply knowledge learned in the classroom and explain their logic and understanding.
Specifying the task:
Questions should use clear and specific verbs — such as define, compare, evaluate, etc. — to describe what students ought to accomplish in their responses.
Including suggested response length:
To avoid student uncertainty on how long a short answer or essay response ought to be, consider including a suggested length alongside the question. Ensure that this length is appropriate and achievable given the time allocated for the exam.
Presenting evaluation criteria:
When possible, instructors should share the criteria that they will be using to evaluate student responses. Learn about developing a rubric. To ensure that students have a sense of what to expect and how to prepare, instructors should discuss the format of an exam in class prior to the exam date and explain what the expectations for open-ended responses are.
Potential energy is:
a) the energy of motion of an object.
b) not the energy stored by an object.
c) the energy stored by an object.
d) not the energy of motion of an object.
In this question the good stem is clear, brief, and presents the central idea of the question through positive construction. However, the distractors are confusing: b) and d) are written in negative constructions that force students to reinterpret the stem, while c) and d) have overlapping, inconsistent content that confuses and tests reading comprehension over content recall.
Potential energy is not the energy:
a) of motion of a particular object.
b) stored by a particular object.
c) relative to the position of another object.
d) capable of being converted to kinetic energy.
In this question the poor stem contains the word “not,” which fails to identify what potential energy is, and tests grammar over student understanding. However, the good distractors are written clearly, cover unique content, and follow a logical and consistent grammatical pattern.
Potential energy is:
a) the energy of motion of an object.
b) the energy stored by an object.
c) the energy emitted by an object.
In this example both the stem and the distractors are written well, remain consistent, and test a clear idea.
This list, organized along Bloom’s Taxonomy, is adapted from the Charles Sturt University’s guide on short answer questions.
List three examples of fossil fuels.
Identify the five levels in Maslow’s Hierarchy of Needs.
What is the capital gains tax?
Define chiaroscuro.
Explain the purpose of scaffolding as a teaching strategy.
Why does the demand for luxury goods increase as the price increases?
Use two examples to show how scaffolding can be used to improve the efficacy of teaching and learning.
Provide three reasons to justify or refute the following argument: governments have a moral obligation to mitigate climate change.
Apply the concept of legal rules and standards to explain how speed limits are designed.
Apply modernization theory to a country case study of your choice, presenting evidence in support and opposition of the theory.
Instructors can also design questions that combine one or more of these skills — such as list and define — to form a question, making sure to assign points for each task.
Instructors often use essay questions to evaluate student understanding and ability to apply classroom concepts. Often, these questions will ask students to make an argument and write a response that is three or more paragraphs. Essay questions on exams are particularly common in the humanities and social sciences.
Essay questions may ask students to argue, justify, evaluate, criticize, or compare, among other tasks. Some sample questions are included below:
1. Humanities Essay Question
Write a close reading of the Milton sonnet provided. Remember that close reading is a form of argument. You are making a debatable claim about the meaning of your sonnet, so the essay should have a recognizable thesis statement in its introduction. The essay as a whole should persuade the reader to accept the truth of your claim
by subjecting the text to careful attention. In each body paragraph, you’ll want to quote a passage from the poem itself, and then interpret that passage, in order to show your reader how your interpretation of the sonnet advances your argument.
2. Social Science Essay Question
Using insights from the lectures, readings, and section discussions about the provision of global public goods, identify what factors are likely to be most important in determining whether the world is able to successfully provide the global public good of manageable greenhouse gases. Your essay should state the key features of the climate change challenge, have a clear thesis statement responsive to the question, and develop its argument with reference to relevant empirical examples and/or evidence.
3. Science Essay Question
Based on your reading of the book and the assigned reading as well as what we have learned in class, decide which side you most agree with. Use at least one science example from the book (specific names and chapters are not necessary) and at least one example from the AACR reading or something we learned in class to support your position, and also your opinion about the microarray tests in relation to this question about the purpose of research and the purpose of the results from those tests.
You can use your notes, the class slides or assigned readings to help you support your choice. Your answer should be in the form of a complete short essay and should include specifics about the basic science as well as some examples of a health disparity.
Haladyna, T.M., Downing, S.M., Rodriguez, M.C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education 15(3): 309-334.
McCowan, R.J., & McCowan, S.C. (1999). Item Analysis for Criterion-Referenced Tests.