Item writing guidelines for Subject Matter Experts
1. Purpose
Thank you for your participation! As a subject matter expert (SME) in your field, you will be helping to create items for an upcoming exam. This exam will help determine whether or not test takers have the necessary knowledge and skills to work in a specified role in your industry.
2. Security and Confidentiality
Because these items might be used on an upcoming exam, it is extremely important that you keep the item completely private and unavailable to view by others, including while working on electronic devices such as cell phones.
3. Overall Process
There are three main steps to creating items. Most experts participate in all of these activities, so please note that you might be asked to also review items.
- Write Items: A SME writes an item.
- Item Review: A different SME(s) reviews the item based on research-based item writing principles. The item either “passes” the review or is returned to the item writer for revision. Oftentimes an item is required to pass multiple reviews from different SMEs.
- Revision/Revise Items: The initial SME revises the item while considering the feedback from the reviewer. (Initial feedback is common, so please be aware that you might be receiving input from reviewers.)
4. Choosing a Topic
From the Authoring Portal, select “Write Items” under your exam name.
Each Certiverse exam is created based on a list of pre-determined topics, along with an estimated number of items that need to be written for each topic. Certiverse also includes descriptions, which provide detailed information about the topics.
To start writing items, you must first choose a topic. Make sure to read the description carefully and understand its goal. It is critical that each item aligns with its respective topic.
Select “Write” to choose your topic and begin writing an item.
5. Item Writing
Certiverse has two different ways for you to write your items:
- The authoring wizard, which walks you through the process of writing each part of the item.
- AI-assisted item writing, where an AI will suggest an item for your review and revision.
AI-Assisted Item Writing
We’ve already written most of a prompt for an AI engine. All you need to add is the specific topic of your item. Type exactly what you want the item to be about (e.g., “what SQL statement stores data into a database” or “what civil rights law
The AI engine will then suggest three items. We expect the AI to continue to get better, but our current research suggests that over 80% of items will need at least minor revision, so you should review the suggestions and pick one that is a good start. The AI is good (not perfect) at making up well-formed sentences that are related to your topic, but you should check the correctness of the answer(s) marked as correct (the “key(s)”) and that the distractors are truly incorrect. You may need to select the best key(s) and then edit the items.
You can reject an item and it will be replaced (up to 10 tries). You can also “Try Again” and revise your specific item topic.
When you accept an item, it will be loaded into the Authoring Wizard for your review and revision.
Tip: Make sure you select the answer(s) that is/are the best key. That way, those answers will be loaded into the “Correct Response(s)” field(s) of the authoring wizard.
See here for a step-by-step video tutorial.
Authoring Wizard
The authoring wizard walks you through writing each part of an item. You’ll also use this interface to edit AI-generated content and to review your items. (The authoring wizard can also be switched the the standard item writing page – select “Standard” to switch views. It’s recommended that you use the wizard.)
The exam may contain one or a variety of the following item types:
|
Item Type Name |
Option |
|
Single Correct Response - Four Options |
(1 Correct & 3 Distractors) |
|
Single Correct Response - Two Options |
(1 Correct & 1 Distractor) |
|
Single Correct Response - Three Options |
(1 Correct & 2 Distractors) |
|
Single Correct Response - Five Options |
(1 Correct & 4 Distractors) |
|
Single Correct Response - Six Options |
(1 Correct & 5 Distractors) |
|
Multiple Correct Response - Four Options |
(2 Correct & 2 Distractors) |
|
Multiple Correct Response - Four Options |
(3 Correct & 1 Distractor) |
|
Multiple Correct Response - Five Options |
(2 Correct & 3 Distractors) |
|
Multiple Correct Response - Five Options |
(3 Correct & 2 Distractors) |
|
Multiple Correct Response - Five Options |
(4 Correct & 1 Distractor) |
|
Multiple Correct Response - Six Options |
(2 Correct & 4 Distractors) |
|
Multiple Correct Response - Six Options |
(3 Correct & 3 Distractors) |
|
Multiple Correct Response - Six Options |
(4 Correct & 2 Distractors) |
|
Multiple Correct Response - Seven Options |
(2 Correct & 5 Distractors) |
|
Multiple Correct Response - Seven Options |
(3 Correct & 4 Distractors) |
|
Multiple Correct Response - Seven Options |
(4 Correct & 3 Distractors) |
|
Multiple Correct Response - Seven Options |
(5 Correct & 2 Distractors) |
|
Multiple Correct Response - Eight Options |
(2 Correct & 6 Distractors) |
|
Multiple Correct Response - Eight Options |
(3 Correct & 5 Distractors) |
|
Multiple Correct Response - Eight Options |
(4 Correct & 4 Distractors) |
Additionally, each item requires:
- a rationale for each correct and incorrect answer. The rationale helps the reviewer understand why the correct answer is truly correct and the incorrect answers are truly incorrect.
- your estimation of the item difficulty; by asking what percentage of test takers will answer it correctly. Your response is based on your own evaluation of the item.
- a cited reference, which includes support for the correct answer and rationale. These references help avoid opinion-based items and will be used by reviewers to verify the rationale.
See here for a step-by-step video tutorial about the Item writing.
Writing principles are research-based guidelines authors should follow to help create high-quality items that are more likely to pass the review process. Please ensure that each item:
- focuses on significant, non-trivial information related to the topic.
- is clear, grammatically correct, and free from unnecessary words or details.
- possesses only one meaning (cannot be interpreted multiple ways).
- does not include “trickery” intended to misdirect from the correct answer.
- is inoffensive and free from bias (age, ethnicity, gender, disabilities, etc.).
- contains all the information needed to determine the correct answer and does not require the test taker to make assumptions.
- avoids negative phrasing (see Example A).
Items should focus on higher-order thinking, where test takers apply their knowledge to situations. Items focused on lower-order thinking, such as simple recall of facts, are less useful (see Example B).
Writing Principles for the Correct and Incorrect Answers
In addition to focusing on the question, it is also important to write high-quality answers. Please ensure that the correct and incorrect answers:
- have a similar format and are of similar length to ensure that the correct answer does not stand out (see Example C).
- are independent in content and do not “give away” the correct answer (see Example D).
- avoid specific determiners such as “never,” “always,” or “only” to make them false (see Example E).
- avoid complex multiple-choice items with answers such as: “all of the above,” “none of the above,” “A and D,” etc.
Good incorrect answers include common misconceptions and words that “sound official.” They should help distinguish between test takers who have and have not mastered the topic.
6. Examples
Note: Correct answer is in bold font.
|
EXAMPLE |
INCORRECT |
CORRECT |
|
A
(Item avoids negative phrasing.) |
With what condition is the fluid imbalance known as edema NOT commonly associated?
a) Viral infections. b) Congestive heart failure. c) Allergic reactions. d) Protein deficiency. |
With what condition is the fluid imbalance known as edema commonly associated?
a) Congestive heart failure. b) Viral infections. c) Multiple Sclerosis. d) Asthma. |
|
B
(Item focus on higher- order thinking.) |
Who was the second President of the United States?
a) John Adams. b) Thomas Jefferson. c) James Madison. d) James Monroe. |
Historians characterize the 22nd Amendment to the U.S. Constitution as a reaction to the presidency of which person?
a) Franklin D. Roosevelt. b) Herbert Hoover. c) Calvin Coolidge. d) Warren G. Harding. |
|
C
(Correct and incorrect answers have similar format and length.) |
Which of the following is the best indication of high morale in a supervisor’s unit?
a) The employees are willing to give first priority to attaining group objectives, subordinating any personal desires they may have. b) The employees are rarely required to work overtime. c) The supervisor enjoys staying late to plan the next day. d) The unit gives expensive birthday presents to each other. |
Which of the following is the best indication of high morale in a supervisor’s unit?
a) The employees willingly give first priority to group objectives. b) The employees are rarely required to work overtime. c) The supervisor enjoys staying late to plan for the next day. d) The unit members give expensive birthday presents to each other. |
|
D
(Answers are independent and do not “give away” the correct answer.) |
When conducting library research in education, which of the following is the best source to use for identifying pertinent journal articles?
a) Current Index to Journals in Education. b) A Guide to Sources of Education Information. c) Resources in Education. d) The International Encyclopedia of Education. |
When conducting library research in education, which of the following is the best source to use for identifying pertinent articles?
a) Current Index to Journals in Education. b) A Guide to Sources of Education Information. c) Resources in Education. d) The International Encyclopedia of Education. |
|
E
Avoid specific determiners (e.g., never, always, only.) |
What should you do to avoid infection after receiving a puncture wound to the hand?
a) Ensure that no foreign object has been left in the wound. b) Always go to the immunization center to receive a tetanus shot. c) Be treated with an antibiotic only if the wound is painful. d) Never wipe the wound with alcohol unless it is still bleeding. |
What should you do to avoid infection after receiving a puncture wound to the hand?
a) Ensure that no foreign object has been left in the wound. b) Go to the immunization center to receive a tetanus shot. c) Be treated with an antibiotic when the wound is painful. d) Wipe the wound with alcohol unless it is bleeding. |
Contact Us
If you have any questions or need additional assistance, please contact us by either emailing support@certiverse.com or by submitting a ticket from this article.
References
Burton, S. J., Sudweeks, R. R., Merrill, P. F., Wood, B. 1991. How to Prepare Better Multiple-Choice Test Items: Guidelines for University Faculty. Brigham Young University Testing Services and The Department of Instructional Science. https://testing.byu.edu/handbooks/betteritems.pdf
Haladyna, T. M. & Downing S. M. (1989). A taxonomy of multiple-choice item-writing rules. Applied Measurement in Education, 1, 37-50. https://doi.org/10.1207/s15324818ame0201_3
Rush, B. R., Rankin, D. C., & White, B. J. (2016). The impact of item-writing flaws and item complexity on examination item difficulty and discrimination value. BMC Medical Education, 16, 1-10. Article 250. https://doi.org/10.1186/s12909-016-0773-3
Example items are adapted, with permission, from the referenced Burton, Sudweeks, Merrill and Wood guide (1991). Example C created by Certiverse 2020.