Preparation of Classroom Assessment
Planning the Test
One essential step in planning a test is to decide why you are giving the test. (The word “test” is used although we are using it in a broad sense that includes performance assessments as well as traditional paper and pencil tests.)
Are you trying to sort the students (so you can compare them, giving higher scores to better students and lower scores to poor students)? If so you will want to include some difficult questions that you expect only a few of the better students will be able to answer correctly. Or do you want to know how many of the students have mastered the content? If your purpose is the latter, you have no need to distribute the scores, so very difficult questions are unnecessary. You will, however, have to decide how many correct answers are needed to demonstrate mastery. Another way to address the “why” question is to identify if this is to be a formative assessment to help you diagnose students’ problems and guide future instruction, or a summative measure to determine grades that will be reported to parents.
Four Steps in Planning an Assessment
- Deciding its purpose
- Developing test specifications
- Selecting best item types
- Preparing items
Step 1: Decide the Purpose
What location in instruction?
- limited in scope
- low difficulty level
- serve as basis of remedial work, adapting instruction
- pretest (placement)
- items similar to outcome measure
- but not the same (like an alternative form)
- during instruction
- monitor learning progress
- detect learning errors
- feedback for teacher and students
- limited sample of learning outcomes
- must assure that mix and difficulty of items sufficient
- try to use to make correction prescriptions (e.g., review for whole group, practice exercises for a few)
- enough items needed in each specific area
- items in one area should have slight variations
- end of instruction
- mostly summative –broad coverage of objectives
- can be formative too
Step 2: Develop Test Specifications
- Why? Need good sample!
- How? Table of specifications (2-way chart, “blueprint”)
- Prepare list of learning objectives
- outline instructional content
- prepare 2-way chart
- or, use alternative to 2-way chart when more appropriate
- doublecheck sampling
Table of Specification
|Objectives/Content area/Topics||Knowledge||Comprehension||Application||# items/
% of test
|1. Know the advantages & disadvantages of the major selection-types of questions.|
|2. Be able to differentiate between well and poorly written selection-type questions|
|3. Be able to construct appropriate selection-type questions using the guidelines and rules that were presented in class.|
Why are These Better Specific Learning Outcomes?
- Selects correct definitions
- Lists major limitations of different item types
- Selects proper procedures for assessment purpose
- Distinguishes poor procedures from good ones
- Distinguishes poor decisions/products from good ones
- Applies construction guidelines to a new content area
- Identifies flaws in procedure/product
- Lists major and specific content areas
- Lists general and specific learning outcomes
- Creates a component of the test
- Provides weights for cells in table of specifications
- Judges quality of procedure/product
- Justifies product
- Improves a product
Step 3: SELECT THE BEST TYPES OF ITEMS/TASKS
What types to choose from? Many!
- short answer
- multiple choice
- extended response
- restricted response
- extended response
- restricted response
Which type to use? The one that fits best!
- most directly measures learning outcome
- where not clear, use selection-type (more objective)
- multiple choice best (less guessing, fewer clues)
- matching only if items homogeneous
- true-false only if only two possibilities
Strengths and Limitations of Objective vs. Essay/Performance
- Can have many items
- Highly structured
- Scoring quick, easy, accurate
- Cannot assess higher level skills (problem formulation, organization, creativity)
- Can assess higher level skills
- More realistic
- Inefficient for measuring knowledge
- Few items (poorer sampling)
- Time consuming
- Scoring difficult, unreliable
Step 4: PREPARE ITEMS/TASKS
Strategies to Measure the Domain Well—Reliably and Validly
- specifying more precise learning outcomes leads to better-fitting items
- use 2-way table to assure good sampling of complex skills
- use enough items for reliable measurement of each objective
- number depends on purpose, task type, age
- if performance-based tasks, use fewer but test more often
- keep in mind how good assessment can improve (not just measure) learning
- signals learning priorities to students
- clarifies teaching goals for teacher
- if perceived as fair and useful
Strategies to Avoid Contamination
- eliminate barriers that lead good students to get the item wrong
- don’t provide clues that help poor students get the item correct
General Suggestions for Item Writing
- use table of specifications as guide
- write more items than needed
- write well in advance of testing date
- task to be performed is clear, unambiguous, unbiased, and calls forth the intended outcome
- use appropriate reading level (don’t be testing for ancillary skills)
- write so that items provide no clues (minimize value of “test-taking skills”)
- avoid specific determiners (always, never, etc.)
- don’t use more detailed, longer, or textbook language for correct answers
- don’t have answers in an identifiable pattern
- write so that item provides no clues to other items
- seeming clues should lead away from the correct answer
- experts would agree on the answer
- if item revised, recheck its relevance
Writing Instructional Objectives
Begin with the end in mind…
An objective is a description of a performance you want learners to be able to exhibit before you consider them competent. An objective describes an intended result of instruction, rather than the process of instruction itself.
Course Description, Goals and Objectives
It is important to be able to differentiate the course description from the course objectives. A course description simply tells what the course is about. You might consider the GOALS of the course to be linked to the course description; they are broad educational statements fitting the mission and description of the course. Specific measurable objectives, however, tell what the learner will be able to do upon successful completion of the course. Begin with the end in mind…
Why are well-written objectives important?
First, they provide some basis and guidance for the selection of instructional content and procedures. Second, they help in evaluating the success of the instruction. Third, they help the student organize his/her efforts to accomplish the intent of the instruction.
How Does One Write a Good Objective?
Most of us are fairly proficient at writing a course description and course goals. We have a good sense of what we want the course to be about. Writing behavioral objectives, however, can take some practice. So let’s begin with just two simple rules.
Rule #1: A good objective communicates your intent well and leaves little room for interpretation. There are words that we often use that are open to many interpretations, and there are words that we can use that leave less to the imagination. So, when writing behavioral objectives, stick to the words that leave less room for interpretation. Consider the following:
The verbs below may prove useful as you write your instructional objectives and are based on Blooms’ Taxonomy of cognitive behavior:
- RULE#2: There are three characteristics thathelp communicate intent when writing an objective: Performance, Conditions, and Criterion.
Performance: An objective always states what a learner is expected to be able to DO.
Conditions: An objective often describes the conditions under which a student is able to DO or perform the task.
Criterion: If possible, an objective clarifies how well the student must perform the task, in order for the performance to be acceptable.
…Other ways to think about writing instructional objectives include the following:
ABCDs of Writing Objectives
- A-Audience: The who. “The student will be able to…”
- B-Behavior: What a learner is expected to be able to do or the product or result of the doing. The behavior or product should be observable.
- C-Condition: The important conditions under which the performance is to occur.
- D-Degree: The criterion of acceptable performance. How well the learner must perform in order for the performance to be considered acceptable.
Learner objectives can relate to Cognitive, Affective, and Psychomotor domains.
Learner objectives can relate to Cognitive, Affective, and Psychomotor domains.
Here are some examples of fairly well-written objectives: There is always room for improvement…
By the end of the Neurology curriculum, the learner will describe in writing a cost-effective approach to the initial evaluation and management of patients with dementia (an approach that includes at least six of the eight elements listed on their handout).
By the end of their internal medicine ambulatory clerkship, each third-year medical student will have achieved cognitive proficiency in the diagnosis and management of hypertension, diabetes, angina, chronic obstructive pulmonary disease, hyperlipidemia, alcohol and drug abuse, smoking, and asymptomatic VIV infection, as measured by acceptable scores on interim tests and the final examination.
The student will be able to name the five stages of mitosis. (Not all objectives need all components specified above. This is a nice, clearly measurable
The student will be able to describe protein folding and its relation to protein function.
The student will be able to define stress; discuss typical neurobiological and behavioral responses to stress and their implications for physical and mental functioning.
By the end of the HIV curriculum, all residents will have identified their attitudes and beliefs regarding HIV patients who are drug abusers and will have discussed with their colleagues and attending physicians how these might influence their management of such patients.
Interact with others with honesty, civility, and respect.
Maintain composure when dealing with hostile or difficult patients. These skills can be assessed by direct or videotaped observation by an instructor.
From UCSD’s Curricular Objectives
- Compassion and sensitivity in the care of patients and respect for their privacy and dignity.
- Respect for patient autonomy.
- The ability to build therapeutic relationships with patients.
- Honesty and integrity in all interactions.
- Responsibility and trustworthiness in the execution of all duties.
- The ability to accept criticism and to understand the limitations of one’s own knowledge and skills.
- Adherence to all of the principles of UCSD’s Professionalism Code.
- Demonstrate a commitment to excellence and ongoing professional development.
In addition, graduates of the UCSD School of Medicine should possess the following:
- Dedication to the well-being and needs of patients.
- Dedication to lifelong learning and an appreciation for the role of science in medical advances.
- Dedication to continual enhancement of clinical skills.
- An understanding of the threats posed by conflicts of interest in the practice of medicine and the performance of research.
- A commitment to promote the health and well-being of the communities they serve.
- The willingness to lead when leadership is required.
By the end of the curriculum, all medical students will have demonstrated proficiency in assessing alcohol abuse by utilizing all four of the CAGE questions with one simulated and one real patient. These skills can be assessed by direct or videotaped observation by an instructor.
Demonstrate effective use of rapport-building techniques, including empathic listening, facilitation, clarification, validation, reflection, and appropriate nonverbal cues. These skills can be assessed by direct or videotaped observation by an instructor.
Assess patient’s expectations, values and goals. These skills can be assessed by direct or videotaped observation by an instructor.
Use proper hand-washing technique prior to starting the physical exam. These skills can be assessed by direct or videotaped observation by an instructor.
Perform a basic psychiatric interview, including risk of suicide and potential of harm to self or others. These skills can be assessed by direct or videotaped observation by an instructor.
Some objectives involve more than one domain:
The learner will be able to: orally present (skill) a new patient’s case in a logical manner, chronologically developing the present illness, summarizing the pertinent positive and negative findings as well as the differential diagnosis and plans for further testing and treatment (cognitive).
We end with the Magic Triangle:
If we begin with the end in mind, the objectives inform the learning activities and evaluation confirms that the expected learning did indeed occur.
Instructional objectives should be SMART:
Specific – Use the ABCDs to create a clear and concise objective.
Measurable – Write the objective so that anyone can observe the learner perform desired action and objectively assess the performance.
Achievable – Make sure the learner can do what is required. Don’t, for example, ask the learner to perform complex actions if they are a beginner in an area.
Relevant – Demonstrate value to the learner. Don’t teach material that won’t be used or on which you will not assess.
Timely and Time Bound – Ensure the performance will be used soon, not a year from now. Also, include any necessary time constraints, such as completing a task in “10 minutes or less.”
SOURCE: Official Web Site of the University of California, San Diego. Copyright © 2015
Types of Learning Outcomes
Most of us recognize that there are many different types of objectives. Some are easy, only requiring the simple recall of a definition. Others are more complex, requiring problem solving or evaluation. One popular categorization scheme for types of objectives is Benjamin Bloom’s (1956) Taxonomy of Objectives for the Cognitive Domain, which includes the following levels:
• Knowledge – Primarily concerned with students’ ability to memorize or recall certain specific facts.
• Comprehension – Usually involves the ability to interpret, paraphrase, and extrapolate, thus demonstrating students’ basic understanding of ideas that they did not originate.
• Application – Includes activities in which the student applies concepts and principles to new and/or practical situations.
• Analysis – Concerned with breaking down a piece of information into its constituent parts, differentiating and denoting.
• Synthesis – Involves the blending of elements and parts to form a whole. Students should be able to create a structural pattern that was not previously present.
• Evaluation – At this highest level, students might judge the value of a work, the logical consistency of written data, or the adequacy of someone else’s conclusions.
The Components of Effective Instruction
Teaching is the systematic presentation of content assumed necessary for mastery within a general area of knowledge. Instruction is a general term that means providing knowledge in a systematic way.
The characteristics of effective instruction can be represented in four groups:
1. Planning instruction
2. Managing instruction
3. Delivering instruction
4. Evaluating instruction
All teachers plan, manage, deliver, and evaluate their instruction, whether they are working with students who are gifted, students with disabilities, or students without special needs.
If all students in a class were at the same instructional level and if the goals and objectives of schooling were clearly prescribed and the same for all students, then instruction would consist of doing the same things with all students, in the right order, at the right time. But all students are not alike, and the goals and objectives of instruction are not the same for all students. This is why planning is such an important part of instruction.
Planning means making decisions—about what information to present, how to present the information, and how to communicate realistic expectations to students.
Planning instruction, then, involves three steps:
1. Deciding what to teach
2. Deciding how to teach
3. Communicating realistic expectations Each of these three steps for planning instruction includes specific tasks.
Planning Deciding What to Teach
1. Assessing students’ skills
2. Analyzing the instructional task
3. Establishing a logical instructional sequence
4. Considering contextual variables
5. Analyzing instructional groupings
6. Identifying gaps between actual and expected performance
Deciding How to Teach
1. Setting instructional goals
2. Selecting instructional methods and materials
3. Pacing instruction appropriately
4. Monitoring performance and replanning instruction
Communicating Realistic Expectations
1. Actively involving students in learning
2. Explicitly stating expectations
3. Maintaining high standards
Deciding What to Teach
Deciding what to teach enables teachers to match instruction to each student’s skill level. The process of deciding what to teach includes:
• Assessing students’ skills
• Analyzing the instructional task
• Establishing a logical instructional sequence
• Considering contextual variables
• Analyzing instructional groupings
• Identifying gaps between actual and expected performance
Deciding How to Teach
It is difficult to know ahead of time how best to teach.Teaching is an experimental process:
Effective teachers try approaches and materials until they find the combination that works best in moving students toward instructional objectives. You may have thought, or even been told in courses, that the way to decide how to teach a student is to give a battery of tests, identify the student’s strengths and weaknesses, then remediate weaknesses or build up strengths. But this is not enough. Yes, it is important to take into account a student’s level of skill and to identify learning and behavioral strengths and weaknesses, but simply knowing a student’s score on a test cannot help you decide how to teach the student. In making decisions about how to teach, teachers must make an educated guess about the kinds of approaches that will work, then try those approaches and monitor the results. That educated guess is based on experience, either with a particular student or with others like that student. After that, the only way to decide how to teach is to teach, then to gather data to determine its effectiveness. This does not mean that you should teach blindly. Experience provides a basic understanding of what works and what does not work. Also, check the literature for guidelines of effective instruction like these (Wittrock, 1986):
• Begin with an overview or use advance organizers and lists of objectives to set the stage for a presentation.
• Signal transitions between parts of a lesson and review or summarize subparts as the lesson proceeds.
• Ask questions of varying levels of difficulty throughout a presentation.
• Control the pace and continuity of lessons by regulating the time allowed for students to ask or answer questions. In addition to making an educated guess and consulting the literature, four activities come into play when you are deciding how to teach:
• Setting instructional goals
• Selecting instructional methods and materials
• Pacing instruction appropriately • Monitoring performance and replanning instruction
• Monitoring performance and replanning instruction
Setting Instructional Goals
The process of deciding how to teach begins with setting instructional goals for individual students, then establishing an instructional sequence. Most complex skills consist of combinations of simpler or lower-level skills. These lower-level skills must be taught in a logical sequence in order for students to acquire the complex skill.
Communicating Realistic Expectations
The third component of instructional planning is setting realistic expectations for students and communicating those expectations to them. When teachers do not expect much from their students, they are shortchanging them. If they have the skills to do so, students will, over time, learn to perform at the level of expectation that teachers hold for them (Good & Brophy, 1984). When those expectations are realistically high, students succeed; when they are unrealistically low, students fail.