Here's the truth about assessment: There is no holy grail, no tool that does it all. What exists are trade-offs; every tool has an advantage and a disadvantage.

So explained assessment experts in the edLeader Panel "Building a Comprehensive Assessment System: Overcoming Key Challenges." Making trade-offs, explicitly and intentionally, the panelists emphasized, is essential to creating the ideal assessment system.

A Look at Assessment Items

The panelists described two items on opposite ends of the assessment spectrum to demonstrate how these "trade-offs" play out. 

Performance tasks: These assess depth, rigor, and application: Students must "know their stuff." The tasks reduce test-wiseness and guessing, which are typically associated with multiple-choice questions. The Orbit 1 Space Station Theater, from the New Standards Project, is an example.

However, performance tasks take a lot of time. They also build in invisible unreliability. If a student likes or knows the setting, they're a "lottery winner." If not, they may do poorly, not because they don't know the content, but because of the setting. To fully assess a student's ability, a teacher would need to develop six or seven performance tasks.

Multiple choice: This is the go-to assessment. What makes them good or not so good is the distractors, which are designed to see whether students really know the answer. Multiple-choice questions are well understood, highly reliable, and very transparent. With a fixed amount of time, a test consisting only of multiple-choice questions provides twice as much information as a test consisting only of innovative item types.

But they're not very good at depth. Using Webb's depth of knowledge levels, multiple-choice questions can consistently be written at levels one and two, and, with certain skills, at level three.

An ideal assessment pairs performance tasks with multiple-choice questions. Multiple-choice questions can check for students' background knowledge, and then, if they are ready, they can engage in a performance task.

How a Test Is Built

Designing effective assessments is a bit of a science, which two assessment theories address. 

Classical test theory: Assessments in this category call for identifying the skills to be tested, writing about five to seven items per skill, and scoring how many students get right. At the end of the items, there is one piece of information.

Classical test theory fits mastery measures, designed to show mastery of a very specific skill or a very small set of related skills.

Classical test theory is well known, highly reliable, and has tremendous face validity. But it has floor and ceiling effects. If a student gets all the items right, it's hard to see what they don't know. If they get all the items wrong, it's hard to know how far down they are. And if there are lots of skills to assess, the assessment gets very long.

Modern test theory, by contrast, emphasizes items: write a set of items in the same domain, study their difficulty, and create a difficulty scale. Administer the items through a computer-adaptive test: If a student gets an item right, they get a more advanced item. With four items instead of five, the teacher discovers that the student knows some elements and excels in others.

Modern test theory applies to general outcome measures, which provide broad assessments of many skills. The theory is efficient, highly reliable, and avoids floor and ceiling effects. But teachers can't see the items students missed, and developing it takes a lot of money.

Making Assessment Decisions

The challenge is creating the ideal assessment system and knowing which tool to use at any given time. Conducting a Plus/Minus/Interesting (PMI) analysis on every assessment being considered invites a close review of deficits and benefits:

• Plus: What is a plus about this assessment task?
• Minus: What is a minus about it?
• Interesting: What is something interesting about it?

The ultimate goal? Find the tool that delivers the best results for students, helping them grow as much as possible.

Learn more about this edWeb broadcast, Building a Comprehensive Assessment System: Overcoming Key Challenges, sponsored by Renaissance.

Join the Community

Assessment for Learning is a free professional learning community where educators can learn from top experts about effective assessment practices.

Article by Michele Israel, based on this edLeader Panel