How Can Assessment BeImproved?

Digital Performance Assessment May Be the Answer

By Paul Horwitz

Assume that you are selecting items for an assessment leading to certification of electronic technicians. Which of these would you choose?

• Define the term "digital multimeter" and give two examples of its use.
• An RC circuit consists of a 100-ohm resistor in series with a 10-microfarad capacitor. What is its time constant?
• Using a function generator and an oscilloscope, find out what is wrong with this power supply and fix it.

It's a trick question, of course. The first of these assessment items stresses irrelevant language skills and the second involves plugging numbers into a memorized equation. The third clearly comes closest to simulating the conditions a technician might actually encounter on the job. Nevertheless, the questions on certification exams for electronic technicians are far more likely to resemble the first two items than the third. The reason is simple: "performance assessments" like item three are much more costly to administer and score.

Or are they?

Computers can simulate any electronic circuit or measuring device; moreover, they can monitor and report on the actions of their users. All that's missing, then, is the ability to analyze those actions and make valid inferences from them concerning a student's knowledge, understanding, and skill, and her ability to combine those assets to solve realistic problems.

For several years, the Concord Consortium has been experimenting with computer-based models of realworld phenomena. The computer sets up a challenging problem that involves manipulation of a model to achieve a specific goal. It then monitors what the student does, possibly offering hints or asking questions at critical junctures. Most important, the computer logs the student's actions— answers to questions as well as experimentation with the model. By aggregating such data from thousands of students, we have been able to detect statistically significant patterns in their performance that correlate to their learning gains as measured by other means (see "Interactive Models," page 12). Thus, we can use performance data to make valid inferences concerning students' content knowledge as well as their ability to explore and reason with models.

To date, our work with these models has involved science students, not budding technicians, but that's about to change. With support from the National Science Foundation's Advanced Technological Education Program, we are adapting our technology to assess students as they troubleshoot computer models of faulty electronic circuits, using models of standard measuring equipment. This will enable us to pose challenges similar to the one described above, and to monitor, for instance, whether the student knows where to place the probes of the oscilloscope and how to set its time scale.

We have developed the technology necessary to give teachers those tools, but so far we have concentrated our data reporting tools on the needs of researchers, not teachers. The challenge now is to adapt our technology to create formative assessments that can identify the struggling students and describe their difficulties in enough detail so the teacher can help them, long before they are confronted by that high-stakes, multiple-choice certification exam.

Paul Horwitz (paul@concord.org) is Director of the Computer-Assisted Performance Assessment Project.