What Do We Know About NAEP Performance Standards?

By Russ Allen
WEAC Research Consultant

On November 29, 2006, Education Week reported that the U.S. Department of Education will conduct a comparative study of what it means to score proficient or advanced on state tests compared with tests administered by the National Assessment of Educational Progress (NAEP).

This is an issue of interest because in nearly all states a greater percentage of students score proficient or advanced on state tests (using state standards) than score proficient or advanced on NAEP tests (using standards adopted by the National Assessment Governing Board). In Wisconsin, for example, approximately three-fourths of students score proficient or advanced each year on the Wisconsin Knowledge and Concepts Examinations (WKCE), compared to about 40% on NAEP tests.

The controversy of state versus national standards is a direct consequence of the "No Child Left Behind" law (NCLB). This law requires that all schools test students annually in mathematics and reading (also science in 2007) in grades 3-8 and that a specific percentage of students in each school, district, and “eligible” subgroup (see below) score proficient or advanced on state tests. The law also requires that each state develop tests that are aligned with “challenging” state standards and administer them annually to students in selected grade levels. Finally, NCLB mandates that a representative sample of students from each state participate in testing by NAEP.

Under NCLB, all schools and school districts must meet a standard of adequate yearly progress (AYP). The most basic requirement of AYP is that 95% of students in each school (and the district as a whole) must be tested annually in each content area, and a specified percentage of students must score at the proficient or advanced levels. “Subgroups” of students also must meet the same requirements that have been established for schools and school districts – if there are sufficient numbers of students within the subgroup. For students with disabilities, the minimum number is 50 students. For the following subgroups, at least 40 is required: American Indian/Alaska Native; Asian/Pacific Islander; Black, not of Hispanic Origin; Hispanic; White, not of Hispanic Origin; English Language Learners; and Economically Disadvantaged.

In 2005-06, the targets for districts, schools, and eligible subgroups were 67.5% of students scoring proficient or advanced in reading and 47.5% proficient or advanced in mathematics. Over time, the percent of students who must score proficient or advanced on state tests increases to 100% by 2013-14.

Critics of public education use this type of information to argue that Wisconsin’s standards (and those of most other states) were deliberately set low in order to make the states’ schools appear better than they are. Few ask whether NAEP standards might be the problem. After all, NAEP’s standards are not sacred. As with all standard-setting procedures, human judgment is at the heart of the process. As a result, it should not be surprising to learn that different standard-setting procedures yield different results.

Student motivation on NAEP tests

Some of the difference in test scores between the states and NAEP may be a function of student motivation. That is, research shows that students are less motivated to do as well on the NAEP tests than they are on state tests. This is especially true for older students ("Experimental Studies on Motivation"). A similar conclusion is reached by the Center for Public Education – pointing out that NAEP testing has no consequences for students while state assessments can have consequences, for example, by influencing decisions about grade promotion or high school graduation. The Center for Public Education concludes that research on motivation is by no means conclusive, but does say that the effect of student motivation (or more precisely, the lack of it) is important, especially among older students.

NAEP itself is aware of the problem of lack of motivation among many students. Reckase ("The Controversy over National Standards"), for example, calls NAEP a “drop-from-the-sky test,” meaning that students take the test without any prior warning or preparation. He also reminds us that NAEP tests are low stakes for students, meaning there are no rewards or penalties for poor performance. In 1996, for example, NAEP found that on extended response items 6% of 4th graders, 13% of 8th graders, and 35% of 12th graders left items blank or gave “off-task answers” to questions.

The NAEP Standards

For several years Gerald Bracey ("Where are the Standards?") has criticized NAEP’s standards by calling attention to the following evidence:

In the Third International Mathematics and Science Study, U.S. students ranked third among all countries participating, yet fewer than one-third of U.S. 4th graders were proficient or advanced on NAEP.
Likewise, on the 2002 reading assessment U.S. 3rd graders finished 2nd in the world, yet only 31% scored proficient or advanced on NAEP

Bracey maintains that we should be asking why is it that U.S. elementary level students do so well on international assessments, yet only one-third of them are judged proficient or advanced by NAEP.

Bracey is not the only critic. NAEP also has been severely criticized by the National Academy of Sciences, the U.S. Government Accountability Office, and other testing and measurement experts. In particular, the National Academy of Sciences (NAS) says NAEP’s standards are “fundamentally flawed. . . producing unreasonable results.” Further, NAS concludes that the judgment tasks are “ . . . difficult and confusing; raters’ judgments of different item types are internally inconsistent; appropriate validity evidence for the cut scores is lacking; and the process has produced unreasonable results.”

The most compelling evidence that something may be wrong with NAEP’s standards comes from the National Assessment Government Board itself. It states on its own Web site that its standards should be used on a “trial basis” with “caution” until the Commissioner of Education Statistics determines that the achievement levels are “reasonable, valid, and informative to the public” ("The Status of Achievement Levels").

The National Assessment Governing Board was directed by the "No Child Left Behind" act of 2001 to come up with an alternative. To date, this has not been done (confirmed in an e-mail sent on September 22, 2006, from Susan Loomis, representative of the National Assessment Governing Board, to Russ Allen, WEAC).

Watch for the results of the study by the U.S. Department of Education – it’s due in spring of 2007.

References