The Mysterious Stanford 9

One day last summer, I opened my mailbox to find a thin envelope containing the results of the Big Test. The moment felt, as they say, like deja vu all over again. Many summers before, in my last year of high school, a similar envelope had arrived at our house in Tennessee. Feeling slightly lightheaded, I had carried that one unopened to the backyard, where I sat down under a tree and ran my fingers along its edges. Inside were my SAT scores, numbers that would dictate a great deal about my future.

I did not know then that the SAT was, in the lexicon of exam makers, a “high stakes” exam, but I would have understood the concept. At that moment, the stakes could not have been higher. The test would determine whether I went to a faraway college or remained in a hometown I desperately wanted to escape. As it turned out, I did get away. And though I’m no longer certain about the wisdom of the flight, I was forever impressed by the power of those numbers to propel my life in the desired direction. It seemed like magic, especially because I was convinced my success sprang not from high-octane intelligence but from a sly talent for decoding the test’s mind set and supplying the expected answers.

This new envelope brought results of a different order. The Big Test was the Stanford 9 rather than the SAT. And the scores were not mine but my son’s.

It will come as no surprise to most parents in California that I felt equally lightheaded as I lifted this envelope from the mailbox. If anything, the Stanford 9 looms larger in the lives of California schoolchildren today than the SAT did in mine. The Stanford 9, being administered to grades 2 through 11 now through late May, has become the ubiquitous Big Brother test for every public school kid in the state, assigning scores that will be tattooed on the child’s permanent record.

And those scores will be employed in unprecedented ways. Unlike the achievement tests of yore, administered largely to assess the performances of school districts as a whole, the Stanford scores will be used to rank students individually. The Los Angeles Unified School District is sending out a cheery letter this year informing parents that their children could be refused promotion if they don’t meet certain standards.

And that’s not all.

As Stanford 9 scores begin to compete with classroom grades as a measure of academic performance, they will play a role in all the crucial passages of school life from the assignment of teachers to opportunities for gifted programs and magnet schools.

In other words, the Stanford 9 has blossomed into a very high-stakes test indeed. Even teachers and principals can feel its power. Teachers whose students consistently produce low scores may well find their careers derailed. In January, last year’s scores were converted by the state into something called the “Academic Performance Index.” Let the cumulatives fall low enough, and your school could be taken over by the state.

We have asked for this, of course, in the name of school reform. Education has a habit of falling prey to each generation’s notions about learning, and our generation gravitates toward control. We want to know exactly how our children perform relative to those in Illinois or Maryland, we want to identify deficiencies quickly and we want the power to correct them. Thus we test and test. That may be OK, at least in principle. But here in California, an odd mystery sits at the core of the testing mania. That mystery involves the Stanford 9 itself. Virtually no parent in California has ever seen, or ever will see, the test that now passes judgment on our children. The Stanford 9 is kept as secret as the formula for an atom bomb.

I’m not talking about the necessary security that surrounds the taking of the test, but a much broader, more profound secrecy that envelops Stanford 9 at all times. Even after the test is administered, its contents are hidden from all but a few education bureaucrats. The used copies, millions of them, are locked up and eventually are returned to the publisher, Harcourt Educational Measurement in San Antonio, Texas. The secrecy is so severe that we are even prevented from seeing practice versions of the test produced by Harcourt. The very nature of this test is hidden from us.

Why make a big deal about seeing the Stanford 9? There are several reasons, all flowing from the paramount role that the test now plays in California’s education.

First, the Stanford 9 already exerts a powerful influence over the de facto curriculum in the state’s classrooms, an influence that will only grow. In a recent chat room discussion on the Internet, one teacher asked: “Do we really want to define elementary mathematics by how it prepares students for (the Stanford 9)?” That is what we see happening right under our noses. The general public pronounces the schools with high scores “good” and schools with low scores “bad.” But where, exactly, is the Stanford 9 leading teachers? Without the right to scrutinize the Stanford 9, we can’t know what it emphasizes and what it does not. We have ceded its power to a group of test makers in San Antonio.

Then there’s the issue of test preparation. My sister, a sixth-grade teacher in Virginia, had the chance last year to review a number of prospective standardized tests being considered there. She was amazed at the differences. Some took a hard, tough approach. Others were touchy-feely. Each had its own set of internal rules. To come to grips with any of the tests, my sister says, a student must get comfortable with its idiosyncrasies. Otherwise, it’s like trying to drive a car for the first time. Everything is awkward and much energy gets absorbed by the strangeness.

That’s where California students find themselves. They are denied the opportunity to get comfortable with the Stanford 9 because the publisher refuses to sell practice versions of the tests here. We’ll return to this issue later, but I would argue that the lack of practice tests inevitably produces higher levels of stress among students and lower scores. This secrecy, incidentally, is not practiced in some other states. In Texas, for example, each year’s achievement tests are posted on the state’s education Web site a few weeks after they are administered. Every question is displayed, so anyone can look at the test, see how it works, demystify it. Parents can also see whether the test questions match their school’s curriculum.

Here in California, our only exposure to the Stanford 9 comes each spring, when parents receive something known as the test “preview.” This year the preview I received consisted of four sample questions. That’s right, four questions. They were supposed to help me acclimate my son to an exam that lasts a week, covers a dozen “content clusters” and includes 350 math, reading and science questions.

After the preview, there is nothing. Say you are a parent and you receive scores for your child that put him in the 76th percentile in reading and the 38th percentile in math. You want to know what went wrong in math. Maybe he went “off bubble” on his answer sheets, mismatching his answers with the questions; maybe the exam demanded a level of mathematics he had not been taught. So you ask for a copy of the test. You won’t get it. If teachers ask, they won’t get it. They see it only when giving it to their students and are denied the chance to scrutinize it at leisure and decode its contents. Many principals never see it. And in Los Angeles, even district staff members responsible for answering parents’ questions about the Stanford 9 are refused the chance to see it.

The big question is why. If students would benefit from a basic familiarity with the test that will rule their academic careers, why do we block all opportunities to build that familiarity?

Part of the answer is simple. It’s money. In California, we have decided to trade off the benefits of familiarity for cost savings. The Stanford 9 is kept secret even after it’s administered because we use the same test the next year. And the year after that. “If we released the test each year, we would need a new one to replace it,” says Gerry Shelton, a testing expert at the state’s Department of Education. “And that costs money.”

Shelton estimates that creating new questions for all versions of the test from the second grade through the junior year of high school would cost somewhere between $10 million and $12 million a year, about one-fourth of the $42 million that California spends annually on its standardized test program. It’s an amount that Texas and a few other states obviously have chosen to spend.

“The Texas model is attractive because parents and students can learn a lot from reviewing the questions and the answers,” Shelton acknowledges. “The same model has been used in Europe for a long time.” In France, for example, the country almost closes down on the afternoon after the national test because everyone goes to the bistro and reads the test questions that are published in the newspaper. “Still, it’s expensive,” Shelton says, “and we chose to go the other way.”

Actually, you could argue that California’s expense for re-creating its tests would be lower than that of other states on a per-student basis because we have such a huge student population. If revealing the test resulted in a greater comfort level, higher scores and better understanding of the test’s demands, you also could argue that we would be getting a bargain. An even greater bargain would be the availability of Stanford 9 practice tests, which could provide many of the same benefits as a released test, minus the opportunity for students to compare the actual questions with their answers.

As most parents know, the sale of practice tests has grown into a booming industry as the test craze has swept the nation. Walk into any so-called “teacher store” in Los Angeles and you will find whole shelves filled with practice versions of many achievement exams, including the SAT. But you won’t find a true practice test for the Stanford 9. By “true practice test,” I mean a version produced by Harcourt with the identical style and approach as the test itself. Using another publisher’s product to practice for an achievement test is akin to practicing basketball where the size of the court and the height of the basket have been changed. It’s not the same.

Yet that’s where Harcourt has left us. The company does produce practice tests for the Stanford 9 but refuses to sell them to the public. It offers them only to school districts, for use by teachers in the week of the exam, and demands the return of any unused copies. And here the explanations get murky.

When I approached Harcourt, its representative first contended that California law prohibits the sale of Stanford 9 practice tests to the public. “The legislation that established the [testing] program includes a specific prohibition against practice tests,” said a spokesman, Ed Slawski. “We didn’t write those rules, and we wouldn’t have written them had it been up to us. We are simply obeying regulations laid down by California.”

But the company backed away from this position when the pertinent state code appeared to do just the opposite of what the company claimed. The code, in fact, specifically allows the sale of practice tests to students and parents, and exempts the tests from other security measures. So Slawski offered a second explanation: “As a company, we don’t believe in test prep. We don’t believe in selling products that are only designed to improve performance on tests.”

That sounds closer to the truth. For many years, test publishers, subtly or not, have sought to discourage preparation for their exams. In some cases, their arguments slyly suggest that preparation amounts to a kind of cheating because the preparer is “training for the test” rather than concentrating on schoolwork, and thereby acquiring an unfair advantage. In other cases, they seem to argue the opposite, that preparation is futile and a waste of time.

Both these arguments are bunk. SAT preparation, now a nationwide mini-industry, has demonstrated that most students benefit from the chance to decode a particular test and develop strategies for taking it. The College Board, which offers a slew of practice tests for the SAT, advertises the effectiveness of test study. As for the ethical argument, preparing for standardized tests is no less ethical than preparing for history tests. As tests such as the Stanford 9 come to control curriculum, you could say that studying for the test is the same as studying schoolwork. They cover the same material.

Harcourt continues to make these arguments, I’m convinced, to serve its own ends. Ever since the standardized test industry was founded in the early part of the 20th century, the programs have benefited from the air of remote omniscience that surrounds them. We know best, the publishers imply, and we don’t need to tell you diddly-squat.

If the test makers say your child is lousy at reading comprehension, then he must be lousy. You don’t know enough to challenge them. And the test makers, like the witch doctors of old, aren’t about to tell you their secrets. Even mumbo jumbo--the witch doctor’s favorite trick--is used occasionally to enhance the air of mystery. Take the Stanford 9 score sheet sent to parents sometime in mid-summer. It includes a list of “Content Clusters,” so called because they break down the larger scores into smaller categories, each with a title or short description. Not a bad idea, if the cluster titles actually explained anything.

But clear explanations would remove some of the mystery, no? So Harcourt has created titles that seem to have been borrowed from a software writer’s conference. One cluster in the reading section is called “Process Strategies.” Another is called “Recreational Comprehension.” My personal favorite, in the mathematics procedures section, is titled “Comput./Symbolic Notation.” These clusters came from a score sheet for a second-grader. Out of curiosity, I called the Harcourt help line to see if anyone could explain the titles. A woman named Nancy answered. I identified myself as a parent and went straight for the prize item. What, I asked her, is “Comput./Symbolic Notation?” Nancy paused. She couldn’t help me, she said, because Harcourt’s help line was designed to help professionals, not parents. Helping parents was the job of the school district. I pointed out that Harcourt, not the school district, created the titles, and Harcourt, not the school district, had sent the score sheet. Even so, Nancy couldn’t help.

So I called L.A. Unified and talked to Jim. What, I asked Jim, is “Comput./Symbolic Notation?” Jim didn’t have a clue. “Perhaps you should call Harcourt,” he said. No one could answer, of course, because the phrase is gibberish. There’s more. Even when the titles are not gibberish, it hardly matters. One cluster within the “Reading Vocabulary” subtest is called “Context.” Imagine yourself the parent of a second-grader who answered three of six questions correctly in the “Context” cluster. You try to picture the questions that went into “Context” and stumped your child half the time. You can’t picture them, because you have no idea what Harcourt is talking about, and Harcourt won’t tell you.

Long ago, when I took the SAT, it too operated with arrogance and mystery. Because of that, and because of the stakes involved, taking the SAT was a cold-sweat experience. I remember the moment when the test began, the terrible quiet in the cafetorium where we sat, the proctors stationed along the sides and the test administrator looking at her watch and saying, “You may begin.” I looked down at a test form completely new to me. I read instructions I had never read before. I studied questions and tried to figure out the level of subtlety or cleverness the test makers wanted. I was forced to do that because the College Board sold no preparation materials whatsoever in those days. The board claimed that preparation was useless because the test was beyond the reach of training or practice. The test supposedly extracted the mental talents from your brain in spite of all attempts to trick or distract it, measured those talents and recorded your score. Resistance was futile.

For the SAT, those days are long gone. The College Board now offers several volumes of practice SATs, a videotape featuring SAT teachers demonstrating how to improve your scores and a computer software package that drills students in the nuances of the test. In other words, the College Board admits through its actions that test-taking is a skill that can be learned, at least in part. In doing so, it has reduced the hellishness of the experience of taking the SAT and has raised the scores of thousands, if not millions, of students.

But California and Harcourt Educational Measurement have not yet gotten the message. Our test system treats students, some as young as 8 years of age, the same way the College Board treated high school juniors 30 years ago. Perhaps it’s time to come clean. If California education officials and Harcourt would show us their old tests and make available practice tests, we would show them happier kids and better-performing kids. Call it a bargain for high achievement.