Good Reviews Go Beyond Lab Tests
Computer magazines for years have published product reviews to help readers select hardware and software. Typically, reviews are assigned to independent writers who report their impressions of a single product based on their own criteria.
There isn’t necessarily anything wrong with such reviews--that’s how I work. But some buyers want more hard facts and, perhaps, less opinion. Enter the computer testing laboratory.
Three years ago, Ziff-Davis, publisher of PC magazine and other computer publications, set up its own testing center called PC Labs. PC World, a rival monthly magazine from International Data Group, uses an independent lab in Philadelphia. And InfoWorld, a weekly newspaper also owned by IDG, earlier this year opened the InfoWorld Test Center in Menlo Park, Calif. Other publications are starting to use labs, too.
Although the labs differ in their techniques and philosophies, all perform what is known as benchmark testing. They measure, for example, how long it takes for a word processing program to jump from one page to another, for a spreadsheet program to recalculate a complicated formula or for a computer system to retrieve data from a hard disk.
The labs also try to get a general feel for the quality of a product. Is it easy to set up and use? Will it break easily? Are machines advertised as IBM compatible really able to run a wide range of software written for IBM machines?
When the testing is finished, the labs turn out the data for their publications. But a good review doesn’t end there. The information is turned over to a seasoned reviewer who interprets the data, adds subjective opinion and writes a review.
The numbers provided by the labs tell only part of the story. Slight differences in speed, especially on machines or programs that already are very fast, often are imperceptible to the user. Nevertheless, a battery of tests can help isolate a product’s strengths or weaknesses and provide useful information to prospective buyers.
PC Labs occupies several offices on the fourth floor of Ziff-Davis headquarters on New York’s posh Park Avenue. It’s an expensive and unlikely setting for a cluttered facility with several rows of workbenches and about 40 testing stations.
The lab represents a major commitment for PC magazine, a 550,000-circulation, twice-a-month magazine that covers IBM and compatible computers. It has a director, a manager and four project leaders, along with two technical specialists and three people who keep track of the machines and software packages that arrive daily.
It can cost as much as $100,000 to test one category of products, according to lab director John Dickinson. The lab tests and compares products based on requests from PC magazine’s editorial staffers.
Some of the work is done by free-lance writers who travel to New York for their projects. To avoid individual biases, each writer is given an extensive ‘script” listing specific tests and procedures. The script is developed by a product area expert and approved by the lab’s staff.
For one issue the lab workers evaluated 120 printers. Another project took a close look at modems, using a $30,000 modem-testing machine.
The lab generates lots of data, which it summarizes for the magazine’s readers. The Feb. 16 issue of PC magazine, for example, published a four-page chart, showing features and prices of 21 low-cost computers compatible with the IBM AT. The lab, unlike computer manufacturers, uses consistent standards for reporting such specifications as hard-disk size, available memory, number of expansion slots and other important details.
Although PC Labs relies heavily on test results, its mission goes beyond number crunching. Dickinson said: “We encourage our reviewers to go beyond scripts and do real work--their own work.”
The lab distributes its testing disk so that you can check out your own computer, hard disk and printer, or take it with you when shopping for new IBM compatible equipment. For a free copy, write to PC Labs, One Park Avenue, New York, N.Y. 10016.
InfoWorld is a weekly “controlled-circulation” newspaper that is sent, at no charge, to about 160,000 people who are responsible for making hardware and software purchasing decisions. Others can subscribe for $100 a year (51 issues). It covers personal computers used in business--primarily IBM and compatible machines and the Apple Macintosh. The paper opened its test center about four months ago.
Like PC labs, InfoWorld’s center has about 40 test computers. Lab director Jeff Angus sees his mission as being different from his counterparts at PC Labs. “We do the pure numeric tests, but not to a high degree,” he said. “We feel that most of those figures aren’t relevant to what people do.”
InfoWorld gathers the numbers, Angus said, because “they are easy to accumulate,” but editorial judgments are based on “a wide range of activities” by the testers and the magazine’s reviewers and editors. InfoWorld, according to Angus, generally looks at a smaller number of products in each category, focusing on ones the staff believes are most likely to interest readers.
The test center’s motto, posted throughout the room, quotes philosopher William James, who said: “Wisdom is knowing what to ignore.” InfoWorld rates products on a scale of 0 to 10 based on a weighted average of its battery of tests. Before he begins a test, Angus polls about 1,000 InfoWorld readers to help determine the criteria to use when evaluating the product. The survey also determines how much weight to give to each area in arriving at the overall rating.
When evaluating database software, for example, readers care more about ease of use than small differences in operating speed. As a result, an easy-to-use program is more likely to get a high score than one that is fast but complicated.
InfoWorld publishes its rating criteria as well as the weight assigned to each. That makes it possible for readers to recalculate the results based on their own priorities. The newest version of InfoWorld’s hardware testing disk will be available soon. For a free copy, write to InfoWorld Hardware Benchmarks, 1060 Marsh Road, Menlo Park, Calif. 94025.
San Francisco-based PC World magazine, which claims about 500,000 subscribers, uses National Software Testing Laboratories, an independent lab that publishes its own ratings newsletter. PC World Editor Richard Landry has three main criteria when evaluating any product: performance, features and usability. The latter includes ease of use.
“So what if a program has 40 features, compared to another program’s 38, if those 40 features make the program more difficult to use?” Landry said.
Landry cautions that number ratings can be a “pseudoscientific” attempt to translate subjective judgments into objective measures. His magazine rates products as excellent, good, fair or poor.
Ultimately, the products you buy should depend on your own needs and personal tastes. Thanks to these and other magazines, you don’t have to be a big corporate buyer to get good research, but you still have to be a careful buyer. Read the magazines, but try the machines before you buy and insist on a money-back satisfaction guarantee.
Computer File welcomes readers’ comments but regrets that the authors cannot respond individually to letters. Write to Lawrence J. Magid, 3235 Kifer Road, Suite 100, Santa Clara, Calif. 95051, or contact the L. Magid account on the MCI electronic mail system.
