New Web Rating Service Shows Promise of Avoiding Usual Pitfalls
The old Soviet arms-control motto, “doverai, no proverai”--trust, but verify--could aptly be applied to e-commerce. Every online merchant claims to warrant your trust, but the means for verifying those claims have been primitive, leading millions of consumers and businesses to shun online buying.
Open Ratings, a start-up co-founded by Pattie Maes, one of the Web’s most influential thinkers and a professor at MIT’s Media Lab, suggests a new solution for the verification problem.
To understand the significance of the Cambridge, Mass.-based company’s system for rating Web businesses, to be launched this month, consider the methods of other Web rating sites. Rateitall.com and Deja.com casually poll volunteer users. Bizrate.com takes the process to a higher level by asking consumers of e-tail vendors to complete a questionnaire after completing a transaction, then aggregates the responses to create ratings.
But data from these services suffer from a fundamental problem: A tally of self-selected respondents is not scientifically reliable. Pristine research based on random sampling costs time and money, so many Web-based rating sites substitute large sample sizes for proper sampling.
Sample size, however, has little to do with predictive accuracy, according to Bob Tortora, a polling expert with Gallup Organization, citing a classic case. In 1936, a magazine called Literary Digest mailed 10 million ballots to Americans and used about 2 million self-selected responses to predict the outcome of the presidential race.
The magazine used auto registration lists and phone books to find addresses--at the time, a relatively wealthy and heavily Republican demographic. The Digest predicted that Republican Alf Landon would trounce Franklin Roosevelt.
Roosevelt won by a landslide, as predicted by professional pollsters based on the views of a few thousand randomly selected voters.
“Without random sampling, you might make some bad business decisions, and worse, you can’t even quantify how bad the decision might be,” Tortora said.
This is not to say that ratings based on poor methodology are useless. For a book, a computer, or even a car, thumbs up from a single reliable source might justify the purchase.
But if you’re a buyer for a chemical company and need $2 million worth of liquid oxygen by tomorrow, your job might depend on choosing the right supplier. Would you choose based on anecdotal reports?
Open Ratings, a service designed for business-to-business e-commerce but also applicable to consumer transactions, offers a solution.
The first part of the Open Ratings process works similarly to Bizrate.com’s: Users fill out questionnaires after a transaction, rating conventional standards such as customer service and on-time delivery, as well as custom categories for each type of business. Web artists might be rated on design skill, chemical suppliers on safety procedures.
Sellers also rate buyers, creating a record of credibility at both ends of the transaction--similar to the buyer-seller ratings used by Web auctioneer EBay.
Then Open Ratings adds objective data, such as credit histories and past performance on contracts. The company weights the factors of each transaction--including the historical credibility of each seller--according to what factors are most important in each industry and to each buyer.
Stan Smith, Open Ratings chief executive, says that his company’s technology corrects for “ballot-stuffing” techniques that unscrupulous companies might try to boost their own ratings or sabotage competitors.
To be sure, the sample is still self-selected. But Smith argues that the Open Ratings method combines much more information than standard polling with sophisticated machine intelligence to more than make up for it.
He declines to reveal his company’s secret sauce--you’ll have to trust him that it works as brewed. But Open Ratings recently received a vote of confidence from Cambridge, Mass.-based Forrester Research, a leading Web research group, which has adopted the system for use in its own online ratings of Web sites.
Chris Dellarocas, professor of information technology at MIT’s Sloan School of Management, views the Open Ratings system as a big step forward in the usefulness of Web-based assessments, partly because the technology allows customized ratings of businesses based on each buyer’s needs, priorities and past preferences--all logged by the system.
For example, if a buyer designates product quality as the highest priority in several previous transactions, the system might weight quality over, say, speed of delivery, in rating suppliers for that buyer.
Open Ratings also makes ratings portable.
If your credit rating were owned and used by a single bank or business, you would obviously be crippled in financial dealings. This is akin to how some Web-rating schemes now operate; for example, EBay’s ratings pertain only to EBay transactions.
Just as a person’s credit rating is accessible to any bona fide creditor, Open Ratings creates a business-reputation rating for each company, available on any business-to-business exchange that subscribes to the service. For example, if Acme Inc. sells plastic widgets on WidgetWorld.com and sheet metal on MetalPlanet.com, its ratings as a supplier would be compiled from the experience of buyers at both sites--and provided to both sites.
For Open Ratings to work, of course, it will need cooperation from a large share of such exchanges--a tall order for a start-up.
And it remains to be seen how rapidly and effectively the untested system can adapt to rapid changes in the quality of a product or service--not unusual when a company falls on hard financial times or changes its target market.
Dellarocas also points out that if successful, Open Ratings will amass a gigantic database of buying behavior that will be used to provide intelligence to marketers and might run afoul of privacy-protection sensibilities or legal barriers.
Facing such pitfalls, the company could well fail. But its approach, if handled effectively, could begin to lift the fog of suspicion from the Internet marketplace. Given that state of e-commerce stocks, they could use the help.
*
Times staff writer Charles Piller can be reached at charles.piller@latimes.com.
