Can you trust Gartner's 'Magic Quadrant' or other analysts' reports?

By Bill Snyder

Created 2009-10-22 03:00AM

Talk about David versus Goliath. A tiny Silicon Valley software vendor is taking on mighty Gartner, one of the technology industry's largest and most influential market research and consulting companies. The battle is playing out in a San Jose federal courtroom, where ZL Technologies is asking for $132 million in damages (plus even more in a punitive judgment), saying the research outfit damaged its prospects by ranking it in the bottom segment of its closely watched Magic Quadrant report. The MQ divides technology providers into different classes, with the bottom segment essentially forming a "do not buy" recommendation.

Sure, vendors complain all the time that Gartner or IDC or Forrester and their cousins do a crummy job and unfairly diss their products. Sometimes they have a point; often they don't. There are, after all, a lot of crummy products out there. But the ZL Tech lawsuit raises issues that have bubbled under the surface of the technology industry for years. Most significant, I think, is the common practice of selling research and consulting services to the very companies whose products it evaluates. (Disclosure: IDG, which owns InfoWorld, also owns IDC, a major technology research company that competes with Gartner.)

[ IT faces another information disconnect: Its big vendors' agendas often don't match IT's [1], as InfoWorld's Eric Knorr explains. | Keep up on the day's tech news headlines with InfoWorld's Today's Headlines: Wrap Up newsletter [2]. ]

"It's naïve to think that it doesn't matter [to a research firm] if a vendor delivers a huge chunk of revenue," says ZL Tech CEO Kon Leong. What's more, he says, vendors with big marketing budgets are unfairly favored by Gartner, and that stifles innovation in the industry. "Small innovative guys are running uphill against the Gartner wind," he told me.

Is market research protected by the Constitution?
Citing the company's policy of not commenting on pending litigation, a Gartner spokesman would say only that it believes the suit has no merit. But the company makes a very detailed argument in motions filed with the court [3], most interestingly claiming protection under the First Amendment's guarantees of free speech: "Whether plaintiff's opinions about its product are correct, comprehensible, or sincere has no legal significance; what matters is that the complaint fails to state a claim because it attacks opinions expressed by Gartner."

Not surprisingly, Gartner also disputes the notion that its ratings are arbitrary and biased, saying that Magic Quadrant rankings are based on "more than 1,000 conversations. We learn from these conversations not only why a client is choosing or has chosen a specific vendor, but why it did not choose other vendors that were on its shortlist."

At least one contention raised is beyond argument: Ratings by research firms matter. In particular, the Magic Quadrant is taken very seriously by enterprise IT departments evaluating products and services, and Gartner says a vendor's ability to market is a significant factor weighted in the Magic Quadrant rankings.

Here's an excerpt from Gartner marketing materials included in ZL Tech's complaint: "A vendor that builds a strong product but is unable or unwilling to also invest in marketing and sales to capture a growing base of customers will find itself unable to invest in future development."

Is that fair to startups and innovative smaller players? I'm not sure. But buyers certainly need to believe that technology vendors have staying power. No IT department wants to be stuck with an orphan product.

ZL Tech argues that because its major rival in the mail archiving market, Symantec, has so much more marketing muscle, it is rated higher by Gartner, severely damaging its business. ZL Tech also contends that its product is far superior to Symantec's, and its revenues, currently about $10 million a year, would be much larger if Gartner's rating had not scared off potential customers.

How about full disclosure of conflicts of interest?
In our conversation, ZL Tech's Leong argued that Gartner needs to tell vendors exactly how the Magic Quadrant scores are compiled. "The tech industry would benefit if Gartner were required to disclose more data in its evaluation process and disclose component scores, so vendors know exactly where they are lacking and by how much and take corrective action."

He also stated that there should be a better appeal process for vendors. Gartner does have an in-house ombudsman to handle disagreements, but Leong argues that a Gartner employee has a built-in conflict of interest.

And most important, in my opinion, Leong argues for full disclosure of conflicts of interest. "Gartner generates its revenues from payments made by the same vendors whose products it evaluates. Similar to the new rules now being imposed on financial ratings agencies on Wall Street, Gartner should be required to disclose the revenues received from the vendors it ranks."

The media's hands may not be clean
To be fair, there's reason for readers to be skeptical of product reviews and the like in the technology press. As Galen Gruman, executive editor of InfoWorld and a longtime tech journalist and former consultant, reminded me: "Much of the enthusiast (consumer) and vertical trade press is colored by its relationships with the vendors, from the unintentional 'we're all in the same boat' soft bias to the intentional 'favor the advertisers' hard bias. And we all know that some publishers do stories based on who advertises. That hurts us all, but readers have also gotten good at detecting bias as a result."

Plus, as someone who has worked in the business and technology press for 17 years, I can tell you that big companies like Microsoft, Apple, and Intel, as well as small fry, will on occasion retaliate against publications and writers they deem too critical by cutting off access. That is meant to shape coverage by at least quieting troublesome voices -- and that's harder to detect as a reader. Given such a problem, it's easy to see why in the research context that a small company such as ZL is frustrated that it becomes invisible in customers' buying decisions when a Gartner essentially reduces its voice.

I suspect that ZL Tech will not win its lawsuit, though I could certainly be wrong about that. But much more significant than the fate of one company are the larger issues raised in the case. Considering the importance of research to the health of the IT market, the questions of conflict of interest and implicit bias deserve the light of day. Maybe this case will start that process.

I welcome your comments, tips, and suggestions. Reach me at bill.snyder@sbcglobal.net [4].

This story, "Can you trust Gartner's 'Magic Quadrant' or other analysts' reports? [5]," was originally published at InfoWorld.com [6]. Follow Bill Snyder's Tech's Bottom Line blog [7] at InfoWorld.com.