Do you believe in magic (quadrants)?

Written by on March 20, 2015 in Opinion with 2 Comments

Ethical Vs Profitable Matrix Ethics and Profits ConvergeFor some strange and inexplicable reason, our most popular story ever and one that continues to attract new readers every week, is a piece questioning Gartner’s Magic Quadrants written by Ed Finegold.

Maybe it’s because we had the audacity back in 2013 to question the worth, even credibility, of what many believe is gospel on where companies and their products stand in comparison to their peers based on a dots position in one of four squares?

But these are no ordinary squares, they are Gartner’s trademarked Magic Quadrants with twin axes consisting of ‘completeness of vision’ and ‘ability to execute.’ Whether these two criteria alone should give a potential purchaser the confidence to acquire a particular product, solution or service is questionable, yet many fall back on them in making ‘calculated’ decisions.

Not so long ago, IBM used to feature ads on television where a rather nervous looking executive was told by a superior that his decision to go with IBM was wise “because nobody ever got fired for buying IBM.”

Although Gartner has never showed the same gall as IBM in promoting its Magic Quadrants, there is a belief by many in the market place, especially those that don’t want to waste too much time investigating all potential vendors, that going with those positioned in the top right-hand quadrant is ‘a safe bet.’

Companies pay handsomely for this ‘canned’ information, after all, it saves them time and bother. And you can’t help but admire Gartner that succeeds at making money out of solid research and informed opinion, right?

But just because Gartner says so, should it be considered the gospel truth alluded to earlier? Even questioning Gartner is considered by many as a ‘career threatening’ activity, so powerful is the charm of the Magic Quadrants, yet some ‘brave hearts’ have tried.

Back in 2004, John Cox writing for Network World, said that “plenty of vendors, gored by the quadrant’s twin axes of ‘completeness of vision’ and ‘ability to execute,’ insist it’s all just hocus-pocus.”

Gartner itself says, a “Magic Quadrant is a culmination of research in a specific market, giving you a wide-angle view of the relative positions of the market’s competitors. Contextualization enables you to view a Gartner Magic Quadrant through a specific lens, providing high-impact additional perspectives by key industry, region and company size. This enhancement provides analyst commentary on the market and notable vendors from key client contexts. The interactive Magic Quadrant experience provides you with a customized view of the market based on the criteria most important to you.”

That’s all well and good but raises the question on how Gartner goes about deciding who even qualifies to be in an MQ. Disruptive new players making market inroads might not fit into the Gartner perception of vision and execution, and in today’s market watching out for the disruptors is probably just as important as sticking with the ‘old guards’ most likely to be featured in the top Gartner quadrants.

Being relegated to the ‘Challengers, Niche Players and Visionaries’ quadrants is tantamount to having a flashing neon sign over your front door blazing ‘BEWARE’ to all that pass. That’s despite the disclaimer on the Gartner site that states: “Keep in mind that focusing on the leaders’ quadrant isn’t always the best course of action. There are good reasons to consider market challengers. And a niche player may support your needs better than a market leader. It all depends on how the provider aligns with your business goals.”

If that is indeed the case, why even buy the QM in the first place? Unless buyers of the Gartner reports know exactly what criteria was used to determine positioning then it probably discounts their worth, surely?

In Cox’s article Gartner executives were quick to defend the MQ, and cover off the methodology and reasoning deployed including a final step before publication to circulate the conclusions to the vendors to catch any inaccuracies.

One irate vendor that approached us had sent a formal letter expressing concerns about Gartner’s lack of thorough research and consequent lack of understanding of the market in the most recent for Integrated Revenue and Customer Management for CSPs MQ. The vendor had done its own detailed analysis of the current MQ and found blatant flaws in comparing results against the previous MQ.

Another interesting observation made was that Gartner criticizes a lot of vendors for having a relatively weak SI partner ecosystem. “Gartner has missed a key trend – big software vendors no longer require big SI partners as a channel. Operators increasingly want the big software vendor to prime the project, and employ SI partners, as required, to get the project done. If you look closely, the companies Gartner criticizes for lacking SI partners are much more successful than the ones which rely on partners as a channel. Just another indication that Gartner doesn’t know very much about how our market works.”

Our reaction to the anomalies our irate vendor highlighted was to do our own research. We surveyed the purchasing decision makers of fifteen leading telcos in Europe and asked them to score the influence that Gartner and the Magic Quadrant had on them. We decided not to score Gartner against completeness of vision or ability to execute.

The results, it turned out, were not magical and certainly not if you were Gartner. In the survey, a score of ‘one’ meant that Gartner’s information had absolutely no influence on them and ‘five’ meant it was ‘gospel.’ Only one decision maker gave them a ‘four’ and the rest gave them a ‘three’. This means that when it comes to it the smoke and mirrors are replaced by a sensible consensus that Gartner’s information is just part of the information gathering process.

The bad news for vendors is that there is the fear factor and the perception is that if you – as a vendor – are not at least somewhere in the quadrant you will not be invited to tender when RFP time comes around. And that is a lot, perhaps too much, influence in a market.

One other vendor that had reduced its spend with Gartner found it had slipped in the MQ rankings and was convinced the two were related. Alex Leslie reported another similar claim back in August. Heaven forbid, that surely could not be a factor, could it? But that’s the sort of gossip that can damage the MQ standings and is part and parcel of owning something that is so successful.

Until something else comes along that gives a decent snapshot of market players, easy enough for decision makers to digest, or purchasers put more effort into vendor selection themselves, then the Gartner Magic Quadrants will continue to thrive. Hopefully, the same can be said for those vendors that don’t make into one of those hallowed squares.

Tags: , ,

About the Author

About the Author: Tony is a freelance writer, regular speaker, MC and chairman for the telecoms and digital services industries worldwide. He has founded and managed software and services companies, acts a market strategist and is now Editor of DisruptiveViews. In June 2011, Tony was recognized as one of the 25 most influential people in telecom software worldwide. .


If you enjoyed this article, subscribe now to receive more just like it.

Subscribe via RSS Feed

2 Reader Comments

Trackback URL Comments RSS Feed

  1. Ed Finegold says:

    I’d be happy to write another critical analysis of any vendor ranking from a research house. All the original article did was measure the MQ by its own criteria with a dose of common sense. Having since seen the evaluation process up close, I can add to my original statements. It is not that the process isn’t thorough, it’s that its qualitative criteria are based on biased assumptions that necessarily skew the end result. In other words – it’s opinion driven, not fact based. The portions that are quantitative make sense, though Id be curious to go back to the 2012 MQ and see whether the quantitative results matched the stated criteria as I first argued – Can a vendor in a telco MQ receive “points” for customers outside the telco domain? The bottom line for any CSP is simple – do your own homework. You have plenty of experts in house who know 2 things 3rd party analysts can’t know as well – your environment and your culture. No vendor ranking system can account for those two critical factors.

  2. Andy Tiller says:

    AsiaInfo actually improved its position quite a bit in this year’s MQ, but the analysis was flawed in so many ways, containing numerous factual inaccuracies and false assumptions, and based on simple questionnaire responses rather than proper, rigorous analysis. I contacted Gartner’s ombudsman to suggest that the MQ process needs to be improved. For example, requiring the analysts to interview some of the vendor’s customers would be a good start (even if it’s rather inconvenient because the customer doesn’t speak English). Requiring the analysts to request briefings on areas they judge, and pro-actively check their facts, would also be a good idea. In our case it looks like the analysts just guessed some of the answers or went on gut feel. The ombudsman’s department was very defensive of the MQ process and placed the onus squarely on the vendors to pro-actively provide briefings. Of course we should (and do) do that. But only Gartner knows what they want to assess, and if we miss a topic surely that’s no excuse for Gartner to publish un-validated guesswork.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: