So you’re in the market for a new phone, tablet, laptop, gaming rig, or other device and you want to get the most powerful model you can afford. Tech reviews can be rather subjective, so while there are plenty of opinions on the internet, you might be looking for hard data from benchmark tools which are designed to see exactly how one devices compares to another when performing the same CPU or GPU-intensive tasks.
There’s just one problem: You can’t really trust those scores, because device makers are cheating. AnandTech dug up evidence that most major Android device makers are taking steps to inflate their scores on popular benchmark tests.
Benchmarks are generally tricky because they’re not always indicative of real-world performance. An app that tries to perform a complex calculation or render graphics in real-time doesn’t always tell you if your phone will feel sluggish while performing day-to-day tasks.
But apps such as AnTuTu, Vellamo, and GFXBench are widely used to at least see how various Android devices compare with one another. If everything worked the way it was supposed to, you’d at least expect an Android phone or tablet with higher scores than its peers to feel faster in everyday usage.
Unfortunately device makers know that tech reviewers, hobbyists, and others use these tests. So they include software on their devices that boosts performance when you’re running popular benchmark tools. Theoretically this means your device is capable of achieving the level of performance reported… but it probably won’t. That’s because your phone or device is effectively overclocked when running the test and then returns to normal speed when the test is over.
This summer Samsung was caught inflating the performance of the Galaxy S4 smartphone on benchmarks this way. But after spending a few months looking at various devices, the folks at AnandTech found that almost every major company that ships Android devices in the US fiddles with the software to inflate benchmark scores.
Not all companies are currently optimizing devices for every benchmark — but Asus, HTC, LG, and Samsung are all guilty of optimizing their devices to perform well on at least some benchmark tests.
Motorola’s recent devices don’t do this, and neither do the NVIDIA Shield or the Google Nexus 4 or Nexus 7. But odds are if you’ve bought an Android device that’s not running Google’s stock version of Android, it’s been rigged to get higher benchmark scores than it should.
AnandTech notes that the same sort of shenanigans have been taking place in the PC world for a while, and people haven’t stopped relying on benchmarks as a way to compare PCs. I doubt it’ll happen in the Android device space anytime soon either.
But I’m all for public shaming of the companies involved in an effort to get them to stop trying to convince customers that their products are more powerful than they really are. That’s no way to make customers happy.
Most smartphone shoppers probably don’t care about benchmark scores at all, and those who do definitely don’t want to find out that the scores are meaningless.