UL Delists MediaTek Powered Devices Due To Benchmark Whitelistingby Andrei Frumusanu on April 16, 2020 9:00 AM EST
- Posted in
Last week we detailed an article covering MediaTek’s seemingly widespread default inclusion of a benchmark whitelist in their chipset BSP (board support package) – a mechanism that enables more aggressive performance tuning of a device’s power management once it detects that a benchmark application is running.
Yesterday, UL, the developers of the PCMark and 3DMark benchmarking suites, have followed up on our investigation and analysed a wider range of devices, and have made the decision to temporarily delist all devices powered by a wide range of MediaTek SoCs, a list of over 50 devices from over 25 different vendors.
We had worked with UL early on in the investigation, with them providing us alternative anonymised versions of the benchmarks which bypass the whitelist detection, thus exposing the cheating behaviour.
The UL news post states:
“Using hidden mechanisms to detect benchmarking apps by name and make app-specific performance optimizations is not an "accepted industry standard." It is, in fact, the very opposite of the accepted standard.
Likewise, benchmark scores based on hidden app-specific optimizations and settings that are enabled by default and not available to the user do not accurately reflect a device's true performance in everyday use.
Simply put, a device must run a benchmark as if it was any other application. Performance gains must come from reacting to the nature of the workloads in the test rather than the name of the app itself.
As it has with similar cases in the past, we hope this delisting will help persuade MediaTek to change its approach and join the rest of the industry in adopting benchmarking best practices.”
The statement is a harsh rebuttal of MediaTek’s public response to our article, again pointing out that the practice is anything but an “industry standard” – especially damning since it’s coming from one of the major benchmark developers in the industry.
As UL stated in their blog post, and what we also pointed out in our original piece, we hope that the negative reactions to the matter will convince MediaTek to abandon such practices and reconsider their view of what the “industry standard” is.
- Mobile Benchmark Cheating: When a SoC Vendor Provides It As A Service
- UL Delists Huawei Devices Due To Inaccurate Benchmarks
- Huawei & Honor's Recent Benchmarking Behaviour: A Cheating Headache
- Futuremark Begins Delisting "Cheating" Mobile Devices from 3DMark Database
- They're (Almost) All Dirty: The State of Cheating in Android Benchmarks
Source: UL News Post
Post Your CommentPlease log in or sign up to comment.
View All Comments
zmatt - Thursday, April 16, 2020 - linkOuch. Good to know Underwriter's Labs still isn't anyone's fool.
Jorgp2 - Thursday, April 16, 2020 - linkIs it actually Underwriters Lab?
FunBunny2 - Thursday, April 16, 2020 - linkyes
shabby - Thursday, April 16, 2020 - linkI preferred madonion...
JKflipflop98 - Friday, April 17, 2020 - linkTo be fair, I'm sure they have more than one lab. . .
Hamm Burger - Thursday, April 16, 2020 - linkTwo thumbs up, Andrei!
ballsystemlord - Thursday, April 16, 2020 - link++
Makaveli - Thursday, April 16, 2020 - linkShame them into changing I like it.
NICOXIS - Thursday, April 16, 2020 - linkNice, a good old slap in the hands
The_Assimilator - Friday, April 17, 2020 - linktl;dr of UL's statement: Get f**ked, MediaTek cheaters.