They say the best camera is the one you have with you, and most of us are never without our phones.
But how do you know which smartphone has the best camera? Every manufacturer claims to have a great camera, they throw around megapixel counts and aperture sizes, but none of this tells you how good your shots will be in the real world.
Many of us turn to independent photography lab, DxOMark, for in-depth reviews that boil down the camera quality of each flagship smartphone to a single number out of 100. Since the Parisian company began to test mobile phone cameras in 2012, it has been the gold standard. The scores it bestows are trusted enough to be widely referenced in the tech press.
“When Google achieved top marks [for the Pixel], they covered London in ads because they were proud,” Clement Viard, Senior Director, Image Quality Evaluation at DxOMark told Digital Trends.
Collectively we shared around 2.5 trillion photos online last year and 90 percent of them were captured on smartphones, according to a report from Deloitte. That number is going up every year. The Millennial generation is particularly interested in mobile photography, with 24 percent saying it’s the most important function of their smartphone, and 57 percent saying it’s very important, according to Osterman Research.
Little wonder, then that a good DxOMark is so coveted. But how does the firm arrive at these widely quoted scores?
“Today, the differentiator for the phone is probably the software more than the hardware,”
“For smartphones, we always use the auto mode. We take the phone out of the box, turn it on, start the camera app, and take pictures,” explains Viard. “Camera users are used to tweaking settings, but smartphone users expect to be able to just push a button, so the testing has to be completely different.”
Every smartphone camera is subjected to the same battery of tests, broken down into sections that cover things like exposure and contrast, color, and autofocus. Each camera is meticulously tested with a mix of specialized equipment in the lab.
“We also have to get out of the lab and test in the real-world. We go to the same spots around Paris, around the same time of day. We try to find places where the colors will always be the same, where there is dark and light to see how it handles shading.”
They take 1,500 photos with every smartphone camera that’s tested and then break them down and grade them. To account for changes in light and weather, they also snap shots with a reference device for comparison, so they can be sure they’re grading the phone and not the weather. Each photo is cross-checked by at least two experts from the 40-person team. The result is a 100-page report for each review, highlights of which are published on the DxOMark website.
Viard says they’re usually in agreement, but when there’s a difference of opinion the rest of the team will be called in for a debate. There’s no doubt they take the scoring system seriously, but separating opinion from objective fact can be tricky. You’ll often see people saying that Samsung’s Galaxy phones are oversaturated, for example, but some people prefer the way they look. Where is the line on quality?
“There was a lot discussion in trying to work out what would be the best way to assess color quality,” explains Viard. “We take pictures of a scene with colors and we map colors into a color space that will look at saturation and bias. What we found is that if the distance between the real color and the color from the camera is within an area, then it’s really a matter of taste. Different people can argue for weeks and they’re not going to come to the same conclusion.”
What DxOMark does in this situation is to stop factoring it into the score beyond a certain point. It does the same thing at the bottom end of the scale. There’s a cut-off point where a photo is so bad that you’ll discard it anyway.
Because DxOMark also offers consulting services and specialized equipment to help manufacturers to improve their camera performance, it has been subject to occasional claims of bias. The idea being that manufacturers who engage its consulting services will get a better score.
“The exact score calculation is confidential and the reason we keep it confidential is because we don’t want manufacturers trying to optimize for the score,” says Viard. “We are very aware that the way we design the score formula may influence some choices companies make, but the testing protocol will be exactly the same for every single phone, with all the same objective metrics, real world results, and human grading.”
DxOMark recently updated its testing protocol for the first time, to add two new categories to every score: Zoom and bokeh. The new protocol also places greater emphasis on low light conditions, and DxOMark introduced more motion into its testing, to see how cameras perform when both the subject and photographer are moving. This is all intended to better reflect what consumers want and the real-world conditions we take photos in.
The new testing protocol has been a long time in the making. DxOMark is keen to ensure that it only tests features that really matter to people. It considered the impact of 3D capabilities a few years ago, but decided it was a fad. It’s already testing front-facing cameras with the rise in popularity of selfies, and watching 360-degree camera technology very carefully, but it will be a while, if ever, before these make into the score.
“We always try to focus on usage rather than technology,” says Viard.
Apple’s iPhone 7 Plus bokeh effect, where the subject is in sharp focus and the background is blurred, is a good example, as it sparked a demand for bokeh and got everyone talking about it.
“There are many ways of creating the bokeh effect,” Etienne Knauer, Senior VP, Sales and Marketing, Business Solutions, DxOMark explained to Digital Trends. “But you require at least two different images, so with a single camera, you don’t know it, but actually you’re taking multiple images, even if it’s from the same sensor, or they use a dual camera. You need two to make an estimation of the depth and then software blends them.”
Under the old test protocol, there was no distinction between the single-lens iPhone 7 and the dual-lens iPhone 7 Plus, but the introduction of zoom and bokeh, enabled the 7 Plus to creep up the chart with a score of 88, while the iPhone 7 scored 85. The rescored Google Pixel managed 90 on the new scale, putting it joint top with the HTC U11 , but there are some very big phones on the horizon that might challenge the current champions.
“We only test the flagships and the very best phones,” says Viard. “We look at the specs and what the manufacturer is saying to find the phone where they’ve put most emphasis on photography and that’s the one we will pick.”
Sadly, DxOMark lacks the resources to test every phone that comes out, though Viard says they’d like to. There’s no doubt that the field has widened in the last couple of years.
“For the first few years Apple, Nokia, Sony, or Samsung was always the best. Today everybody is putting emphasis on the camera. HTC, Huawei, OnePlus, and Google have made huge improvements to the camera, so the number of companies we test has grown.”
If you’re into smartphone cameras, you’ll know that Sony makes most of the sensors that the flagships use, yet its own phones don’t tend to score as highly as other handsets with the same sensor.
“Today, the differentiator for the phone is probably the software more than the hardware,” says Viard. “Camera tuning makes such a big difference. You can have two manufacturers with exactly the same hardware, even the same chipsets, but the image quality is very different because of the algorithm.”
Even with the enormous processing power smartphone cameras now possess and all the clever software tricks, there are some things a DSLR does better. DxOMark’s bokeh test, for example, is designed to catch out these tricks. It’s a face with triangles that go out to a fine point, wires on the same plain, and lots of details and broken images in the background, to see if the camera gets confused and blurs the wrong parts, which it invariably does.
“On a DSLR, if it’s on the same plain, it will have the same sharpness,” says Viard. “Smartphone cameras have a long road before they reach DSLR level, but I think the gap is closing, the progress has been huge in the last five years.”
All the major manufacturers are investing serious resources into improving camera technology, trying to make it easier for average people to capture perfect shots. That’s great news for us, because at the end of the day, mobile photography is all about capturing precious memories. If you want to know which smartphones will give you the best chance of catching those spontaneous moments, then it’s always worth checking out what DxOMark has to say.
