About testing camera sensors
Setting the stage
Nowadays, we have a bewildering choice of free resources and websites which analyze every aspect of digital cameras. Every photographer worthy of the name can make an informed decision about his/her next acquisition in detail. Mostly, we are looking at image quality, which is considered the prime most important attribute of a camera. Whether with the current stand of technology, the minute differences between cameras of similar sensor size and same generation justify choosing one over the other, is debatable, though.
The reference website in the field is undoubtedly the leading, reputed website Digital Photography Review (hereafter DPR). Their comprehensive reviews cover every detail of a camera and their cleverly-designed interactive widget allows you to easily compare image quality of up to 4 cameras side-by-side. These images are peeped at by thousands of people quarreling about the superiority of the one or other camera.
While DPR controls environmental variables such as lighting, humidity, temperature and setting, there are a few things that are strikingly unequal among pictures. Take the example of the just-published review of the Olympus OM-D E-M5 (no matter how long the name).
Here, the raw images of the Olympus EM5 are compared with its most direct competitors: the Panasonic G3, the Sony NEX7 and the Samsung NX200.
It appears that at any high ISO setting, the EM5 output is cleaner than the G3, although they probably share the same sensor. Both the APS-C sized sensors of the Sony and the Samsung also appear noisier at pixel-level (bear in mind they have higher MP counts), while the Samsung looks slightly better than the Sony. It might differ slightly depending on which part of the image you are looking at, but generally, these are the trends.
The erroneous conclusion would be, image noise quality-wise: EM5 wins in first place, then the G3 or the NX200 (depending on your taste) and finally the NEX7 is the big loser.
The big problem about that comparison is that all 4 cameras received a different exposure (1).
If exposure is unequal among cameras, so is the signal to noise ratio, which underlies most of the technical attributes of image quality, like dynamic and tonal range. When shooting in RAW, other technical aspects of image quality like sharpness, colour and contrast, do not depend on the camera but much more on the photographer, which is not what we are interested in when testing cameras.
Pixels aplenty: ACR-converted raw from Sony NEX-7 at ISO 6400.
Notes and references:
(1) The exposure is the total light per area (photons / mm²) that falls on the sensor while the shutter is open.
(2) Note that for a comparison of image quality at equal display size, only the total amount of light would determine image quality
(3) If you choose even bigger full-frame sensors from the interactive widget, they use F11
The Truer Story: EM5 vs. G3
So why the EM5 images look better than the G3? Because they received more light.
Dial in ISO 3200 in the widget for all cameras for example. Download the files and check the EXIF: while both the G3 and EM5 have the same F-number, the G3 picture is taken at 1/1250s and the EM5 at 1/800s. That means that the Olympus sensor was exposed 2/3 stops higher than the Panasonic, and is equivalent to a visual difference between ISO 800 to 1250. This was necessary to get the same apparent exposure (ie. brightness) as the G3, because Olympus’ ISO values (the gain) are much lower than indicated.
Now feel free to lower the ISO value of the G3 to 1600 and you will see nearly identical results now for an image that was exposed much closer to the EM5. At ISO 1600, the G3 is exposed 1/3 stop higher than the EM5 and shows correspondingly slightly finer-grained noise. So is Olympus cheating with their ISO values? Before fanboys attack: every manufacturer does. The problem is that it leads us to false conclusions. In reality, the EM5 image quality raw quality is not different from the G3’s.
The Truer Story: EM5 vs. APS-C
What about the bigger APS-C peers from Sony and Samsung? This is a different story, because larger sensors inherently capture more total light at the same F-number, and thus show better image quality.
You will see that both the Samsung and Sony have a higher F-number of 8. That was probably necessary to achieve the same depth of field from those bigger sensors, as to be able to compare their sharpness to other-sized sensors (3). But it means that here, the APS-C sensors are artificially bottle-necked at the aperture to provide the same depth of field as smaller sensors. Consequently, they get a lower exposure that should be compensated with longer shutter speeds. But even by taking the latter into account, at F8 and 1/640s, the Samsung receives 1/3 stop less than the Olympus, and the Sony at F8 and 1/1000s, receives one whole stop less!
Clearly, in this comparison, the EM5 is advantaged compared to its peers as it receives 1/3 EV more light than the NX200, 2/3 EV more light than the G3 and 1 EV more light than the Sony. Now you can lower the ISO of the Sony by one stop to find out that even with a double pixel count, it has similar pixel-level noise as the Olympus.
Note that even though there are such differences in exposure, the apparent exposure of the images stays comparable and is probably evened out by the automatic conversion paramaters of Adobe ACR's raw engine that tries to put images from different cameras on the same level.
It looks as if DPR’s studio image comparison widget is biased somehow. What objective conclusions can you draw from it? Besides, what is really the point in comparing images shot with such artificial settings? It does not tell us anything about the camera’s performance in the real world. Very few people shoot exclusively in manual mode, and in all the other modes, the decision of how much light the camera will actually capture lies in its own electronic brain.
Currently, no website I know addresses all those shortcomings except maybe the thorough tests at DxOMark, but they do not tell you about the camera’s exposure behavior either.
Page created: 03/05/2012