Why you shouldn't trust DXO lens testing

[Max the Dog]

I recently added to my Sony V Nikon page but was reminded once again of the inaccuracy of what appears to be calculated scores over at DXO.  One of the best examples I am aware of is the insanely low calculation on the FE55 when used on the NEX7 vs the A7r.

 

DXO shows the FE55 getting half the resolution on the NEX7 as the A7r.  In hundreds of tests I performed, the FE55 on the NEX7 was typically within 5-6% of the A7r scores.

 

DXOs 'guess-testing' isn't even in the ballpark.    More here - https://sonyvnikon.wordpress.com/2015/03/16/nikon-d810-v-sony-a7r-with-zeiss-otus-85mm/

 

[Vivek]

A general Q to all those testers out there (pro or amateur testers):  What the heck are you trying to accomplish?

[Guest all8]

Well it makes sense. A 36MP FF sensor against a 24MP APSC sensor WITH an AA filter. So 30MP to 15MP sounds plausible.

 

Its not possible for the NEX-7 to resolve at more than 24P-Mpix without an AA filter, let alone with one ... 

[Max the Dog]

A general Q to all those testers out there (pro or amateur testers):  What the heck are you trying to accomplish?

 

Not to just be suckered by the claims of companies that don't deliver.    

[Max the Dog]

You really have no idea what you're talking about.

 

 

The photosite density of the NEX7 is 25& greater than the A7r so that means more pixels on the subject and with a lens capable of resolving 51mp (the calculated FF density of the NEX7 and other 24mp APSc sensors) then other than sensor evolution (it's well documented the NEX7 had a weak AA filter) and the A7r has very poorly designed thick sensor glass that causes astigmatism)  the 5-6% difference is realistic and a reality. 

 

At least I actually tested the system -  DXo just calculated the #s. 

 

Well it makes sense. A 36MP FF sensor against a 24MP APSC sensor WITH an AA filter. So 30MP to 15MP sounds plausible.

 

Its not possible for the NEX-7 to resolve at more than 24P-Mpix without an AA filter, let alone with one ... 

[Austerror]

Astigmatism?

 

I'm not trying to be picky, but my understanding of that word is that it applies to spheres (eyes) and not a flat piece of glass.  Could you elaborate a little please?

 

Also a link to the weak Nex-7 AA filter please. I've not found any information about that.  It would be interesting for me to see the results of with AA and without AA on my NEx-7.

[Vivek]

You really have no idea what you're talking about.

 

 

The photosite density of the NEX7 is 25& greater than the A7r so that means more pixels on the subject and with a lens capable of resolving 51mp (the calculated FF density of the NEX7 and other 24mp APSc sensors) then other than sensor evolution (it's well documented the NEX7 had a weak AA filter) and the A7r has very poorly designed thick sensor glass that causes astigmatism)  the 5-6% difference is realistic and a reality. 

 

At least I actually tested the system -  DXo just calculated the #s. 

 

Enlighten me.  How is the D810's filter stack is designed and how thick it is compared to that of A7r's or that of the NEX-7?  Did you measure anything of the stack to make claims about astigmatism?

 

I don't disagree that these are not well designed cameras.

[Vivek]

Not to just be suckered by the claims of companies that don't deliver.    

 

 

What do they claim?

[iul]

@MaxtheDog

 

Hi Max

 

I think you don't have to worry - it seems to be just a misunderstanding.

 

Let me quote from the dxomark page:

 

"The number of P-Mpix of a camera/lens combination is equal to the pixel count of a sensor that would give the same sharpness if tested with a perfect theoretical optics, as the camera/lens combination under test."

 

The maximum P-Mpix value that a 24 MP camera-lens combination can get is 24.

 

The maximum P-Mpix value that a 36 MP camera-lens combination can get is 36.

 

So the dxo result that you quoted means nothing else than that 9 of 24 MPs are lost on the NEX 7 while 7 of the 36 MPs are lost on the A7r.

[andyfromboston]

Yes, exactly. To put it another way, it's a synthetic rating of a number of pixels that represents the amount of detail that a given camera+lens combination could get under ideal circumstances. It's not a measure of photosite density. 

[Max the Dog]

What do they claim?

 

14 bit color.    (as recently as a week ago and since annoucement)

 

[Max the Dog]

@MaxtheDog

 

Hi Max

 

I think you don't have to worry - it seems to be just a misunderstanding.

 

Let me quote from the dxomark page:

 

"The number of P-Mpix of a camera/lens combination is equal to the pixel count of a sensor that would give the same sharpness if tested with a perfect theoretical optics, as the camera/lens combination under test."

 

The maximum P-Mpix value that a 24 MP camera-lens combination can get is 24.

 

The maximum P-Mpix value that a 36 MP camera-lens combination can get is 36.

 

So the dxo result that you quoted means nothing else than that 9 of 24 MPs are lost on the NEX 7 while 7 of the 36 MPs are lost on the A7r.

 

The main point is that DXO is wrong - the 'theoretical' limits have been exceeded by practically every APS-c user that put any decent lens on to their camera. 

[iul]

You should contact them and discuss this - they will be happy

[Max the Dog]

We're way out in left field now.  My point is that a 24mp APSc has a FF density of 50+mp.  If you have a lens that can resolve that well and all other factors equal then the denser APSc sensor can resolve better than a FF.  One only needs to look at the A7r (36mp) A7 (24mp) and A7s (12mp) to see the linearity in the relationship between resolution and photosite density.  When one actually tests the A7r and NEX7 with a real lens the facts speak clearly.  They are very close in Imatesting which is indistinguishable by the naked eye.

 

 

Yes, exactly. To put it another way, it's a synthetic rating of a number of pixels that represents the amount of detail that a given camera+lens combination could get under ideal circumstances. It's not a measure of photosite density. 

[Max the Dog]

I did - Jan 30, 2014 - No response yet from DXO yet.   

 

http://www.dxomark.com/Lenses/Sony/Sony-FE-Carl-Zeiss-Sonnar-T-STAR-55mm-F18 

 

You should contact them and discuss this - they will be happy

[Max the Dog]

Enlighten me.  How is the D810's filter stack is designed and how thick it is compared to that of A7r's or that of the NEX-7?  Did you measure anything of the stack to make claims about astigmatism?

 

I don't disagree that these are not well designed cameras.

 

Roger did.

 

http://www.lensrentals.com/blog/2015/01/a-thinner-sensor-stack

 

And Diglloyd 

 

http://www.fredmiranda.com/forum/topic/1340474/2

[Max the Dog]

Dude - this is why god invented google. 

 

Just google the terms - NEX7  weak AA filter  and get the result

 

http://www.fredmiranda.com/forum/topic/1340474/2

 

http://www.sonyalpharumors.com/sony-nex-7-moire-test-and-negative-impression-by-ricehigh/

 

What happened to intelectual curiosity ? 

 

Astigmatism?

 

I'm not trying to be picky, but my understanding of that word is that it applies to spheres (eyes) and not a flat piece of glass.  Could you elaborate a little please?

 

Also a link to the weak Nex-7 AA filter please. I've not found any information about that.  It would be interesting for me to see the results of with AA and without AA on my NEx-7.

[andyfromboston]

We're way out in left field now.  My point is that a 24mp APSc has a FF density of 50+mp.  If you have a lens that can resolve that well and all other factors equal then the denser APSc sensor can resolve better than a FF.  One only needs to look at the A7r (36mp) A7 (24mp) and A7s (12mp) to see the linearity in the relationship between resolution and photosite density.  When one actually tests the A7r and NEX7 with a real lens the facts speak clearly.  They are very close in Imatesting which is indistinguishable by the naked eye.

 

But Max, those are measurements of different things. FF equivalent pixel density is not what DXO is claiming to be representing. The P-Mpix number is not meant to be a representation of detail per unit area or dimension of sensor, which is what Imatest measures. Imatest numbers come in LW/PH, which means "line widths per picture height". If you look at, say, photozone.de, that's what you're seeing. If you read DXOMark's explanation of P-Mpix, it is clear that they are talking about the total amount of detail captured. It is not a "per picture height" measurement.

 

Please do look at these things before jumping to conclusions. Not doing so will lead to errors such as telling a person "You really have no idea what you're talking about," when really that person does understand the subject and ought to be given some credit.

[Max the Dog]

I've done thousands of Imatests and I know and understand what the data shows.   You've done how many ?

 

Additionally, I have both Otii and the Apo-Sonnar for my D810, so I am pretty clear on what sharpness, resolution, transmission and CA are both in lab and real world enviroments.   

 

You're so clueless I have no idea what you're even talking about. 

 

My main points

 

DXO doesn't test

DXO calculates

DXO gets it wrong

And with all things being equal and using capable lenses smaller photosites (denser sensors) can resolve better than larger ones.  One only needs to look across the A7r,A7 and A7s to see the linearity of this point. 

 

 

But Max, those are measurements of different things. FF equivalent pixel density is not what DXO is claiming to be representing. The P-Mpix number is not meant to be a representation of detail per unit area or dimension of sensor, which is what Imatest measures. Imatest numbers come in LW/PH, which means "line widths per picture height". If you look at, say, photozone.de, that's what you're seeing. If you read DXOMark's explanation of P-Mpix, it is clear that they are talking about the total amount of detail captured. It is not a "per picture height" measurement.

 

Please do look at these things before jumping to conclusions. Not doing so will lead to errors such as telling a person "You really have no idea what you're talking about," when really that person does understand the subject and ought to be given some credit.

[andyfromboston]

Say what you will, the fact is that you are talking about sharpness in per-linear-measurement-of-sensor terms and DXO is talking about the amount of detail resolvable by the entire sensor. The NEX7 scores lower in DXO than the A7R because it is a smaller sensor with fewer pixels. Under the same conditions it might score higher than an A7R in Imatest because it has more pixels per mm. Those are different metrics. It is possible to have more pixels per mm, yet still have fewer pixels total, by having fewer mm. That is very, very simple stuff. It doesn't take an internet forum bully / troll to understand that "number per unit of distance" is not the same as "total number".

 

The question of how many pixels you'd have if you made the NEX7 sensor full-frame is not relevant.

[PCA]

Why do I care if DxO is right or wrong?  

[Max the Dog]

Because photographers like to make purchases based on research and that's all good provided that the results aren't just speculation calculated from a very poorly designed set of assumptions that can be off by 10 standard deviations.  That's why.

 

 

 

Why do I care if DxO is right or wrong?  

[PCA]

Because photographers like to make purchases based on research and that's all good provided that the results aren't just speculation calculated from a very poorly designed set of assumptions that can be off by 10 standard deviations.  That's why.

What did we do before DxO existed?  I've never bought a camera or lens based on what DxO said, or any test chart.  I'm more interested in the pictures.  I'm much more interested in going to Flickr or another site and looking up images tagged for a given lens and seeing the actual results than scores from test charts.  I've never had someone buy a print of a test chart.  Is there a big market for A3+ sized prints of MTF charts?  Must be where all those Fine Art photographers are making all their money.  I'll put one in my portfolio and see if it works.

[Max the Dog]

Found the test  - Jan 2014

 

[Erp]

TLDR: OP either doesn't understand or is willfuly ignorant of the difference between total resolution and resolution per spatial unit.