How bad is sample variation of lenses? A test using three sample of the Panasonic Lumix G 25mm f/1.7.
Lens reviews are often criticized for using only one sample of a lens. Some people claim that the differences between samples are so big that a bad lens might look good and vice versa. Is this true? How strong is sample variation really?

Method of the review

I bought three sample of the Panasonic Lumix G 25mm f/1.7 new. I bought them from two different shops with 6 months in between. This is why I am pretty sure they are from at least two different batches of production. The serial numbers also support this assumption.
The test pictures were made with and Olympus OM-D E-M5 Mark II. The camera was mounted on a tripod, the image stabilization was turned off, the electronic shutter was used, the camera set to ISO200 and I released it via wifi. Therefore shuttershock or camera shake had no influence on the results. To make sure I took three pictures each time and checked them for blur and focus errors. All images are 100& crops from the out-of-camera jpgs.

Sample variation at F/1.7

Image center

There are no differences between the three samples visible.

Image border

The lenses with the serial numbers …049 and ….519 show identical results, but the picture taken with the lens with the serial number …048 is slightly worse. We have found a slight case of sample variation.

Sample variation at F/5.6

Image center

Again there is no difference between the lenses.

Image border

At F/5.6 the lens number …048 catches up with the other two lenses, there is no difference between the three.

Conclusion about lens sample variation

There was only one case of slight sample variation (at the image border at F/1.7). We are talking about lenses with thousands of copies made in mass production. Of course sample variation exists. But is it bad enough to affect lens reviews? Can it make a bad lens look good or a good lens look bad? After this test I honestly doubt that. Sample variation can be measured, but only in extreme cases it is visible in real life shooting. My conclusion is therefore, that it is perfectly fine to use just one copy of a lens for a review.

The Panasonic Lumix G 25mm F1.7 is available at amazon.com, amazon.co.uk and also

An addition about copy variation

I received some very angry comments about this article. People are claiming that sample variation matters greatly and that every time they buy they get three to five lenses, test them and keep only the best. However, when I asked for the images of such a comparison I got no reply or the reply was, that the pictures were already deleted.
I am not claiming to have done a scientific study and as far as I know none exists. The closest thing to such a study is a test by Lensrentals, who compared hundreds of lenses. (You can read about it in this article on dpreview.) Their conclusion is, that copy variation does exists and can be measured, but that it is not strong enough to be noticeable in real world usage. They add that there are always some lenses, which are truly defective, but according to lens rentals that has nothing to do with sample variation and can usually be detected without comparing the lens with other copies.
Why are so many people claiming they notice huge differences in copies of the same lens? Let’s be frank, testing a lens is not easy. There are many factors that can make one copy seem sharper than another. Not just the obvious ones like focus errors and shutter shock. There are many not so obvious sources of vibrations or changing lightning conditions (like the movement of clouds and reflections).
I am not perfect and I am not saying that my article is the last word about the topic. But I simply ask one thing, if you think Lensrentals and I are wrong, do not just post a rude comment claiming that I am wrong, show us some proof. I would be very interested in seeing some comparison images (but please tell us how exactly you made them) of a case of huge sample variation.