This work addresses the difference in pair-comparison image quality experimental results when the tests are conducted with expert vs. non-expert observers. The study concludes that the expertise of observers has a statistically significant impact on the results, and that the use of expert observers leads to reduced variability in the results.