A new benchmark called Vision Q asks AI models to judge computer vision images on one named quality, such as edge sharpness. The authors tested 20 AI models acting as judges. The best one was right 63.1 percent of the time, a little under two times out of three. Random guessing would be right 32.2 percent of the time. The best model is clearly better than guessing, but it still misses more than a third of the time.
That gap is the reason for this article. If the best AI judge makes this many mistakes, you should test your own judge before trusting it. These scores matter outside the lab too, because people already ask AI assistants to look at photos and answer. Imagine uploading two photos of a medicine box and asking how much to take, then uploading them in the opposite order. If the answer changes, you need to know whether the label was hard to read or the image order affected the result.
The author tested whether image order could affect a model's answers. In one test, Claude was shown two pictures and asked which had fewer compression artifacts. Over 24 trials, Claude picked the second picture in 23 of them, whatever the pictures were. That suggests a strong position bias, though the result came from the author's own test images and may not hold for every kind of picture.
The author then built a smaller test of original images, each pair differing by one known change, and logged every answer to Weights & Biases. The goal was a short script that checks three things about an image judge: whether it looks at the detail you ask about, whether it favors the first or second image, and whether it makes up an answer.
Source: Towards Data Science · Summarized by HeadlinesBriefing