Traditional retrieval metrics often penalize systems for finding meaningful similarities across different categories. Using CLIP embeddings on 9,315 architectural images, this study reveals that semantic similarity and categorical agreement are not always aligned. The article explores how CLIP identifies visually related buildings despite differing architectural labels, challenging rigid evaluation criteria like Precision@k. By analyzing roof types, materials, and symmetry, the authors argue that current benchmarks may misclassify insightful retrievals as errors.
The research introduces a dataset spanning 23 architectural styles to test whether CLIP's apparent mistakes are random or systematic. Through cosine similarity measurements and feature-based analysis, the study shows that CLIP frequently retrieves architecturally related styles even when labels mismatch. This suggests that relevance should not be a binary label-based decision but a nuanced assessment of visual and structural relationships.
The findings advocate for evolving retrieval evaluation beyond simple categorical matching. By embracing embedding-driven insights, systems can better capture the complex, multi-dimensional nature of similarity that human perception naturally recognizes.
Source: Towards Data Science · Summarized by HeadlinesBriefing