Data is the most valuable part of machine learning, but collecting and labeling task-specific data is often difficult. Manual capture risks introducing bias, and verifying annotations from third parties or from vision-language models used for pre-annotation adds further effort. Synthetically generated data offers an alternative, since ground truth annotations come built in by design. The key challenge is whether models trained on synthetic images can generalize to real-world data.
This project explores whether a state-of-the-art pre-trained feature extractor can produce rich representations that let lightweight task heads, trained only on synthetic data, generalize to real images. The chosen domain is the card game SET, where players find triplets of cards in a 4x3 field in which each feature (shape, color, shading and count) is either identical or different across all three cards. Detecting these cards is a manageable generation problem that remains engaging.
The author selected DINOv3 from Meta for its strong localized feature extraction and its distilled smaller variants. Given limited hardware, the smallest distilled convolutional model, ConvNeXt Tiny with about 29M parameters, was chosen, with task-specific heads added for bounding boxes, keypoints or segmentation masks. As a first step, the project localizes cards without yet classifying their type.
For synthetic data, OpenCV draws basic shapes, building cards from rounded rectangles and composites of rectangles and circles. Procedural generation gives unlimited control over card appearance, allowing parameters to be varied widely to improve transfer to real photographs.
Source: Towards Data Science · Summarized by HeadlinesBriefing