How Well Does GPT-4o Understand Vision? Evaluating Multimodal Foundation Models on Standard Computer Vision Tasks

Prompt chaining is a technique designed to help MFMs break down complex tasks into simpler, manageable sub-tasks. We develop prompt chains for all of the tasks we evaluate, namely, object detection, semantic segmentation, grouping, depth prediction, and surface normal prediction. To guide the choice of how to split each task into sub-tasks, we rely on our early key observation that most MFMs are relatively strong at image classification, and therefore try to split each task into multiple classification sub-tasks.

We divide the task into two stages. First, the model identifies all objects in the image. Then, it localizes each object by recursively zooming in. We divide the image into grid cells and ask the model to check if any part of the object is in each cell. The model discards empty cells, narrowing the search area. By using both coarse and fine grids, we quickly downsample and refine the object's edges, pinpointing its location.

In semantic segmentation, the goal is to assign a class label to each pixel in an image. Instead of querying each pixel individually, we group pixels using SLIC, removing the need for per-pixel queries. Superpixels segment the image into smaller, homogeneous regions based on features like color and texture. We then classify these superpixels in batches, leveraging the strength of MFMs in image classification. To improve accuracy, we include previous batch predictions in the chain and provide multi-scale crops of each superpixel, which enhances the model's ability to capture fine details.

In the grouping task, given an image and a query point, the goal is to find other pixels that belong to the same object or background. Unlike semantic segmentation, this task has no fixed set of classes, making it more challenging. We use superpixels and leverage the MFM's ability to assess visual similarity. Each superpixel acts as a node in a graph, with edges connecting neighboring superpixels. Starting from the query point, the model evaluates adjacent superpixels to determine if they belong to the same object. This process continues, merging relevant superpixels, until no more are added.

Predicting 3D depth from a single 2D image is inherently ambiguous, so we perform relative depth prediction by having the model rank different regions of the image based on their distance from the camera. Instead of querying individual pixels, we segment the image into superpixels and sample pairs of superpixels for comparison. The model ranks these pairs by relative depth. These pairwise rankings are then globalized using an objective function that assigns larger values to deeper superpixels. We assume all pixels within a superpixel share the same depth, allowing us to extend superpixel-level rankings to a pixel-wise depth map across the image.

For surface normal prediction, we use a similar ranking approach as with depth. We select standard basis vectors (right, up, and forward) as reference directions and query the model to compare randomly sampled superpixel pairs based on their alignment with each vector. The pairwise comparisons are then globalized using the same algorithm as for depth, resulting in three separate surface normal maps. As before, we assume uniformity within each superpixel, assigning the same rank to all pixels within a superpixel group.

Here, we showcase the ability of MFMs to perform multiple tasks on the same image, using prompt chaining. The visuals are generated using GPT-4o, and demonstrate a strong semantic understanding of the images, and a non-trivial but relatively weaker understanding of geometric properties.

Here, we provide a visual comparison between the predicted outputs of various models and the ground truth across all tasks.

An RGB input is shown on the left and the model's predictions on the right, obtained for each frame of a video. Users can use the slider to navigate through frames, observing how the model changes its predictions over time.

Recent updates to GPT-4o allow it to generate dense image outputs instead of just text, which is a promising development for vision tasks. However, we found that the model often creates "semantic recreations" rather than accurate, pixel-perfect edits. This can lead to hallucinations and spatial errors, making it challenging to use for precise vision tasks without further refinement. The figure below highlights some of these failure cases.

Here, we quantitatively explore how different MFMs perform across various visual tasks. As described earlier, we tested GPT-4o, o4-mini, Gemini 1.5 Pro and 2.0 Flash, Claude 3.5 Sonnet, and Qwen2-VL-72B, comparing their capabilities to specialized vision models.

Our classification testing revealed interesting results across all datasets. While MFMs didn't quite match the performance of specialized vision models like Model Soups ViT-G and OpenCLIP H, they showed impressive capabilities. GPT-4o emerged as the standout performer, followed by Gemini 2.0 Flash, Gemini 1.5 Pro, Claude 3.5 Sonnet, Qwen2-VL, o4-mini and Llama 3.2. Notably, these models demonstrated good resilience to image corruptions and distribution shifts.

In object detection tests, we compared the MFMs against specialized vision models like DETR and Co-DETR. While all MFMs performed below these specialized models, GPT-4o achieved the highest performance among the MFMs, significantly outperforming its peers. Interestingly, even when testing Gemini 1.5 Pro and Qwen2-VL with direct bounding box regression, they still couldn't match GPT-4o's performance using the chain algorithm.

For semantic segmentation, the MFMs achieved notable but not state-of-the-art performance. While they showed promising capabilities, they still fell behind specialized models like OneFormer.

In our grouping task evaluation, which built upon semantic segmentation, we saw varying levels of success among the MFMs. GPT-4o emerged as the top performer, showing good overall performance, though still not matching the capabilities of the specialized SAM model.

Our depth prediction testing revealed that while MFMs performed better than random guessing, they still showed significant limitations compared to specialized models like Omnidata. Quantitatively, their geometric abilities appear relatively weaker than their semantic abilities. We evaluated performance using both standard metrics and relative measurements like Spearman correlation coefficients and pairwise depth comparison accuracy.

The surface normal prediction tasks revealed some significant limitations in the MFMs' 3D understanding capabilities. Most notably, several models struggled with left-right direction correlation, and Gemini 1.5 Pro performed below random chance in 2 out of 3 directional components. These results suggest that MFMs currently have limited 3D visual understanding capabilities.

We defer a detailed discussion of several design choices and experimental results to the main paper. Topics covered include:

• Sensitivity analysis of various prompts.
• Performance on images beyond standard datasets.
• A detailed discussion of the algorithms.
• Insights into the blind guessing baseline.
• Preliminary investigations with GPT-4o Image Generation.
• Cost analysis of prompting.