You trained a model. It spits out images that look okay. But are they actually good? If you rely on just one number to tell you, you're lying to yourself. The field of multimodal generative AI is messy, and no single metric captures the full picture of quality, alignment, and diversity. You need a suite of tools.
Most developers make the mistake of chasing a low FID or a high CLIPScore and calling it a day. That’s dangerous. A model can have a perfect FID score while completely ignoring your text prompt, or a high CLIPScore while failing to draw three cats when you asked for three cats. This guide breaks down what these metrics actually measure, where they fail, and how to build an evaluation dashboard that doesn’t fool you.
The Core Problem with Single-Metric Evaluation
Think about what makes a generated image "good." It needs to match the prompt (alignment). It needs to look realistic (quality). It needs to vary across samples (diversity). And it needs to be safe. Traditional metrics were built for different eras. Inception Score, introduced in 2016, checked if an image looked like a clear class from ImageNet. It ignored text entirely. FID, created by Martin Heusel and colleagues in 2017, improved this by comparing feature distributions, but it still doesn't care about your prompt.
A 2023 survey of 37 text-to-image papers found that many studies relied solely on automatic measures like FID, often describing human evaluation poorly or not at all. The result? Papers claiming state-of-the-art results that humans couldn't even agree were better. Automatic metrics lag behind human preference. Correlation between simple scores and what people actually like is often weak. You have to triangulate.
CLIPScore: Checking Alignment, Not Truth
CLIPScore became popular after OpenAI released CLIP in 2021. Jack Hessel and his team introduced CLIPScore as a reference-free metric for image captioning. The math is straightforward: it takes the cosine similarity between the image embedding and the text embedding from the CLIP model. Higher is better. If the image and text sit close together in CLIP's vector space, the score goes up.
This is great for ranking checkpoints quickly. It tells you if the image broadly matches the concept in the text. But here is the trap: CLIPScore is essentially a bag-of-words check. It struggles with counting, spatial relationships, and negation. If you ask for "a dog without a collar," CLIP might give a high score to an image of a dog with a collar because the word "collar" is present. It rewards superficial semantic similarity. It also inherits biases from the internet data CLIP was trained on. A high CLIPScore proves compatibility, not factual correctness or compositional accuracy.
FID: Measuring Distribution, Not Prompt Adherence
Fréchet Inception Distance is the standard for image quality. It works by extracting features from real images and generated images using the Inception-v3 network. It then fits Gaussian distributions to both sets of features and calculates the distance between them. Lower is better.
FID answers one question: Does this batch of generated images look like it came from the same distribution as the real dataset? It does not answer: Did the model follow the instructions? You can have a very low FID with a model that ignores prompts and just generates random pretty pictures from the training set. Conversely, a model that follows prompts perfectly but generates slightly less realistic textures might have a higher FID.
FID is also fragile. It depends heavily on the reference dataset, image resolution, preprocessing steps, and sample size. Comparing FID scores across different papers is often meaningless if they didn't use identical pipelines. NVIDIA’s research on precision and recall highlights that a low FID can hide mode collapse-a generator might produce high-quality images but lack diversity, yet FID won't clearly distinguish this from a diverse, high-quality model.
Beyond the Basics: KID, Precision, and Recall
If FID feels too blunt, consider Kernel Inception Distance (KID). Proposed by Bińkowski et al. in 2018, KID uses a polynomial kernel to estimate the maximum mean discrepancy between feature distributions. Unlike FID, KID is unbiased at small sample sizes. This makes it much more reliable for smaller datasets or when you don't have millions of images to test against.
To separate quality from diversity, use Precision and Recall. Precision measures how many generated samples fall within the manifold of real data (fidelity). Recall measures how much of the real data manifold the generator covers (diversity). Improved versions of these metrics help diagnose whether your model is hallucinating weird artifacts (low precision) or just repeating the same few styles (low recall).
Compositional Correctness: TIFA and GenEval
Standard metrics miss the details. Did the model draw two red cars or one blue truck? TIFA (Text-to-Image Faithfulness) tackles this by using a language model to break down the prompt into questions. For example, "A cat on a mat" becomes "Is there a cat?" and "Is there a mat?" and "Is the cat on the mat?" Then, a visual question answering (VQA) model checks the image against these questions. The score is the percentage of correctly answered questions.
This approach is interpretable. You can see exactly which part of the prompt failed. GenEval takes a similar object-focused approach, testing for existence, counting, color, position, and relations. These metrics expose failures that CLIPScore hides. A model might get a high CLIPScore for "two dogs" but only generate one dog. TIFA would catch that.
| Metric | Primary Purpose | Strengths | Major Weaknesses |
|---|---|---|---|
| CLIPScore | Text-Image Alignment | Reference-free, fast, good for broad matching | Poor at counting/spatial logic, biased by CLIP training data |
| FID | Distributional Quality | Industry standard, correlates with realism | Ignores prompts, unstable at small samples, sensitive to preprocessing |
| KID | Distributional Quality | Unbiased at small sample sizes, stable | Less common, requires careful kernel tuning |
| TIFA/VQAScore | Compositional Accuracy | Interpretable, tests specific attributes/counts | Depends on VQA model accuracy, slower computation |
| Precision/Recall | Fidelity vs. Diversity | Diagnoses mode collapse and artifacts | Requires careful threshold setting, complex implementation |
Building a Defensible Evaluation Pipeline
Don't just run one script. Build a dashboard. Here is a practical checklist for evaluating a new multimodal model:
- Freeze Your Setup: Lock down the prompt suite, generator checkpoint, sampler settings, resolution, and random seeds. Variance kills reproducibility.
- Report Uncertainty: Never report a single FID number. Run multiple seeds and report the mean and standard deviation. One lucky seed can skew results.
- Combine Metrics: Use CLIPScore for alignment, FID/KID for quality, and TIFA/GenEval for composition. If these disagree, investigate why.
- Keep Failed Outputs: Don't silently delete broken images or unsafe generations. Include them in your stats. They reveal robustness issues.
- Add Human Validation: Automatic metrics are proxies. Conduct pairwise comparisons with human raters. Ask: "Which image better matches the prompt?" Report inter-rater agreement.
For a quick sanity check, start with CLIPScore and FID. But if you want to publish or deploy seriously, add TIFA or VQAScore to verify compositional logic. Remember, GenAI-Bench showed that VQA-based scores align better with human ratings than CLIP-based ones for complex tasks. Trust the data that mirrors human perception.
Why is my CLIPScore high but the images look wrong?
CLIPScore measures semantic similarity in a shared embedding space, not logical correctness. It often fails with counting, spatial relationships, and negations. An image of a "dog with a cat" might score highly for "cat with a dog" even if the arrangement is nonsensical. Use TIFA or GenEval to check compositional accuracy.
Can I compare FID scores from different papers directly?
Usually, no. FID is highly sensitive to the reference dataset, image resolution, preprocessing pipeline, and sample size. Unless the papers used identical implementations and datasets, the numbers aren't comparable. Always re-calculate FID using your own consistent pipeline if you need to compare models.
What is the difference between FID and KID?
Both measure distributional similarity using Inception features. FID assumes the features follow a Gaussian distribution, which can be inaccurate. KID uses a kernel method that is unbiased at finite sample sizes, making it more reliable for smaller datasets or when you have limited computational resources for large-scale generation.
Do I still need human evaluation if I have good automatic metrics?
Yes. Automatic metrics struggle with aesthetics, safety, subtle artifacts, and cultural context. Humans detect uncanny valley effects, bias, and relevance nuances that algorithms miss. Pairwise human comparison remains the gold standard for final validation before deployment.
How many images do I need for a reliable FID score?
FID becomes statistically unstable with small sample sizes (e.g., under 10,000 images). Most benchmarks use 50,000 or more generated images compared against a large real dataset. If you have fewer images, KID is a better choice due to its lower variance at small scales.
