Khoi P. N. Nguyen

PhD student in NLP at UT Dallas. Interested in maths, programming, and humanity.

2 minute read ml

Evaluation has become a major topic in AI research and development. As the number of AI systems grow, the need to know “how good they are” also grows, for multiple reasons: to select the model to release in production, to know which types of capabilities needs improvements in a model, and eventually to feel confident about its ability (especially after a model upgrade.)

In this post, I want to highlight two dimensions of an evaluation, which has helped me in high-level reasoning about what evaluation to use for a task given some resource constraints.

The dimensions

The two dimension are:

  1. AUTOMATEDNESS: whether the evaluation is done automatically, or manually via some human labor. This gives rise to auto eval and human eval.
  2. GROUND-TRUTH NEED: whether the evaluation can or cannot be done without the ground truth labels of the tasks on some dataset. This gives rise to reference-free eval and reference-based eval.

These two dimensions are orthogonal: the automatedness of an eval does not cause it to need or not need ground truths. Therefore, these dimensions give rise to 4 subtypes of evals in the following chart:

The two dimensions of an eval — automatedness (auto vs. human) and ground-truth need (reference-based vs. reference-free) — forming a 2x2 matrix of four eval subtypes.

What did my papers used?

MemeIntent (2024).

  • Task: Given a meme, generate a natural-language sentence describing its final intent.
  • Eval type: reference-based auto and human eval.
  • Why reference-based? Because without it, we need to rely on an AI model to reference-freely judge the described intent. We did not trust AI models for their ability to do that.
  • What auto eval metrics were used? Because this is reference-free auto eval, it is automatic comparison. We used popular sentence comparison metrics.

MemeQA (2025).

  • Task: Given a meme and a multiple choice question about its meaning, select one or all of the correct options.
  • Eval type: reference-based auto eval.
  • Why? Because (1) it is trivial to auto-eval multiple choice questions and (2) collecting labeled answers for multiple choice questions is trivial (due to the ease in measuring inter-annotator agreement.)

MemeInterpret (2025).

  • Tasks: Given a meme, (1) generate a sentence describing the surface level details observed, including the image and the text, (2) generate a list of sentences describing the background knowledge needed for someone to understand the meme, (3) generate a sentence describing the final meaning of the meme, and (4) classify whether the meme is hateful or not.
  • Eval types: Left as an exercise for the reader

A pending Meme[Something] paper (2026?)

  • Task: Given a meme, generate a certain structured representation of its meaning
  • Eval types: reference-based human and auto eval
  • Why? Even thought 2 years have passed since MemeIntent, AI models have got much smarter to be trusted with doing ref-free eval, our task here is so sophisticated that evaluation requires both ground truths and human comparison.

Updated:

Comments