Good news: Every Eval Ever (EEE) and Hugging Face Community Evals now speak the same language. The two projects are fully intercompatible, meaning you can cross‑post evaluation results, interpret them consistently, and ha ve every score link back to open models, leaderboards, and one standardized metadata store.

EEE started in February 2026 as part of the EvalEval Coalition—the first cross‑institutional push to fix how AI evaluation results are reported, whether by first‑party or third‑party evaluators. Around the same time, Hugging Face launched Community Evals to decentralize benchmark score reporting on the Hub. Put together, they close the gaps that make it hard for users, researchers, and policymakers to trust, understand, and pick the right evaluations and models.

Evaluation results are how we measure model capabilities, compare models, and reason about safety and governance—yet they remain scattered and hard to compare across papers, leaderboards, blog posts, and harness logs, each in its own format. The same model on the same benchmark can return wildly different scores depending on who ran it and how. Take LLaMA 65B: it has been reported at both 63.7 and 48.8 on MMLU. Those gaps often come from evaluation settings that, as it turns out, are commonly left unreported.

EEE tackles the reporting side. It defines a single JSON schema for an evaluation result that records:

• who ran it
• which model
• how it was accessed
• generation settings
• what the metric actually means
• [recommended] companion JSONL file for per‑sample outputs.

The schema was built with input from researchers and policy researchers, and it ingests results from any source—so harness logs, leaderboard scrapes, and paper numbers all end up in the same shape. Converters, examples, and a contributor guide live in the GitHub repository. Since launch, the EEE datastore on Hugging Face has grown to around 229,000 evaluation results across more than 22,000 models and 2,200 benchmarks, sourced from 31 different reporting formats. To reproduce those runs from scratch would cost somewhere in the hundreds of thousands of dollars—a strong argument against letting the data scatter once someone has already paid to generate it.

Learn more about the schema and how to contribute here.

Now the integration and attribution are tighter. Contributors can send EEE results directly to Hugging Face Community Evals. A converter takes your EEE records and writes the small YAML files that Hugging Face expects—no need to manually maintain the same result in two formats.

This is new functionality for anyone reporting or reading evaluations, not just existing EEE contributors. First‑party evaluators reporting on their own models and third‑party evaluators reporting on someone else’s can both submit to Community Evals and to EEE. Anyone browsing the Hub gets results that trace back to a full record. When you submit data through your organization’s official Hugging Face account, your results appear with a verified checkmark on EvalEval—a clear signal that the numbers come straight from the source. The rest of this post walks through Community Evals and the converter in more detail.

How Hugging Face Community Evals works together with EvalEval

Hugging Face Community Evals works on two fronts.

Each benchmark lives in a dataset repo that self‑registers by adding an eval.yaml. Once registered, that dataset page automatically aggregates and displays a leaderboard of every score reported against it across the Hub. The set of official benchmarks expands over time.

Model scores are stored in .eval_results/*.yaml files inside the model repo. They appear on the model card and feed into the corresponding benchmark leaderboard. Results from the model author and from anyone else via pull requests are all aggregated, with each score carrying a badge indicating whether it’s author‑submitted, community‑submitted, or independently verified. Anyone can submit a score to any model by opening a PR with the correct YAML file, and the model author has control to close PRs or hide results on their own repo.

Here’s an example leaderboard:

Community Evals Leaderboard for Humanity's Last Exam on the Hub

This is where EEE and Community Evals connect. When you send a result to both, two things happen. First, your score appears on the Hugging Face model page and gets included in the benchmark’s leaderboard. Second, it carries a source badge that links directly back to the full EEE record—containing the generation config, harness version, reproducibility notes, and any instance‑level data.

An Evaluation (MMLU-Pro) from EEE Datastore (a) cross‑linked at the file level to a Hugging Face model card (b). The Source EvalEval badge links to the full JSON record.

The two destinations serve different roles toward the same goal. Hugging Face puts your result right where people browse models, with a link back to the source. EEE keeps the full structured record that makes the result interpretable, and powers Eval Cards on top of it. Send your data to both, and the same evaluation becomes both visible and legible at once—which is the whole point of reporting in the first place.

You can see that cross‑compatibility below. The same GPQA scores that surface on the model card above also render in Eval Cards, which compose the EEE run data with benchmark and model metadata into one interpretable record. Same evaluation, different surface:

How it works

Hugging Face stores evaluation scores in the model repo as YAML files under .eval_results/. Only three fields are required: the benchmark dataset, the task, and the value. The source block is what creates the backlink to EEE.

- dataset:
    id: openai/gsm8k
  task_id: gsm8k
  value: 96.8
  date: '2024-07-16'
  notes: '8-shot CoT'
  source:
    url: https://huggingface.co/datasets/evaleval/EEE_datastore/blob/main/flat/objects///.json
    name: EvalEval

The converter automatically fills this in from your existing records. It maps source_data.hf_repo to dataset.id, evaluation_name to task_id, score_details.score to value, and evaluation_timestamp to date, then adds the datastore object URL as the source link to the per‑record EEE JSON. For now, it supports four official benchmarks: MMLU‑Pro, GPQA, HLE, and GSM8K.

The converter does more than just reshape fields. Point it at an EEE datastore collection, and it downloads that collection along with the referenced records, checks object hashes, and finds scores that map to a supported benchmark. Before writing anything live, it audits what already exists: it reads every .eval_results YAML on the model’s main branch and in open PRs, comparing by dataset and task rather than filename. If a score already exists, it’s marked already_present. If a different score exists, it’s flagged as score_conflict. If the model repo can’t be found on the Hub, it’s marked missing_hf_model. Everything else is marked ready.

Nothing gets pushed without your explicit sign‑off. The tool writes local YAML previews and a review file for inspection, shows a report of what’s ready and what needs attention, and only opens PRs after you type OPEN PRS and enter a commit message. Reruns reuse cached results for a collection unless you pass --force.

The converter's review step. Excluded entries (here, models with no matching Hub repo) are listed with their EEE source URLs, and the ready PRs wait on an explicit OPEN PRS confirmation.

Start here

Start by submitting your full records to the EEE datastore.

Using the EEE converter adds only one extra step, and the converter automates most of it. The tool is a vailable in the GitHub repository. To process a collection, run:

uv run tools/hf-community-evals/community_evals_converter.py MMLU-Pro --datastore evaleval/EEE_datastore@main

Review the previews and the report it generates, then type OPEN PRS when you’re ready to submit. Full documentation for the schema, CLI, and converters is at evalevalai.com/every_eval_ever/hf-community-evals.