Model Arena

How reliably do language models extract the data from your documents? Measured with Lector: the same documents, the same expected values, every field checked individually.

  • – models compared
  • – generated documents
  • – fields checked

Leaderboard

Which model extracts most reliably?

Ranking by field accuracy …

As of …
Model ranking by field accuracy.
Rank Model
Loading ranking …

Field accuracy = correct ÷ expected fields. More under methodology.

Document examples

Where do the modelsget it wrong?

Generated delivery notes, invoices and claim notifications from the test set with expected values and the models’ actual responses. A selection, not the full dataset. Document content and field names remain in German.

Actual model response from the evaluation

Document

Generated document from the test set. Expected values and responses come from the evaluation in Lector.

Document

Model result

…

    Header data

      Line items

        A field is correct if its value matches the expected value exactly. A missing value counts as an error.

        Methodology

        How We Measurethe Models

        Every model extracts the same documents in Lector.
        Each field is compared with its expected value.

        Task

        • Extraction of every field, including line items on delivery notes and invoices.

        • Classification and more document types will follow.

        Dataset

        Generated documents with expected values:

          Scoring

          • Field accuracy = correct ÷ expected fields. Missing, incorrect and invented values count as errors.

          • Values must match exactly; the order of line items doesn’t matter.

          Evaluation CenterBeta

          Your Documents,Your Leaderboard.

          The leaderboard above comes from Lector’s Evaluation Center. There you measure models, prompts and settings on your own documents, with the same metrics.

          1. Pick a Dataset

            Verified documents with expected values, for example everything marked as gold.

          2. Let Variants Compete

            Each variant, whether a different model or a revised prompt, runs on its own branch against the same dataset.

          3. Compare and Roll Out

            Field accuracy per variant, field and document type side by side. The best variant goes live.

          More about the Evaluation Center
          Example data

          Test the Models on Your Data

          Curious how well the models perform on your data and with your prompts? Sign up now and try it out.