Invention Analysis Benchmark

Question

Can AI systems produce invention analyses that are accurate, complete, and reliable enough for an attorney to use as a genuine starting point — not just plausible-sounding text?

Task Definition

TBD — the precise task definition (what input a system receives, and what output it must produce) is still being designed.

Dataset / Inputs

The benchmark is built around a focused set of patent publications — currently planned at approximately 12 — selected to represent a range of invention types. The exact publications, selection criteria, and dataset structure have not yet been finalized.

Gold Standard

TBD — the Laboratory is still determining how gold-standard invention analyses will be created and by whom. This section will eventually distinguish the reference answer used for comparison from the evaluation rubric used to judge a system's output.

Method

TBD — the evaluation procedure, including how a system's output is compared against the gold standard and how disagreement between evaluators is resolved, has not yet been finalized.

Limitations

TBD — like any benchmark, this one will have limitations: dataset size, subject-matter coverage, evaluator agreement, and how well performance here predicts performance on other legal tasks. Those limitations will be documented explicitly once the benchmark exists in a form that has them.