Skip to content

Reserve a held-out set for evaluating a coding rule

Protect a separate review set from iterative rule development so an apparent improvement is not merely better fitting the examples already inspected.

Guides

How should a team check a revised classification rule?

A team classifying public comments about app accessibility may refine its categories after every difficult example. That is useful development work, but performance on those same examples no longer provides an independent check. Reserve a separate eligible set before revising the rules, and keep its contents outside the rule-writing discussion until evaluation.

Example research prompt
Apply the frozen accessibility coding rule to the supplied held-out conversations. Keep development examples separate and report unresolved category disagreements without universal accuracy claims.
01 / Guides

Partition before refinement

Assign eligible records to a development set and a held-out set using a declared procedure. Keep replies from the same conversation together to reduce leakage of near-identical context. Exclude records lacking essential context from both sets under the same rule, documenting those exclusions rather than removing them only from evaluation.

02 / Guides

Develop with visible cases

Use the development set to clarify categories such as navigation barrier, text legibility and assistive technology interaction. Preserve examples and counterexamples in the rulebook. Do not preview held-out passages to solve a category dispute, because that would convert them into development material even if the team avoids changing their formal label.

03 / Guides

Evaluate the frozen rule

Apply the final definition to the held-out set once and retain disagreements. Inspect whether errors cluster around a particular category or source format. If the set is small or manually selected, describe the results as a local evaluation rather than a universal accuracy claim about every accessibility conversation.

04 / Guides

Handle another revision honestly

Deliver the rule version, partition identifiers and evaluation limits. If evaluation leads to a substantive revision, mark the current held-out set as consumed and seek a new independent set for a later check. Repeated tuning against the same examples cannot remain an independent validation merely because the original split still exists.

Keep the boundaries clear.

  • Conversations should stay together when splitting closely related evidence.
  • Platform badges describe research context, not guaranteed access to evidence on that platform.

Start with a read-only check: ask your agent to report actual source coverage before collecting records or exporting an audience.

Primary references

Documentation reviewed 2026-11-13. Product capabilities, client availability and plans can change. Third-party marks identify their products, not an affiliation.

Know what social media shows about you.
Start with a question.

Get started
montage