How should a team check a revised classification rule?
A team classifying public comments about app accessibility may refine its categories after every difficult example. That is useful development work, but performance on those same examples no longer provides an independent check. Reserve a separate eligible set before revising the rules, and keep its contents outside the rule-writing discussion until evaluation.
Apply the frozen accessibility coding rule to the supplied held-out conversations. Keep development examples separate and report unresolved category disagreements without universal accuracy claims.Partition before refinement
Assign eligible records to a development set and a held-out set using a declared procedure. Keep replies from the same conversation together to reduce leakage of near-identical context. Exclude records lacking essential context from both sets under the same rule, documenting those exclusions rather than removing them only from evaluation.
Develop with visible cases
Use the development set to clarify categories such as navigation barrier, text legibility and assistive technology interaction. Preserve examples and counterexamples in the rulebook. Do not preview held-out passages to solve a category dispute, because that would convert them into development material even if the team avoids changing their formal label.
Evaluate the frozen rule
Apply the final definition to the held-out set once and retain disagreements. Inspect whether errors cluster around a particular category or source format. If the set is small or manually selected, describe the results as a local evaluation rather than a universal accuracy claim about every accessibility conversation.
Handle another revision honestly
Deliver the rule version, partition identifiers and evaluation limits. If evaluation leads to a substantive revision, mark the current held-out set as consumed and seek a new independent set for a later check. Repeated tuning against the same examples cannot remain an independent validation merely because the original split still exists.
Keep the boundaries clear.
- Conversations should stay together when splitting closely related evidence.
- Platform badges describe research context, not guaranteed access to evidence on that platform.
Start with a read-only check: ask your agent to report actual source coverage before collecting records or exporting an audience.
Primary references
Documentation reviewed 2026-11-13. Product capabilities, client availability and plans can change. Third-party marks identify their products, not an affiliation.