Detect encoding problems before a research table is imported — what should the operator record?
An encoding mismatch can corrupt titles, alter identifiers or hide a special marker in the first column name. The resulting table may look almost correct while joins fail. Validate the file's declared representation and retain an untouched permitted copy so a decoding fix can be explained rather than improvised.
Check decoding and header interpretation without guessing lost characters from context.
Capture declared charset, observed byte markers and export format in the manifest without assuming every text file uses the host's default encoding. A declared encoding and observed byte marker can disagree; preserve that diagnostic fact and stop ambiguous decoding rather than choosing whichever output looks most readable.
Record decoding_revision and unresolved failures, withholding a file whose identifiers cannot be recovered faithfully from permitted evidence. A checked artifact should carry its encoding contract so a later operator does not introduce the same corruption by importing it under an incompatible machine default.Record file encoding evidence
Capture declared charset, observed byte markers and export format in the manifest without assuming every text file uses the host's default encoding. A declared encoding and observed byte marker can disagree; preserve that diagnostic fact and stop ambiguous decoding rather than choosing whichever output looks most readable.
Decode explicitly
Use the known export encoding when available and treat invalid sequences as an error or review state rather than silently replacing characters. Replacement characters can conceal irreversible information loss, especially in opaque identifiers; a successful parser with replacement enabled is not sufficient evidence of faithful recovery.
Test header behavior
Include a labeled fixture with a leading byte-order marker and verify the first logical column name matches the contract after valid decoding. After valid decoding, compare the first parsed header against the exact required schema path, catching invisible prefix characters that can break joins despite an apparently correct display.
Compare recovered values
Check identifiers and non-ASCII text against the authorized source representation, avoiding manual guesses about what corrupted characters probably meant. If the only available copy is already corrupted, do not infer original names from topic context or neighboring rows; leave those values unresolved until legitimately revalidated.
Release the verified artifact
Record decoding_revision and unresolved failures, withholding a file whose identifiers cannot be recovered faithfully from permitted evidence. A checked artifact should carry its encoding contract so a later operator does not introduce the same corruption by importing it under an incompatible machine default.
Keep the boundaries clear.
- Check decoding and header interpretation without guessing lost characters from context.
- Operational record: Record file encoding evidence. Context badges do not certify access.
Start with a read-only check: ask your agent to report actual source coverage before collecting records or exporting an audience.
Primary references
Documentation reviewed 2026-11-15. Product capabilities, client availability and plans can change. Third-party marks identify their products, not an affiliation.