Skip to content

Deduplicate social research records

Separate repeated records, repeated authors and cross-posted messages while preserving original evidence and platform-specific identities.

Guides

Are you counting records, messages or distinct people?

Duplicate handling changes the meaning of a research count. Two retrieved copies of one post are different from two comments by the same account, and neither establishes that matching handles on different platforms are one person. Choose the unit before deduplicating, then preserve enough provenance to explain every merge or exclusion.

Deduplication prompt
Deduplicate repeated source records using platform and original identifiers where available. Report records and distinct platform accounts separately, preserve original references and ranks, and do not merge cross-platform handles without evidence or fetch replacement accounts.
01 / Guides

Define the counting unit

State whether the output counts source records, content items, platform accounts or independently established people. An audience usually needs distinct account identities, while a content brief may legitimately keep several posts by one author. Do not remove repeated authors merely because you intended to remove duplicate rows; those are separate operations with different effects.

02 / Guides

Use identifiers before text similarity

Where available, compare the platform and original record identifier or canonical source URL to identify repeated retrieval of the same item. Retain the observation context when it matters. Similar wording can be a quote, a common question or a separate event, so do not merge independent records solely because an agent finds their text alike.

03 / Guides

Keep cross-posts and identity links explicit

Label cross-posted content as related evidence without inventing shared ownership between accounts. Require inspectable evidence before collapsing cross-platform identities. Keep the original source references even if the presentation groups repeated wording into one theme. This allows reviewers to distinguish a theme's message frequency from the number of independently observed authors.

04 / Guides

Publish an auditable count

Report the starting record count, the counting unit and why exclusions were made. Keep original ranks or references when the task requires a frozen ordering. Do not fetch replacements just to recover the old count after removing a duplicate. If deduplication creates a shortfall against a requested audience size, state the new actual count and the remaining gap.

Keep the boundaries clear.

  • Repeated authors and duplicate records are not the same thing.
  • A deduplicated shortfall must remain visible.

Start with a read-only check: ask your agent to report actual source coverage before collecting records or exporting an audience.

Primary references

Documentation reviewed 2026-11-13. Product capabilities, client availability and plans can change. Third-party marks identify their products, not an affiliation.

Know what social media shows about you.
Start with a question.

Get started
montage