Find duplicate evidence before it fills your context window.

Paste a query and its retrieved chunks. ContextDeduper estimates lexical overlap, identifies likely near-duplicates, and prepares a compact draft for human review.

This is a lexical preflight, not semantic truth. Similar wording can carry different meaning, and paraphrases can evade lexical checks.

Analysis & results

LEXICAL / LOCAL

Add at least two chunks, then run the preflight. Results stay in this tab.

Token estimate 0

Flagged token share 0%

Recommendation 0 keep / 0 remove

Chunk recommendations

Thresholds: Jaccard 0.58 or cosine 0.82

Pairwise lexical overlap

Top 30 highest-overlap pairs; additional pairs are omitted

Pair	Jaccard	Cosine	Signal

Unique terms in kept chunks

None found.

Cleaned context draft

0 estimated tokens removed

The flagged token share counts the full estimated size of every remove candidate, not only its overlapping words. Always inspect removed chunks: lexical overlap is not proof that two chunks are semantically interchangeable, accurate, or relevant.

How the preflight works

Text is normalized into lexical terms inside your browser.
Every chunk pair receives Jaccard and term-frequency cosine scores.
A query-aware greedy pass keeps distinct chunks and flags high-overlap candidates.
The tool estimates removed context and composes an editable draft.

What it cannot decide

Whether a paraphrase carries the same claim.
Whether a source is correct, current, or trustworthy.
Whether domain jargon makes useful chunks look repetitive.
Whether the cleaned context improves model answers.

Frequently asked questions

Does ContextDeduper use embeddings or an LLM?

No. The current version uses local lexical token overlap and term-frequency cosine similarity. That makes it fast and private, but it cannot detect every semantic duplicate.

Is the token count exact?

No. It is a rough character-based estimate across languages. Use your model provider's tokenizer for billing or hard context limits.

Can I paste confidential retrieved text?

The current tool processes inputs in your browser and does not send them to our server. Still follow your organization's device and data-handling policy.

Should every flagged chunk be removed?

No. Review each recommendation. Similar language may contain a distinct date, exception, source, or claim that the lexical analysis cannot understand.