The idea, in plain words
Clean without changing the facts
Copied documents often contain repeated footers, broken spacing, and navigation text. Those extra words can compete with the passage that answers a question.
Remove recognizable noise while preserving the facts: X200, prices, dates, and words such as ‘not’. Cleaning is not the same operation as an embedding model's own tokenization.
1
Read the original source, including its repeated boilerplate.
2
Normalize spacing and remove the known repeated footer.
3
Keep product identifiers, table values, negation, and source identity.
4
Compare the useful text with the original before indexing it.