Important Notice: Our web hosting provider recently started charging us for additional visits, which was unexpected. In response, we're seeking donations. Depending on the situation, we may explore different monetization options for our Community and Expert Contributors. It's crucial to provide more returns for their expertise and offer more Expert Validated Answers or AI Validated Answers. Learn more about our hosting issue here.

How are near duplicates determined?

April 26, 2017duplicates near

0

Posted

How are near duplicates determined?

1 Answer

0

Posted

Briefly, the process involves finding how similar each pair of documents is. Documents are broken into “shingles” where a shingle is a set of consecutive words. For example, if the shingles are three-words long, then words 1-3 would be the first shingle, 2-4 would be the second, and so forth. The similarity of two documents is measured as the number of shared shingles divided by the number of unique shingles in the two documents. More similar documents are those that share more shingles.