Showing posts with label Google Process Detecting Duplicate Content. Show all posts
Showing posts with label Google Process Detecting Duplicate Content. Show all posts
Tuesday, February 26, 2008
Google Process Detecting Duplicate Content
New Google Process for Detecting Near Duplicate Content { On Your Page }
A new patent application on near duplicate content from Google explores using a combination of document similarity techniques to keep searchers from finding redundant content in search results.
The Web makes it easy for words to be copied and spread from one page to another, and the same content may be found at more than one web site, regardless of whether its author intended it to be or not.
How do search engines cope with finding the same words, the same content, at different places?
How might they recognize that the pages that they would want to show searchers in response to a search query contain the same information and only show one version so that the searchers don’t get buried with redundant information?
Duplicate and Near-Duplicate Documents on the Web
Sometimes a creator of content might show that content on more than one page on purpose, such as when:
1. They provide a “mirror” of a site - A site, or pages on a site may be copied at different domains to stop potential delays that happen when lots of people attempt to request the same document at the same time, or to keep the delivery of pages from being slow.
2. There are different formatted versions of the document - plain text or HTML or PDF or a separate printable version of the same document may be available for viewers to choose from, and special versions for phones and PDAs may also be available.
3. Sometimes content is shared with other sources, such as news wire articles, or blog posts that are published at both a group blog and an individual’s blog.
Sometimes content may be duplicated at other pages regardless of its creator’s intent, such as when:
1. Someone took some or all of the content for republication pursuant to fair use, or in violation of copyright.
2. The publishing system used shows the content at more than one address on the same site, so that it may appear to be unique based upon being located at a different address.
3. Content was aggregated or incorporated into another source on the Web, such as in a mashup or search results, or in some other form.
There are other instances where content is duplicated on more than one page, or where documents are very similar. It makes sense for a search engine to try not to show the same content over and over again to a searcher in a list of search results.
It’s a challenge that search engineers need to meet carefully, because there are instances where duplicated content is legitimately on the Web, and other times when it is duplicated without permission and regardless of its creator’s copyright.
Recent Google Efforts towards Duplicate and Near Duplicate Content
One of the more interesting papers from Google employees last year gave a very good overview of processes to detect duplicate and near duplicate processes on web pages - Detecting Near Duplicates for Web Crawling (pdf).
In that paper, one of the processes described in detail was developed by Moses Charikar, a Princeton professor, who has worked for Google in the past. Moses Charikar also is listed as the inventor of a Google patent granted early last year, which discusses ways to detect similar content on the Web - Methods and apparatus for estimating similarity
This past week another Google patent application, from Monika H. Henzinger, explores how duplicate and near duplicate content might be detected at different web addresses. The patent application includes references to a number of different previous methods, including Dr. Charikar’s.
Detecting duplicate and near-duplicate files
Invented by Monika H. Henzinger
US Patent Application 20080044016
Published February 21, 2008
Filed August 4, 2006
The patent application explores how some different existing methods for detecting near duplicate content could be used together to try to identify near duplicates on the Web.
It provides citations to a number of documents on the Web that explore the topic of duplicate, and near duplicate content, including the following:
* Finding similar files in a large file system
* Scalable Document Fingerprinting
* Copy Detection Mechanisms for Digital Documents
* Syntactic Clustering of the Web
* Similarity Estimation Techniques from Rounding Algorithms (pdf)
* Similarity search system with compact data structures
* Methods for identifying versioned and plagiarised documents
From those documents, Dr. Henzinger tests and explores the documents from Andrei Z. Broder (Syntactic Clustering of the Web) and Moses Charikar (Similarity Estimation Techniques from Rounding Algorithms), and compares the approaches from each.
While there were differences in how effective these approaches were according to tests run, the conclusion about their effectiveness in the patent application was that “neither of the algorithms worked well for finding near-duplicate pairs on the same Website, though both achieved high precision for near-duplicate pairs on different Websites.”
We’re also told that:
In view of the foregoing, it would be useful to provide improved techniques for finding near-duplicate documents. It would be useful if such techniques improved the precision of the Broder and Charikar algorithms. Finally, it would be useful if such techniques worked well for finding near-duplicate pairs on the same Website, as well as on different Websites.
Using Multiple Similarity Techniques Together
Techniques similar to those described in the documents from Broder and Charikar could possibly be combined to work in sequence, to improve the dectection of similar documents. The patent filing provides a nice overview of how that combined process would work, and why it would be an improvement.
Boilerplate, Similarity Techniques, and Fingerprints
One reason why some of the near-duplicate document detection algorithms perform poorly on pairs of pages from the same site, according to Dr. Henzinger, is because of boilerplate text that appears on those pages. A boilerplate detection process might be used to remove or ignore that boilerplate content in near-duplicate document analysis. I wrote about other reasons why Google might look for boilerplate on pages recently in Google Omits Needless Words (On Your Pages?)
This patent application explored the possibility of using the Broder and Charikar processes together, but it could use other, or additional document similarity techniques.
One approach used to “fingerprint” the content on pages is in creating “tokens” from the content as described in Rabin’s Fingerprinting By Random Polynomials. We’re told that different fingerprinting approaches might also be used, such as those described in the Hoad and Zobel paper Methods for identifying versioned and plagiarised documents.
Conclusion
I wrote about some of the problems around duplicate content on web pages in Duplicate Content Issues and Search Engines, and the post includes links to a number of white papers and patent filings about duplicate content.
The process described in this new patent application doesn’t so much introduce a brand new method of identifying near duplicate content on pages as it comes up with an approach that takes advantage of other detection methods in a new way.
I didn’t go into a lot of specific details on how the different similarity processes work because those are detailed fairly deeply in the papers that I linked to, and the process described in this patent application doesn’t necessarily rely completely upon any one of those processes, but rather on the idea that multiple processes could be used together intelligently.
Seo Updates
Subscribe to:
Posts (Atom)
Subscribe
Labels
- SEM (4)
- SEO updates (3)
- Search engine optimization (3)
- Increase online sales (2)
- Keyword density (2)
- Page Rank (2)
- SEO Activities to Promote your Site (2)
- SEO Tips (2)
- SEO Tips Search Engine Rank (2)
- SEO guidelines (2)
- Search engine marketing (2)
- google seo (2)
- how increase traffic on website (2)
- search engine updates (2)
- seo (2)
- 404 pages Google Toolbar Beta 5 (1)
- AdCenter keyword optimization (1)
- AdSense - Mobile ASense Available (1)
- AdSense skip domains bought (1)
- AdWords Preview results (1)
- Blogs Improve Inbound Links (1)
- Cartel SEO (1)
- Changing Contents without Changing Theme (1)
- Co-citation in SEO google (1)
- Co-citation in SEO language (1)
- Come out from Duplicate Content Filter (1)
- Create Closed SEO (1)
- Create Closed SEO Cartel (1)
- Critical Components Search Engine Ranking (1)
- Demerits of PPC (1)
- Demerits of SEO (1)
- Determining Relevance and Variety (1)
- Do Follow Blog List (1)
- Do-Follow Blog List (1)
- DoFollow (non-nofollow) Forums List (1)
- Domain Tasting (1)
- Drive Traffic Business Web Site (1)
- Duplicate Content Filter Google (1)
- Evolution of Natural Linking (1)
- Fight Duplicate Content (1)
- Free DoFollow Forums site List (1)
- Free Press Release Site list (1)
- Future Of SEO (1)
- Generate High Pagerank Links (1)
- Generate Free High Pagerank Links (1)
- Google AdSense - Mobile AdSense (1)
- Google Adsense (1)
- Google Adsense Secrets (1)
- Google Adsense Tips (1)
- Google Adsense Tricks (1)
- Google Algorithm Vs Seo webmasters (1)
- Google Analytics (1)
- Google Analytics Update (1)
- Google Approach Indexing Stopwords (1)
- Google Omits Needless Words (1)
- Google Page Rank (1)
- Google Process Detecting Duplicate Content (1)
- Google Ranking (1)
- Google SEO Algorithm Problems (1)
- Google Search Update (1)
- Google Semantically Related (1)
- Google Sitelinks Update February 2008 (1)
- Google Sitelinks Updates (1)
- Google Yahoo fight with feds (1)
- Google and Yahoo fight with the feds (1)
- Google and yahoo will work on mobile programs (1)
- Google dance started Feb march 2008 (1)
- Google dance started PR updates (1)
- Google free advertising (1)
- Google kill Domain Tasting (1)
- Google on Reading Text in Images (1)
- Google supplemental results (1)
- Google targets (1)
- Google updates - search results (1)
- Google updates pricing apps (1)
- Google updates pricing for Postini apps (1)
- Google's "Knol" Affect SEO (1)
- Google: SEO (1)
- Google: Webmaster Central (1)
- Hidden Text (1)
- Hidden content (1)
- High PR Social Bookmarking Sites list (1)
- High PR links (1)
- High PR links are shite (1)
- How Blogs Improve Inbound Links (1)
- How Do I Use RSS (1)
- How Search Engines Index Pages (1)
- How do RSS feeds work (1)
- Importance of Link building (1)
- Importance of Linking (1)
- Increase Alexa Ranking (1)
- Increase conversion rate (1)
- Increasing Your Traffic (1)
- Internet cable cut (1)
- Is RSS a Substitute For E-mail (1)
- Key words (1)
- Keyword Matching to Phrase-Based Indexing (1)
- Keyword Proximity (1)
- Keyword prominence (1)
- Keywords seo (1)
- Landing page optimization (1)
- Latent Semantic Indexing (1)
- Link Building Techniques (1)
- Link Building Trick - Get Free 1 Way Links (1)
- Link Exchange Seo Tips (1)
- Link Popularity Process (1)
- MICROSOFT GETS SUSPICIOUS (1)
- Make Money Online Using SEO (1)
- Make User Friendly Website (1)
- Merits of SEO (1)
- Microsoft AdCenter keyword (1)
- Microsoft AdCenter keyword optimization (1)
- Mistakes Search Marketing (1)
- Mistakes Search Marketing Firms (1)
- More info on updates (1)
- More info on updates in February Google (1)
- MySpace Layouts vs Boilerplate Crapola (1)
- New Google Process for Detecting Near Duplicate Content (1)
- Online Message Boards (1)
- Onpage offpage optimization tips (1)
- Organic Search Engine Optimization (1)
- PPC (1)
- PPC Merits (1)
- PR (1)
- PR updation january 2008 (1)
- Page Rank Export List History (1)
- Page Rank Export Table (1)
- Page Rank Export Table Definitions (1)
- Page Rank Updates (1)
- Page rank updation (1)
- PageRank Assumptions (1)
- Paid Inclusion (1)
- Pay per Click Campaign (1)
- Pay per click (1)
- Press Release Distribution Site list (1)
- Promote RSS Feed (1)
- Promote your Business Free (1)
- Quality Web Directories for SEO (1)
- RSS op[timization (1)
- RSS Feed Tips (1)
- RSS Feeds (1)
- RSS and SEO (1)
- RSS for search engine (1)
- RSS for SEO (1)
- RSS optimization (1)
- Reverse Engineering Search Engine Algorithms (1)
- Reverse Engineering algorithms (1)
- Reverse Engineering benefits (1)
- Reverse Engineering pitfalls (1)
- SEO Copywriting tips google yahoo (1)
- SEO Friendly Flash (1)
- SEO Friendly Flash adobe announced (1)
- SEO Impacts online Sales (1)
- SEO Planning (1)
- SEO Strategic Planning (1)
- SEO for You (1)
- SEO services that promise (1)
- SEO suggestions (1)
- SEO suggests (1)
- SEO tricks (1)
- SEO vs PPC (1)
- SEOs Will Game Wikia Search (1)
- SERP updates (1)
- SMO activities (1)
- Search Engine Marketing Skills (1)
- Search Engine Optimization experts (1)
- Search Engine Rank- Google Ranking (1)
- Search Engines Trust Older Websites (1)
- Search Marketers (1)
- Seo google (1)
- Seo progress plan (1)
- Site Analysis (1)
- Sitemaps Protocol Cross Host Support (1)
- Sitemaps Protocol Scaled For Cross-Host Support (1)
- Social Media Optimization activities (1)
- Stop Negative SEO (1)
- Tips Link Builder (1)
- Tips Link Builders Hurt Themselves (1)
- Tools for Monitoring Your Website (1)
- Traffic and Link Building Techniques (1)
- Understanding SEO (1)
- Updates in seo (1)
- Use Digg Get More Traffic (1)
- Web Traffic (1)
- Website Maintenance Services Need (1)
- Website Optimization Process (1)
- Website Promotion Process (1)
- What Are Podcasts (1)
- What Can RSS Help Me Do (1)
- What is RSS (1)
- What is a Feed (1)
- What is a Post (1)
- What is a Squeeze Page (1)
- Where peoples go on google.com (1)
- Who Benefits From RSS (1)
- Who Publishes Content in RSS (1)
- Why SEO (1)
- Wiki Search Engine (1)
- Wikia Launches Search Engine (1)
- Wikia is ever to rival Google (1)
- Wikipedia Vs Knol (1)
- X-Robots tag (1)
- Yahoo Launched Ymail and Rocketmail (1)
- Yahoo Phrase Based Indexing in a Nutshell (1)
- Yahoo Replaces Google PageRank Assumptions (1)
- Yahoo Replaces PageRank (1)
- Yahoo Testing Relevance (1)
- Yahoo on Segmenting Web Sites into Topical Hierarchies (1)
- Yahoo-Google can work together (1)
- adobe announced SEO Friendly Flash (1)
- but don’t deliver (1)
- cable cut network slowdown (1)
- contribute knowledge (1)
- dofollow Social Bookmarking Site list (1)
- domain testing (1)
- eCommerce SEO Tips (1)
- flash index crawl search engine (1)
- flash index crawl search engine google yahoo msn (1)
- free Social Bookmarking Site list (1)
- free advertisement by google (1)
- free adward advertising (1)
- google X-Robots tag (1)
- google Approach Indexing (1)
- google Landing page optimization (1)
- google Stop Negative SEO (1)
- google algoritm problem (1)
- google optimization (1)
- google read image text (1)
- how to Get Free one Way Links (1)
- how to Use Digg Get More Traffic (1)
- improve Alexa Ranking (1)
- in (1)
- keyword analysis (1)
- knlol (1)
- links from blogs (1)
- long keyphrase optimization (1)
- long tail seo (1)
- network slowdown (1)
- offpage optimization (1)
- online advertising network (1)
- onpage optimization tips (1)
- optimization web directories (1)
- pay per click advertisement (1)
- pr updation jan 2008 (1)
- rediff.com (1)
- relative absolute interlinking linking (1)
- relative absolute interlinking seo tips (1)
- search Engine Updation (1)
- search engine position using RSS (1)
- search results updates google (1)
- sem firm india (1)
- seo Landing page optimization (1)
- seo algoritm problem (1)
- seo guideline (1)
- seo india (1)
- services promise but dont deliver (1)
- trafffic Building Techniques (1)
- traffic building (1)
- wikia launch search engine (1)
- will knol affect seo (1)
- x Robots tag (1)
- yahoo.com (1)