Search Engine Robots - How They Work, What They Do (Part I)

Automated robots search engines, sometimes called spiders or crawlers , they are looking for Web pages. How do they work? What do they really do? Why are they important? One might think that all the stories about indexing Web pages to add a search engine databases, that robots will be large and powerful creatures. Wrong. Robots search engines have only basic functionality so that the first readers compared to what they can understand on a Web page. In early readers, the robots can not do certain things. Robots do not understand frames, Flash animations, images or JavaScript. Around who do not want to enter password protected areas and do not click the buttons on everything you have in your site. They can be stopped cold while indexing dynamically generated URL and slowed to a stop with JavaScript navigation. Robots that search engine work? Do robots search engines like automated data applications Trip of the Internet to find information and links. When you submit a site to a search engine for Send the URL of the page, the new has been added to the tail of robot sites to go to the next incursion on the Web. Even if you do not live on a page, many robots find your site because of links to other sites that link to your own. This is one reason why it is important to build his popularity and the link to find links to other sites are now back to you. Upon arrival at the site, the automated robots first check to see if you have a robots.txt file. This file is used to tell robots which areas of the site is outside the boundaries of them. Typically, these May be directories containing only binaries or other files that the robot does not need to worry. Robots collect links to each page they visit, then follow the links to other pages. In this way, follow the links from one page to another. The entire World Wide Web is made up of links, the original idea is that you can follow links from one place to another. That's how robots move. The intelligence to index pages on the Web comes from the search engine engineers, who believe that the methods used to evaluate the information that the search engine to retrieve the robots. When introduced into the search engine databases, information is available for users to find search engines. When a search engine user enters a query into the search engine, is a series of quick calculations to ensure that the search engine shows only the right to deliver results to its visitors the most relevant to answer your search. You can see the pages of your site search engine robots have visited looking for their server logs or the results of his program Register of statistics. Robots show identification when they visited your site, the pages you visit and how often they visit. Some robots are easily identifiable by their user agent, such as Google Googlebot , the second is a bit more obscure, like Inktomi for Slurp . But other robots can be listed in their records, which are not easily identified, and some even appear to be power of man browsers. Along with the identification of different robots, and count the number of visits, the statistics also show that bandwidth aggressive illegal robots, robots May or do not want to go to your site. In the resources at the end of this article, you find sites that lists the names and IP addresses robots search engines to help you identify them. How to read the pages on your site? When the search engine robot visits your page, you can see in the visible text on the page, the content of different labels on the page source code (title tag, meta-tags, etc.) and hyperlinks on your page. In other words, and links that the robot is found, a search engine that determines what the page is about. There are many factors used to determine the issues and each search engine has its own algorithm to evaluate and process information. According to the robot is created in a search engine, information is indexed and then forwarded to the search engine database. The information provided to databases will be part of a search engine and directory ranking process. When the search engine visitor submits their query, the search engine through its Digs database to give the final list displayed on the results page. The search engine update databases at different times. When you're on the search engine databases, robots, claims that go periodically to reflect changes to your pages, and to ensure that you have the latest information. The number of times they are visited depends on how the search engine has its meaning, depending on the search engine. Sometimes the visit robots can not access the Web page you visit. If your site is down, or you experience enormous amounts of traffic, the robot May not be able to access your site. When this occurs, the site can not be re-indexed, depending on the frequency of visits by the robot on your site. In most cases, robots can not access your pages, try again later, in the hope that his site will be available later. * Identify resources Spider - Search Engine Watch http://searchenginewatch.com/webmasters/spiders.html Robotstxt.org * List of robots and protocols for the establishment of a robots.txt file. http://www.robotstxt.org/ * Spider-Mat training, discussion forums and articles on spiders search engines and Search Engine Marketing. * Http: / / Spider-food.net / Spiderhunter.com articles and resources on monitoring robots search engines. http://www.spiderhunter.com/ Sim * Search Engine Search Engine Spider Simulator Robot World that simulates a spider robots that search engines to read your page. Http://www.searchengineworld.com/cgi-bin/sim_spider.cgi Daria Goetsch is the founder and Search Engine Marketing Consultant for Search Innovation Marketing, a Search Engine Optimization http://www.searchinnovation.com serves small businesses. It specializes in marketing search engines since 1998, including three years as a specialist search engines O'Reilly Media, Inc., a technical publishing company. Copyright? Search Marketing Innovation 2002-2005. http://www.searchinnovation.com http://www.searchinnovation.com All rights reserved. Permission to write this article is granted if the article is reproduced in its entirety, without editing, including the exchange of information. Include a hyperlink to http://www.searchinnovation.com http://www.searchinnovation.com when you use this article in newsletters or online.