A Web spider is a computer program that browses the WWW in a disciplined, automated manner. Other terms for Web spiders are ants, regular indexers, bots, and worms[1] or Web spider, Web robot, or—especially in the FOAF community.
This method is called Web crawling. Many sites, in particular search engines, use spidering as a means of offering up-to-date data. Web spiders are mostly used to create a copy of all the visited pages for later processing by a search engine that will index the downloaded pages to offer fast searches. Spiders can also be used for automating repairs tasks on a Web site, such as checking links or validating HTML code. Also, spiders can be used to gather specific types of information from Website pages, such as harvesting e-mail addresses.
A Website crawler is one type of software agent. In general, it starts with a list of URLs to visit, called the seeds. As the spider visits these URLs, it identifies all the hyperlinks in the page and adds them to the list of URLs to visit, called the crawl frontier. URLs from the frontier are recursively visited according to a set of policies.
Subscribe to:
Post Comments (Atom)
Very well explained. Your blog provides a good learning material for all the SEO beginners.
ReplyDeleteThis comment has been removed by the author.
ReplyDeleteFrom the scratch,your subjects briefed in the article "much attractive" and great...
ReplyDeleteIndia's No.1 Textile Technicians Webportal
Hat’s off. Well done, as we know that “hard work always pays off”, after a long struggle with sincere effort it’s done.
ReplyDeleteYour Post is very informative and easy to learn... keep posting...
ReplyDeleteseo india
I appreciate the effort of the author. I found it is an informative post on bootstrap technology. I hope you shared this kind of blog in future to help both the developer and business people.
ReplyDeleteWoocommerce Development Company
Convert HTML to Wordpress Theme
PSD to Wordpress Conversion Services
Woocommerce Development Services
Wordpress Customization Service