Web Crawler is also known as

Question

UnlockIAS · Accepted Answer

Correct — C, Web Spider. A web crawler is a program that automatically and systematically browses the World Wide Web, following hyperlinks from page to page so a search engine can index them. Because it moves along the 'web' of links, it is also called a web spider (or spiderbot). Google's crawler, for example, is named Googlebot.

Why the other options are wrong — (a) A link directory is a human-curated, categorised list of websites (like the old Yahoo Directory or DMOZ) — it does not automatically crawl and index pages.  (b) Not a term for a crawler. It hints at SEO (search engine optimisation) — a practice for improving a site's ranking, not the program that indexes the web.  (d) A vague, made-up label; it is not a recognised synonym for a web crawler.

Concept: A web crawler (web spider / bot) is the first stage of how a search engine works. Starting from a list of seed URLs, it downloads pages, extracts the hyperlinks on them, and queues those links to visit next — building an index of the web that the search engine later ranks against a query.

State PCS 2019 Q73 — Science & Technology PYQ with Answer

State PCS 2019 Q73 — Science & Technology PYQ with Answer