Web Crawler is also known as
- (a)Link Directory
- (b)Search Optimizer
- (c)Web Spider
- (d)Web Manager
Correct — C, Web Spider. A web crawler is a program that automatically and systematically browses the World Wide Web, following hyperlinks from page to page so a search engine can index them. Because it moves along the 'web' of links, it is also called a web spider (or spiderbot). Google's crawler, for example, is named Googlebot.
- (a)Link Directory — A link directory is a human-curated, categorised list of websites (like the old Yahoo Directory or DMOZ) — it does not automatically crawl and index pages.
- (b)Search Optimizer — Not a term for a crawler. It hints at SEO (search engine optimisation) — a practice for improving a site's ranking, not the program that indexes the web.
- (d)Web Manager — A vague, made-up label; it is not a recognised synonym for a web crawler.
A web crawler (web spider / bot) is the first stage of how a search engine works. Starting from a list of seed URLs, it downloads pages, extracts the hyperlinks on them, and queues those links to visit next — building an index of the web that the search engine later ranks against a query.
The name 'spider' is a metaphor: the program travels across the 'web' of interlinked pages. Ignore the SEO-flavoured decoy ('Search Optimizer') — crawling (indexing the web) and SEO (improving rankings) are different things.
- Web crawler = web spider = spiderbot — an automated program that indexes web pages
- It follows hyperlinks from page to page, starting from seed URLs
- Crawling → indexing → ranking is the core search-engine pipeline
- Google's crawler is called Googlebot
- Start from a list of seed URLs
- Fetch (download) a web page
- Parse the page and extract its hyperlinks
- Queue the new links and index the content, then repeat
The crawler loops across the 'web' of links — hence the nickname 'spider'.
- Choosing 'Search Optimizer' — that points at SEO, not the crawler itself
- Confusing a curated 'link directory' with an automated crawler
IT-terminology questions ask for synonyms or definitions ('X is also known as…'). Lock the synonym set: web crawler = web spider = bot.
No directly related past PYQ was found.
- practice — not a real PYQ
Googlebot is an example of a:
- (a)Firewall
- (b)Web crawler
- (c)Spreadsheet program
- (d)Operating system
Answer(b) Web crawler — Google's program for indexing web pages.
- practice — not a real PYQ
The main purpose of a web crawler is to:
- (a)Protect a computer from viruses
- (b)Systematically browse and index web pages for a search engine
- (c)Design web pages
- (d)Compress large files
Answer(b) Systematically browse and index web pages for a search engine.