Web Crawler: What Is a Web Crawler?

A web crawler is a computer program that browses and indexes content on the World Wide Web. Search engines use web crawlers to provide up-to-date search results. Also known as a spider.