The leading search engines, Google, Yahoo! and MSN Live, use crawlers to find the pages for their algorithmic search results. Pages that are already linked from another search engine's indexed pages do not need to be re-submitted because they will be found automatically. Some search engines, notably Yahoo!, operate a paid directory submission service that guarantee crawling for either a set fee or cost per click. Such programs usually guarantee quicker inclusion in the database, but do not guarantee any specific ranking within the search results, just that your site will be indexed.

Yahoo's paid inclusion program has drawn much criticism from both advertisers and competitors. Two major directories, the Yahoo Directory and the Open Directory Project both require manual submission and human editorial reviews before being accepted. Google offers Google Webmaster Tools, to which an XML Sitemap file can be created and submitted for free online to ensure that all pages are found, especially pages that aren't easily discoverable by automatically following links between pages, such as dynamic content.

Search engine crawlers (or robots) take into account a variety of different factors when crawling a site. Not every page is added or indexed by the search engines. Distance of pages (depth) from the root directory of a site may also be a factor in whether or not pages get crawled and indexed.