The pervasive searches on the Web are various methods of cataloging and indexing documents by topic. The biggest problem with thematic catalogs is the inability to track a huge number of variable resources on the Internet. Thematic catalogs are an inevitable way of searching the Internet, which is only complemented by the use of search engines.
Search engines are used to find specific content that interests users. Each of these machines maintains its own database, and the user searches for that database and gets addresses that mention the information they are interested in. The success of the search depends on how many of the key words the user has given match the words in the documents in the database. The programs of these machines, computerized robots called spiders (spiders, Crawlwers, Web robots), wander continuously over the Web and find new sites, update existing ones, delete obsolete and classify the pages found. They require minimal human intervention.
The search engine consists of three main modules: a spider, an indexer, and a server that answers queries.
The spider starts with the initial set of pages (URL list). Pages from the URL list are supplied by one supplier who finds the appropriate document on the Web, downloads it and puts it in the list of processing documents. The documents in this list are supplied by one by one a parser that analyzes them and the terms contained on it along with the address are given to the indexer who indexes them by terms and forms the database with the notions of their addresses. The addresses of the new documents found in the clearing process are placed in the address list by the cleaver. (URL list)
A good URL definitely attracts Googlebot! In addition, it makes it easy to search and find your site and content, and the URL also appears as part of the search result itself along with the page title. Pay attention to this:
– URL should contain words
– use the directory structure to let users know where they are on your site
– Do not use long URLs with unnecessary parameters
– do not use generic namespaces such as “page1.html”
– Do not repeat the keywords in the URL
– Do not appoint pages or directories to words that have nothing to do with what is in them
– Do not use the same URL for different pages
– Do not use strange capitalization of letters within a URL
Some pages on the Internet cannot be searched in this way. Pages that can not be downloaded by spiders can be divided into 3 main categories
– Pages protected search standard
– Pages that can not be reached by links from other parties
– Pages hidden by protective walls