History of Technology

History Of Web Crawling

The origins of web crawling began with the first search engines, and their need for indexing Web pages, assigning importance to the relative number of integration points. Search engines refer to this aspect of their architectures by different names. PageRank is what Google uses to describe this technology while DistanceRank is often used to define the spatial and orthogonal aspects of relative search engines (Bidoki, Yazdani, 2008). The intent of this paper is to evaluate of web crawlers are changing society economically, environmentally, ethically and medically.

The Economic Changes from Web Crawling

Most significant have been the changes to how online advertising models are structured and executed. This is certainly the case with how Google has created AdWords, which is predicated on PageRank of specific inquiries or words. An advertiser can "lease" the access of the word when typed in a search query. C Depending on how the web crawler ranks the specific term is how much the advertiser is charged. This has turned into a $1B a year business for Google in just seven years (Thelwall, Stuart, 2006). The greatest economic impact web crawlers are having on society today are the business models they enable. The focus on how to create a web crawling system that can consistently and reliably rank pages with respect to specific pages is critical to the long-term growth of online advertising (Cothey, 2004). There are also the ethical implications of business models based on the intersection of search data and the content of web pages most often shown (Thelwall, Stuart, 2006). Web crawlers can for example be configured to deliver only a specific company's results or to greatly exaggerate the importance of a given term or concept in search results over time. This is where the ethicacy of web crawlers comes into play and is discussed in a later section of this paper. Finally the economic aspects of how web crawlers enable more effective forms of communication in shared networks and over Wi-Fi networks is just beginning to be studied (Bidoki, Yazdani, 2008). The growing reliance on smartphones, Wi-Fi networks and contextual advertising will result in entirely new business models based on web crawler technology as well.

Environmental Considerations of Web Crawling