The Design and Implementation of web crawler in web-station

• 1、School of Computer Science and Technology, China University of Mining and Technology
• 2、School of Mines, China University of Mining and Technology

Abstract：Search engine has always been focused on improving user’s experiences; the degree of user’s experiences is reflected in three aspects: accuracy, completeness and quickness. Use a jargon saying: precision ratio, the recall ratio and the search speed (i.e. searching-time). One of the most easily achieved aspects is searching speed, because it is difficult for visitors to recognize the speed of a system whose searching time is below 1 second, then it is not being discussed here. For the accuracy of Chinese search engine, the save market must ensure the wanted information will be given to the users ahead of the search results items. This area refers to “the sequence of web pages”, so it is not being discussed in this paper, too. For completeness, some important information must be ensured no missing and latest web pages can be found. This requires the search engine owns a powerful web page collector, which, usually called “web crawler” or “web spider”.This paper presents a method of designing inside-station web crawler. This method which based on certain strategy of crawling various network resources from the Internet can analyze the crawled web pages precisely and then store them into local web page library for searching. This kind of web crawler provides resource bases for realizing the thematically-based, forth-generation search engine.

Keywords： web crawler crawler downloader crawler updater