An Effective Approach to Enhancing a Focused Crawler Using Google

Cited 0 time in webofscience Cited 0 time in scopus
  • Hit : 46
  • Download : 0
In this paper, we share our experience in augmenting a focused crawler of our vertical search engine designed to work with academic slides. The goal of the focused crawler was to collect Microsoft PowerPoint files from academic institutions. A previous approach based on a general web crawler can fail to collect a sufficient number of files mainly because of the robots exclusion protocol and missing hyperlinks. As a remedy to these problems, we propose a combinatory approach in which the indexing information maintained by a general web search engine such as Google is utilized for target URL list generation through our query generator, further then complemented by our URL extractor and file downloader. Because Google has already crawled billions of web pages, it will be more cost-efficient and potentially effective to systematically retrieve the desired information from Google than to redo crawling from scratch by ourselves. Our focused crawler, which we call SlideCrawler, has been used for our vertical search engine CourseShare since the fall of 2011. The capability of SlideCrawler was verified for the top-500 world wide universities. SlideCrawler collected about one million files from the top-500 universities. Further, the study results show that SlideCrawler outperforms Nutch, collecting 3.7 times more slide files.
Publisher
SPRINGER
Issue Date
2020-10
Language
English
Article Type
Article
Citation

JOURNAL OF SUPERCOMPUTING, v.76, no.10, pp.8175 - 8192

ISSN
0920-8542
DOI
10.1007/s11227-019-02787-9
URI
http://hdl.handle.net/10203/276610
Appears in Collection
CS-Journal Papers(저널논문)IE-Journal Papers(저널논문)
Files in This Item
There are no files associated with this item.

qr_code

  • mendeley

    citeulike


rss_1.0 rss_2.0 atom_1.0