(A) client-oriented distribution architecture for web search agent

Abstract

Agent-assisted information retrieval has attracted much attention in recent years as the amount of information available on the World Wide Web has increased at a staggering rate. Among the agents deployed on the Web, search agents which queries multiple search engines are gaining popularity among end-users who would rather delegate the task of searching to competent agents instead of searching on their own. Current search agents speed up their search process by exploiting potential parallelism available in accessing distributed databases by submitting the queries in parallel. However, as the number of users and search engines increase, establishing multiple connections to remote search engines has the potential to saturate the network and increase the load on the server running the search agent. Our study focuses on resolving these issues by having multiply replicated agents distributed throughout the network initiate the queries while localizing the network traffic to the client``s machine. Special attention has been made on the design and implementation of a search agent that limits its use of network resources, prevents skewing the logs on the remote sites, and act in accord with the guidelines suggested for robot writers.