In order to achieve high performance visual servoing, substantial efforts must be made for the calibration of intrinsic parameters of the camera and the transformation matrix as well. In order to avoid the tedious and difficult calibration work, a Kalman filtering-based estimation algorithm is proposed to estimate the composite image Jacobian or the homogeneous transformation matrix on-line, which can reduce the influence of noise. Using the estimated Jacobian matrix, a PD visual controller is used to make features converge to desired values with satisfactory dynamic performance, without a priori knowledge of the kinematic structure and system parameters. The proportional and differential gains are tuned using a genetic algorithm to obtain optimal controller parameters. A series of experiments are performed on peg and hole assembly to investigate the feasibility and effectiveness of this method. (c) 2007 Elsevier Inc. All rights reserved.