A major problem of optical microscopes is their small depth-of-field (DOF), which hinders automation of micro object manipulation using visual feedback. Wavefront coding, a well-known method for extending DOF, is not suitable for direct application to micro object manipulation systems based on visual feedback owing to its expensive computational cost and due to a trade-off between the DOF and the image resolution properties. To solve such inherent problems, a flexible DOF imaging system using a spatial light modulator in the pupil plane is proposed. Especially, the trade-off relationship is quantitatively analyzed by experiments. Experimental results show that, for low criterion resolution, the DOF increases as the strength of the mask increases, while such a trend was not found for high criterion resolution. With high criterion resolution, the DOF decreases as the mask strength increases when high-resolution images are required. The results obtained can be used effectively to find the optimum mask strength given the desired image resolution. (C) 2007 Optical Society of America.