In this paper, a vision based tracking controller with adaptation to uncertainty in depth information is presented. Depth uncertainty plays a special role in visual tracking as it appears nonlinearly in the overall Jacobian matrix and hence cannot be adapted together with other uncertain kinematic parameters. We propose a novel parameter update law to update the uncertain parameters of the depth. It is proved that system stability can be guaranteed for the visual tracking task in presence of uncertainties in depth information, robot kinematics and dynamics. Simulation results are presented to illustrate the performance of the proposed controller.