Joint flexibility is an important factor to consider in the robot control design if high performance is expected for the robot manipulators. In this paper, we propose an adaptive tracking control method which can deal with the kinematics uncertainty and uncertainties in both link and actuator dynamics of the rigid-link flexible-joint (RLFJ) robot system. Adaptive observers are designed to avoid acceleration measurements due to the fourth-order overall system dynamics. Convergence of both end-effector tracking errors and observing errors are proven and sufficient conditions are presented to guarantee system's asymptotic stability.