In past decades, much progress has been obtained in vision-based robot control theories with traditional image processing methods. With the advances in deep learning based methods, Convolutional Neural Network (CNN) has now replaced the traditional image processing methods for object detection and recognition. However, it is not clear how the CNN-based methods can be integrated into robot control theories in a stable and predictable manner for object detection and tracking, especially when the aspect ratio of the object is unknown and also varies during manipulation. In this paper, we develop a vision-based control method for robots with an eye-in-hand configuration, which can be directly integrated with existing CNN-based object detectors. The task variables are generated based on parameters of the bounding box from the output of any real-time CNN object detector such as You Only Look Once (Yolo). To address the chattering problem of bounding box, Long Short-Term Memory (LSTM) is used to provide smoothed bounding box information. A vision-based controller is then proposed following task-space motion control design formulation in order to keep the object of unknown aspect ratio in the center of field of view of the camera. The stability of the overall closed-loop control system is analyzed rigorously using Lyapunov-like approach. Experimental results are presented to illustrate the performance of the proposed CNNbased robot controller.