This paper contributes to the field of physical human-robot collaboration. We present a complete control framework, which aims at making humanoid robots capable of carrying objects together with humans. First, we design a template identifying the primitive subtasks necessary for collaborative carrying. Then, these subtasks are formulated as constrained optimization problems for controlling the whole-body motion of a humanoid robot. The subtasks include two walking pattern generators that account for physical collaboration, as well as posture and grasping controllers. Finally, we validate our framework in a variety of collaborative carrying experiments, using the HRP-4 humanoid robot.