In current interactive television schemes, the viewpoints should be manipulated by the user. However, there is no efficient method, to assist a user in automatically identifying and tracking the optimum viewpoint when the user observes the object of interest because many objects, most often humans, move rapidly and frequently. This paper proposes a novel framework for determining and tracking the virtual camera to best capture the front of the person of interest (PoI). First, one PoI is interactively chosen in a segmented 3D scene reconstructed by space carving method. Second, key points of the human torso of the PoI are detected by using a model-based method and the human’s global motion including rotation and translation is estimated by using a close-formed method with 3 corresponding points. At the last step, the front direction of PoI is tracked temporally by using the unscented particle filter (UPF). Experimental results show that the method can properly compute the front direction of the PoI and robustly track the best viewpoints. However, there is no efficient method, to assist a user in automatically identifying and tracking the opportune viewpoint when the user observes the object of interest because many objects, most often humans , move rapidly and frequently. This paper proposes a novel framework for determining and tracking the virtual camera to best capture the front of the person of interest (PoI). First, one PoI is interactively chosen among a segmented 3D scene reconstructed by space carving method Second, key points of the human torso of the PoI are detected by using a model-based method and the human’s global motion including rotation and translation is estimated by using a close-formed method with 3 corresponding points. At the last step, the front direction of PoI is tracked temporally by using the unscented particle filter (UPF). Experimental results show that the method can properly compute the front directio n of the PoI and robustly track the best viewpoints.