This is the last part of the series, using MediaPipe in TouchDesigner. The following example is a continuation of the last post of pose tracking. This version will use a Script CHOP to output the position information of the torso tracked in the film sequence. The output window will display four numbers (11, 12, 23, 24) on the four corners of the torso. The four numbers are the indices of the pose landmarks corresponding to the torso of the body.

The Script CHOP will output 3 channels

• pose:x
• pose:y
• pose:visibility

Each channel has 33 samples, corresponding to the 33 pose landmarks. The visibility channel will indicate how likely the landmark is visible in the image. The following code segment describes how it is done.

xpos = []
ypos = []
visb = []

if results.pose_landmarks:
    for p in results.pose_landmarks.landmark:
        xpos.append(p.x)
        ypos.append(p.y)
        visb.append(p.visibility)

    tx = scriptOp.appendChan('pose:x')         
    ty = scriptOp.appendChan('pose:y')         
    tv = scriptOp.appendChan('pose:visibility')
         
    tx.vals = xpos         
    ty.vals = ypos         
    tv.vals = visb
         
    scriptOp.rate = me.time.rate         
    scriptOp.numSamples = len(xpos)

The final TouchDesigner project folder MediaPipePoseCHOP is now available in the GitHub repository.

The following example illustrates the Pose Tracking solution in the Google MediaPipe, using TouchDesigner. It will display the tracking result in a Script TOP. Instead of using the live Video Device In TOP, it uses the Movie File In to track the dancing movement from 2 film clips. The project also makes use of a Keyboard In CHOP to switch between the 2 film clips.

The project does not resize the original film clip with a Resolution TOP. It performs the resize function within the Python code in the Script TOP with the OpenCV function cv2.resize(). Each pose detected will generate 33 pose landmarks. The details can be found from the following diagram.

Together with the original video image, the drawing utility will generate the pose skeleton with the following code segment.

mp_drawing.draw_landmarks(
    image, results.pose_landmarks, mp_pose.POSE_CONNECTIONS)

The final TouchDesigner project is available in the GitHub folder as MediaPipePoseTOP. Owing to file size and copyright concerns, the two film clips are not included in GitHub.