The piece of audio visualisation artwork traced and responded to a piece of disappearing music/song in Hong Kong recently. It was a month long exercise(14 June – 13 July 2023) to create simple visualisations of the music using the various operations in TouchDesigner, every day before the music disappears in the public domain.
Unlike musical notations, audio visualisation is an irreversible process. It is quite impossible to recover the original music/song given a few visualisations of the same piece. Nevertheless, the variation in loudness, pitch and perhaps timbre can be visible.
The two artworks are the re-interpretation and re-development of the original computational cinema artwork. The original inspiration came from the legend that when the great Chinese calligrapher, Zhang Xu watched the sword dance of the Lady Gongsun, he invented the wild cursive style calligraphy. With the use of computer vision and basic machine learning technologies, I speculated on the possibility to automatically transform a martial art fighting sequence into cursive style Chinese calligraphy.
Movement in Time, Part 2, 2016 new version
It analyses the fighting sequences from two classical martial art films, Burning of the Red Lotus Monastery, 1963 and Dragon Inn, 1967 and transforms those sequences into cursive style Chinese calligraphy with characters generated from the famous text, Thousand Character Classic.
I developed a custom software in Python to extract the motion data with optical flow analysis. Ten frames of the motion data are summarised into 10 3×3 grid, similar to the one we used to learn Chinese calligraphy in our childhood. The motion data from the fighting sequences is matched against a database of 1,000 Chinese characters written in cursive script style using the same 10x3x3 model. An animation of the matched character is displayed on the screen according to the proper calligraphic stroke order and thickness.
Movement in Time, Part 2, 2022 version
It is an interactive edition of the former one, interpreting the live movement of the audience member and transforms it into cursive style Chinese calligraphy in real time to be shown in the exhibition venue in the Hong Kong Museum of Art.
The underlying mechanism of the interactive version is similar to the cinematic one. Instead of using optical flow to analyse the motion data, the interactive version uses the Google MediaPipe machine learning library for gesture tracking. The captured data is also matched against the same database of 1,000 characters with the 10 frames of 3×3 grid model. In addition to showing the cursive style writing of the Chinese character, the software also simulates the dancing movement of two ribbons attaching to the two hands of the audience member. It creates the intersection of virtual dance movement and real physical body movement, with reference to another performance piece from the Taiwanese Cloud Gate Dance Theatre. Below is a test of concept using the dance sequence from a similar work by the Cloud Gate Dance Theatre related to Chinese calligraphy.
The cursive style Chinese calligraphy of the Thousand Character Classic was written by Ms Lam Lai-sha, Lisa based on the version of Yu Youren.
The tutorial is an updated version of the MediaPipe Pose using the new segmentation mask function to identify the human body tracked. It used the Script TOP to generate the mask image. Users can further enhance the image with the Threshold TOP for display purpose. Similar to the previous tutorials, it assumes the installation of Python 3.7 and the MediaPipe library through Pip.
The source TouchDesigner project file is available in my TouchDesigner GitHub repository. The Python code is relatively straighforward. The pose tracking results will include an array (segmentation_mask) of the size of the tracked image. Each pixel will have a value between 0.0 to 1.0. Darker value will be the background while brighter value will likely be the tracked body. Here is the full listing.
# me - this DAT
# scriptOp - the OP which is cooking
import numpy as np
import cv2
import mediapipe as mp
mp_drawing = mp.solutions.drawing_utils
mp_pose = mp.solutions.pose
pose = mp_pose.Pose(
min_detection_confidence=0.5,
min_tracking_confidence=0.5,
enable_segmentation=True
)
def onSetupParameters(scriptOp):
return
# called whenever custom pulse parameter is pushed
def onPulse(par):
return
def onCook(scriptOp):
input = scriptOp.inputs[0].numpyArray(delayed=True)
if input is not None:
image = cv2.cvtColor(input, cv2.COLOR_RGBA2RGB)
image *= 255
image = image.astype('uint8')
results = pose.process(image)
if results.segmentation_mask is not None:
rgb = cv2.cvtColor(results.segmentation_mask, cv2.COLOR_GRAY2RGB)
rgb = rgb * 255
rgb = rgb.astype(np.uint8)
scriptOp.copyNumpyArray(rgb)
else:
black = np.zeros(image.shape, dtype=np.uint8)
scriptOp.copyNumpyArray(black)
return
Here are two documentation videos of the artwork iFaceDQ. The first one is the demonstration of the use of the Dlib facial landmarks and triangulation. The second one is the demonstration of face swapping with the members of the local band, Mirror.
It is a site specific interactive installation created for the 2nd UNSCHEDULED Art Fair, Karin Weber Gallery. The artwork made use of the mirror of the former changing room of the Top Shop, Central. It performs a face detection to convert the visitor’s face into an animated figure, resembling the yesterday cyberspace creature trapped inside a mirror box. The exhibition period is from 02 to 06 September 2021 in the G/F, 1/F, Asia Standard Tower, 59-65 Queen’s Road, Central.
The example will continue to use a Script CHOP, Python and TouchDesigner for a face detection function. Instead of using the MediaPipe library, it will use the Dlib Python binding. It refers to the face detector example program from the Dlib distribution. Dlib is a popular C++ based programming toolkit for various applications. Its image processing library contains a number of face detection functions. Python binding is also available.
The main face detection capability is defined in the following statements.
This is the last part of the series, using MediaPipe in TouchDesigner. The following example is a continuation of the last post of pose tracking. This version will use a Script CHOP to output the position information of the torso tracked in the film sequence. The output window will display four numbers (11, 12, 23, 24) on the four corners of the torso. The four numbers are the indices of the pose landmarks corresponding to the torso of the body.
The Script CHOP will output 3 channels
pose:x
pose:y
pose:visibility
Each channel has 33 samples, corresponding to the 33 pose landmarks. The visibility channel will indicate how likely the landmark is visible in the image. The following code segment describes how it is done.
xpos = []
ypos = []
visb = []
if results.pose_landmarks:
for p in results.pose_landmarks.landmark:
xpos.append(p.x)
ypos.append(p.y)
visb.append(p.visibility)
tx = scriptOp.appendChan('pose:x')
ty = scriptOp.appendChan('pose:y')
tv = scriptOp.appendChan('pose:visibility')
tx.vals = xpos
ty.vals = ypos
tv.vals = visb
scriptOp.rate = me.time.rate
scriptOp.numSamples = len(xpos)
The final TouchDesigner project folder MediaPipePoseCHOP is now available in the GitHub repository.
The project does not resize the original film clip with a Resolution TOP. It performs the resize function within the Python code in the Script TOP with the OpenCV function cv2.resize(). Each pose detected will generate 33 pose landmarks. The details can be found from the following diagram.
Image from the Google MediaPipe
Together with the original video image, the drawing utility will generate the pose skeleton with the following code segment.
The final TouchDesigner project is available in the GitHub folder as MediaPipePoseTOP. Owing to file size and copyright concerns, the two film clips are not included in GitHub.
The following example presents a more general approach to obtain the hand tracking details in a Script CHOP. We can then use other TouchDesigner CHOPs to extract the data for visualisation.
For simplicity, it also detects one single hand. For each hand tracked, it will generate 21 landmarks as shown in the diagram from the last post. The Script CHOP will produce 2 channels, hand:x and hand:y. Each of the channel will have 21 samples, corresponding to the 21 hand landmarks from MediaPipe. The following code segment describes how it is done.
detail_x = []
detail_y = []
if results.multi_hand_landmarks:
for hand in results.multi_hand_landmarks:
for pt in hand.landmark:
detail_x.append(pt.x)
detail_y.append(pt.y)
tx = scriptOp.appendChan('hand:x')
ty = scriptOp.appendChan('hand:y')
tx.vals = detail_x
ty.vals = detail_y
scriptOp.numSamples = len(detail_x)
scriptOp.rate = me.time.rate
The TouchDesigner project also uses Shuffle CHOP to swap the 21 samples into 21 channels. We can then select the 5 channels corresponding to the 5 finger tips (4, 8, 12, 16, 20) for visualisation. The final project is available for download in the MediaPipeHandCHOP2 folder of the GitHub repository.
