https://docs.google.com/presentation/d/1GyFYRkxcsIRNr8WHu6ocMHV7bU7v2Nalz0PuoTZoBrg/edit?usp=sharing

Kinetic Connections

→ Dynamically controlling a digital marionette through physical hand gestures

Abstract

This project leverages p5.js and ml5.js HandPose to create an interactive marionette simulation, where a puppet is controlled from above by strings or wires attached to its body. Using real-time hand and finger tracking, certain positions of fingertips are connected via strings to the joints of a marionette. The marionette responds naturally to the user’s hand gestures, with specific finger movements and hand configurations controlling specific parts of the arms and legs. The marionette’s motion is lifelike, responding naturally to the forces and dynamics of the strings for an immersive and satisfying experience. This piece encourages users to create their own movements, mimicking a dance or telling a story through the marionette’s gestures.

Inspiration

One of my favorite scenes in The Sound of Music is the marionette performance to “The Lonely Goatherd.” I’ve always been impressed by how well puppeteers bring inanimate objects to life through precise movements. This inspired me to think about the different types of art and performances we create with our hands, and how those gestures could be reimagined in a digital space. I wanted to replicate a satisfying tangible experience while exploring gesture-based interaction + how we can interact with digital environments using physical input.

Audience:

• Artists/performers: people interested in interactive storytelling and performance art
• Casual users: people curious about playful interfaces
• The audience interacts with the piece by holding up their hands. As they move their fingers, the marionette responds dynamically, creating a sense of control and connection, as if physically manipulating the puppet

Challenges: What is your biggest technical and/or conceptual challenge you anticipate?

• Accurate HandPose detection when hand is placed palm down
• Simulating smooth + realistic string dynamics and body movement
• Creating an intuitive interaction that feels natural and responsive

References

For figure + movement:

https://editor.p5js.org/mufaromukoki/sketches/-uxx0lAE0

For string/limb movement, using verlet physics:

https://editor.p5js.org/aatish/sketches/r1oMEgq_G