Reading

1. Mixing movement and machine by Maya Man.

   -Using technology, PoseNet, as a means to convey storytelling and meaning

2. Humans of AI by Philipp Schmitt.

   -COCO: Common Objects in Context

   -Dataset with 328000 annotated images containing “91 object types that would be easily recognizable by a 4 year old.”

   -Images scraped from Flickr

   -Avoided images of single objects in isolation, wanted context

   -AI is not “magical”… COCO restores context to images, crediting the photographers and workers behind AI systems

3. Open Sourcing the Origin Stories: The ml5.js Model and Data Provenance Project by Ellen Nickles

   • What questions do you still have about the model and the associated data? Are there elements you would propose including in the biography?
     • Potentially add more details on the geographic/demographic diversity of the collectors/developers
   • How does understanding the provenance of the model and its data inform your creative process?
     • Adds context on how the model might influence/limit creative output
     • Allows artists to understand any ethical concerns before incorporating into their work

Making

For this week’s assignment, I was inspired by the filters that rearrange your face(ex. nose for eyes, eyes for nose). I decided to make a filter that replaces your eyes with your mouth and your mouth with your eye.

I started out trying BodyPose before realizing that it was pretty limited, with only 5 facial keypoints. BlazePose, on the other hand, has 11 facial keypoints.

I first had to detect the positions of the eyes and mouth. With BlazePose, I was impressed by the precision(3 keypoints for eyes and 2 for mouth).

I then did some research to find the copy() function, which can copy rectangular regions of an image/video and draw them in specified positions.

// mouth to left eye
    copy(video, leftMouth.x, leftMouth.y-mouthHeight*mouthScale/3, mouthWidth, mouthHeight, leftEyeInner.x - mouthWidth/2, leftEyeInner.y-mouthHeight*mouthScale/2, mouthWidth*mouthScale, mouthHeight*mouthScale)
    
    // mouth to right eye
    copy(video, leftMouth.x, leftMouth.y-mouthHeight*mouthScale/3, mouthWidth, mouthHeight, rightEyeOuter.x - mouthWidth/2, rightEyeInner.y-mouthHeight*mouthScale/2, mouthWidth*mouthScale, mouthHeight*mouthScale)
    
    // right eye to mouth    
    copy(video, rightEyeInner.x-rightEyeWidth, rightEyeInner.y-rightEyeHeight*scale/4, rightEyeWidth, rightEyeHeight, rightMouth.x, leftMouth.y-rightEyeHeight*scale/2, rightEyeWidth*scale, rightEyeHeight*scale)
    
    

The most difficult part was figuring out the positioning and doing all the math. The copied areas move with your face.

Mouth for eyes

Eye for mouth

I also added a slider(scale 1 to 3) to adjust the size of the mouth.

Link to Code

Since copy() can only take rectangular sections, I’d want to next work on isolate the shapes of my mouth/eyes more, sort of what BodySegmentation can do but I’m not sure it’d be precise enough? Overall I had fun experimenting this week. It’s cool too see how precise the keypoints are, especially with BlazePose.