Sketch Link: Clear the Fog

Clear the Fog is an interactive web application using machine learning to detect facial expressions and hand movements, simulating fogging up and drawing on a mirror in real time.

For this project, I was inspired by the simple, nostalgic experience of breathing on a cold window/mirror and drawing or writing messages in the fog. I wanted to translate that playful, tactile experience into a digital form in a way that feels fun and satisfying.

Interactions: When users blow on the screen, the screen gets foggy. Users are then able to hold up their hand to wipe the fog away. Over time the fog gradually fades and disappears.

I started out using FaceMesh to track mouth movements. To detect when my mouth was making a blowing motion, I calculated mouth height using points 26 and 36. I then set a threshold(30) to determine if the distance was big enough for my mouth to be open.

I then started to work on creating a fog effect. I started out using simple ellipses but wanted a more organic wavy shape.

I did some research and found these sources…

https://www.youtube.com/watch?app=desktop&v=rX5p-QRP6R4 (sketch)

https://editor.p5js.org/Rico2022/sketches/Gyq9o1U5t

https://editor.p5js.org/jixuansun/sketches/HkKFVvRke

function createFog(mouthCenterX, mouthCenterY) {
  radius += 1;

  if (radius >= 50) {
    radius = 49;
    console.log("fade out fog");
  }

  for (let i = 0; i < 200; i++) {
    let alpha = map(i, 0, 20, 255, 0);
    let weight = map(i, 0, 10, 1, 20);
    fill(255, 255, 255, 100);
    strokeWeight(weight);
    stroke(255, 255, 255, alpha);

    beginShape();
    let xoff = 0;
    for (var a = 0; a < TWO_PI; a += 0.1) {
      let offset = map(noise(xoff, yoff), 0, 1, -25, 25);
      let r = radius + offset;
      let x = mouthCenterX + r * cos(a);
      let y = mouthCenterX + r * sin(a);
      vertex(x, y);
      xoff += 0.1;
    }
    endShape();
  }

  yoff += 0.01;
}

I started out using the Blobby code and combining it with the fade blur edges sketch to create a wavy, misty shape. I then realized I’d want to have multiple blobs, so turned it into a Fog class.

The fog blob grows to a certain radius before slowing losing opacity and disappearing.

class Fog {
  constructor(x, y) {
    this.x = x;
    this.y = y;
    this.radius = 10;
    this.maxRadius = 190;
    this.yOff = random(0.0, 1.0);
    this.alpha = 40;
  }
  // increase size until hitting max size
  // after hitting max size, start fading 
  grow() {
    this.radius += 1;

    if (this.radius > this.maxRadius) {
      this.radius = this.maxRadius;
      this.alpha -= 0.1;
    }
    
    this.alpha = constrain(this.alpha, 0, 40)
  }
  // create shape + blurred edges for fog-like appearance
  display(layer) {
    // blur edges
    layer.fill(255, 255, 255, this.alpha);
    layer.stroke(255, 255, 255, this.alpha);
    layer.strokeWeight(1);
    
    // create organic shape with noise
    layer.beginShape();
        let xoff = 0;
    for (var a = 0; a < TWO_PI; a += 0.1) {
      let offset = map(noise(xoff, yoff), 0, 1, -25, 25);
      let r = this.radius + offset;
      let x = this.x + r * cos(a);
      let y = this.y + r * sin(a);
      vertex(x, y);
      xoff += 0.1;
    }
    layer.endShape();
    
    this.yoff += 0.01;
  }
  // check if fog is faded out
  isFinished() {
    return this.alpha <= 0;
  }

}

Screen Recording 2024-10-16 at 2.41.09 PM.mov

The next step was integrating HandPose. I wanted to simulate how you’d use your hand to wipe glass, so I targeted the upper parts of 3 fingers. I connected those points to create a shape that’d be the “eraser.”

Figuring out how to use my hand eraser to clear parts of a shape was definitely the hardest step. I first tried with a red ellipse.

I figured out that I needed to put the red ellipse on a separate graphics buffer.

I also finally found the erase() function, where erased areas become transparent and reveal the content beneath. This took a very very long time…

Finally, it was time to combine my HandPose and FaceMesh sketches. This is where I ran into trouble getting the erase effect to work. Unlike the red ellipse I drew in setup(), the fog is continuously drawn in draw(). As a result, I couldn’t figure out how to fully erase sections since every frame would redraw the fog over the erased areas.

So… in the current version, users are able to create one fog blob at a time. The fog blob will fade over time. When users hover over the fog with their hand, there is an eraser cut out that reveals the live webcam feed underneath.