📄 Published in SoftwareX (Elsevier)
HandWind-Sim: Real-time web-based cloth simulation with gesture-driven wind interaction
🔗 DOI: https://doi.org/10.1016/j.softx.2025.102442

---

HandWind-Sim

A real-time, web-based cloth simulation framework with gesture-driven wind interaction.

Overview

HandWind-Sim is an interactive framework that recognizes user hand gestures via a standard webcam to interact with a 3D cloth simulation in real-time, directly in the browser. This project aims to increase the accessibility of complex physics-based interactions for applications in HCI research, interactive art, and virtual prototyping.

Features

• 🖥️ Real-time Web Simulation: Provides a real-time cloth simulation that runs directly in modern web browsers without any installation required.
• 👋 Intuitive Gesture Interaction: Recognizes user hand movements via a standard webcam and translates them into an intuitive wind vector to control the simulation.
• 📊 Interactive Visual Feedback: Offers a visual feedback system for the generated wind (direction and strength) and the tracked hand position, providing an immersive and understandable experience.

How It Works (Architecture)

HandWind-Sim utilizes a pipeline architecture where modules for gesture input, physics simulation, and rendering are organically connected.

1. Gesture Input & Processing: Extracts hand landmarks from the webcam video stream using MediaPipe and analyzes the movement to generate a wind vector.
2. Physics Simulation: Simulates the deformation of the cloth mesh based on the generated wind vector and physical constraints, using a Position-Based Dynamics (PBD) algorithm.
3. Rendering & Visualization: Renders all 3D elements, including the deformed cloth, tracked hand, and wind effects, using Three.js to provide visual feedback to the user.

Requirements

• A modern web browser with WebGL support (e.g., Chrome, Firefox, Edge).
• A webcam.

Installation

1. Clone this repository:
2. Navigate to the project directory:
3. Open the index.html file in your web browser.
   • For the best experience and to avoid potential browser security issues with local files, it is recommended to use a local server. A simple way is to use an extension like Live Server in Visual Studio Code.

Demo Video

handwind_sim_mov1.mp4

See HandWind-Sim in action.

Acknowledgements

• The cloth simulation physics is based on the work of Matthias Müller et al. on Position-Based Dynamics.
• Hand tracking is powered by Google's MediaPipe.
• 3D rendering is done with the amazing Three.js library.