Deep Dive: Audio-Visual Physics & WebAssembly
This laboratory sandbox demonstrates complex client-side engineering by merging real-time binary parsing, high-performance WebAssembly audio synthesis, and the HTML5 Canvas API into a single, cohesive, sub-millisecond synchronized application.
1. Synthesizing Audio in the Browser
Standard web browsers cannot natively render complex MIDI files utilizing heavy sample libraries. To solve this, we compiled the open-source FluidSynth C engine into WebAssembly (WASM). Running inside an isolated AudioWorklet thread, this engine directly reads the raw `.sf2` SoundFont and `.mid` sequence files from your device's memory, generating CD-quality audio buffers on the fly without causing the main UI thread to stutter.
2. Translating MIDI to Visual Physics
To create the visuals, a custom JavaScript parser performs a "Hex Dump" on the binary `.mid` file before playback begins. It analyzes the Variable-Length Quantities (VLQs) to build an exact "Tempo Map." As the WebAssembly audio engine plays, it sends telemetry data (the current "MIDI Tick") back to the main thread. We use this clock to spawn HTML5 Canvas particles exactly when a "Note On" or "Pitch Bend" event occurs, resulting in frame-perfect audio-visual choreography.
3. Real-Time Hardware Synchronization
One of the biggest challenges in web-based media is A/V sync. The audio hardware clock (WebAudio API) and the visual frame clock (requestAnimationFrame) operate at different speeds depending on the user's monitor (e.g., 60Hz vs 144Hz). This tool employs a dynamic interpolation algorithm that calculates the drift between the CPU's performance.now() clock and the audio engine's tick count, adjusting the speed of the visual particles dynamically so they strike their targets at the precise millisecond the audio transient is heard.
4. Client-Side Processing & Privacy
Adhering to strict privacy and performance standards, this entire physics simulation operates strictly on a Bring Your Own Content (BYOC) basis. When you select a file via the input fields, the FileReader API loads it directly into your device's RAM. No audio files, metadata, or logs are ever uploaded to a remote server. The heavy lifting is done entirely by your local CPU and GPU.