This is a video I created to demonstrate a concept for a live video system I’d like to use in collaboration with a dance performance. (Actually, I have an HD production version implemented, but it doesn’t illustrate the concept as well as this side-by-side video.)
But in the video here we see a live capture of the screen. On the right, you can see the camera input of the real time motion. This video is captured into a rolling buffer at 175fps, but played out on the right in real time at 60 fps. (Some frames get skipped to keep it live.) On the left, we see the more interesting output: We scrub through the buffer forwards and backwards, fast or slow-motion. The changes in speed are triggered in sync with the music, creating an effect that I really love. Some effects are also synchronized with the music.
Slowing down a live camera input is by no means a new idea. Heck, I implemented such a system with Jitter at the Prada Beverly Hills store more than 10 years ago. But I’ve never seen this done as elegantly and crisply as I’d like.
Typically in the A/V world, as a video signal passes through multiple pieces of gear, we add latency. That means that a live camera shot of someone clapping on stage might correspond to a live video behind them (IMAG, or “Image Magnification”) of that person clapping 1/10 of a second later. Usually this isn’t such a big deal, but for the Live Time Stretching system, I want the sync between audio and video to be much tighter. The great thing about this system is that it uses a live camera input, but it uses video triggers that can be anticipated earlier than the music actually plays. So while it may be that the video passes through a number of processors on its way to a screen, and the video itself is delayed by maybe 1/10 of a second, the changes in playback speed and direction get triggered 1/10 of a second early, so the effect appears to be perfectly synchronized.
We’re still on the lookout for a dance company that would like to collaborate on a piece that uses this technology. If you think it’s interesting, I hope you’ll let us know!