An awesome HTML5 interactive music video

Designers and developers are pushing the envelope with what’s possible in interactive design on a continuous basis. And awesome new examples come out all the time.

One of the newest examples is an interactive music video for Evelyn, by ABBY.

It’s a fantastic site that gives you the opportunity to mix different instruments and vocal styles together, while the song is playing, for a completely custom experience.

We asked the developers how they created such an awesome interactive video, and got their tips for working on projects of this type.

1. Where did the idea for the video come from? What was the creative process?

Although we were not directly involved in the creative process we will give you a short summary on how it was created.

We, Steffen & Dominik, just founded a web development agency in Berlin called bleech specializing in HTML5 and other modern web technologies. We have our office in a shared space together with a recording studio which is run by the members of the band ABBY. The space is completed by a booking agency and an iOS agency.

Most of us have known each other since college and we have worked together since then on many different projects in many diverse and creative environments.

The actual idea for the video was developed by a friend of us who studied design. He needed a topic for his thesis and had the vision of creating an interactive experience which lets the user experiment with the different components of a modern song.

2. Can you give a quick overview of the process for actually creating the video, the steps involved, etc.?

One of the biggest challenges for the band was to select the adequate instruments which differ in their sound and characteristics but at the same time have a pleasant sound and harmonize with each other. Additionally they had to ensure that the rhythms and harmonics of the newly integrated tracks did not, at any time, go against each other.

We believe they did a really great job on this.

The entire idea was developed without dealing with any technical requirements regarding web technology. So they started recording one video for each audio track in a historic sound studio in Berlin. This resulted in a total of 20 individual videos, which then had to be merged to make the idea become reality. For this task they asked a befriended Flash developer to make a website where you can control the different tracks and mix your own version of the song. Unfortunately, he faced some difficulties with the syncing process over a network connection, so that early draft never made it to a final release.

That’s when we came in. We felt challenged to prove that it was possible to bring their idea to life with the latest technologies that nobody has yet used in this exact way.

We started building the foundation in a test-driven JavaScript environment to ensure a consistent communication between the videos, our mediacontrollers, and the global timeline module. The mediacontrollers take care of switching the videos and displaying only the currently selected track. The timeline module serves as a reference time for each mediacontroller and syncs them if necessary.

3. What unexpected challenges presented themselves during the project? What advice would you give a developer who wanted to create a project like this?

A tough part in the development process was to keep the videos in sync without doing to many computations and to ensure that even on older computers a good, responsive user experience is possible.

The methods we found out to be most effective were an aggregation of several algorithms which keep the videos in sync and adapt to the machine’s performance by increasing the threshold and the frequency of how often the syncing is triggered.

The biggest challenge though was fine tuning in the millisecond area, as for a song with 120 beats per minute, an offset of 50ms of any of the tracks would clearly be noticed by the listener. Finally, we managed to get all of the audio and video tracks in sync by less than 10ms on high performance computers (like a 2011 MacBook Pro / Air).

If you’re planning on developing a media-driven HTML5 project, be prepared for sleepless nights optimizing small pieces of code, unforeseen browser bugs, and a million possible ways to implement a single function.

4. Where do you see this kind of content going in the next couple of years?

We hope that more developers start to experiment with media-related web projects and hope to see new frameworks being created for media rich applications. Up to this point Java (processing) and Flash still have some advantages for certain use cases.

The most prominent developments in HTML5 at the moment are definitely the audio and video capabilities and we are looking forward to features like the mediacontroller or device element being implemented in new browsers.

With modern server technologies such as websocket, we are waiting to see the delivery of real-time events to the user instead of only serving static, prerecorded content. Additionally, it would be great to see more large scale interactive content that integrates social experience with existing media channels. The usual stuff that everybody is talking about right now…

We are glad to be part of this project and to work with great artists with various expertise. The project has become our little baby and we are really overwhelmed by the positive response.

BTW, we’re looking for people to support our growing team.

What other awesome HTML5 projects have you seen recently? Let us know in the comments!

WDD Staff

WDD staff are proud to be able to bring you this daily blog about web design and development. If there's something you think we should be talking about let us know @DesignerDepot.