Works in Progress | Category

Web Audio Editor (update)

Title

Web Audio Editor: A simple audio editor that works in the browser, designed for kids age 7+ (URL // source code)

Description

I teach music workshops at an afterschool program where kids build their own sample libraries from scratch. They find the default audio editor (Audacity) really frustrating, and I’m with them! So I decided to build an audio editor that does all the stuff that kids like to do (like reverse sounds, change the speed, and add reverb). I wanted an easy way to make good sounds, and learn about sound in the process.Even though it was designed for an audience who has never worked with digital sound before, the Web Audio Editor can be of use to anyone who wants effects or just a quick edit. I wanted to do this in the browser, rather than as a standalone application, after researching music education software’s transition to the web through my Technology for Music Education course (at Steinhardt). I also drew inspiration from Pixlr, the online photo editor that I choose over Photoshop for quick little edits. Web audio is still in its early stages, so the pieces necessary to build this weren’t available until very recently. But HTML5, the WebAudio API, and Canvas open up a lot of potential for audio playback and visualizations.

Much of the Web Audio Editor’s functionality comes from SoX, a command line utility that has been around for over 2 decades. Shawn Van Every introduced me to SoX in CommLab Web, and helped me set it up on my server so that I can call it with PHP. The frontend is built off of Wavesurfer, a JavaScript library that visualizes the waveform in a JavaScript Canvas. I also added a Canvas to visualize the frequency spectrum, inspired by ICM and Marius Watz’ Sound as Data workshop. I am continuing to develop this project and would be thrilled to present it at the ITP show.

Personal Statement

My primary inspiration was seeing kids struggle to use Audacity. I wanted to see if I could build something simpler that functions in the browser. I developed the idea in a course at Steinhardt called ‘Technological Trends in Music Education: Designing Technologies and Experiences for Making, Learning and Engagement.’ I made it a project in CommLab Web (just to get the PHP / SoX running) and ICM (to visualize the sound data and build a user interface).

Background

My course at Steinhardt included readings from MIT’s Lifelong Kindergarten group, and I became very interested in Scratch, a visual programming environment for kids. Scratch very effectively teaches programming concepts by giving a space to create and readymade pieces, like lego blocks. Scratch has an audio editor that has been inspirational for this project.On a more technical level, I’ve found several attempts to build a Web Audio Editor, including plucked.js, a thesis project called Audiee (http://www.stud.fit.vutbr.cz/~xmyler00/audiee/#), and a thesis project by Ryan Viglizzo here at ITP. I’ve been in touch with Ryan. I have dug deep into PHP, SoX, CSS, HTML5 / JavaScript / Web Audio / Canvas to figure out how to do a lot of things I had never done before.

The Web Audio Editor was designed for kids in my after-school workshop, so 2nd-8th graders who have short attention spans and haven’t necessarily worked with digital audio before. But I think it’s of use to anybody who needs to make a quick audio edit, or add some fun effects.

I’ve learned a ton! I had never heard of SoX and was new to PHP. Initially I thought I’d be able to use Processing.js and Minim to visualize and scrub through the audio. But I quickly learned that Minim uses JavaSound, and Java is going out of fashion for web browsers. I also learned more about how Processing.JS works and why it won’t really be able to help me in this case. So I dug into HTML5/JavaScript, where I discovered that much of what works in Processing can be recreated in a Canvas, and the WebAudio API works a bit like Minim. I’ve also learned a lot about user testing with different audiences, including the kids in my after-school program as well as my classmates at ITP.

References

Built with SoX, and Waveform.JS a library by Katspaugh. Other libraries include Bootstrap and jquery. The frequency spectrum visualization draws from Boris Smus’ Web Audio API ebook (O’Reilly 2013). Many thanks to Shawn Van Every, Daniel Shiffman, Alex Ruthmann, Sam Brenner, Sam Lavigne, Luisa Pereira, Ryan Viglizzo, Danne Woo, Beam Center and everyone else who has helped me figure out how to do this!

Audio Transformer update

Audio Transformer, the Web Audio Editor, is online in a functional demo mode. It’s not ready for public testing until I prepare my server, and there are many features (and bug fixes) yet to come. But if you’d like to check it out, here it is (source code here)…

I had the chance to user-test with my ICM class, and observed that graduate students tend to start by pressing the button on the left and moving to the right. Only those who really knew what they were doing with audio editing would jump straight to the fun stuff (effects). A few people also thought to press Spacebar for play/pause, but most didn’t think to do that, so I added a “spacebar” text hint to this button. At that point, there was a button to place markers at a spot on the waveform, and everyone wanted click’n’drag to select an area, so I’ve begun to implement this. I also added Loop mode, with a symbol that shows up when loop is on, though if I have the time I’d like to develop my own buttons that look like physical buttons.

“Speed Up/Down” has little effect on the waveform, so there needs to be a way to show that the length of the file is changing, otherwise it doesn’t look like those buttons do anything. I added a timer in the top-right but would like to visualize this more clearly by showing the full waveform in a skinny timeline at the top, and the selected area in the bottom. As the file shortens, the zoom-level stays the same, so the selected area will grow in proportion to the full file. This’ll make a lot more sense if I can just show you what I mean. Other comments were that the frequency spectrum colors didn’t seem to correlate to the waveform colors, and if the colors that represent sound should be link.

Before presenting this to kids in my workshop, I need to indicate when WAE is processing audio, and grey out the buttons so that they can’t freeze the program by overloading the “speed up” button.

I am subbing for my friend’s Scratch video-game workshop, and I had the chance to work on sound effects using the Scratch interface:

Scratch has been a big influence on my approach to “designing for tinkerability”, as has a lot of the projects and research coming out of MIT Media Lab’s Lifelong Kindergarten Group. Its audio editor has a concise, efficient design. They don’t overload the user with too many parameters, for example, they leave things as “louder” and “softer” rather than a volume slider. This is the way I’ve implemented my effects, though I think that in the name of tinkerability, I should not only provide a preset starting point, but also provide an advanced mode for users who wish to dig deeper.

Scratch gives kids three options for sounds: 1. record your own, 2. choose from preset sounds, 3. upload a sound. The kids wanted to try each of these. One group wanted to record their own voices saying “crap!” Another went to YouTube trying to figure out a way to bring the Dr Who theme into their project. And others explored the library of existing sounds. I think that offering all of these starting points would strengthen the web audio editor.

Designing for kids as young as 2nd grade is difficult because they aren’t all able to read at the same level. This applies to words, but it also applies to symbols. For example, when I asked the kids to play a sound in Scratch, some didn’t know what button to press. They hadn’t all been exposed to a sideways triangle symbol as a play button. Even if it said “play” they might not know what it means to “play” a sound. I don’t know if there’s a better way to convey the meaning of these abstract audio concepts, but I think that using the most simple, conventional names and symbols will help establish meaning that will stick with them later in life.

As my Physical Computing teacher Tom Igoe says, there’s no such thing as ‘intuitive’, just learned behavior. So in an educational setting for kids who’ve never worked with audio before, it will be necessary to point out some things.

Just this morning, I had the opportunity to present this project to a 5-year old. At first, thanks to her guide pointing out the foam chair, she was more interested in picking up the foam chair than in working with the Audio Transformer. When she sat down, I gave a short explanation that this is a way to listen to sounds and change the way they sound. I showed her how to click and drag a file from a desktop folder into the browser, then pressed buttons to change the sound. She was much more interested in dragging the sounds than in modifying them. Click’n’drag is a difficult skill for novice computer users, but she told me she’s been working on it with her dad, and she seemed intent on mastering it now. The dragging distance proved too far for her to manage, so I helped load the sound and then encouraged her to try pressing the buttons. She didn’t understand which button to press to play the sound until I pointed it out, and from there she successfully slowed down and reversed the sound and played it back. She was on a tour of ITP so my project had a lot of competition for her time, but afterwards she said that the project was “fun.” I asked if there was anything that wasn’t fun and she said no. I think this is a good sign, but I’d like to try to make it easier to load readymade sounds—perhaps within the browser itself the way Scratch does—without the need to click and drag.

As things stand, I have several features I hope to implement:

~~Don’t afford the ability to press buttons while audio is processing (because it causes errors)~~ (but could be done more elegantly)
Allow Edits w/ better highlighting of selected area
Zoom mode w/ additional waveform view update, highlight selection
Spiff up interface with symbols that can help bridge a child’s current level of understanding with audio-related symbols that’ll carry meaning later on in life.
~~Allow Record (WebRTC?) https://github.com/muaz-khan/WebRTC-Experiment/tree/master/RecordRTC/RecordRTC-to-PHP~~ (but stops recording properly [gets glitchy] after ~three recording sessions or if a file is played until the end…why??)
More options for starting sounds (preload a range of cool sounds and waveforms)
Oscilloscope ( http://stuartmemo.com/wavy-jones/ ) because the wavesurfer plugin isn’t as preceise to illustrate the concept of a sine wave, triangle wave etc they just look like big blocks of sound…
Better Undo/Redo (download page with all files at end of session then delete them?) ///// on close, delete all of the files. Filesize limit. These are important before making the website public so as not to overload my server.
“Advanced Mode” allowing user to tweak effect parameters. Audacity has too many parameters, Scratch has too few, WAE should provide a simple starting point but allow tinkering for those who wish to dig deeper and explore

[ Dec 7th update: crossed two items off the list)

ICM: Audio Transformer Documentation

Here’s my Audio Transformer so far. Source code is on github.

More info to come!

Audio Transformer

My final project, “Audio Transformer,” will be a simple audio editor that functions in the browser.

I’m designing for kids age 7 and up who have never worked with audio. The idea came from teaching afterschool music workshops, and seeing how frustrated kids get by the defacto freeware program Audacity. With so many features, Audacity can be overwhelming even for adults—here’s a screenshot.

audacity-macosx

I want to make this work in the browser because I’ve also been researching this topic for my class Technological Trends in Music Education: Designing Technologies and Experiences for Music Making, Learning and Engagement. So much of the technology we’ve looked at in that course is happening in the browser. Some technology, like Scratch, incorporates audio editing features. But I have yet to find any dedicated audio editor that works in the browser. I think there is a need for this sort of specialized tool, something like the pixlr photo editor but for audio. I could even see myself using Audio Transformer in favor of pro audio programs when I just need to make a simple edit.

My goal is to keep things simple by focusing on the features that interest the kids in my workshop. They want to hear sounds backwards, slowed down, and sped up. They need to be able to delete, crop, cut, copy, paste in order to make their edits. Fades, normalizing file volume, and determining the file’s starting point (ie. cropping out initial silence) are not so exciting for kids, and often neglected, so these should all happen automatically by default. Undo and Redo are essential. I’m going to model these functions after Scratch’s audio editor with two cursors for selecting a segment, and dropdowns to choose effects/edits.

The sound data needs to be presented in a visually meaningful way. I think it’s important to introduce the concept of a waveform, and to visualize the frequency spectrum. I’ve begun to experiment with visualizations that can represent frequency and attack/decay in meaningful ways. For example, I’d like to have a visual method of changing the bass/treble frequencies, something like this:

I went to the Sound as Data workshop and was very inspired by the possibilities for visualizing sound, but all of the examples use Minim, which uses java.sound and therefore won’t work in the browser. Maxim is a sound library that has a javascript version, so I made the above using the Maxim library because I thought it would work in the browser…

…but it doesn’t work in the browser. There are a lot of issues with Processing, sound, and the web. So I may just have to use JavaScript. I’m excited about the idea of delving into the WebAudio API which has a lot of the stuff I’d need.

Here’s a javascript demo that visualizes the waveform and has the beginnings of the UI that I’m going for.

Here’s a functional mockup that actually edits an audio file (using PHP and SoX, running on a server)

Next, gotta figure out how to pull this all together! My big question is:

How can I get JavaScript talking with PHP? I need to send variables from one to the other. From my research and experiments so far, I think the answer involves AJAX, POST and html forms, but nothing is working so far.

Pixel Experiments

So much to play with this week! Here are my two favorite experiments.

Bearpocolypse – loops through each pixel in the bear image, compares it to every other pixel, and changes each pixel based on “distance” of the RGB.

source: http://itp.jasonsigal.cc/icm/pixels/bearpocalypse/bear.pde

In Motion Dot-tector – built off of the Learning Processing Motion Sensor, and inspired by something I saw at the Exploratorium as a kid. Ellipse size decreases if there is a lot of motion, and ellipses only show up at pixels where there is motion (a difference between current capture and the most recent capture). This one took a lot of memory to run.

source: http://itp.jasonsigal.cc/icm/pixels/inmotion/inmotion2js.pde

I got caught up in all of the examples and libraries, but wasn’t sure where to take them. I’d like to come back to Pixels to make music out of shapes and/or movement.

Particle Systems, PVectors, Simulation

For this ICM assignment, I tried to incorporate a bunch of the concepts that have come up in class recently and also some Nature of Code stuff that I really wanted to wrap my head around. These include ArrayList, Polymorphism, Inheritance, Perlin Noise, Particle Systems, oscillation and physics simulation. I messed around with the physics engine Box2D over the summer and thought it was cool but I’d rather do physics on my own terms so I made my own version of forces like gravity. This sketch is a work in progress but was fun to organize everything into ArrayLists of ArrayLists and I’m glad I was able to wrap my head around the structure.

unfortunately this didn’t translate well in javascript mode but here’s source code on Github.

Pond 3D – Processing + Max/MSP

Inspired by Oskar Fischinger’s Optical Poem (video below), I created an array of rippling circles that would symbolize sounds. I used A 3D bouncing ball that changes direction to control the playback speed of the sounds. Both of these classes have bounce off of each other when they collide.

I started off messing around with Minim and Maxim, two sound libraries for Processing. But these libraries were not the most fun. I decided to try using Max/MSP since it’s been a while and Andy Sigler introduced OSC in a flyby as a way to send messages between programs. I got way too caught up in making the Max patch, and the Processing sketch doesn’t come across without sound, so unfortunately if you want the full experience you’ll need to download this Processing sketch and also run this standalone application created in Max/MSP.

Download Rippling_Pond3d_2.pde + Ripple.pde (class) + Frog.pde (class)

*bonus* download ripples_audioplayer.app (zip)

I have run into some processing questions: How is Z determined in P3D? Is there any way to angle ellipses or can they not be rendered in 3D? Gladys gave me the 3D idea, she thought it’d be cool if the ball made a ripple every time it hit the side of a 3d cube, and I started to go in that direction but gave up because I don’t understand 3D yet. I am psyched about OSC though.

This week I extended the rippling pond from last week. I was inspired by

ICM Week 3: Many Ripples

I wanted to make circles that expand repeatedly from the center of a mouse click, so I made a ‘Ripple’ class that has the variables x,y, initial size, speed of expansion, and color. It had a function to reset itself at its initial size, to create an endless loop. The problem was that for every new mouse click, the old ripple would either disappear (if background() was in draw) or stop moving. I had a vague idea that an Array could solve this, but wasn’t sure how, and I thought it looked kind of cool just rippling out from the center of the screen anyway.

I passed the code along to Adarsh, my buddy for this week’s assignment. He introduced an ArrayList to contain each ripple, and saw some areas where the code could be improved. He used an ArrayList instead of a plain old Array because in the latter, we’d need to know how many ripples there are, but we can keep adding/removing from a list. With the ArrayList in place, he got MousePressed working.

Where we’re at so far: MANY RIPPLES (view online // source code).

if you click, 5 ripples are added to the arraylist. On KeyPressed, the last five ripples are removed. I talked with Adarsh about some ways we could keep going with this, he’s interested in trying P

ICM Week 1 HW – “Blinds”

For this week’s project, we were asked to make some static image like a self portrait using some basic shapes in Processing.

I was inspired by the idea that Processing makes it possible to do things that aren’t possible in programs like Photoshop. I had been trying to use Photoshop to make lines that increase in size while the spaces between them decrease. It was an annoying process…kind of like making each bar of a prison cell one at a time (this was an example that Shiffman mentioned in class). So I wanted to tackle this with a For loop. From there, I went wild with the shapes.

View the source code here or processing.js version here