“Time Traveler” piano song

It’s time to start going out of my comfort zone and pursuing music a bit more! This is a song that I wrote earlier this year for the harpsichord and dulcimer, performed on the electric keyboard, and mastered in Logic. It’s dedicated to my sister, who is living in Tanzania the next few years teaching elementary school. It’s already tomorrow where she lives.


admin“Time Traveler” piano song

Color Piano 2.0

Learn how to play piano songs by watching notes fall towards the keyboard as color-blocks; similar to how Guitar Hero works, but with a real instrument. Color piano Theory (CPT) ties together chords, scales, inversions, octaves, key signatures, and play-by-play examples of classical compositions, getting you started with playing piano the easy way =)

Color Piano is free to use, please share!

Getting your feet wet with Color Piano (features);

  1. Drag & Drop MIDI files into your browserto view/play them in CPT. Helpful links;
  2. Seek to a specific location in the song, or replay parts your having troubles with;
    • To do this, use your MouseWheel, or Scroll with two fingers using a Trackpad, or Use the Scrollbar on the right of the Piano.
  3. Play the keyboard with your computers keyboard =)
    • `1234567890-=  and  asdfghjkl;’  are all black keys.
    • qweryuiop[] and zxcvbnm,./ are all white keys.
  4. Configure the Piano to play slowerwhen learning a new song, then slowly increase the speed as you get better!
    • Step 1. Click on the Configure cog on the right of the Piano.
    • Step 2. Use the range slider to configure the Speed.
  5. Configure the Synesthesia, aka color-to-note mapping, that you relate with. My personal favorite is D.D. Jameson, but there are a lot of other interesting options created by people throughout history, starting with Issac Newton. To configure;
    • Step 1. Click on the Configure cog on the right of the Piano.
    • Step 2. Use the select-menu to configure the Synesthesia.
  6. Configure whether you want to see the notes before they happen, or afterthey happen. Before is default for learning to play piano, after is a mode to be used strictly as a visualizer (much harder to learn from);
    • Step 1. Click on the Configure cog on the right of the Piano.
    • Step 2. Use the select-menu to configure the Visualization.


  • 2.1—1/6/13
    • Download links on MIDIs.
    • Circle of Fifths Synesthesia modes.
    • Cache MIDIs in FileSystem for quicker loading.
  • 2.0.0–9/22/12
    • 2,000 MIDI songs from Disklavier.
    • MIDI-browser w/ search engine.
  • 1.5.1–4/23/12
    • Previous + next song buttons.
    • Faster rendering. Fix bugs in Chrome 18.
    • Play + Theory modes.
  • 1.4.2–12/23/11
    • Load MIDI from remote URls in configure pane.
    • Improved MIDI reproduction.
  • 1.3.8–12/18/11
    • Speed controls, and ability to scroll through midi.
    • Steinway grand piano synth.
  • 1.3.0–12/11/11
    • Tie into Web Audio API for more accurate playback in Chrome.
    • Tie into localStorage to save settings.
    • Preview notes before they happen.
  • 1.2.0–12/6/11
    • Using base64 soundfonts.
    • Now displays all 88-keys of a standard piano.
    • Watch notes falling towards the keys before the note plays!
  • 1.1.0–11/27/11
    • HTML5 <audio> is used for sound-output.
    • Color Piano Theory is available on the Chrome Webstore.
  • 1.0beta
adminColor Piano 2.0

HTML5: SoundFonts

Color Piano Theory is now available on the Chrome Webstore. There haven’t been any major UI overhauls since last reported, but there has been a lot of work going on the back-end! Most importantly moving from the Java interface to native HTML5 <audio> tag (as Java isn’t supported in the Chrome Webstore). Although this sounds like a simple task, there’s a lot of steps involved; hopefully this will save someone else a bit of trouble!

Generating your own soundfont files;

  • JSMIDI will allow you to generate MIDI files with the MidiWriter package;
	var key = 0x45; // the note A4
	var noteEvents = [];
	Array.prototype.push.apply(noteEvents, MidiEvent.createNote(key));
	var track = new MidiTrack({ events: noteEvents});
	var song  = MidiWriter({ tracks: [track] });
  • Saving the MIDI files to disk; File Writer API allows you to save those generated MIDI files to your hard-disk, or, alternatively (and a bit more simple in terms of programming), you could POST the base64 from an embedded <iframe> to .PHP, and write to the file-system;
	var iframe = document.createElement("iframe");
	iframe.src = "index.php?midi=" + (song.b64) + "&key=" + key;


if ($_REQUEST['midi']) {
	$myFile = "./midi/".$_REQUEST['key'].".mid";
	$fh = fopen($myFile, "w") or die("can't open file");
	fwrite($fh, base64_decode(str_replace(' ','+',$_REQUEST['midi'])));
  • Getting out of MIDI format; At this point, we have a bunch of MIDI files. We need to eventually get these MIDI’s -> OGG format, by mapping it to a high-quality SoundFont;
    • Older versions of iTunes allows you to batch convert from MIDI’s -> MP4’s. That was very nice feature that seems to have disappeared…
    • Online app, such as SolMire, allow you to convert from MIDI’s -> MP3’s and other formats, one at a time. I especially like that SolMire allows you to choose the desired SoundFont to use on the .MIDI.
    • MIDI2MP3 is a command line application available for Window and Mac OSX that enables you to use specific SoundFonts in your encodings, and allows you to use the command line… and therefore the ability for batch MIDI -> WAV conversion! FluidSynth Soundfont GM is a good .SF2 file to get you started 😉
  • Getting into the OGG format;
    • Switch (for Mac) allows you to convert from WAV’s, MP4’s, and MP3’s -> OGG’s.
    • oggenc from Vorbis, allows you do batch conversion of WAV’s -> OGG’s using a bash script. The calls are like this:
      •  ./oggenc -m 64 -M 128 audio.wav
  • Converting the OGG’s -> base64, and storing them in .js or .jgz file(s):
The following code will allow you to take those MIDI files we created with the JSMIDI package (step #1) and convert them from WAV to OGG to JS to JGZ in seconds! Presenting a solution for the batch conversions of multiple MIDI’s into base64 soundfonts;

# gzip     - http://www.gzip.org/
# base64   - http://www.fourmilab.ch/webtools/base64/
# oggenc   - http://www.rarewares.org/ogg-oggenc.php
# midi2mp3 - http://www.audiosoftstore.com/downloads.html

# from MIDI to WAV to OGG to JS to JGZ, and beyond!

find ./directory -name '*.mid' -print0 | while read -d $'\0' file
		# from MIDI to WAV
		./inc/midi2mp3 $file -sf ./sf2/FluidSynth_1.43.sf2 -e wave
		# from WAV to OGG
		./inc/oggenc -m 64 -M 128 $file.wav
		# from OGG to base64 embedded in Javascript
		echo "if (typeof(Soundfont) === 'undefined') Soundfont = {};" > $file.js
		echo "Soundfont['`basename $file`'] = 'data:audio/mpeg;base64,`base64 -i $file.ogg -o -`';" >> $file.js
		# gzipped version
		gzip $file.js -c > $file.jgz
Now you’re ready to create your own custom Soundfont =)
adminHTML5: SoundFonts

Music Box

Last weekend I got a chance to mess around with Mr. Doob’s rendering framework Three.js and Daniel van der Meer’s amazing MIDIBridge.  I’ve been a big fan of both Ricardo and Daniel for some time now, so it was fun to work with their codebases.

Three.js is a lightweight 3D engine that can render in <canvas>, <svg> and WebGL.  Ricardo (Mr. Doob) claims this framework is written “for dummies”, which is perfect for me, because that is exactly where I’m at with 3D programming 😀  Creating this wall of interactive color blocks was simple with Three.js.

MIDIBridge allows Javascript to connect to Java’s internal MIDI synthesizer. The author, Daniel, is working on a Javascript to Flash bridge that uses SoundFont2 to map amazing libraries of sound.  There are thousands of SoundFont2 patches available, which opens up the doors to all sorts of new creativity.  The framework is easy to tie into by sending direct midi messages to specific channels.  For instance, the message code 128 turns the note off whereas 144 turns it on.

Matt West’s MIDI file reader jasmid was extremely useful to parse the MIDI files from the server on the client-side using Javascript.  I’ve tweaked the script slightly to allow for pausing, and resuming of songs.  The collection of royalty free classical piano pieces comes from Disklavier World.  Thank you to all, amazing contributions!

Here’s the WebGL Music Box that reads a selection of classical music such as Tchaikovsky’s Waltz of the Flowers from the Nutcracker. And then there is the interactive version that you can control the notes with your mouse.

This demo has been tested on Mac in Google Chrome, Safari, and Firefox.

adminMusic Box

Color Piano v1

UPDATE: There is a more recent post on Color Piano.

Color Piano Theory (CPT) was inspired by an interest in building an educational application that utilizes colors in teaching piano theory.  CPT ties together chords, scales, inversions, octaves, and key signatures.  CPT is a visual interface for learning the keyboard.

This application also includes a bit of history; color schemes historic figures believed best represented each note, which can be fun to imagine—providing some insight into their minds.

Visual/audial memory recognition

To improve memory recognition, colors are mapped to the sounds on the keyboard, creating a synesthetic experience. By picking a color-mapping that works best for you, these colors will give you a visual cue to the note you’re playing.

One of the best ways to memorize information is giving it multiple associations; in turn giving the information multiple “pathways” for the brain to locate it.  With color added to the mix, we are building a memory recognition triangulation:  sound (measured in hz), color (in RGB), and space (the XY coordinate of key on the keyboard).

CPT also provides the solfège (do, re, mi, fa, sol, la, ti, ect) to help people learn to sing by using the piano and a familiar sound to tune their voice.

Historic mapping of color to sound

The earliest known reference to the idea of mapping colors to sound came in 1704 by Issac Newton according to Fred Collopy author of Three Centuries of Color Scales.  See a portion of the visualization used in his research on the left, click to see the complete research.

This leads me to a question brought to me recently, “Why do so many of these people associate ‘red’ with ‘C’, ‘orange’ with ‘D’, ‘yellow’ with ‘E’, ‘green’ with ‘F’ and so on?”  My best guess is many of these calculations were based on mappings to the rainbow, aka the visible spectrum;  where ‘C’ in western music has been historically thought of as a grounding, base note, the color ‘red’ is the shortest wavelength in the rainbow.

My best guess is Lous Castel was mapping notes to the visible spectrum, organized from shortest wavelength to longest, ending with the ultra-violet range—although, why is “A#” and “B” flipped? Perhaps a sign of dyslexia? Alexander Schriabin declared that “D#” sounds “steely with the glint of metal”, and “E” sounds “pearly blue the shimmer of moonshine”, and who can argue with that?  What does sound look like to you?

Color Piano Project

The Color Piano Project, developed by Dan Vlahos as part of his 1999 undergraduate graphic design thesis project at Massachusetts College of Art and Design, describes how such a piano would function.  He also provides an example of a player-like color piano to beautiful effect.

Creating “MIDIPlugin”

Being a big HTML5 fan, I decided to program the application in Javascript—the first hurdle was getting MIDI working in the browser to synthesize sound.

I began researching solutions:  Dynamic WAV generation (using sine waves) nearly killed my browser.  Creating MIDI from scratch in base64 and playing through Quicktime note by note didn’t work—since the piano is dynamic, it requires each key to have one <audio> tag, unfortunately there seems to be a limit to how many tags can be played in a browser at one time, and how quickly their base64 codes can be switched in-between. Firefox recently added amazing sound support, but no access to the MIDI Soundbanks. Perhaps someday Google will provide a Native Client MIDI solution 🙂 …until then…

Javascript <-> Java communication

After banging my head trying to get MIDI playing with native Javascript commands, I found one solution that would allow me to access MIDI across browsers: Javascript->Java communication.  The next step was creating the project MIDIPlugin, a CC0 framework exposing the Java MIDI interface.  Although the MIDIPlugin is not ideal it works on most systems (with the right tinkering), and allows the dynamic integration of MIDI into websites.

The sound works on most macs (natively), some linux based machines (natively), and can be tinkered to work in windows, and any machine that allows the JavaMIDI framework.  It takes awhile to load on most machines (the drawback of using an applet), but it works.  Read more on how to tie the MIDIPLugin into your application.

Presenting a synesthetic educational experiment

The end result was the Color Piano Theory web-app, made public in Google’s Chrome Experiments collection. Play around with the application—I hope it helps you create something beautiful.

Synesthesia on the web:


adminColor Piano v1