AUMI: Adaptive Use Musical Instrument
Musical improvisation for everyone by turning motion into sound.

The Adaptive Use Musical Instruments (AUMI) software interface enables the user to play sounds and musical phrases through movement and gestures. This is an entry to improvisation rather than "hitting the right notes" or playing set pieces of music. Instead, the software uses music as a way for participants to express a range of affects, both by themselves and in response to, or in conversation with, others.

While the AUMI interface can be used by anyone, the focus has been on working with children who have profound physical disabilities. In taking these participants as its starting point the project attempts to make musical improvisation and collaboration accessible to the widest possible range of individuals. This approach also opens up the possibility of learning more about the relations between ability, the body, creativity and improvisation, from within a cultural context that does not always acknowledge or accept people with disabilities.

AUMI continues to be revised and improved with input from the technologists, students, therapists and with feedback from registered users. An on-site training program is now available.

If you see a word with a dotted underline, it's a Glossary item. Touch it to get a quick hint about the term. Touch it again to dismiss the window.

AUMI for iOS was developed for the Deep Listening Institute by Henry Lowengard and adapted from the desktop version of AUMI, which is also available from the Deep Listening Institute, Ltd.

AUMI websites:

You can get software support by emailing:

This link points to a PDF copy of these instructions. You can export them to iBooks or print them out for reference.

AUMI was developed with the support of the The Gould Family Foundation.

This is iOS AUMI version


AUMI uses the front-facing video camera to track motion and control which sounds are played. For this to work optimally, good lighting, a neutral background, and a steady support for the iPad, iPhone, or iPod Touch being used are needed. The more neutral the background, the better AUMI will be able to find intentional motion. The Apple iPad folding case is a pretty good support, but you can also use commercially available stands and mounts, and in a pinch, some big rubber bands, a chair or a pole, and some creativity will also do.

AUMI runs on all iOS devices - iPhones, iPod touches, iPad, iPad mini, iPad PRO - running iOS 8.2 and above which have front facing cameras. You will need to confirm that you want to use the camera in a dialog the first time you use AUMI.
For a better experience, you may want to:

You can play AUMI right away when it starts up, or by using the Play AUMI tab bar button to change the screen to the play screen. You will see a live video image, a circle with a red dot in the center (the cursor), and a number of rectangles (the "sound_boxes"). You can rescale the sound_boxes by touching the corners of the enclosing box with two fingers.

If the cursor goes to the edge of the screen for 2 seconds, it will automatically recenter itself!
You can then put the cursor anywhere on the screen with a single touch. You can change various aspects of these interface elements with the Looks button. You can hide and show all the toolbars, and make a completely clean (and safe to touch) screen by tapping with three fingers.

To change the _instrument , the number of available notes, scales, and other aspects of the audio and video of AUMI, use the _Instruments button, which is on the video screen for iPads and in the tab bar on smaller devices.

AUMI has a Login screen. This is a place to identify a user, and other metadata useful in data in the reports AUMI makes. If you are not interested in those reports, you can leave those spaces blank. To help with filling out a name, you can pick one from the Address Book. You can organize users into Groups in the Address Book app (outside of AUMI, that is), and use that to help set the name without a lot of typing. The first time you access the Address Book, AUMI will ask permission to use it.
A name is also used to identify saved instrument _setups. If you have already saved some setups, you can load one from a list with the "List Setups" button.

The Login page in no way affects the ability to play AUMI, nor is this information sent to any server.

There are a number of adjustments that can be made to the tracker software that can help it work best for your device and the environment it will be working in. These adjustments are found in the Settings app, in the AUMI configuration screen.

AUMI can refine the way it turns movement into moving the cursor. Read the _Instruments section to find out more.


Some AUMI users are only capable of extremely limited voluntary motion. AUMI can be adjusted to accommodate their needs in a number of ways:


I have a setup that lets me record AUMI as it's operating - it takes some money and an external computer. You might be able to do it with hardware: an HDMI adapter and cable to a video recording device. Modern iPads and iPhones can transmit video and audio over WiFi to an AirPlay(™ Apple) equipped device, such as a flat screen TV or video recorder. My solution is in software: it transmits a lower-res video (and audio) over to a computer that can pick it up and record it, using a program called AirServer. Sometimes it's a little finicky, but that's how I've made some videos of AUMI.

I'd suggest just using an external video camera to record both AUMI and the real performer using it, as you would in the past. You have a little more control over it that way!


This is a quick description of the toolbar and interface buttons. Click on them for more details!

The interactive video interface.
Instruments Instruments
Pick _Instruments, set the volume, and adjust the tracking features. (Not here on iPads)
Configure AUMI's MIDI playing features. (Not here on iPads)
Login Login
Register who is currently using AUMI, so activities can be recorded, Load and Save instruments, send and recieve instruments, and Create Reports.

Looks Looks
Change aspects of how AUMI looks, such as colors, sound_box layouts, and sizes. This is launched from the "Looks" "button at the left of the main video screen.
Sounds Sounds
Assign sounds to sound_boxes. This is launched from the "Sounds" button at the left of the main video screen.
The user interface is different on iPads and iPhones. On iPhones, the Instruments and MIDI buttons are on the MAIN TOOLBAR. On iPads, the Instruments control button is part of the AUMI PLAY PAGE TOOLBAR, and contains the MIDI controls as well.

AUMI works on iPhones and iPod touches that have front-facing cameras, but is best on iPads and iPad Minis. Newer models will be faster!

The AUMI Web site at is here, featuring more information, videos, support, and tips and hints.

The Login page is where you can set up a name and other metadata here used in recording activity during an AUMI session.

You can use the address book button to get access to the device's contacts to find names more easily. When you select a name, it is put in the name field.

The report button brings up an interface for exporting the AUMI report files. These files are a comma separated value files (CSV), supported by most spreadsheet software. See the reports section for more details.

You can load and save an _instrument and other configuration data, which is called a "_Setup"," with the List Setups,Load Setup and Save Setup buttons. The Save Setup button saves the current instrument and its layout, colors, and sound order to a file named the same as the user's name, e.g. [user name].aumi.

You can save other instrument setups for a user by adding a note to the user name, e.g. Clara K. sitar.
The idea is to load a Setup appropriate for that user, so Load Setup looks up the Setup file using the current user name. If there is no name, it tries to load a Default Setup, which you can create by saving a Setup with no user name.
You can also pick the Setup out of a list using the List Setups button, which will also pick the user.
Setup list
List Setups will also let you rename or delete existing setups by touching the appropriately named buttons. Done will dismiss this dialog.
You can also find and export these setup files or import them by using the iTunes Document interface for AUMI. They will have the suffix ".aumi". "An ".aumi" file is an Apple plist file, which is a kind of XML file, and so is legible by humans.

Sending setups is an experimental feature!
You can set up device running AUMI so that you can send a setup from one to the other. This allows a therapist's device to change aspects of the player's device without needing to touch the screen.
This is using Apple's "Multipeer Connectivity" feature. That means all the devices involved should be on a local network, or have Bluetooth enabled, or it can also communicate peer to peer (that is, via WiFi without actually being connected to a WiFi network).
You can designate a device to listen for setups by touching the "Listen for Setups" button. You can actually do this with up to 7 devices!
On another device running AUMI, you can then send the current setup on that device to the listening device(s) by touching "Send Setup". You'll be presented with a list of "nearby" devices that are listening as it discovers them. You can now touch the device names of the devices you wish to send the setup to. It should connect and change the setup on the listening device, and put up a notice that tells you so. If it fails for some reason, there will be another notice telling you that.


This is the main interface to be used while playing AUMI (iPad version).
Over a live video image, AUMI shows a set of rectangular "sound_boxes" and a circular cursor, which is what initiates sounds as it moves into the sound_boxes.

The sound_boxes for a vertical layout actually stretch to the edge of the top and bottom of the screen. The horizontal layout behaves similarly with respect to left and right.
You can also set the position of the cursor by touching the screen.
When the cursor enters a sound_box, the corresponding sound is played, and the box highlights relative to the volume of the sound that is playing.

AUMI works by detecting and tracking the movement of "interesting" parts of a video image. "Interesting" in this case usually means high contrast and angular shaped parts of the image. The tracker works best when there is clear lighting, and a blank background behind the user.

The relative motion of the tracking_spots moves the cursor on the screen. The cursor has no direct relationship with the tracking_spots. This allows any motion it detects in the image to control the cursor, and thereby initiate the sounds.
There's a toggle button, "Show Spots," in the Instruments screen, that makes the tracking_spots visible when clicked. Click it again to hide the spots.

You can change the size of the rectangle the sound_boxes are laid out in by touching the screen with two fingers. The touch points move the top left and bottom right corners of the box.
The layout of the rectangles can also be changed, except for Loop instruments, by using the Looks control. The layouts are Horizontal, Vertical, Boxed, and Circular. The Boxed and Circular layouts enable users to play notes out of their usual order.
That same control can change the color, transparency, and thickness of the sound_box outlines, and the color, transparency, and size of the cursor.

On an iPad, the "Instruments" button appears on this screen, while on an iPhone, that button is on the navigation bar.

You can hide and show all the toolbars, and make a completely clean (and safe to touch) screen by tapping with three fingers.


On iPads, the Instruments screen is accessed with a button near the top of the Play AUMI screen. On iPhones and iPod touches, it's in the tool bar.
The Instruments screen is where you select an instrument and its properties, set the general volume, and set video tracking options.
A big scrolling instrument picker is divided into four parts. Some instruments don't use all the parts, so they are blanked out.
Instruments break down into four types, each colored differently:

  1. Melodic instruments (green). They have a range of chromatically tuned samples that can be used to play melodies. Most melodic instruments have a range from MIDI note 36(C2) to MIDI note 96 (C7).
  2. Sample instruments (white): They simply play samples that are not tuned or in any particular order. The order can be reassigned by using the Sounds button.
  3. Looped instruments (yellow) : All the sounds play simultaneously in a continuous loop. The closer the cursor is to a sound's sound_box, the louder that sound will play. If you make the size of the cursor bigger, it will cover more sounds and you will hear more of them simultaneously in the mix. The choices for loop sounds and the number of them on the screen can be changed by using the Sounds screen.
  4. MIDI instrument (blue): Makes no sound by itself. It sends MIDI messages to other programs or devices.

The parts of this control are as follows:

Other controls are: These parameters are saved in the user's _setup file (see Setups).


Set Looks

This control lets you change the sizes, layouts, and colors of some interface elements.
Pick which element (Cursor or Guide) you want to change from the selector button.

By touching in the array of colors, you change the color of the selected element.
The box on the bottom left of the color array will show you the current color setting.
There is an "Opacity" slider which lets you change the color's opacity (transparency).
For Guides, this also help choose the color that the sound_box is colored.
The "Size" "slider is used to change these sizes:

The Lock switch locks the shape of the sound_box. This disables the ability to rescale the sound_box with a two-fingered touch.

The layout buttons select whether the sound_boxes are laid out horizontally, vertically, in a grid, or in a circle.

When you are using the MIDI instrument, and you are not using the actual cursor velocity, and if your layout is vertical, the position between top and bottom of the layout grid is used as the note's "velocity," so that near the bottom of the grid is "low velocity" and near the top of the grid is "high velocity". For a horizontal layout, left is "low velocity" and right is "high velocity"."

To dismiss this control on an iPad, tap anywhere outside the popup. To dismiss this control on an iPhone, tap "Done".


This screen is accessed by touching the "Sounds" button on the "Play AUMI" screen.
The Sounds dialog allows assign specific sounds to sound boxes. It's only enabled for Percussion, Noises, Loops, and Relative Motion instruments, because the sounds of these instruments do not have to be in a particular order

To quickly remove a number of sounds from active use, select their names and touch the "Down Arrow". Similarly, if you want to use instruments from the middle of the list, select them and touch the "Up Arrow". Now you can easily re-order them!
Pressing the Sounds button shows you an interactive list of all the sounds available for that instrument. The ones currently playing are in black, and the ones that are currently not playing are shown grayed out.

The Loop instrument, for example, can play from 4 to 12 sounds, but more sounds are to be found in the list which can be moved into place so that they can be heard.
Similarly, if you set up a Percussion or Noises instrument with just two sounds to play, you can choose those sounds by moving them into place using the sounds screen.

This alternative order of sounds is saved with the instrument if you use the "Save Setup" button on the Login screen.



This screen is included as part of the Instruments screen on iPads.

If the current instrument is a MIDI instrument, you can use these controls to select a MIDI channel and MIDI program settings.

Because most MIDI instruments are not percussive, and need to be told to stop playing a note, there's a slider to set the delay time until a "note off" event is emitted. But if AUMI is set to be in monophonic mode, MIDI notes will stay on until the cursor is in a new sound box.
When a MIDI instrument is selected, a big red CLEAR MIDI button appears, in case there are stuck notes. You can also enable and disable MIDI transmissions with a switch.
You can select a program with the picker on the right. The names of the programs are the standard General MIDI names, and may not correspond to sounds that are actually played by the MIDI synthesizer you are connecting to.
Different synths try to discover MIDI sources in different ways, and sometimes, it's trickier than it looks. AUMI will try to connect to every MIDI receiver that it sees, even if they are added while AUMI is running. Apple's "Camera Connection Kit", when connected to a hardware USB-to-MIDI adapter, will let you attach AUMI to an external hardware MIDI synthesizer. There are other MIDI interface available as well.
Here's one way I've gotten another synthesizer running on the same device to work:

  1. Start AUMI.
  2. In Instruments, choose a MIDI instrument. Pick a scale and note range too, if you like.
  3. Make sure the MIDI switch is turned on.
  4. Click the home button to put AUMI in the background (it's still running)
  5. Start an iOS MIDI synthesizer like Bismark, DXi, Animoog, or another MIDI compatible iOS app.
  6. Make sure the synthesizer can play using background audio. (For example, in DXi, it's a switch in the MIDI section of its settings and information popup)
  7. Click the home button to put the synth away, and click on AUMI's icon again.
  8. It should be able to play the MIDI synth now.
You can also use a local network to play instruments on a nearby laptop or desktop. That's a little more involved; I'd suggest you look at tutorials about the Apple Audio MIDI Setup utility, like this one by Joe Stallings.


AUMI generates two kinds of reports in the form of CSV files. Each time AUMI is started, a new file is created in the format AUMI-YYMMDD-HHMMSS.csv, so that the date and time are part of the file name, e.g.: AUMI---121206-092201.csv .

Every 15 seconds, a record is cut with some information that may be used to make reports in an external spreadsheet program like Numbers or Excel. This information may be useful for tracking the progress of individual AUMI users in different programs at various institutions.

Also, when the program transitions from foreground to background, it will also log this event. In this record, the timestamp and identifying information are retained, and the position related fields will be blank, but the Scale field will say FOREGROUND or BACKGROUND.

There's a second kind of report that is enabled with the optional switch "Log Sound Box Events".
It tracks the entry and exit from sound boxes, and record the time spent in each one, and its file has a slightly different name from the usual report file:
AUMI-EL-[institution]-[program]-[date and time].csv

List Reports
The report button will present you with a list of all the report files AUMI has made. They are ordered by date, and the two types of files are in different sections of the list.

When you are ready to send out the reports, tap Send. Any program, like Apple's Numbers, that can read a text file will be listed in the dialog that pops up. You can also use Air Drop to send the files directly to any other Apple device that supports Air Drop, for instance, a desktop machine. You can also mail or post the files to various services, and you can also cancel the dialog.

All these files, are also available in the iTunes Document interface for AUMI. The main log file is also written whenever you leave the program.

Each record has the following information:

and the event report fields are these: Thus, you can tell users apart, and send files to different institutions.
Note that the Instrument name has the "Emoji" stripped off of it.

For example:


  Using the iTunes Document Browser

If your device doesn't have mail set up, then of course it can't mail anything out! You can get your reports by:

  1. plugging in the iPad to a computer that knows about it. Usually, that's the one that was used to set it up or back it up.
  2. open iTunes
  3. find the device in iTunes's top bar:
    finde device
  4. click on that and click on 'Apps' in this list of buttons:
    find app
  5. scroll down to where it says "File Sharing"
    find file sharing
  6. scroll down that list and click on AUMI:
    find AUMI
  7. The box on the side marked "Documents" will show all the sessions it has recorded:
    find Documents

All the personal set-ups are also in the Documents Directory.
You can select them all and click "Save to..." to save them in a directory that you can remember. They don't take up a lot of space, and you can also just delete them after you've saved them.


In the actual AUMI app, you can only configure some aspects of the trackes and video from this section of the instructions.

AUMI uses software called a "tracker" to analyze the video images in order to know how to move the cursor.
These tracker settings may help you optimize AUMI in when you have a faster iOS device, or want to experiment with the responsiveness of AUMI.

Feature Tracker parameters

Each kind of device and working environment has its own best parameters for working with a Feature Tracker.

Velocity to loudness controls

The range of loudness controlled by velocity when the "Use Velocity" switch is on can be set here. Velocity loudness can be thought of as ranging from 0.0 to 1.0. The high level is probably more important; if you set it to a fraction like 0.1, it will take more velocity to make it play loudly (and it will be more sensitive!), whereas setting it higher, like to 8.0, means nearly any movement will be loud.

Camera Resolution

AUMI's camera resolution is set low to accomodate slower devices, but on new fast devices, you can use a higher resolution if you like. The "Zoom" feature will still let you reduce the resolution and lower the latency of the tracker.

Front or Back Camera

iOs devices have a front facing and a back facing camera. In some situations, the rear camera might be more useful.


Somewhat adventurous AUMI users or administrators can add their own sounds into AUMI in a limited fashion. By using the iTunes Document interface for AUMI, you can drop in MP3 sound files and have them automatically turn into instruments. Here's how it works:

The "User Sounds" Instruments can be saved in a Setup (using "Save Setup"), but because of the possibility of sound files being added or removed using the iTunes Documents interface, if it is loaded on a device that does not have one of these files, it will put a tilde (~) in front of the name and play Silence instead.



You can register for news about AUMI by filling in your email address here:

Email & Social Media Marketing by VerticalResponse (


AUMI for iOS was developed for the Deep Listening Institute, now called "The Center for Deep Listening at Rensselaer" by Henry Lowengard, and adapted from the desktop version of AUMI, which is also available from the Rensselaer Polytechnical Institute (RPI). Thanks to Pauline Oliveros, Leaf Miller, Jaclyn Heyen, Emily Halstein, Lisa Barnard Kelley, and Al Margolis of DLI, and the desktop developers: Zane Van Duzen, Zevin Polzin, Doug Van Nort, Ian Hattwick, Aaron Krajeski, Ivan de Almeida Soares Franco, John Sullivan
For the AUMI Consortium: Prof. Sherrie Tucker (University of Kansas), Prof. Eric Lewis (McGill University, Montreal), Thomas Cuifo (Mt. Holyoke)
For RPI, Thanks to Prof. Jonas Braasch, Prof. Tomie Hahn, and Dean Mary Simoni.

Some audio was generously provided by these musicians:

iOS, iTunes, iPad, iPhone, iPod Touch, GarageBand are registered trademarks of Apple, Inc.
AUMI was developed with the support of the The Gould Family Foundation.
The feature tracking software is derived from KLT (Kanade-Lucas-Tomasi) from Stanford University.
The tutorial on Apple's Audio MIDI Setup Utility is "Using Your iPad as a Wireless MIDI Controller" by Joe Stallings.

AUMI websites:

AUMI (c) 2012-2014 Deep Listening Institute, Ltd., and 2015-17, Center For Deep Listening at RPI
Developed by Henry Lowengard
build version: 1.1.6