AUMI: Adaptive Use Musical Instrument
Turning Motion into Sound. Musical Improvisation for Every Body.

version 2.2.0

The Adaptive Use Musical Instruments (AUMI) software interface enables the user to play sounds and musical phrases through movement and gestures. This is an entry to improvisation rather than "hitting the right notes" or playing set pieces of music. Instead, the software uses music as a way for participants to express a range of affects, both by themselves and in response to, or in conversation with, others.

The original focus of the AUMI interface was on children with profound disabilities and their teachers, therapists, caregivers, and families. In taking these participants as its starting point the project attempts to make musical improvisation and collaboration accessible to the widest possible range of individuals and communities. This approach also opens up the possibility of learning more about the relations between ability, the fullest possible range of bodies and minds, creativity and improvisation, from within a cultural context that does not always acknowledge or accept people with disabilities. The AUMI user base includes adults with and without disabilities, seniors, participants of independent living centers, artists with and without disabilities, and participants of mixed-ability, cross-generational community jam sessions, and many others.

AUMI continues to be revised and improved with feedback from anyone who uses it. We want to hear from you. Send feedback to .

If you see a word with a dotted underline, it's a Glossary item. Touch it to get a quick hint about the term. Touch it again to dismiss the window.

AUMI for iOS was developed for the Deep Listening Institute by Henry Lowengard, and adapted from the desktop version of AUMI, which is also available for download from

AUMI websites:

You can get software support by emailing:

This link points to a PDF copy of these instructions. You can export them to iBooks or print them out for reference.

AUMI was developed with the support of the The Gould Family Foundation, Improvisation, Community, and Social Practice (ICASP), and International Institute for Critical Studies in Improvisation (IICSI).
Further AUMI development is supported by a grant from the Craig H. Neilsen Foundation.

This is AUMI for iOS version 2.2.0

Screenshots used in these instructions are taken from various models and simulations of iOS devices and may not look exactly like what is displayed on your device!

AUMI for iOS is also available for MacOs 12.4 and later on Apple Silicon devices in a somewhat untested but workable capacity.


This section is a broad overview of setting up and playing AUMI. Many more details are covered in specifically related sections.

AUMI uses either the front- or back- facing camera to track motion to control which sounds are played. For this to work optimally, good lighting, a neutral background, and a steady support for the iPad, iPhone, or iPod Touch being used are needed. The more neutral the background, the better AUMI will be able to find intentional motion. The Apple iPad folding case is a pretty good support, but you can also use commercially available stands and mounts, and in a pinch, some big rubber bands, a chair or a pole, and some creativity will also do.
Good lighting means your subject is not backlit, which will obscure details that need tracking for motion and face tracking and dulls colors for color tracking. Make sure there is no prominent coloration in your lighting, which can happen if the AUMI player is shadowed by a colored tent or window.
By turning on the "See Spots" feature of the Motion tracker, you can visualize what that tracker thinks is "interesting" and make adjustments so that an AUMI player's motion is emphasized. More hints for this are covered in the Motion Tracker section of these instructions.

AUMI runs on all iOS devices - iPhones, iPod touches, iPad, iPad minis, iPad PRO - running iOS 9.3 and above which have cameras. You will need to confirm that you want to use the camera when a permissions dialog box appears the first time you use AUMI.
For a better experience, you may want to:

You can play AUMI right away when it starts up, or by using the Play AUMI tab bar button to change the screen to the play screen. You will see a live video image, a circle with a red dot in the center (the Cursor), and a number of rectangles (the "sound_boxes").
The Cursor moves when AUMI detects movement in the video image. When the Cursor enters a sound box, a sound plays. You can put the Cursor anywhere on the screen with a single touch, and play AUMI with a single touch to move around the Cursor just as you can with video tracking.
The sound boxes can be rescaled by touching the corners of the box that encloses them with two fingers.

If the Cursor goes to the edge of the screen for 2 seconds, it will automatically recenter itself!
You can change various aspects of these interface elements with the Looks button.
You can hide and show all the toolbars, and make a completely clean (and therefore safe to touch) screen by tapping with three fingers.
There is an optional notice that warns you that it's about to hide the screen. You can cancel that action with "DON'T", approve it with "YES", or keep the optional notice from coming back with "Don't show this notice again". The notice will time out after 6 seconds and "Not approve" the action. If the notice is set to not show again, it will approve the hiding action.

Two important buttons are also on the play space:

There are four major sections that configure AUMI's processing and UI:


A Setup stores all the parameters for an AUMI session: the instrument, its looks, a fully configured tracker, and any timing and chords that are to be used.
Setups are maintained by using the Setup screen.
This includes naming, renaming, loading, deleting, and sharing setups either in a peer-to-peer fashion or as a shared document.


AUMI has the ability to store some data associated with its use, and export this data as CSV files to be used with reporting software. This reporting is associated with the Setup name, which could be set to name a user and a session name for particular tests, and other metadata useful in data in making the reports.
By default, all reporting is disabled. If you are considering the use of reports, please see the LEGAL document. If you are not interested in reporting, you can leave the metadata information spaces blank.

The name is also used to identify saved instrument _setups. If you have already saved some setups, you can load one from a list with the "List Setups" button.

The Setup page in no way affects the ability to play AUMI, nor is this information sent to any server.

AUMI can refine the way it turns the Cursor position relative to the sound_boxes into sound. Read the _Instruments section to find out more.


Some AUMI users are only capable of extremely limited voluntary motion. AUMI can be adjusted to accommodate their needs in a number of ways:


As of iOS11 and Yosemite (Mac OS X 10.10) or later on a Mac OS system, you can record directly from a connected iPhone or iPad by using the QuickTime Player app. Simply select "New Movie Recording" and in the on-screen control, pick the devices name in the drop down menu (the "v") for both video and audio. Then record and trim as usual.
If you have another kind of computer that can record from an HDMI stream, you might be able to do it by using hardware: an HDMI adapter and cable to a video recording device. Modern iPads and iPhones can transmit video and audio over WiFi to an AirPlay(™ Apple) equipped device, such as a flat screen TV or video recorder.
In MacOS Ventura and later, there is built in AirPlay emulation on your Mac. See "AirPlay Receiver" in the "AirDrop & Handoff" section of the System Setting menu. There is also AirPlay emulating software for older versions and other operating systems: the device transmits video (and audio) over to a computer that can pick it up and record it, using a program called AirServer, among others. Sometimes it's a little finicky, but that's how I've made some earlier videos of AUMI.

I'd suggest just using an external video camera to record both AUMI and the real performer using it, which gives a better idea about what is happening!


This is a quick overview of the toolbar and interface buttons. Click on them for more details!

The interactive video interface.
Instruments Instruments
Pick _Instruments, set the volume, pick a scale and other instrument related features. On iPhones, this is in the toolbar, and on iPads, this is accessed from the side menu.
Configure AUMI's MIDI playing features. On iPhones, this is in the toolbar, and on iPads, this is part of the instruments dialog box.
Info Instructions
Leads to these instructions
Setup Setup
Load and Save AUMI settings like scales, instruments, video and timing, send and receive Setups to other devices, set logging metadata, enable logging to create reports, and read and export reports.

Looks Looks
Change aspects of how AUMI looks, such as colors, sound_box layouts, and sizes. This is launched from the "Looks" "button at the left of the main video screen.
Instruments Instruments
Pick _Instruments, set the volume, pick a scale and other instrument related features. On iPhones, this is in the toolbar, and on iPads, this is accessed from the side menu.
Sound Order Sound Order
Assign sounds to sound_boxes. This is launched from the "Sound Order" button at the left of the main video screen. This button does not appear when using melodic instruments, which order sounds by using scales.
Video Camera and Tracker Conf Video Camera and Trackers Settings
Change the size, resolution of the video cameras, choose tracker, and adjust tracking parameters.
Timing and Chords Conf Configure Timing & Chords
Change the options and speeds of AUMI's timing quantization feature. It also configures chords and arpeggios.
Clear MIDI
Immediately stops the playing sound, whether it's from AUMI or via MIDI. Sliding off the button will lock it, it will rename itself to "RESUME", and a "SILENCED" message will appear on screen. Tap it again to resume playing.
Will load the Setup named Default. This will help you reset the system to a known state. You could also go to Setups and load any other Setup you like, but this is right there on the play screen. If there is no setup named Default, one will be created that is a Piano with a Motion Tracker.
Touching this button will show an optional notice that warns you that it's about to reload the Default setup. You can cancel that action with "DON'T", approve it with "YES", or keep the optional notice from coming back with "Don't show this notice again". The notice will time out after 6 seconds and not approve the action. If the notice is set to not show again, it will approve the resetting action.
The user interface is different on iPads and iPhones. On iPhones, the Instruments and MIDI buttons are on the MAIN TOOLBAR. On iPads, the Instruments control button is part of the AUMI PLAY PAGE TOOLBAR, and contains the MIDI controls as well.

AUMI works on iPhones and iPod touches, but is best on iPads, iPad Pros, and iPad Minis. Newer models will be faster, especially for tracking!

AUMI occasionally puts up notices which can be dismissed by tapping the "done" button. Most of these informational notices go away by them selves after two seconds, and the "optional" notices go away after 6 seconds. Notices about movement and event reporting, though, require a manual confirmation.

The official AUMI Web site at is here, featuring more information, videos, support, news, and tips and hints.
iPad Setup
iPad setup screen
iPhone Setup
iPhone setup screen

On the Setup Screen, you can save and load _instrument and other configuration data, which together is called a _Setup , by using the "Save Setup" button and the list of already saved setups.
The Save Setup button saves the current instrument and its layout, colors, and sound order to a file named [setup name].aumi. It is in Apple "plist" format, related to XML.

You can organize instrument setups for an AUMI user by adding a user name to the setup name, e.g. Clara K. sitar.
That saved file is visible in AUMI's section of the Files app, and also seen if the device is connected to a MacOs computer and seen as a device in the Finder. Older devices may need to use the iTunes Document interface.
The list of setups has an automatically created description line that tells

Listening and Sending
AUMI can be configured so that you can send a Setup from the device it's running on to another copy of AUMI running on a different device. This allows one device to change aspects of another device without needing to touch the screen.
This feature works even if the receiving device has a screen in Installation Mode that can't be easily unlocked.
Listening and Sending is using Apple's "Multi-peer Connectivity" feature. That means all the devices involved should be on a local network, or have Bluetooth enabled.
You can designate a device to listen for setups by touching the "Listen for Setups" button. You can actually do this with up to 7 devices!
On another device running AUMI, you can then send the current Setup on that device to the listening device(s) by touching "Send Setup". You'll be presented with a list of "nearby" devices that are listening as it discovers them. Touch the device names of the devices you wish to send the Setup to. It should connect and change the Setup on the listening device, and put up a notice that tells you so. If it fails for some reason, there will be another notice telling you that.

AUMI can tell the listening device to load an AUMI Instrument that it doesn't have if it knows where to get it from. "Send Setup" has either a "smiley face" or an "X smiley face" next to the name if the Setup is using a user loaded instrument. If that instrument is an "AUMI Instrument" file loaded from a website, there will be a "smiley face", meaning that the receiver can attempt to download it from the original source if it's not on the device already. If there's an X, that instrument cannot be sent. AUMI will not send audio data in a Setup, it only sends the URL of where to find audio data. If you send a setup that has
User Sounds in it, it will work if the receiving device has those sounds installed under the same name. If the sender is sending an AUMI_Instrument that was loaded on that sending device, it will tell the receiver to load that instrument (because it remembers where it was loaded from), but otherwise, it will use the Piano.
Once the setup is received, it can be saved on the device that received it by tapping "Save Setup".

Another way to distribute setups is to use the "Share Setup" button. This is a way to send a Setup through messaging, email, AirDrop, or third party tools that share documents. Since they are special temporary documents, they are named with the suffix "_EXPORT.aumi". These in turn can be collected as setups in a cloud-based directory, or on a website, so that people using AUMI remotely together can easily load Setups used for their performances.

The Setup page is also where those who are using the reports feature may set up the metadata used in recording activity during an AUMI session.
The report button brings up an interface for exporting the AUMI report files. These files are a comma separated value files (CSV), supported by most spreadsheet software. See the reports section for more details. See the legal section prior to enabling the reports feature.


This is the main interface to be used while playing AUMI (iPad version).

Over a live video image, AUMI shows a set of rectangular "sound_boxes" and a circular Cursor, which is what initiates sounds as it moves into the sound_boxes.
You have the choice of either showing the live video on the screen, or replacing it with a neutral background. Some users prefer not to see themselves while playing, while others enjoy the visual feedback of watching their movements interact with the Cursor and Sound Boxes. Others go back and forth, sometimes turning off the video to reduce the distraction of trying to make a movement make a particular sound, in order to focus more on listening, or to turn the attention of a group from watching individual screens to relating to one another.
You can also set the position of the Cursor by touching the screen, and use this method exclusively for controlling AUMI. There is a tracker designed solely for use by touches, that does not use video at all, called the Touch Tracker.

For most instrumetns, when the Cursor enters a sound_box, the corresponding sound is played, and the box highlights relative to the volume of the sound that is playing.
For Loop instruments, the nearness of the Cursor to a Sound Box controls the mix of a sound loop playing in that box.
For Relative Instruments, the relative motion of the Cursor chooses which sound to play.

AUMI's Motion Tracker works by detecting and tracking the movement of "interesting" parts of a video image. "Interesting" in this case usually means high contrast and angular shaped parts of the image. The Motion Tracker works best when there is clear lighting, and a blank background behind the user.

The relative motion of the tracking_spots moves the Cursor on the screen. The Cursor has no direct relationship with the tracking_spots. This allows any motion it detects in the image to control the Cursor, and thereby initiate the sounds.
There's a toggle button, "Show Spots," ("Show Areas" for Color Trackers) in the Video Camera and Tracker Settings screen, that makes the tracking_spots visible when clicked. Click it again to hide the spots.
For the Face Tracker, little eye and mouth icons show the position of the eyes and mouth that it finds.
For the Color Tracker, you can see the areas that are the selected group of colors highlighted in paler versions of themselves.

The Face Tracker detects faces with machine learning and put the Cursor right on the face's nose. It's not as flexible as the motion tracker, but it is pretty fast - which might make it the right tracker for your situation!

The Color Tracker detects colors and put the Cursor in the middle of where the largest block of that color is onscreen. While it's very fast, you need to select the color to track by double tapping the screen and sliding your finger around to pick up variants of that color.

The Distance Tracker is the same as the Color Tracker, except the Y axis is replaced with value derived from the area of the largest block of color. That roughly corresponds to distance, and so morion towards and away from the screen (camera) can be used!

The Touch Tracker uses just a single touch to play the screen, with no video used at all. The video image is replaced with the background color.

You can change the size and shape of the rectangle the sound_boxes are laid out in by touching the screen with two fingers. The touch points move the closest corners of the top and bottom of the containing rectangle.
The layout of the rectangles can also be changed, except for Loop and Relative instruments, by using the Looks screen controls. The layouts are Horizontal, Vertical, Gridded, and Circular. The Gridded and Circular layouts enable users to play notes out of their usual order.
The Looks" screen controls can change the color, transparency, and thickness of the sound_box outlines, and the color, transparency, and size of the Cursor.

On an iPad, the "Instruments" button appears on this screen, while on an iPhone, that button is on the navigation bar.

There is a big red "SILENCE" button in case you want to stop the sound that's playing and clear the queued Timing based notes. While it's held down, the trackers and sound will be disabled and even loops will stop playing. This is called "Silent Mode". You can lock it in "Silent Mode" by sliding off of the button and then lifting your finger. When you let go, or tap it again if it's locked, the trackers will work again and the loops will play. While in "Silent Mode", the button label will change to "RESUME"
If the instrument is a MIDI instrument, this button will change its label to "CLEAR MIDI" and will silence MIDI notes.

To reset AUMI to a predictable configuration, tap the "RESET" btton. It actually loads a Setup named "Default". "Default" is automatically added to the list of Setups, but you can save any setup you like under that name, and the Reset button will load it.

You can hide and show all the toolbars, and make a completely clean (and safe to touch) screen by tapping with three fingers.

Note: in iOS 13 and on, a three finger tap is now interpreted as a part of a cut, copy, or paste operation. To do a three finger tap that will be recognized in AUMI, you may have to put each finger down slowly (one-uh-two-uh-three-uh) for it to be detected properly. This was fixed in iOS 13.1 and later!
When AUMI is used in a public installation there's a chance that you may want to lock its control interface. One way would be to put it behind glass, but if the device is exposed, it needs another method.
There is an Installation Mode which locks the screen from revealing the toolbars. This is enabled and disabled by touching down and holding two fingers - possibly resizing the Sound Boxes' bounding rectangle - while moving a third finger a long distance of more than 100 pixels. When enabled, it prevents a "normal" three-finger tap from revealing the controls, and also disables a four-finger tap from locking or unlocking the size of the bounding rectangle. This is designed to be a little obscure! It also locks the Sound Boxes' rectangle size, even when you go back to showing the normal interface.
There is an optional notice that warns you that it's about to go into installation mode. You can cancel that action with "DON'T", approve it with "YES", or keep the optional notice from coming back with "Don't show this notice again". The notice will time out after 6 seconds and "Not approve" the action. If the notice is set to not show again, it will approve going into installation mode.

iPhone Instruments

On iPads, the Instruments screen is accessed with a button on the left of the Play AUMI screen. On iPhones and iPod Touches, it's in the tool bar at the bottom of the screen.
The Instruments screen is where you select an instrument and its properties, and set the general volume, and a few other options.
A big scrolling instrument picker is divided into five parts. Some instruments don't use all the parts, so they are blanked out.
Instruments are collections of sounds that AUMI can play in a number of ways. AUMI comes with a good variety of sounds, some conventional, some unconventional. AUMI does not currently play audio though Apple's Inter App Audio or Audio Unit interfaces, nor with Audiobus, but it can send MIDI to apps which do support these features.
Instruments are categorized into sections, each with its own icon and background color, accessed by the scrolling control on the far left. The instruments associated with those categories are correspondingly colored.

User instruments are made by importing sound files. If they follow certain naming conventions, they can be organized into individual instruments. If there are no User instruments, the background color for those options are gray.
The other parts of this control are as follows:

Other controls are: These parameters are saved in the user's _setup file (see Setups).


Set Looks

The Looks screen controls lets you change the sizes, layouts, and colors of some interface elements.
Pick which element you want to change from the "Cursor", "Boxes", or "Backgr. (background)" selector button.

By touching in the array of colors, you change the color of the selected element.
The box under the Opacity slider will show you the current color setting.
The "Opacity" slider lets you change the color's opacity (transparency).
For Boxes, this chooses the color that the sound_box is colored. Boxes are always transparent, but you can make them more transparent with the Opacity slider.
The top line is purely black and white, the line underneth shows very subtle shades of color!
The "Size" "slider is used to change these sizes:

It's quite possible to set all the colors the same for a really minimal and rather mysterious kind of screen.

The Lock switch locks the shape of the layout. This disables the ability to rescale the bound box of the layout with two-fingered touches. This switch can also be interactively turned on and off with a four-finger touch, if the screen is not in Installation Mode. The best way to do this is to put one finger down on the screen and use the other hand to do a three finger tap! This way, you don't accidentally resize the layout.
Another way to toggle the layout lock is to resize as usual with two fingers, then keep holding them down until more than 5 seconds has passed. This is also true if the layout is already locked, just touching and holding for 5 seconds will unlock it.
When the layout is locked, the Looks button will show a Lock icon on it to remind you of that fact.

The Video switch lets you choose between showing live video or just using the background color. It has the same function as the "Hide Video" switch in the video camera and tracker configuration dialog.

The layout buttons select whether the sound_boxes are laid out horizontally, vertically, in a grid, or in a circle.
The direction arrow is used to reverse the order that the sounds are laid out in for the Horizontal, Vertical, and Circular layouts.

Play Vertical
Play Horizontal
Play Grid
Play Circle

The Box Width slider can be used to make the sound boxes skinnier, thus putting more space between them. This way, it's easier not to play an adjacent sound box by mistake! AUMI will play the sound if you cross over a box quickly as well, even if it's only 1 pixel wide, so being skinnier is not a disadvantage that way.

When you are using the MIDI instrument, and you are not using the Cursor's velocity, and if your layout is vertical, the position between top and bottom of the row of sound_boxes is used as the note's "velocity," so that near the bottom of the grid is "low velocity" and near the top of the row is "high velocity". For a horizontal layout, left is "low velocity" and right is "high velocity"." These values are taken from the low and high velocity settings in the Instrumnet control.

To dismiss this control on an iPad, tap anywhere outside the popup. To dismiss this control on an iPhone, tap "Done".

Notes about the Grid
gridded layout This gridded layout assigns notes in a special order that tries to associate nearby pitches with nearby Sound Boxes. This picture gives an idea of how the notes are ordered chromatically.
 gridded layout
In this image, you can see they make more sense in a diatonic scale.
There can be up to 121 sounds available!

Sound Order

This screen is accessed by touching the "Sound Order" button on the "Play AUMI" screen.
The Sound Order dialog allows assign specific sounds to Sound Boxes. It's only enabled for Percussion, Noises, Loops, and Relative Motion instruments, because the sounds of these instruments do not have to be in a particular order

To quickly move a number of sounds from active use, select their names and touch the "Down Arrow". Similarly, if you want to use instruments from the middle of the list, select them and touch the "Up Arrow". Now you can easily re-order them!
Pressing the Sound Order button shows you an interactive list of all the sounds available for that instrument. The ones currently playing are in black, and the ones that are currently not playing are shown grayed out. You control the number of sounds that are playing with the slider at the top of the screen.

The Loop instrument, for example, can play from 4 to 32 sounds, but more sounds are to be found in the list which can be moved into place so that they can be heard.
Similarly, if you set up a Percussion or Noises instrument with just two sounds to play, you can choose those sounds by moving them into place using the sounds screen.

This alternative order of sounds is saved with the instrument if you use the "Save Setup" button on the Setup screen.

MIDI controls are a separate screen on iPhones:


The MIDI screen is included as part of the Instruments screen on iPads.

If the current instrument is a MIDI instrument, you can use these controls to select a MIDI channel and MIDI program settings.

Different synths try to discover MIDI sources in different ways, and sometimes, it's trickier than it looks. AUMI will try to connect to every MIDI receiver that it sees, even if they are added while AUMI is running. Apple's "Camera Connection Kit", when connected to a hardware USB-to-MIDI adapter, will let you attach AUMI to an external hardware MIDI synthesizer. There are other MIDI interfaces available as well.
To have AUMI play another synthesizer or synthesizers on the same device:
  1. Start AUMI.
  2. In Instruments, choose a MIDI instrument. Pick a scale and note range too, if you like.
  3. Make sure the MIDI switch is turned on.
  4. Click the home button (on devices that have them) or swipe on the Home indicator (those without a Home button) to put AUMI in the background (it's still running)
  5. Start an iOS MIDI synthesizer like Garage Band, Bismark BS-16i, KQ Dixie, Animoog, Thumbjam, or another MIDI compatible iOS app.
  6. Make sure the synthesizer can play using background audio. This is usually a switch in the MIDI section of the app's settings and information section
  7. Click the home button (on devices that have them) or swipe on the Home indicator (those without a Home button) to put the synth in the background, and click on AUMI's icon again.
  8. It should be able to play the MIDI synth now.
AUMI is not an Audio Unit, nor does it support Inter App Audio. AUMI's sounds cannot be played via MIDI either.
You can also use a local network to play instruments on a nearby laptop or desktop. That's a little more involved; I'd suggest you look at tutorials about the Apple Audio MIDI Setup utility, like this one by Joe Stallings.


AUMI is capable of generating two kinds of reports if this feature is enabled. If you are considering the use of the reports feature, please see LEGAL. These reports are in the form of CSV files. .

privacy notice

There are two kinds of events that AUMI optionally logs: Movements and Sound Box Events. These are enabled by switches on the Setup screen.

When you enable logging, you'll see a privacy notice in a red box appear with the text:

                AUMI collects usage data on this device when this
                switch is turned on, data which is only stored
                locally on this device.
                You can switch this option off at any time. You can
                also either export or remove this stored data.
                For more information, read the Records and Legal
                sections of the Instructions.
You can dismiss this notice by tapping the "DONE" button.

Saved Setups does not save the state of these switches. This is to make sure these switches are always set manually. This is true for remotely sent and initially loaded setups as well.

info for reports, iPad version These fields open up when you choose to enable reporting. (iPad version) They are included in the csv files that the reporting makes, so that you can track several players (entered as part of the setup names) and programs on the same device. For instance, you could call a setup "Janet's Kitchen Percussion".

Movement event logging

When logging is turned on, a tiny logging notice appears in the upper left corner of the screen to remind you that it's logging.
logging mark

When movement logging is enabled with the optional switch "Log Movements", every 15 seconds, a record is cut with some information that may be used to make reports in an external spreadsheet program like Numbers or Excel. This information may be useful for tracking the progress of individual AUMI users in different programs at various institutions.

The movement file is created with a name in the format AUMI-YYMMDD-HHMMSS.csv, so that the date and time are part of the file name, e.g.: AUMI---121206-092201.csv

Also, when the program transitions from foreground to background, it will also log this event. In this record, the timestamp and identifying information are retained, and the position related fields will be blank, but the Scale field will say FOREGROUND or BACKGROUND.

Sound Box event logging

There's a second kind of report that is enabled with the optional switch "Log Sound Box Events".
It tracks the entry and exit from Sound Boxes, and records the time spent in each one, and its file has a slightly different name from the movement report file:
AUMI-EL-[institution]-[program]-[date and time].csv

Reviewing, deleting, and exporting logs

List Reports
The Reports button will present you with a list of all the report files AUMI has made. They are ordered by date, and the two types of files are in different sections of the list.

When you are ready to send out the reports, tap SEND. Any program, like Apple's Numbers, that can read a text file will be listed in the dialog that pops up. You can also use Air Drop to send the files directly to any other Apple device that supports Air Drop, for instance, a desktop machine. You can also mail or post the files to various services, and you can also cancel the dialog.

All these files, are also available in the iTunes Document interface for AUMI. The main log file is also written whenever you leave the program.

Each record has the following information:

and the event report fields are these: The "Version" is currently "lv_1.0", and the version also records the device type and AUMI's release number. This way, you can tell users and instruments apart, and send files to different institutions.

For example:




If for some reason you don't want to Send or Mail the reports externally, you can find them in the Files app, in "On My iPad"'s AUMI folder. But if you are running on on older operating system, you can get your reports by:
  1. plugging in the device into a computer that knows about it. Usually, that's the one that was used to set it up or back it up.
  2. open iTunes
  3. find the device in iTunes's top bar:
    find device
  4. click on that and click on 'Apps' in this list of buttons:
    find app
  5. scroll down to where it says "File Sharing"
    find file sharing
  6. scroll down that list and click on AUMI:
    find AUMI
  7. The box on the side marked "Documents" will show all the sessions it has recorded:
    find Documents

All the personal set-ups are also in the Documents Directory.
You can select them all and click "Save to..." to save them in a directory that you can remember. They don't take up a lot of space, and you can also just delete them after you've saved them.

Video Video Configuration Dialog: motion
Video Camera Settings There are many options to customize the video camera here:

Front or Back Camera: AUMI can use the front camera or the back camera. The video image of the front camera is flipped like a mirror, but the rear camera remains the same.

Hide Video: This switch lets you control whether you see the live video on the screen or not. Some players may not want to look at themselves while playing, or you may just prefer a solid color instead of video for some performances. When Hide Video is off, the screen will show what the camera sees and display the live video image behind the Sound Boxes, controls, and Cursor. When Hide Video is on, the screen will show a solid colored background instead. This background color can be set using the Set Looks screen controls.The Cursor, sound_boxes, and tracking_spots are unaffected by this switch. Select the segment "No" or "Yes". There is an identical switch in the Set Looks screen control dialog.

AUMI's camera resolution is initially set low to accommodate slower devices, but on newer, faster devices, you can use a higher resolution if you like. The "Zoom" feature will let you zoom in to the center of the image, and reduce the resolution, thereby significantly lowering the latency of the tracker. In fact, if you set the camera to a high resolution setting, and zoom way in, you can more easily track the motion of eye blinks and tongues.


  • Lowest resolution (192x144)
  • 352x288
  • 640x480
  • 1280x720

Zoom: If the camera is too far from the user, Zoom can help make limited motion more effective. It also speeds up the interactivity a lot, because there is a smaller image to track!


AUMI uses software called a "tracker" to analyze the video images in order to know how to move the Cursor.
These tracker settings may help you optimize AUMI when you have a faster iOS device, or want to experiment with the responsiveness of AUMI.

  • The Motion Tracker looks for high contrast parts of the image (features or tracking spots) and tries to correlate their movement from frame to frame. The average direction of this movement is used to move the Cursor. Some settings related to the Motion Tracker are detailed below.
  • The Face Tracker looks for a face in the image and puts the Cursor right on the nose. When it cannot find a face, it leaves the Cursor where it is.
    Video Camera Configuration Dialog: face
  • The Color Tracker looks for a specific color (really, a set of colors) in the video image and tracks the center of the largest contiguous area of those colors that it finds.
    There's a debugging switch, "Color Box", that draws a bounding box around the largest contiguous area that matches the color set. It also changes the video's colors to show the limited color set that it uses for tracking instead of the vastly larger number of colors normally seen. This can help you pick out colors sets.
    The Color Tracker is very fast and can be used with high resolution video! A good tip is to put a brightly colored sticker or button on a stick, a hat or a glove, and set it to track that color! Holding a colored ball or putting on a bright red clown nose also works well.
    Video Camera Configuration Dialog: color
  • The Distance Tracker is a variant of the Color Tracker that uses the area of the largest contiguous color to control the Y-axis placement of the cursor. Since that area can be correlated with the distance from the camera, it's a good way to track objects that are approaching and receding. The X axis is unaffected. With carefully thought out layouts, this can be a very efective way to play AUMI.
  • The Touch Tracker is an official way to play the screen with just a touch. No video is processed at all. As a side effect, the recentering of the cursor feature is turned off, and you can tap on a sound box repeatedly to play it.

Next to the tracker selector is a toggling button SHOW SPOTS / HIDE SPOTS (SHOW AREAS/ HIDE AREAS for Color Trackers) that shows or hides the Tracker's tracking spots for debugging purposes.

  • For the Motion Tracker, each little numbered square on the screen is one of the tracker's tracking_spots. The dark blue ones are spots that have moved enough to contribute to the motion of the Cursor, the paler ones are being ignored until they move more. It's useful to find out what it is looking for, based on the tracker settings.
  • The Face Tracker also shows icons for the eyes and mouth with little eye and mouth icons. This gives feedback on whether the Face Tracker has detected a face or not.
  • The Color and Distance Trackers will not "show spots" (there are none), but will color the areas that match the tracked color a lighter shade of whatever color it is.
  • The Touch Tracker shows no spots - it's not tracking any video!

Each kind of device and working environment has its own best parameters for working with this kind of tracker.
 The Motion Tracker has these configuration options:

Motion Tracking Settings Hints

These setting can work in combination to make the visual environment emphasize motion by the AUMI player rather than motion of objects in the background, shadows, or other things out of your control. For example: The Face Tracker has no configuration options.
The Color Tracker has no configuration options per se, but it has a preset collection so you can save and load colors to track without otherwise needing to select them. However, it takes an extra step to set up which color to track. The Touch Tracker has no configuration options.

Timing Configuration Dialog
"Timing Quantization" is the process of playing sounds on even beats. Normally, when a Cursor touches a Sound Box, it immediately produces a sound. Sometimes, you can have more control if playing the sound related to that contact is delayed until it appears on a specific beat. AUMI decides what to do with the events that come in and schedules the playing of sounds to occur only on the beat.
Different styles of quantization let you change the timing of the beats and schedule more than one note to play from a single entry into a Sound Box. You can choose from these by choosing a "Note Pattern". If the sounds are spread out in time and a new sound box is touched, usually this will restart the sequence of sounds when the pattern has finished playing.

Here are the Note Patterns:
Characteristics of these "Beats" can be configured with the sliders, and small changes can be set accurately by using the "<" and ">" buttons next to each slider.
The sliders control the following things:

AUMI makes it easy to add pre-packaged sets of sounds as optional instruments. This way, custom sounds for a particular user or group can be easily installed on a number of devices. Since they are optionally loaded, they don't enlarge the original AUMI download or clutter up the lives of people who might not be interested in them.
A collection of pre-made AUMI instruments is available on's AUMI Instrument page ( Just tap on one and it will ask to launch AUMI, and install it right away and let you play it. This page is constantly growing with a collection of unusual instruments and sound sets.
If you are connected to the internet, these instruments are also easily available from within AUMI by touching the "More" button on the Instruments screen. Read about MORE here There are two ways to delete a User Instrument: One is to use the "Delete Instrument" button in the instruments controls, or you can delete individual sounds in the Sounds Control. If you choose to delete all the sounds, the option to delete the whole instrument will present itself.
An AUMI Instrument can be prepared by the AUMI staff out of your sound files and put in this list for the whole AUMI community, just contact! Or you can make one yourself:

or, if you are on Windows, or, on Linux or other POSIX systems: These files can be loaded into AUMI from a web page by using the HTTP scheme "aumiinst:", for instance,
<a href="aumiinst://path/to/whatever.aumiinst">Whatever</a>
You can also text or email an aumiinst file as an attachment, click on it on a device, and it should launch AUMI and load it up in the same way.

Somewhat adventurous AUMI users or administrators can add their own sounds into AUMI in a limited fashion.

By using some naming conventions, you can organize the files into separate melodic and percussive Instruments. If the instrument you created has more than 4 sounds, it can also be used as a Looping instrument or Relative instrument.

Here's how to add sound files from a desktop Mac using the Finder (after MacOs 13/Catalina): Here's how to add sound files from iTunes File Sharing interface (before MacOs 13/Catalina):
You can also share sounds in an iCloud Drive folder so they are available for several devices.

Naming Conventions

The sounds can be turned into different kinds of Instruments if they follow this naming convention: If you want to organize your sound files into separate instruments, you can follow this naming convention of having then all start the same way, but with an underscore separating the name from the "MIDI note number" or other distinguishing string. For instance: You can also put them in a directory with an instrument name. This is what actually happens with "AUMI Instrument" files. The "User Sounds" Instruments can be saved in a Setup (using "Save Setup"), but because of the possibility of sound files being added or removed if it is loaded on a device that does not have one of these files, it will put a tilde (~) in front of the name and play Silence instead if it can't find sound files when it is loaded. The same goes for sharing sounds to other devices via the Listen and Send interface.

Many apps like "Audio Share" or other editing, recording, and synthesis programs, have a sharing button that can send a sound file to other programs. AUMI is now a destination for this kind of sharing. The supported sound formats are : mp3, aif, aiff, wav, and m4a. I'd suggest you keep the sounds under 30 seconds long.

AUMI now can read extra metadata in a file included along with the AUMI Instrument sounds named "metadata.json". If this file is missing, generic information is provided, but this gives a little more flexibility to the AUMI Instruments. At some point, all the official AUMI Instruments will have this metadata included!

The metadata file is a JSON file which may expand in future releases. An example is this one:
                    "uiname": "Bamboo Clarinet",
                    "desc": "Bamboo Clarinet in the Widow Jane Mine",
                    "tags": [
                    "types": [
                    "created": "20220613T194521",
                    "icon": "\ud83c\udf8b",
                    "author": "jhhl",
                    "user": true,
                    "source": "",
                    "version": "1.0",
                    "znotes": "melodic"
As you can see, it gives user interface names and descriptions, tags (that aren't used yet), a list of types so it knows what type of instrument it is, where the AUMIInst can be found online, an icon (usially an Emoji or two), and some other metadata that may turn out to be useful. The Icon and Description show up in the official listing for the instrument, replacing the generic icon for it.
The choices for "types" are as follows: melodic,perc,loop,rel and so can make sure an AUMI Instrument can only be found, say, in the User Loops section. The metadata file is future-proofed with a Version number string.
If you are in the User Sounds and AUMI Instruments screen, a "More" button becomes available. If you are connected to the internet, you will see this control:
More Aumi Instruments Screen
This list of instruments comes from the Official list of AUMI Instruments hosted on, which gets updated from time to time.

This list has a lot of information in each line, and you can scroll through it, reorder it, or directly search for instruments.

The sorting options are: The Search field will search the names and decriptions, and pop those results up to the top of the list (with colorful backgrounds), and the rest of the sounds will follow. It's case insensitive.
To select one of these for download, tap on the line with the file info in it. It will start downloading, and sometimes this takes a while, but when done, it will load it up and put up a notice telling you of succcess (or failure, should that be the case). Sometimes it had downloaded, but wasn't installed for some reason, so tapping it again might work.
This mechanism of automated downloading is now part of the Setups, so if the setup knows where the sound came from, it can load it up from there if sent to another AUMI instance.

There may be a case where you want to access an unofficial sound collection through the More Button. This is a bit of an advanced feature, since you need to set up the API call that provides thisinformation, but it is possible. Email if you wish to try this out. Otherwise, you can always make a web page with saved instruments and setups on it that can load themselves into AUMI with a click , as long as there's an AUMI on the same device. This is detailed in the User Sounds and AUMI Instruments section of these instructions.



You can register for news about AUMI by clicking on this MailChimp link or visiting the AUMI website's "Get Involved" page at


AUMI for iOS collects data about how the app is used. To learn more about what kind of information is being stored, see Reports. This information is stored only on the device AUMI is running on. It is not shared or uploaded to any service.
Using AUMI's reporting interface, these usage records can be reviewed and exported for further reporting needs, such as research on the progress of a group of AUMI users. The export can be done in a number of ways to any apps which accept text files. These records can have metadata included in them that can identify the user, program, administrator, and institution if the person filling out that metadata wishes. Each file can also be deleted from within the app.

It is the responsibility of the person transferring the data to ensure that the privacy of the AUMI Users is in accordance with the laws in their jurisdiction, and the wishes and consent of the people whose actions are recorded in the data which is being collected and retrieved.

A privacy policy document is currently under review. Contact us at if you have any questions. A preliminary version is visible at:


AUMI for iOS was developed by Henry Lowengard for the Deep Listening Institute, now called "The Center for Deep Listening at Rensselaer" and headquartered in the Center for Cognition, Communication, and Culture. AUMI for iOS adapted from the desktop version of AUMI, also available from Thanks to Pauline Oliveros, Leaf Miller, Jaclyn Heyen, Emily Halstein, Lisa Barnard Kelley, and Al Margolis of DLI, and the AUMI desktop developers:
Zane Van Duzen,
Zevin Polzin,
Doug Van Nort,
Ian Hattwick,
Aaron Krajeski,
Ivan de Almeida Soares Franco,
John Sullivan

For the AUMI Consortium: Prof. Sherrie Tucker (University of Kansas), Prof. Eric Lewis (McGill University, Montreal), Thomas Cuifo (Mt. Holyoke), Dr. Ellen Waterman, Gillian Siddall, and Jesse Stewart.
For Rensselaer, thanks to Prof. Jonas Braasch, Prof. Tomie Hahn, and Dean Mary Simoni, John Fisher.

Some audio was generously provided by these musicians:

Many more "AUMI Instruments" can be found online's AUMI Instrument page ( .
iOS, iTunes, iPad, iPad Pro, iPad Air, iPhone, iPod Touch, GarageBand are registered trademarks of Apple, Inc.
AUMI was developed with the support of the The Gould Family Foundation, Improvisation, Community, and Social Practice (ICASP), and International Institute for Critical Studies in Improvisation (IICSI).
Futher AUMI development is funded through a grant from the
Craig T. Nielson Foundation.
Audiobus is © Audiobus,
The feature tracking software is derived from KLT (Kanade-Lucas-Tomasi) from Stanford University.
The tutorial on Apple's Audio MIDI Setup Utility is "Using Your iPad as a Wireless MIDI Controller" by Joe Stallings.
The unzip code "SSZipArchive" is licensed as follows:
                    Copyright (c) 2010-2015, Sam Soffes,

                    Permission is hereby granted, free of charge, to any person obtaining
                    a copy of this software and associated documentation files (the
                    "Software"), to deal in the Software without restriction, including
                    without limitation the rights to use, copy, modify, merge, publish,
                    distribute, sublicense, and/or sell copies of the Software, and to
                    permit persons to whom the Software is furnished to do so, subject to
                    the following conditions:

                    The above copyright notice and this permission notice shall be
                    included in all copies or substantial portions of the Software.


AUMI websites:

AUMI © 2012-2014 Deep Listening Institute, Ltd., and 2015-2021, Center For Deep Listening at Rensselaer
Developed by Henry Lowengard
compile date:   build version: