AUMI: Adaptive Use Musical Instrument
Musical improvisation for everyone by turning motion into sound.
The Adaptive Use Musical Instruments (AUMI) software interface enables the user to play sounds and musical phrases through movement and gestures. This is an entry to improvisation rather than "hitting the right notes" or playing set pieces of music. Instead, the software uses music as a way for participants to express a range of affects, both by themselves and in response to, or in conversation with, others.
While the AUMI interface can be used by anyone, the focus has been on working with children who have profound physical disabilities. In taking these participants as its starting point the project attempts to make musical improvisation and collaboration accessible to the widest possible range of individuals. This approach also opens up the possibility of learning more about the relations between ability, the body, creativity and improvisation, from within a cultural context that does not always acknowledge or accept people with disabilities.
AUMI continues to be revised and improved with input from the technologists, students, therapists and with feedback from registered users. An on-site training program is now available.
If you see a word with a dotted underline, it's a Glossary item. Touch it to get a quick hint about the term. Touch it again to dismiss the window.
AUMI for iOS was developed for the Deep Listening Institute by Henry Lowengard and adapted from the desktop version of AUMI, which is also available from the Deep Listening Institute, Ltd.
You can get software support by emailing: firstname.lastname@example.org
This link points to a PDF copy of these instructions. You can export them to iBooks or print them out for reference.
AUMI was developed with the support of the The Gould Family Foundation.
This is iOS AUMI version
SETTING UP THE AUMI PLAY SPACE
AUMI uses the front-facing video camera to track motion and control which sounds are played. For this to work optimally, good lighting, a neutral background, and a steady support for the iPad, iPhone, or iPod Touch being used are needed. The more neutral the background, the better AUMI will be able to find intentional motion. The Apple iPad folding case is a pretty good support, but you can also use commercially available stands and mounts, and in a pinch, some big rubber bands, a chair or a pole, and some creativity will also do.
PREPARING YOUR DEVICE
AUMI runs on all iOS devices - iPhones, iPod touches, iPad, iPad mini, iPad PRO - running iOS 8.2 and above which have front facing cameras. You will need to confirm that you want to use the camera in a dialog the first time you use AUMI.
For a better experience, you may want to:
- Adjust the screen brightness for the ambient lighting in the room
- Turn off multitouch gestures - although sometimes I find them useful to switch back and forth to MIDI instruments running in the background
- Turn off key clicks
- Turn off the notification pulldown
- If it's an iPhone and you are in a performance situation, go into "Airplane mode" and it won't spend time looking for cell service or WiFi, which AUMI doesn't need
- Start a MIDI instrument first if you want AUMI to play it
You can play AUMI right away when it starts up, or by using the Play AUMI tab bar button to change the screen to the play screen. You will see a live video image, a circle with a red dot in the center (the cursor), and a number of rectangles (the "sound_boxes"). You can rescale the sound_boxes by touching the corners of the enclosing box with two fingers.
If the cursor goes to the edge of the screen for 2 seconds, it will automatically recenter itself!
You can then put the cursor anywhere on the screen with a single touch. You can change various aspects of these interface elements with the Looks
button. You can hide and show all the toolbars, and make a completely clean (and safe to touch) screen by tapping with three fingers.
To change the _instrument , the number of available notes, scales, and other aspects of the audio and video of AUMI, use the _Instruments button, which is on the video screen for iPads and in the tab bar on smaller devices.
has a Login screen
. This is a place to identify a user, and other metadata useful in data in the reports AUMI
makes. If you are not interested in those reports, you can leave those spaces blank. To help with filling out a name, you can pick one from the Address Book. You can organize users into Groups in the Address Book app (outside of AUMI
, that is), and use that to help set the name without a lot of typing. The first time you access the Address Book, AUMI
will ask permission to use it.
A name is also used to identify saved instrument _setups. If you have already saved some setups, you can load one from a list with the "List Setups" button.
The Login page in no way affects the ability to play AUMI, nor is this information sent to any server.
There are a number of adjustments that can be made to the tracker software that can help it work best for your device and the environment it will be working in. These adjustments are found in the Settings app, in the AUMI configuration screen.
AUMI can refine the way it turns movement into moving the cursor. Read the _Instruments section to find out more.
SOME TIPS AND HINTS
Some AUMI users are only capable of extremely limited voluntary motion. AUMI can be adjusted to accommodate their needs in a number of ways:
- You can rescale the sound_boxes by "pinching" the screen. If you put a lot of sound_boxes in a small space, smaller motion is more likely to initiate a sound!
- You can zoom in to get faster response. It's not really "zooming" as much as taking an excerpt of the middle of the image, but that may help AUMI find the motion it needs to play sound.
- The "Mult" slider in the _Instruments screen can multiply the amount of motion that the tracking software finds, so that it can move the cursor more with each motion.
- You can get feedback on what AUMI is "interested" in tracking by turning on the "Show Spots" button in the Instruments screen. The screen will display little labels at the positions of the tracking_spots it has found. The blue ones are points that have moved enough to be considered for moving the cursor, while the paler ones are currently being ignored until they move more later.
- AUMI can play over other music making programs and synthesizers if they can play in the background. Improvise with AUMI over an iTunes track!
- AUMI's' MIDI playing feature allows a user to play anything that understands MIDI "note on" and "note off" commands, which are not only synthesizers and samplers, but lighting consoles and other media devices. AUMI can control other MIDI instruments on a laptop connected via Wi-Fi, for instance, in Apple's Garage Band, and also other apps running on the same device. With Inter-App Audio and Audio bus, those other apps can add effects and recording and sharing features missing from AUMI.
I have a setup that lets me record AUMI as it's operating - it takes some money and an external computer. You might be able to do it with hardware: an HDMI adapter and cable to a video recording device. Modern iPads and iPhones can transmit video and audio over WiFi to an AirPlay(™ Apple) equipped device, such as a flat screen TV or video recorder. My solution is in software: it transmits a lower-res video (and audio) over to a computer that can pick it up and record it, using a program called AirServer. Sometimes it's a little finicky, but that's how I've made some videos of AUMI.
I'd suggest just using an external video camera to record both AUMI and the real performer using it, as you would in the past. You have a little more control over it that way!
This is a quick description of the toolbar and interface buttons. Click on them for more details!
The interactive video interface.
Pick _Instruments, set the volume, and adjust the tracking features. (Not here on iPads)
Configure AUMI's MIDI playing features. (Not here on iPads)
Register who is currently using AUMI, so activities can be recorded, Load and Save instruments, send and recieve instruments, and Create Reports.
AUMI PLAY PAGE TOOLBAR
Change aspects of how AUMI looks, such as colors, sound_box layouts, and sizes. This is launched from the "Looks" "button at the left of the main video screen.
Assign sounds to sound_boxes. This is launched from the "Sounds" button at the left of the main video screen.
The user interface is different on iPads and iPhones. On iPhones, the Instruments
buttons are on the MAIN TOOLBAR. On iPads, the Instruments
control button is part of the AUMI PLAY PAGE TOOLBAR, and contains the MIDI controls as well.
AUMI works on iPhones and iPod touches that have front-facing cameras, but is best on iPads and iPad Minis. Newer models will be faster!
The AUMI Web site at deeplistening.org is here
, featuring more information, videos, support, and tips and hints.
The Login page is where you can set up a name and other metadata here used in recording activity during an AUMI session.
You can use the address book button to get access to the device's contacts to find names more easily. When you select a name, it is put in the name field.
The report button
brings up an interface for exporting the AUMI report files. These files are a comma separated value files (CSV), supported by most spreadsheet software. See the reports
section for more details.
You can load and save an _instrument and other configuration data, which is called a "_Setup"," with the List Setups,Load Setup and Save Setup buttons. The Save Setup button saves the current instrument and its layout, colors, and sound order to a file named the same as the user's name, e.g. [user name].aumi.
You can save other instrument setups for a user by adding a note to the user name, e.g. Clara K. sitar.
The idea is to load a Setup appropriate for that user, so Load Setup
looks up the Setup file using the current user name. If there is no name, it tries to load a Default Setup, which you can create by saving a Setup with no user name.
You can also pick the Setup out of a list using the List Setups
button, which will also pick the user.
will also let you rename or delete existing setups by touching the appropriately named buttons. Done
will dismiss this dialog.
You can also find and export these setup files or import them by using the iTunes Document interface
. They will have the suffix ".aumi". "An ".aumi" file is an Apple plist file, which is a kind of XML file, and so is legible by humans.
Sending setups is an experimental feature!
You can set up device running AUMI
so that you can send a setup from one to the other. This allows a therapist's device to change aspects of the player's device without needing to touch the screen.
This is using Apple's "Multipeer Connectivity" feature. That means all the devices involved should be on a local network, or have Bluetooth enabled, or it can also communicate peer to peer (that is, via WiFi without actually being connected to a WiFi network).
You can designate a device to listen for setups by touching the "Listen for Setups" button. You can actually do this with up to 7 devices!
On another device running AUMI
, you can then send the current setup on that device to the listening device(s) by touching "Send Setup". You'll be presented with a list of "nearby" devices that are listening as it discovers them. You can now touch the device names of the devices you wish to send the setup to. It should connect and change the setup on the listening device, and put up a notice that tells you so. If it fails for some reason, there will be another notice telling you that.
This is the main interface to be used while playing AUMI (iPad version).
Over a live video image, AUMI shows a set of rectangular "sound_boxes" and a circular cursor, which is what initiates sounds as it moves into the sound_boxes.
The sound_boxes for a vertical layout actually stretch to the edge of the top and bottom of the screen. The horizontal layout behaves similarly with respect to left and right.
You can also set the position of the cursor by touching the screen.
When the cursor enters a sound_box, the corresponding sound is played, and the box highlights relative to the volume of the sound that is playing.
AUMI works by detecting and tracking the movement of "interesting" parts of a video image. "Interesting" in this case usually means high contrast and angular shaped parts of the image. The tracker works best when there is clear lighting, and a blank background behind the user.
The relative motion of the tracking_spots moves the cursor on the screen. The cursor has no direct relationship with the tracking_spots. This allows any motion it detects in the image to control the cursor, and thereby initiate the sounds.
There's a toggle button, "Show Spots," in the Instruments screen, that makes the tracking_spots visible when clicked. Click it again to hide the spots.
You can change the size of the rectangle the sound_boxes are laid out in by touching the screen with two fingers. The touch points move the top left and bottom right corners of the box.
The layout of the rectangles can also be changed, except for Loop instruments, by using the Looks control. The layouts are Horizontal, Vertical, Boxed, and Circular. The Boxed and Circular layouts enable users to play notes out of their usual order.
That same control can change the color, transparency, and thickness of the sound_box outlines, and the color, transparency, and size of the cursor.
On an iPad, the "Instruments" button appears on this screen, while on an iPhone, that button is on the navigation bar.
You can hide and show all the toolbars, and make a completely clean (and safe to touch) screen by tapping with three fingers.
On iPads, the Instruments screen is accessed with a button near the top of the Play AUMI screen. On iPhones and iPod touches, it's in the tool bar.
The Instruments screen is where you select an instrument and its properties, set the general volume, and set video tracking options.
A big scrolling instrument picker is divided into four parts. Some instruments don't use all the parts, so they are blanked out.
Instruments break down into four types, each colored differently:
- Melodic instruments (green). They have a range of chromatically tuned samples that can be used to play melodies. Most melodic instruments have a range from MIDI note 36(C2) to MIDI note 96 (C7).
- Sample instruments (white): They simply play samples that are not tuned or in any particular order. The order can be reassigned by using the Sounds button.
- Looped instruments (yellow) : All the sounds play simultaneously in a continuous loop. The closer the cursor is to a sound's sound_box, the louder that sound will play. If you make the size of the cursor bigger, it will cover more sounds and you will hear more of them simultaneously in the mix. The choices for loop sounds and the number of them on the screen can be changed by using the Sounds screen.
- MIDI instrument (blue): Makes no sound by itself. It sends MIDI messages to other programs or devices.
The parts of this control are as follows:
An instrument selector. Each instrument name has a small image in front of it to help identify it.
The instruments are:
Melodic sounds: these can use scales in different keys.
For reference: MIDI note numbers for these pitches:
c2=36, c3=48, c4=60, c5=72, c6 = 84, c7 = 96
- Piano: piano, C2-C7
- Guitar long: synthetic strings, C2-C7
- Guitar short: synthetic strings, shorter than before, C2-C7
- Tines: a sampled music box, C4(60) to C7(96)
- Reedy: synthetic reed instrument, C2-C7
- Bowed: synthetic string instrument, C2-C7
- Fifths: Looping Fifths, pulsing every second, C2-C7
- Steel Drum: C2-C7
- Kalimba: C4-C7
- Synth Drum: C2-C7
- Woodblock: C2-C7
- Orchestra Hit: C4-C6
- Sitar: C2-C7
- Timpani: C2-C4
- Tubular Bells: C2-C7
- Glockenspiel: C5-C7
- Tenor Recorder: C3-Ab4
- Non-Melodic sounds: these can be arranged in any order, but do not have scales.
- Percussion: sampled percussion sounds.
- Noises: experimental rock band noises (Dr. Nerve's Nerve Events).
- Kitchen Percussion: sounds of pots and pans (The Institute of Contemporary Music).
- Cartoon Sounds: cartoon sound effects. (Screwy Squirrel, MGM)
- String Quartet: The Soldier String Quartet and vocalist Shelley Hirsch.
- Dogs: Dogs barking (http://www.freesfx.co.uk).
- Cats: Cats meowing (from a YouTube compilation).
- Waterphone: Waterphones from Jesse Stewart's Xenophora: Music for Solo Waterphone.
Loops: these are a circle of continuous sounds that can be mixed together.
The closer the cursor is to a box, the louder the loop it represents will play.
This way, slow movements can still play along with other musicians playing loops as the same rate, although some loops are more ambient and do not mesh synchronously with other loops.
There can be from 4 to 12 sound_boxes to play with loops. You can adjust the number with the slider in the "Sounds" popup screen.
- Loops: An instrument where the mix of 2 second long loops is controlled by the cursor position.
- Marsh Loops: The sounds of a marsh, from Dave Soldier's Marsh Fugue
- Fifth Loops: Looping Fifths, pulsing every second which are arranged so they harmonize
- Tran Phu: Recorded in Vietnam by Kala Pierson
- Pin Pia: A traditional Thai instrument recorded by Dave Soldier
- Synthy Loops: A collection of 2 second electronic loops.
- LP Lock Grooves: A collection of loops that will synch with vinyl lock grooves at 33 1/3 RPM. (1.8 second loops)
- Glyptodont: Sounds from a recording of Skip La Plante's Glyptodont.
- Troglodytes: Sounds from Deep Listening Band's recording Troglodyte's Delight.
- Widow Jane Mine: Tenor Recorder played in the Widow Jane Mine.
- Waterphone Loops: Waterphones from Jesse Stewart's Xenophora: Music for Solo Waterphone.
Relative Motion: All the sample playing instruments - melodic or not - can also be used in relative mortion mode. Relative Motion versions of these instruments are marked with a "REL" sign at the end of their names.
Relative motion uses the tracker to determine whether the motion was left, right, up or down, and plays a sound corresponding to that motion. The same "sounds" dialog is used to assign sounds to the directions (in the order Down, Right, Up Left).
- MIDI: This will send MIDI events to either a connected MIDI device (via the "Camera Connection Kit" or other MIDI interface), or to a software MIDI synth running on the same device or within the network.
Read more in the MIDI section.
Adventurous users can add their own sound files to AUMI to be turned into instruments. Read the User Sounds section for details.
- The base note if the instrument is melodic. This is like the tonic of the "key" of the selected scale. Note that many of the scales are a little more complicated than usual major, minor, and chromatic scales.
- The kind of scale to use if the instrument is melodic:
The scales are built dynamically. They start on the "base note" and create the scale by adding a cycle of intervals to that base note.
If that base note is lower than the lowest note available in an instrument (like C4 for the Tines), the scale will be played in the base note's key, but only using notes within the range of the instrument.
- Chroma: a chromatic scale
- Major: a major scale
- Penta: a pentatonic scale
- Minor: a minor scale
- Maj Triad: a major triad I-III-V scale
- Min Triad: a minor triad I-IIIb-V scale
- 7th: a seventh I-III-V-VIIb scale
- up down: a melodic scale that goes in a pattern of up a major third, up a minor third, down a fourth, and up a minor third again.
- Blues: a bluesy scale
- Pelog: a beautiful scale based on Pelog Selsir, a Javanese scale.
- The number of sounds to put on the screen in "sound_boxes". Note that AUMI skips over some of the higher choices to save a little time and space. Loops can play from 4 to 12 sounds or the maximum number of sounds available if less than 12. You can also set the number of sounds from within the Sounds control.
Other controls are:
- Show/Hide Spots: A debugging feature that shows what the tracker is tracking! Each little numbered square is a tracking_spot that the tracker is interested in. The dark blue ones are spots that have moved enough to contribute to the motion of the cursor, the paler ones are being ignored until they move more. It's useful to find out what it is looking for.
- Show/Hide names: This controls whether labels show up in sound_boxes.
- Zoom: If the camera is too far from the user, Zoom can help make limited motion more effective. It also speeds up the interactivity a lot, because there is a smaller bitmap to track!
- Multiplier (Mult): When the Multiplier is at its lowest setting, all "meaningful" motion is directly added to the cursor's position. "Mult" takes that motion offset and multiplies it up to 4 times. This increases the effect of motion, so that a very slight motion can have a bigger effect at moving the cursor.
- Volume (Vol): The volume for playing internal instruments.
- Cutoff Time: Some sounds play for a long time, and overlap each other. This can keep a user from hearing that their motion has started a new sound. By setting the cutoff time, the sound will stop playing after that time in seconds has elapsed. Cutoff time is roughly set in tenths of a second, and runs from 0.1 to 3 seconds. Setting it to 0 will let the sound play its natural length.
- Use/Ignore Velocity: Normally, the cursor's motion doesn't control the loudness of a sound. If this switch is set on, melodic instruments and sampled instruments will play more loudly if the cursor moves into it faster!
Normally, MIDI instruments will use the direction perpendicular to the layout direction the sound areas to control MIDI velocity. If the switch is set on, the speed of the cursor will be reflected in the MIDI Velocity of the played notes.
You can set the low and high range of the loudness that is controlled by velocity in the SETTINGS section. The setting for "volume" is also figured into the "velocity"
- Show or Hide Video: This switch lets you control whether you see the live video on the screen or not. Normally, AUMI shows what the camera sees on the screen. Some players may not want to look at themselves while playing. You can still see the cursor, sound_boxes, and tracking_spots.
- See the video.
- Don't show the video. Instead it shows a neutral background.
These parameters are saved in the user's _setup file (see Setups
This control lets you change the sizes, layouts, and colors of some interface elements.
Pick which element (Cursor or Guide) you want to change from the selector button.
By touching in the array of colors, you change the color of the selected element.
The box on the bottom left of the color array will show you the current color setting.
There is an "Opacity" slider which lets you change the color's opacity (transparency).
For Guides, this also help choose the color that the sound_box is colored.
The "Size" "slider is used to change these sizes:
- For Cursor, you can change the size of the cursor with the slider. It can be set from 70 to 300 pixels.
- For Guide, you can change the size of the sound_box borders with the slider, from 1 to 20 pixels.
The Lock switch locks the shape of the sound_box. This disables the ability to rescale the sound_box with a two-fingered touch.
The layout buttons select whether the sound_boxes are laid out horizontally, vertically, in a grid, or in a circle.
When you are using the MIDI instrument, and you are not using the actual cursor velocity, and if your layout is vertical, the position between top and bottom of the layout grid is used as the note's "velocity," so that near the bottom of the grid is "low velocity" and near the top of the grid is "high velocity". For a horizontal layout, left is "low velocity" and right is "high velocity"."
To dismiss this control on an iPad, tap anywhere outside the popup. To dismiss this control on an iPhone, tap "Done".
This screen is accessed by touching the "Sounds" button on the "Play AUMI" screen.
The Sounds dialog allows assign specific sounds to sound boxes. It's only enabled for Percussion, Noises, Loops, and Relative Motion instruments, because the sounds of these instruments do not have to be in a particular order
To quickly remove a number of sounds from active use, select their names and touch the "Down Arrow". Similarly, if you want to use instruments from the middle of the list, select them and touch the "Up Arrow". Now you can easily re-order them!
Pressing the Sounds button shows you an interactive list of all the sounds available for that instrument. The ones currently playing are in black, and the ones that are currently not playing are shown grayed out.
- The slider at the top sets the number of sounds that this instrument will use, which will be from 1 to 25. On the iPad, you can see the layout of the sound_boxes change as you move it.
- The Reset button restores the order of the sounds to what it was when the control was first displayed.
- You can change the order of the sounds by dragging the sound using the 'dragging' button on the right of each line.
- You can select any number of sound names and send them either to the top of the list or the end of the list using the appropriate buttons. This makes it easier to move sounds you want to hear into place.
- You can add silence to this list by touching the "Silence" button.
- You can duplicate a sound (and later move it) by touching the "Copy" button.
instrument, for example, can play from 4 to 12 sounds, but more sounds are to be found in the list which can be moved into place so that they can be heard.
Similarly, if you set up a Percussion
instrument with just two sounds to play, you can choose those sounds by moving them into place using the sounds screen.
This alternative order of sounds is saved with the instrument if you use the "Save Setup" button on the Login screen.
This screen is included as part of the Instruments screen on iPads.
If the current instrument is a MIDI instrument, you can use these controls to select a MIDI channel and MIDI program settings.
Because most MIDI instruments are not percussive, and need to be told to stop playing a note, there's a slider to set the delay time until a "note off" event is emitted. But if AUMI is set to be in monophonic mode, MIDI notes will stay on until the cursor is in a new sound box.
When a MIDI instrument is selected, a big red CLEAR MIDI button appears, in case there are stuck notes. You can also enable and disable MIDI transmissions with a switch.
You can select a program with the picker on the right. The names of the programs are the standard General MIDI names, and may not correspond to sounds that are actually played by the MIDI synthesizer you are connecting to.
Different synths try to discover MIDI sources in different ways, and sometimes, it's trickier than it looks. AUMI will try to connect to every MIDI receiver that it sees, even if they are added while AUMI is running. Apple's "Camera Connection Kit", when connected to a hardware USB-to-MIDI adapter, will let you attach AUMI to an external hardware MIDI synthesizer. There are other MIDI interface available as well.
Here's one way I've gotten another synthesizer running on the same device to work:
- Start AUMI.
- In Instruments, choose a MIDI instrument. Pick a scale and note range too, if you like.
- Make sure the MIDI switch is turned on.
- Click the home button to put AUMI in the background (it's still running)
- Start an iOS MIDI synthesizer like Bismark, DXi, Animoog, or another MIDI compatible iOS app.
- Make sure the synthesizer can play using background audio. (For example, in DXi, it's a switch in the MIDI section of its settings and information popup)
- Click the home button to put the synth away, and click on AUMI's icon again.
- It should be able to play the MIDI synth now.
You can also use a local network to play instruments on a nearby laptop or desktop. That's a little more involved; I'd suggest you look at tutorials about the Apple Audio MIDI Setup utility, like this one
by Joe Stallings.
AUMI generates two kinds of reports in the form of CSV files. Each time AUMI is started, a new file is created in the format AUMI-YYMMDD-HHMMSS.csv, so that the date and time are part of the file name, e.g.: AUMI---121206-092201.csv .
Every 15 seconds, a record is cut with some information that may be used to make reports in an external spreadsheet program like Numbers or Excel. This information may be useful for tracking the progress of individual AUMI users in different programs at various institutions.
Also, when the program transitions from foreground to background, it will also log this event. In this record, the timestamp and identifying information are retained, and the position related fields will be blank, but the Scale field will say FOREGROUND or BACKGROUND.
There's a second kind of report that is enabled with the optional switch "Log Sound Box Events".
It tracks the entry and exit from sound boxes, and record the time spent in each one, and
its file has a slightly different name from the usual report file:
AUMI-EL-[institution]-[program]-[date and time].csv
The report button will present you with a list of all the report files AUMI has made. They are ordered by date, and the two types of files are in different sections of the list.
You can select a file for export by tapping on it. You can unselect it by tapping it again. You can select all the files by tapping all. Whenever all the files are selected, the "All" button turns into a "None" button, so all the selections can be cleared. The header for each section keeps track of the number of items and how many are selected.
Each line has an "eye" icon on it. By tapping that, a kind of precis of the data shows up on a new screen, which you can examine to make sure it's the one you want. Tap the title bar on this screen to return to the list.
You can delete the all the selected files by touching the delete button. There will be a confirmation dialog just in case you hit it accidentally!
- By dragging the name of a file to the left, a delete button will show up. You can tap this to delete a file individually.
When you are ready to send out the reports, tap Send. Any program, like Apple's Numbers
, that can read a text file will be listed in the dialog that pops up. You can also use Air Drop to send the files directly to any other Apple device that supports Air Drop, for instance, a desktop machine.
You can also mail or post the files to various services, and you can also cancel the dialog.
All these files, are also available in the iTunes Document interface for AUMI. The main log file is also written whenever you leave the program.
Each record has the following information:
- "Date Time Stamp",
- "Notes Played",
- "min X",
- "max X",
- "min Y",
- "max Y",
- "avg Vel X",
- "avg Vel Y",
and the event report fields are these:
- "Date Time Stamp",
- "Name" (available in version elv_1.1)
Thus, you can tell users apart, and send files to different institutions.
Note that the Instrument name has the "Emoji" stripped off of it.
Using the iTunes Document Browser
If your device doesn't have mail set up, then of course it can't mail anything out! You can get your reports by:
- plugging in the iPad to a computer that knows about it. Usually, that's the one that was used to set it up or back it up.
- open iTunes
- find the device in iTunes's top bar:
- click on that and click on 'Apps' in this list of buttons:
- scroll down to where it says "File Sharing"
- scroll down that list and click on AUMI:
- The box on the side marked "Documents" will show all the sessions it has recorded:
All the personal set-ups are also in the Documents Directory.
You can select them all and click "Save to..." to save them in a directory that you can remember.
They don't take up a lot of space, and you can also just delete them after you've saved them.
In the actual AUMI app, you can only configure some aspects of the trackes and video from this section of the instructions.
AUMI uses software called a "tracker" to analyze the video images in order to know how to move the cursor.
These tracker settings may help you optimize AUMI in when you have a faster iOS device, or want to experiment with the responsiveness of AUMI.
- The Feature Tracker looks for high contrast parts of the image (features) and tries to correlate their movement from frame to frame. The average of this movement is used to move the cursor.
- More trackers will be in future versions of AUMI
Feature Tracker parameters
Each kind of device and working environment has its own best parameters for working with a Feature Tracker.
- Tracking controls the number of tracking spots that AUMI uses to detect motion. The more there are, the better the chance that the motion will be accurate. However, lower powered devices may need to keep this number low. A good range is from 50-1000. The default is 50, to accomodate slow devices.
- Tracking Minimum Distance: This controls how close together the tracking spots will be located in pixels. It's good to keep them about 6 pixels apart.
- Motion Sensitivity: This number sets a limit on how much motion a tracking_spot must have for it to be considered "meaningful motion". A setting of 150 is good, but in visually noisier environments, a higher setting may be needed. For more sensitivity, make it low, like 10.
Velocity to loudness controls
The range of loudness controlled by velocity when the "Use Velocity" switch is on can be set here. Velocity loudness can be thought of as ranging from 0.0 to 1.0. The high level is probably more important; if you set it to a fraction like 0.1, it will take more velocity to make it play loudly (and it will be more sensitive!), whereas setting it higher, like to 8.0, means nearly any movement will be loud.
AUMI's camera resolution is set low to accomodate slower devices, but on new fast devices, you can use a higher resolution if you like. The "Zoom" feature will still let you reduce the resolution and lower the latency of the tracker.
- Lowest resolution (192x144)
Front or Back Camera
iOs devices have a front facing and a back facing camera. In some situations, the rear camera might be more useful.
Somewhat adventurous AUMI users or administrators can add their own sounds into AUMI in a limited fashion. By using the iTunes Document interface for AUMI, you can drop in MP3 sound files and have them automatically turn into instruments. Here's how it works:
- Plug in your device and open iTunes.
- When the device becomes available in iTunes, click on it.
- Pick "apps" in the button menu.
- Scroll down below the listing of Apps to the "File Sharing" section
- Find AUMI in that section and click on it. It should look like the image shown below.
- You can now drag files in (or out) from this window. Or you can use the "Add ..." button.
- Drag in any mp3 files you want to use. They must have a suffix of ".mp3", ".aif", ".aiff", ".m4a", or ".wav" (lower case).
They should also follow this naming convention:
- You can also remove files from AUMI using this interface. You can also transfer out the .csv files for report purposes.
- Once the files have been transferred, you can close iTunes.
- When you now restart AUMI, you can see the new instruments appear at the bottom of the list named My Melodic, My Instrument, and My Loops.
If there are no samples in that particular category, the instrument won't be created.
The "User Sounds" Instruments can be saved in a Setup (using "Save Setup"), but because of the possibility of sound files being added or removed using the iTunes Documents interface, if it is loaded on a device that does not have one of these files, it will put a tilde (~) in front of the name and play Silence instead.
You can register for news about AUMI by filling in your email address here:
AUMI for iOS was developed for the Deep Listening Institute, now called "The Center for Deep Listening at Rensselaer" by Henry Lowengard, and adapted from the desktop version of AUMI, which is also available from the Rensselaer Polytechnical Institute (RPI).
Thanks to Pauline Oliveros, Leaf Miller, Jaclyn Heyen, Emily Halstein, Lisa Barnard Kelley, and Al Margolis of DLI, and the desktop developers:
Zane Van Duzen,
Doug Van Nort,
Ivan de Almeida Soares Franco,
For the AUMI Consortium: Prof. Sherrie Tucker (University of Kansas), Prof. Eric Lewis (McGill University, Montreal), Thomas Cuifo (Mt. Holyoke)
For RPI, Thanks to Prof. Jonas Braasch, Prof. Tomie Hahn, and Dean Mary Simoni.
Some audio was generously provided by these musicians:
- Nick Didkovsky and Dr. Nerve for the Nerve Event noises from "Beta 14: OK"
- Dave Soldier and the Soldier String Quartet for the Marsh Loops, Pin Pia, and Soldier String Quartet Excerpts from "Sequence Girls", "Paris 1919" (featuring Shelley Hirsch on vocals), and "The Impossible String Quartet".
- Skip La Plante for Glyptodont .
- Deep Listening Band for Troglodyte's Delight.
- Kala Pierson for Tran Phu.
- The Institute of Contemporary Music for Kitchen Percussion. (via ccMixter.com)
- Free SFX for the Dogs
- Funny Cats and Kittens Meowing Compilation 2013 [HD](You Tube) for the Cats
- Jesse Stewart's Xenophora: Music for Solo Waterphone
- and Henry Lowengard of jhhl.net supplied the Music Box, Synthy Loops, LP Lock Grooves, Tenor Recorder, Widow Jane Mine, Fifths, and prepared or synthesized the other samples and loops.
iOS, iTunes, iPad, iPhone, iPod Touch, GarageBand are registered trademarks of Apple, Inc.
AUMI was developed with the support of the The Gould Family Foundation
The feature tracking software is derived from KLT (Kanade-Lucas-Tomasi) from Stanford University.
The tutorial on Apple's Audio MIDI Setup Utility is "Using Your iPad as a Wireless MIDI Controller" by Joe Stallings.
AUMI (c) 2012-2014 Deep Listening Institute, Ltd., and 2015-17, Center For Deep Listening at RPI
Developed by Henry Lowengard
build version: 1.1.6