Archive for the ‘intro to computational media’ Category

Cats vs. Bat Game – My Final ICM Final

Tuesday, December 22nd, 2009

Now that finals are finished I finally have some time to devote to this journal. Needless to say, a lot has happened during the last month that I have not yet shared here. Let’s begin with an update regarding the ICM final.

Over the weekend I finished my final project for ICM, which is an updated version of the fangs invaders game that is titled “Cats vs Bats”. This updated version of the game is almost unrecognizable from the previous one. Here is a link to an online version of the game (please note that you need to have a fast internet connection to play it because it is not web optimized), followed by a brief description of the game coupled with a list of the latest updates.

You can also download the game here: mac version | windows version.

Project overview: cats vs bats is simple space invader-like game. In this game, a small oriental shorthair cat named Sasha is our only hope to save the world against a swarm of diseased bats.

Though currently this project is screen-based only, I am working on developing a cat toy that will be used in conjunction with this application. A single joystick controller has been created that can be used to control either or both of these projects (so that you can play the game and with your cat at the same time). Also, the proximity of the cat to the toy will enable a player to earn bonus points in the Cats vs. Bats game.

Main updates: Here is an overview of the main updates that I have been working on over the past several weeks. Over the winter break I plan on delving deeper into a few of these areas (such as use of vectors to drive bat motion).
• Improve game scoring logic and add score to game play environment.
• Create an online high-score database.
• Improve animation by making flight patter more random and natural.
• Update look and feel of the game by adding back images and improving design.
• Include bonus points opportunities.
• Add stages with increasing levels of difficulty.

High Score List – Finally Solved

Monday, November 23rd, 2009

Over the past week I have been grappling with a simple problem: determining how to create a high score list for the Fangs Invader game. This seemingly simple problem gave me a complete mental block that took a week to thaw. I am happy to report that on Friday night I was finally able to solve this problem. Here is a simple sketch that displays the functionality that I created followed by an overview of my solution.

The Architecture
From an application architecture perspective, I decided to do most of the heavy lifting in Processing because I am know it better than PHP. Hence the PHP script only has two functions: (1) saving the high scores to a text file; (2) displaying the high scores from the text file. Processing takes care of the following functions: (1) Determining if current player’s score qualifies him to be added to the high score list; (2) capturing the player’s name, if he has a high score; (3) re-sorting the high score list to include the new score and name.

PHP Script
Two weeks ago I struggled with PHP to create a script that was able to read data from a query string and save that data to a text file, and read data from the same text file and display it appropriately. After much trial and error I successfully put together a short script that is able to accomplish this feat.

I found PHP to be a very temperamental language. I am not used to working with un-typed languages – I am not skilled enough to take advantage of the additional flexibility they provide while I seem to get caught up in many unknown and unexpected quirks and behaviors. Nonetheless, I have not been dissuaded from learning this language and hope to build my skills over the next three semesters.

Processing Sketch
Shortly after I finished the PHP script I started working on the Processing sketch. In no time I had the Processing application reading from and writing to the PHP script. I quickly lost steam when I tackled the sorting of the highscore list. This is not to say that this problem is particularly hard, however, in the road to solving it I encountered a major a blank.

At first the problem seemed simple, I created a short algorithm that seemed to work fine. I was actually very satisfied with myself for having solved this problem so quickly and simply. I had even started to integrate the solution into my fangs game.

So my solution did work fine unless there was more than one entry on the list with the same score. When this occurred the algorithm would select the name of one entry and apply that name to all others with the same score. Another issue was that the logic for the Fangs high score list is reversed – the better scores are the lower scores. Therefore, I needed to create an algorithm that is able to locate the lowest score that is above zero.

This is when I hit into a hard brick wall, head-on. The problem was back and I did not know where to even start. My attempts at writing pseudocode were totally unhelpful at first. My mind was so focused on integrating the functionality into the game and doing other things to the game that it just did not want to deal with this problem anymore.

In retrospect, I see how my initial approaches to this problem were overly complex (I won’t even try to explain it here). At first I had a hard refocusing and “seeing the forest from the trees” so that I could tackle the problem from a different perspective. It took me 6 days to figure out a different approach to solve this problem. Part of this process required that I stop working on this problem for a few days so that my mind could disassociate from it.

Here is a pseudocode-like overview of the solution that I developed for sorting the highs core list. All of the functionality outlined below is encapsulated in the changeList() function that is part of a class called PHPconnect. To support my algorithm I declared three array variables, each one holds eleven variables: a position for each from the current high score list and one position for the current player’s score.

The newLocation array is used to store the new location of each element on the top score list with references to their current location on the scoreList array, which holds the pre-sorted rank of each element on the old top ten list along with the new player’s score, saved in the eleventh position in the array. All elements in the newLocation array are initialized to -1 when the changeList() function is called. This is relevant because the newLocation array plays an important role in processing lists that feature multiple players with the same score.

The tempScore and tempName variables are used to temporarily hold the re-sorted top score and top name lists. At the end of the chageList() function the scoreList and nameList arrays updated by being assigned the values from these temporary arrays.

There are two other variables that play a crucial role in this function: locCounter and locCounter reverse. These two variables are used to determine the new location of each element on the top score list. These variables are incremented each time that a score is found which is lower than the current one being processed. The locCounter variable is used to hold the location of valid scores (any number higher than zero). This variable holds a score’s position from the first location in an array. The locCounterReverse variable is used to hold the location of invalid scores (0’s and -1’s). This variable holds a score’s position from the last location in the array.

Now let’s take a look at the algorithm. An outer loop cycles through 11 times to go through each score in the highscore array. A secondary embedded in the outer loop is used to compare each score with the others on the highscore list. The sketch determines the pre-liminary new location of each highscore list item by counting the number of scores that are greater than the current score.

Once the preliminary location is determined, the next step is to check if the current player’s score is repeated, and if so to adjust its position accordingly. To determine if a player has a repeated score we check the position “locCounter” in the newLocation array. If the position is not available (it is not equal to “-1”) then we loop through subsequent positions, using a while loop, until we find one that is available. This available position is then set as the new position for this list element in the newLocation array.

Once each position in the newLocation array has been assigned, a loop is used to input data into the tempName and tempScore arrays. Then these arrays are used to reset the scoreList and nameList arrays with the updated top score list.

Here is the code for the PHPconnect class that I created:

Fangs Resurrected – ICM Final Project Idea

Thursday, November 19th, 2009

After much brainstorming and indecision I have finally settled on my final project for the Intro to Computational Media final. My project will be to create a tricked out (or pimped out) version of the fangs invaders game. Below you will find the fangs game in its current state.

Since I last shared it in class, and on this blog, I have made a few updates to this sketch. Namely, I have added functionality to calculate and display a score, which is shown on a new page at the end of the game. This is a very small improvement in comparison to the changes I will integrate over the next two weeks.

For this project there are a bunch of updates that I plan to make. Since I may not be able to implement all of the changes I have created a prioritized list to guide my work over the past several weeks:
  • Improve game scoring logic and add score to game play environment.
  • Create an online high-score database (or file) using PHP scripting.
  • Improve animation by adding wing movement, and making it more natural.
  • Develop group behavior for the bats so that they interact and flock occasionally.
  • Update look and feel of the game by adding back images and improving design.
  • Add multiple stages with increasing levels of difficulty.
[picture taken from flickr user furryscalyman - Creative Commons License ]

ICM Class Notes – Using PHP – November 5, 2009

Friday, November 6th, 2009
In today’s ICM class we reviewed basic concepts associated to PHP, and discussed when to use PHP (as a stand alone solution, or in conjunction with Processing).
PHP – Hypertext Pre-Processor Language
PHP was designed for server-side scripting – for development of applications that run on servers. In contrast, Processing (and Java on which it is built) was developed for client side applications – for development of programs that run on client-side computers. Originally designed to serve dynamic web content. In short, PHP enables us to store and capture information from a server more effectively and efficiently than Processing.
Here is an example of an application where PHP is used to store data for a Processing sketch. This code was written by Dan Shiffman, my professor at ITP, and it features a shared white board, where users can add new coordinates that can be viewed by all other users (anyone can also clear the board). PHP can also be used to develop a multi-user high score feature for a game developed in Processing.
How to Run PHP?
In order to run PHP you need a server or computer that is properly set-up. Most web servers support PHP, though the ones that are based on Microsoft solutions tend not to. That said, even servers that support PHP need to be properly set-up. Luckily for us, the ITP server is already set up for PHP. Many desktop and laptop computers can also be set-up to run PHP scripts.
PHP code is usually embedded in HTML documents. To embed PHP code into an HTML document the following identifiers are used: “” ends a code block. It is important to note that PHP code will not be present on the HTML source available via web browers. The reason being, the PHP code on the server is used to dynamically generate the HTML code that is sent to your computer, and viewable as source.
The Basic Elements of Coding – PHP Style
The basic concepts associated to PHP coding are similar to those used in Processing. PHP is an object-oriented language that features all of the same attributes: variables, conditionals, loops, functions, classes, etc. Here is a quick overview of some basic similarities and differences between these two tools.
Primitive Variables
All variables in PHP need to be defined/initiated by being assigned a value. To identify a variable the variable name needs to be preceded by the “$” character. However, unlike Processing, PHP does not require or support the typing of variables. This is a more flexible approach than Processing but it also opens the doors to more mistakes not being caught by the compiler.
Example variable definition code: “$ x = 5;”
To make an array in PHP you just assign an “array(arrayElements)” to any variable name. As with primitive variables, there is no need to define the array type. To add elements to an array is simple simpler than in Processing, as shown below. It is important to note a PHP array can also hold different types of data.
  • Creating an array: “$arraySample = array(0,1,2,3);”
  • Adding an element to an array: “$arraySample[] = 5;”
PHP also supports associative arrays as well. These arrays indentify each data element by a name as opposed to an index number. These arrays can be used in interesting ways. They are available in Processing through Java.
  • Using an associative array: “$arraySample[“fred”] = 50;”
Conditionals and Loops
The syntax for loops and conditional statements is identical to that found in Processing.
Functions and OOP in PHP
In PHP to define a function it name needs to be preceded by the identifier “function”. Similar to variables definitions, functions are not typed. Class definitions feature the same overall structure, though the constructor name is identified as “function _constructor()”. When working with instances of objects the “->” in PHP replaces the “.” in Processing.
Query Strings
Query string refers to HTML urls that reference PHP scripts and contain information that can be processed by the PHP script to dynamically generate content (HTML or other text-based documents). Since PHP primarily runs on web servers, this is the main way in which information is passed to PHP scripts. Here is an example where the identifier “name” contains data element “Julio”, “nationality” contains “brazil”, and “residence” contains “nyc”.
Online, these query strings are most often used to transmit data captured from users via a web forms. Processing applications can generate query scripts that both request data and initiate other activities on the server side.
Reading Query Strings
To read values from query strings an associative array is used. PHP creates an associative array that pairs each identifier from the query string (which is always consistent) with a data element (which is variable). This array is called $_GET. Here is a sample line of code to read my name from the query string above into the variable $name:
“$name =$_GET[“name”];”
PHP Resources
  • Dan shared with us Coda, a development environment that enables programmers to access and edit PHP and HTML files on the server. Coda is a text editor and ftp application combined in one.
  • In today’s class we got to see Josh K’s project, a sketch that generates a line that evolves in a 3D space using the principles of Perlin noise. It is a great looking visual, unfortunately, he has not yet posted the sketch online.

ICM Class Notes – Using Images and Video – October 22, 2009

Friday, October 30th, 2009

Here are my notes from last week’s ICM class, where we learned how to use pictures and video in Processing. It’s taken me some time for me to get around to posting these notes because I have been focused on developing the media controller for physical computing, and working on my mid-term project. Without further ado here are my notes.

Using Images in Processing
There are two main types of activities related to using and manipulating pictures in Processing:
1. Loading and displaying images
2. Reading and manipulating the pixels

Loading and Displaying Images
In processing there is a class that handles images that is called PImage – in many ways this class is similar to the PFont object. For example, to create a new instance of an image object the loadImage(URL) function is used rather than a “new” object declaration. Here is sample code:

PFont cat;
cat = loadImage(“cat.jpg”);

The image() function is used to draw onto the screen images that are loaded. This method requires from 3 to 5 arguments, here is the complete set: image(PImageVar, xpos, ypos, width [optional], height [optional]). We can assign an image to the background() function as well, in which case the image would be displayed as the background. Here are some useful methods associated to displaying images:

  • The imageMode() function allows setting of image alignment. Setting options include CENTER, CORNER (standard setting) and CORNERS.
  • The tint() function enables changing the color of the image. It does not color the picture but rather removes the other colors. So if you change the tint to red, processing will set all G and B value to “0”. Tint also supports setting the transparency of an image.
  • The PImage.get(xpos, ypos) function returns the color from the pixel at the coordinate xpos ypos. Get() can be used without an image to get the color of the pixel at coordinates xpos and ypos of the processing screen.
  • The PImage.set(xpos, ypos, color) function sets the pixel at location xpos ypos to the color that is passed as the third argument.
  • The red(color) (green() and blue()) functions determines how of that color (red, green or blue) is included in that pixels.

Loading and Displaying Videos
Processing handles videos in the same way that it handles images (it actually views videos as a bunch of standalone images. Using live video in processing is done through the Capture class. To use this class you need to add the video library to your sketch. Note that you need to use a different class to include movies, which are pre-recorded videos. Unfortunately, Processing does a bad job at handling movies though it works well with live video feeds.

To declare a Capture object you need to use the standard “new” object declaration: “myVideo = new Capture(this, xpos, ypos, refresh rate)”. When you initiate a video object processing will default to using the default system camera. The name of an alternate video source can be added between the ypos and refresh rate variables.

Before you display an image you need to read the image using the function. The same image() that is used for PImage objects will display images from the camera onto the screen. All of the image processing functions discussed above can also be used to analyze video input.

Media Controller Project (and ICM Mid Term) – Phase 3

Friday, October 30th, 2009

During that last several days I have been working on setting up a Processing sketch that can work with my Physical Computing media controller and serve as my mid-term project for the Introduction to Computational Media course. Long before arriving at ITP I have been interested in the design and development of media controllers. This project provided the opportunity for me to start some hands-on explorations.

In my previous post I already discussed the process for choosing the solution for playing and controlling our audio – we have decided to use Processing (and the Arduino) to control Ableton Live. Today I will provide an overview of how I developed the code for this application and some of the interface considerations associated to designing a software that could work across physical and screen-based interfaces.

My longer term objective is to create MIDI controllers using for audio and video applications using touchscreen and gestural interfaces. The interfaces that I am designing would ideally be evolved to work on multi-touch surfaces. In regards to my interest in gestural interaction, this I hope to explore through my current physical computing project and future projects.

Developing the Sketches
Since the physical computing project requires three basic types of controls that are the foundation of the media interface for my computational media mid-term, I decide to start with a focus on writing the code for these three basic elements. I set out to create code that could be easily re-used so that I could add additional elements with little effort. Here is a link to the sketch on, where you can also view the full code for the controller pictured below (v1.0).

The process I used to create these sketches included the following steps: (1) creating the functionality associated to each element, separately; (2) creating a class for each element; (3) integrating objects of each class in Processing; (4) testing Processing with OSCulator and Ableton; (5) creating the Serial protocol to communicate the Arduino; (6) testing the sensors; (7) writing the final code for the Arduino; (8) testing Serial connection to Arduino; (9) calibration of the physical computing interface (whenever and wherever we set it up).

I have already made two posts on this subject (go to phase 1 post, go to phase 2 post), however, today I can attest that I have completed the vast majority of the work. The last processing sketch that I shared featured a mostly completed Matrix object that included functions for OSC communication. The serial communication protocol had also been defined.

The many additions to the sketch include creation of button and slider elements (each in its own class), a control panel (that holds the buttons and sliders), and a version of the application that features multiple button and sliders. The main updates to existing features include changes to Serial communication protocol to support additional sliders and matrices), and OSC communication code updates to ensure that messages are only sent when values change rather than continuously.

For the slider object I used the mouseDrag() function for the very first time. I had to debug my code for a while to get the visual slider to work properly. The button was easy to code from a visual perspective. The challenge I faced was in structuring the OSC messages so that I was able to send two separate and opposing messages for each click. The reason why this is important is that Ableton Live uses a separate buttons for starting and stopping clips. So I had to find a way to enable a single button to perform both functions.

The serial communication protocol update was easy to implement, so I will not delve into it here. To change the OSC communication protocol required a bit more work. I created a previous state variable in each object class to be enable verification of whether a change had occurred. The logic was implemented an “if” statement in the OSC message function.

Evolving the Controller
Here is an overview of my plans associated to this project: I plan to expand the current media controller with a few effect grids and the ability to select individual channels to apply effects. In order to do this I have to create new functions for the matrix class that enables me to set the X and Y matrix map values. I also want to work on improving the overall esthetics of the interface (while keeping its minimal feel).

From a sketch-architecture perspective I am considering creating a parent class for all buttons, grids and sliders. It would feature attributes and functionality that is common amongst all elements. Common attributes include location, size and color; common functionality requirements include detection of mouse location relative to object, OSC communication.

Questions for Class
Here is a question that came up during my development of this sketch (Dan, I need your help here). Can I use the translate, pop and pushMatrix commands to just to capture the current mouse location? This would be an easier solution to checking whether the mouse was hovering over an object.

Media Controller Project – Phase 2

Wednesday, October 28th, 2009

This weekend I spent a lot of time working on solving the issue of how to control and manage the music clips that we want to use in our project. Our requirements are pretty straight forward, which is not to say easy to address.

Requirements for Audio Controls
We need a solution that can handle playback of looped samples and dynamic control of at least two effects to be applied on the sample (such as tempo and pitch). Ideally we would like the solution to be scalable (so we can add multiple sounds) and be able to support quantization and other techniques to ensure that the resulting sound is of good quality.

Since we are a creating a prototype to run off of a single computer we do not need this solution to be easily portable (e.g. it does not need to be easy to run on different computers).

Initial Assumptions
Due to the expertise of the team members we are using a combination of Arduino and Processing to do the heavy lifting in the areas of input sensing and data handling. After researching all of the options available in processing to manage sound, we have decided to use Ableton Live instead. Processing’s role will be relegated to interpreting the data from the Arduino to control Ableton Live via OSC.

Below I provide a more in-depth overview of my research and the solution that I have chosen. I have also posted an updated version of my sketch along with a link to the file I have created in Ableton Live for this application. Please note that you will need to set-up OSCulator in order for the sketch to work properly.

Latest Version of the Sketch
Note that I am only sharing the code for the sketch because no updates were made to the look, feel and interaction of the applet. All updates are related to enabling the sketch to communicate with Ableton via OSC.

/* IPC Media Controller Project, October, 2009
* This sketch is the first draft of the processing portion of our media controller project
* in its current state, this sketch only focuses on reading input from serial ports,
* processing this input to determine location on the virtual matrix, then provide these
* coordinates to other objects (such as the music generation object that we will create in the future)

import processing.serial.*;
Serial arduino;

import oscP5.*;
import netP5.*;
OscP5 oscComm;
NetAddress myRemoteLoc;

boolean isStarted = false;
// Matrix-Related Variables
Matrix matrix;
final int x = 0; final int y = 1; // variables to use with matrix size array
int [] cellSize = {50, 50};
int [] screenPad = {25,25}; // define padding between grid and screen border
int [] screenSize = new int [2]; // define screen size, note that we only add volSize to width, since volume knob will be place to right of screen

// Volume-Control Related Variables
int [] volSize = {0,0};

void setup() {
// initialize the matrix object
matrix = new Matrix(screenPad[x], screenPad[y], cellSize[x], cellSize[y]);

// instantiate the serial variable
arduino = new Serial(this, Serial.list()[0], 9600);

// set frame rate
// start osc communication, listening for incoming messages at port 12000
oscComm = new OscP5(this,12000);
// set destination of our OSC messages (set to port 8000, which is the OSCulator port)
myRemoteLoc = new NetAddress("",8000);

// set screen size related variables
screenSize[y] = int(matrix.getMatrixWidth() + (screenPad[y] * 2));
screenSize[x] = int(matrix.getMatrixHeight() + volSize[x] + (screenPad[Y]*2));
size(screenSize[x], screenSize[y]); // draw the window 100 pixels wider and 50 pixels taller

void draw() {
matrix.sendOscMessage(oscComm, myRemoteLoc);

void serialEvent(Serial arduino) {

void oscEvent(OscMessage theOscMessage) {
/* print the address pattern and the typetag of the received OscMessage */
print("### received an osc message.");
print(" addrpattern: "+theOscMessage.addrPattern());
println(" typetag: "+theOscMessage.typetag());

* this class holds a virtual matrix that will mimic the real world matrix.
* It contains functions that read input from a serial port or mouse, then use that
* input to determine the location of the object or mouse on the grid

class Matrix {

// general variables used accross class
final int x = 0; final int y = 1; // variables to use with matrix size array
final int mouseControl = 0; final int serialControl = 1;

// matrix and cell related variables
final int [] cellNumber = {5, 5}; // number of cells on the horizontal axis of the matrix
final float [] cellSize = new float [2]; // width and height of each cell of the matrix
int [] matrixLoc = new int [2]; // location of the overall matrix
final float [] matrixSize = new float [2]; // the total height and width of the matrix
float [] xCellLoc = new float [cellNumber[x]]; // location of each cell on the grid
float [] yCellLoc = new float [cellNumber[y]]; // location of each cell on the grid
Boolean [][] cellState = new Boolean [cellNumber[x]][cellNumber[y]]; // holds whether the mouse or serial object is hovering over a cell
color activeColor = color (255,0,0); // holds color of active cells
color inactiveColor = color (255); // holds color of inactive cells
int [] previousState = {0,0}; // holds prevous state of the cell

// variables for reading serial input
int mainControl = mouseControl;
float [] serialLoc = {0,0}; // holds Y reading from the serial port

// Matrix Object Constructor
Matrix (int XLoc, int YLoc, int cellWidth, int cellHeight) {
matrixLoc[x] = XLoc; // set X and Y location of the virtual matrix
matrixLoc[y] = YLoc;
cellSize[x] = cellWidth; // set the size of each cell on the grid of the virtual matrix
cellSize[y] = cellHeight;
matrixSize[x] = cellNumber[x] * cellSize[x]; // calculate width of the matrix
matrixSize[y] = cellNumber[y] * cellSize[y]; // calculate height of the matrix

// sets the location of each cell on the grid
for (int xCounter = 0; xCounter < xCellLoc.length; xCounter++) {
xCellLoc[xCounter] = xCounter * cellSize[x]; }
for (int yCounter = 0; yCounter < yCellLoc.length; yCounter++){
yCellLoc[yCounter] = yCounter * cellSize[y]; }

// sets the status of each cell to false
for (int xCounter = 0; xCounter < cellState.length; xCounter++) {
for (int yCounter = 0; yCounter < cellState[xCounter].length; yCounter++) {
cellState[xCounter][yCounter] = false; }
} // close the constructor

// function that returns the height of the matrix
float getMatrixHeight(){
return matrixSize[x];

// function that returns the width of the matrix
float getMatrixWidth(){
return matrixSize[y];

// function that draws the matrix on the screen
void drawMatrix() {
for (int xCounter = 0; xCounter < xCellLoc.length; xCounter++){ // loop through each element in the xCellLoc array
for (int yCounter = 0; yCounter < yCellLoc.length; yCounter++){ // loop through each element in the yCellLoc array
if (cellState[xCounter][yCounter] == true) { fill(activeColor); } // if the cellState is true then change the color of the cell
else { fill(inactiveColor);} // if the cellState is false then don't change the color of the cell
rect(xCellLoc[xCounter]+matrixLoc[x], yCellLoc[yCounter]+matrixLoc[y], cellSize[x], cellSize[y]); // draw rectangle
} // close drawMatrix() function

// function that reads the input from the serial port
void readSerialInput (Serial Arduino) {
if (!isStarted) { // if this is the first time we are establishing a connection
isStarted = true; // set isStarted to true
arduino.write("n"); // respond to arduino to request more data
} else { // if this is NOT the first time we have received data from the arduino
String bufferString = arduino.readString(); // read the buffer into the bufferString variable
if (bufferString != null) { // if bufferString holds data then process the data
bufferString = bufferString.substring(0, bufferString.length() - 1); // trim the string
String[] serialValues = splitTokens(bufferString, " "); // separate the two values from the string and save them in the serialValues variable
serialLoc[x] = float(serialValues[x]); // assign value to serialLoc[x]
serialLoc[y] = float(serialValues[y]); // assign value to serialLoc[y]
arduino.write("n"); // respond to arduino to request more data
} // close readSerialInput() function

// returns an array with the unfiltered x and y locations from the serial monitor (may need to filter data based on range of serial input and requirements of music objects)
int[] getSerialXY() {
return int(serialLoc);

// function for user to set whether main input is serial or mouse based
void setMainControl(int tControlType) {
mainControl = tControlType;

// function that sends OSC messages with input values
void sendOscMessage(OscP5 tOscComm, NetAddress tMyRemoteLoc) {
float messageX = 0;
float messageY = 0;

// open new OSC messages of type x and type y
OscMessage myOscXMessage = new OscMessage("/controlGrid/x");
OscMessage myOscYMessage = new OscMessage("/controlGrid/y");

// determine whether readings that are sent to OSC will originate from serial device or mouse
if (mainControl == serialControl) {
messageX = map(serialLoc[x], 0, width, 0, 1);
messageY = map(serialLoc[y], 0, height, 0, 1);
} else if (mainControl == mouseControl) {
messageX = map(mouseX, 0, width, 0.075, 0.125);
messageY = map(mouseY, 0, height, 0.3, 0.7);

myOscXMessage.add("x "); /* add an int to the osc message */
myOscYMessage.add("y "); /* add an int to the osc message */
myOscXMessage.add(messageX); /* add an int to the osc message */
myOscYMessage.add(messageY); /* add a float to the osc message */
tOscComm.send(myOscXMessage, tMyRemoteLoc);
tOscComm.send(myOscYMessage, tMyRemoteLoc);

print("X: " + messageX + " ");
print("Y: " + messageY + " ");

// returns an array with the unfiltered x and y locations from the mouse-based interface (may need to filter data based on requirements of music object)
int [] getMmouseXY() {
int [] mouseXY = {mouseX, mouseY};
return mouseXY;

// check if a cell on virtual Matrix is active based on the mouse location
void isCellActiveMouse () {
int XLocMouse = mouseX - matrixLoc[x]; // adjust variable to account for location of Matrix within window
int YLocMouse = mouseY - matrixLoc[y]; // adjust variable to account for location of Matrix within window
isCellActive(XLocMouse, YLocMouse); // call the function to check if the cell is active based on current location of mouse

// check if a cell on virtual Matrix is active based on the current physical location/state of an external object
void isCellActiveSerial () {
int xLocSerial = int(map(serialLoc[x], 0, 1024, 0, matrixSize[x])); // adjust variable to account for location of Matrix within window
int yLocSerial = int(map(serialLoc[y], 0, 1024, 0, matrixSize[y])); // adjust variable to account for location of Matrix within window
isCellActive(xLocSerial, yLocSerial); // call the function to check if the cell is active based on current location of mouse


// function that checks whether a specific cell is Active
void isCellActive (int tXloc, int tYloc) {
int xLoc = tXloc; // set the location of the X coordinate where the mouse or serial object is located
int yLoc = tYloc; // set the location of the Y coordinate where the mouse or serial object is located

for (int xCounter = 0; xCounter < xCellLoc.length; xCounter++){ // loop through each element in the xCellLoc array
for (int yCounter = 0; yCounter < yCellLoc.length; yCounter++){ // loop through each element in the yCellLoc array

// check out what are the mouse or serial object is intersecting
if ( (xLoc > xCellLoc[xCounter] && xLoc < (xCellLoc[xCounter] + cellSize[x])) &&
(yLoc > xCellLoc[yCounter] && yLoc < (yCellLoc[yCounter] + cellSize[y])) ) {
cellState[previousState[x]][previousState[y]] = false; // set previous grid element to false
cellState[xCounter][yCounter] = true; // set current element to active status
previousState[x] = xCounter; // set x number of previous active cell
previousState[y] = yCounter; // set y number of previous active cell

Making Some Noise
When we started working on this project we assumed that we would be able to use one of Processing existing sound libraries to play and modulate an audio loop. However, after doing extensive research into Minim, ESS, and Sonia, I realized that none of these tools offered the feature set that we needed for this project.

The next solution that I investigated was Max/MSP. This programming language/environment is definitely capable of providing the functionality that we are looking for. However, no one on our team has the expertise to use it nor the time to learn it for this project.

using OSC to communicate with an external music application that can provide the features we are looking for. I was happy to find out that there is a simple library called oscP5 that makes it easy to communicate from a sketch using OSC. Equally important, I also found an application called OSCulator that routes and translates OSC and MIDI messages.

Having figured out how to get the sketch to communicate via OSC and MIDI we set out to find the right application. This was an easy task in large part because both Michael and I are familiar with Ableton.

I am happy to report that we already have Ableton up and running with the virtual matrix application developed in processing, though that is not to say the sketch is finished. We still need to add start and stop buttons to the interface, along with a volume control (not to mention other improvements and ideas that have not yet been considered).

In the next day or so I will share with you more updates, including details about how the physical elements of the interface are shaping up.

ICM Class Notes on Text Processing and Serial – Oct 14, 2009

Friday, October 23rd, 2009

Last week’s ICM class focused on parsing strings, picking up where we left off the week before, with a short overview of serial communications. Here a list of the topics that we covered:

  • Processing and XML
  • Additional functions for processing text
  • Serial communications

XML Libraries in Processing
XML has a hierarchical structure similar to a tree. This makes XML files easier to read than other types of content. There are many different ways that you can read XML documents in processing. Here is an overview of these options:

  1. Standard string parsing functions in processing, like the ones outlined below and in my post from last week.
  2. Existing processing XML libraries. Examples of these include simpleML, XMLElement, proXML. The libraries enable processing to navigate the structure of an XML file by finding a data elements by navigating through its children or parent structure.
  3. Application processing interfaces (APIs) from the data source. Examples of sites with APIs include Flickr, Google Maps, etc.

Using Tokens to Parse Text
This week we were introduced to the concept of tokens and splits. Tokens are small chunks of text (these are only one character long). Here is a list of functions that leverage tokens or splits.

  • split(String, SplitStringIdentifier); – This function returns multiple strings. The input copy “String” is split at the instance(s) of SplitStringidentifier. The SplitStringIdentifier is removed from the final strings.
  • splitTokens(s, multiple SplitStringIdentifier); – This function is the same of split, however, it can accept multiple SplitStringIdentifiers. These are input all together within quotation marks. For example using “_.” Would look for instances where either “_” or “.” appear in the string.

Comparing Text

  • equals(); – This function compares the copy contained in a string to a piece of text data. For example “string.equals(“stringData”);” is equivalent to the syntax “intVar == 3”. That said, we cannot use the Boolean operator “==” to match strings. That is why the equals() function is needed. In order to compare words regardless of whether they are lower or upper case we can use the string.toLowerCase() function.
  • Regular Expressions – Regular expressions are special text strings for describing search patterns. These capabilities enable more sophisticated parsing of content than is possible through the standard functionality in Processing. This is the ideal way to perform complex data parsing and cleaning. We did not cover this in class, so research will be needed on this front.

Serial Overview
There are two main approaches to creating protocols for serial communication. The first is called punctuation, and it entails adding tokens to the data being communicated to enable the receiving computer to parse the data once received. The second approach, called handshaking, entails having each computer wait to send a message until they have received data from the other connected device.

I will not review these approaches here in detail because I have covered them in my posts from the Intro to Physical Computing class. The punctuation method is described here. I will soon add a link to the post regarding the handshake method (which is currently a work in progress).

Related notes and concepts:

  • CallBack refer to methods that are called by other applications when a certain event takes place. MousePressed and SerialEvent are types of callback or event functions available in processing.
  • To create a new serial object it is important to always include the Serial library (as it is not a standard processing library). Then to instantiate the object, once you’ve declared it, you need to use the following syntax: “portName = new Serial(this, “serialPortNumber”, baudRate);” The “this” argumen tells the serial object that it is setting up a communication between the serial port and this specific sketch.

Converting Applications to OOP Paradigm

Saturday, October 17th, 2009

Since the etch-a-sketch was a simple program it was quite easy to convert it into an object-oriented structure. I decided to create one class that holds all functionality of the virtual etch-a-sketch. Though this sketch was originally created to controlled by the physical computing interface (consisting of two potentiometers) I have added the functionality to be able to use it with the arrow keys.
Here is a link to the Virtual Etch-a-Sketch app.
NYT Lists
I had issues trying to create a single NYT list object that could handle both types of list that I want to use in this sketch. Therefore, the solution that I decided to pursue was to create a parent class that includes some fully defined functions – the ones I was able to abstract to work with both lists – and two function shells – meant to be defined in the child classes. Here are a few lessons from my work on converting this sketch.
Notes on Using Inheritance
Learned that if you re-declare any variables then the functions that were declared in the parent class, which use the originally declared variable, won’t be able to update the newly declared version of that variable. The reason why I want to re-declare a variable is that it is an array, and each child classes features a different number of elements that will be stored in this array.
In the parent class I created a few functions that cycle through this array, and they seem to work. The problem is that when a function that is defined in the child class attempts to access this variable then the values, which had been updated by functions from the parent classes, are re-initialized (I got 0’s).
The solution that I used for this problem was to make the size of the array in the parent object large enough to accommodate the largest possible array size requirements from the child classes (for the case of this sketch that number was 50). To read the data appropriately, the loops in the parent functions have been set to cycle until they reach a “number of list items” variable that is defined by the user when they create and declare the object.
Using Parameters Within a Function
At first I created several functions that used the input parameters as variables within the function itself.  Though this worked in several instances, I realized that on other occasions the input parameter variables became un-useable. Therefore, now I understand that it is important to create new variables that can hold the values of input parameter for later use and processing.

Here is a link to the NYT List sketch.

IPC Class Notes, Serial Communications – week 6

Friday, October 16th, 2009

This week’s class covered basic principles associated to using serial communication to get the Arduino to communicate with applications on a multimedia computer. The focus of our discussion was on using the Arduino as in input device. I’ve already completed the lab for this week, here’s my post about how it went.

Though I have some experience in this area, having created a joystick for a game that I developed a few weeks ago, I did not understand how to create Arduino applications that leverage serial communications without using an existing library (such as firmata). This was very limiting because I was having issues with firmata.

Basic Concepts Related to Serial Communication with the Arduino
We began our exploration into serial communication by reviewing several key concepts and functions. First off, it is important to always keep in mind that only one application can use a serial port at a time (there will be no sharing of serial ports).

  • ASCII – ASCII is the protocol used to transmit standard display characters, such as letter, numbers, spaces, etc. The acronym stands for American Symbolic Code for Information Interchange.
  • Local versus remote: Computers cannot access each other’s code directly. They can only access messages that are exchanged via ports such as the serial one.
  • FIFO – Messages are sent between the computer and a microcontroller using first in and first out organization. This is helpful because it makes sure that messages are received in the order they were sent.
  • BYTE – is a setting that tells serial communication functions to send the raw value of a number using a single byte. This assumes that the value of the number will be between 0 and 255 since this is the only range of values that can be communicate using a single byte [one byte is composed of 8 bits which is equivalent to 2 to the power of 8, 256.
  • DEC – is a setting that tells serial communication functions to send the value coded in ascii format. In this case the number 255 would be coded by 3 different bytes, each byte holding a separate character.
  • Serial.print() – function that sends information, which it accepts as a string argument, via the serial port. The standard setting transmits information as ASCII characters (in other words, it uses DEC mode).
  • Serial.Println() – similar to print() function but adds two bytes to each transmission, a line feed and a carriage return. It features the same standard setting as Serial.print().

Here is a short list of additional functions that we did not cover in class but are related to serial communication.

  • Serial.write() – similar to the print() function but only sends information as BYTE.
  • – function that reads information from the buffer. It reads the first byte of incoming data. If there is not new communications then it will return a -1.
  • Serial.available() – function that reads the buffer to determine how many bytes are available for reading. The serial buffer can hold up to 128 bytes.
  • Serial.flush() – function that clears the buffer. It essentially erases all data from the buffer.

Levels of agreement for serial communications

  • Physical (which pins to use? how are messages physically transmitted?)
  • Electrical (what level of voltage does each device require?)
  • Logical (what does a pulse mean – is it considered a 1 or a 0? )
  • The data (what do the 1’s and 0’s represent? How should they be grouped)
  • The application (what does the groups of 0’s and 1’s signify to the application?)

Using the serial library – in Processing
On the Arduino the serial library and class is a standard part of the codeset. Therefore, you don’t need to include any additional libraries in order to send and receive data via the serial port. Unfortunately, that is not the case for Processing. That is not to say that it is hard to use the serial port in processing. Here is what you have to do:

  • At the top of your sketch you need to include the serial library, since this class is not part of the core Arduino codeset.
  • Declare an object from the serial class.
  • Create the serial object, using three arguments: (1) “this”, which tells the computer that this program is using the serial port; (2) serial port location; and (3) connection speed.

Reading serial data with processing

  • Option 1- Create a function titled serialEvent. This function is similar to mousePressed, except that it is called whenever the serial buffer fills up. This is where most of the action will happen. Be sure to pass on as an argument the serial port where the data will be available (e.g. serialEvent(Serial serialObjectPort) {code here}).
  • Option 2 – use an if statement in the draw section of the program that checks if there is information available in the serial buffer (e.g. serialObjectPort.available()) {code to execute}). This approach usually works a bit slower because the draw function may be in the middle of a loop, and it will need to wait until the next loop iteration to run the code. Whereas the serialEvent() function will run regardless of the state of the draw function.

Random Additional Notes:
Cornflake: great tool to use when developing applications that leverage serial communications. Allows developer to view the content of the serial port using both byte values and ascii characters. Also makes it easy to shut off connection to port, which is important because only one application can use the serial port at once.
Check out the video and image library on processing to be able to create interfaces similar to the drawing-music one shown here.