I’m writing a little post here about my final project for the Computer Vision and Image Processing class I took last semester. This project was inspired by a Hollywood-style computer interfaces (think Minority Report, or more recently, Avatar) where your computer is controlled by the movement of your hand. Rather than having someone wear a glove equipped with a sensor, I sought out to see how well I could track my hand with a camera.

This implementation uses OpenCV and Python. I used these bindings for Python, which I now realize are somewhat out of date. Still, I’m pretty satisfied with the results, given that it was developed over the course of a few weeks.

It works by first selecting “good features to track”, or points in the image where the largest changes in pixel values occur. So, this works best in front of a background that contrasts well with flesh tones. Then, it tracks the location of points as they change from frame to frame of video. Together, these two techniques are known as KLT tracking. You can read more about it, as well as read the original papers describing it here.

I put up my code on GitHub, though it is under commented and could use a great deal of optimization. Like a lot of my class projects, I think it represents a solid start, though. You’ll need the OpenCV libraries I linked before and Pygame in order to run it.

Here’s a video of me using it:

Computer Vision Tracking Final Project from Paul, who badly needs a haircut.

The left window shows the tracking. Here you can see me press “f” to find these “good features”, which are highlighted in green. Then, I press “r” to tell it that I intend on moving the object to track to the right. I move my hand to the right (my right, anyway), and then the most relevant points are highlighted in red. You can see that as I move, the tracker updates which points it finds the most relevant, discarding points that go against the grain of the average movement of my hand, and picking up new ones. The gray circle represents where the program thinks the center of my hand is.

The right window is a simple “game”. The object is to move the hollow blue circle on the green one, while avoiding the red one. I think this does well to demonstrate the level of precision the system has, although if I had more time, I would have done something more interesting.

It’s also possible to see some of the limitations of this approach. For one, you can only track one object per camera. That isn’t really a problem for what I was trying to do, as a final version of this should have two cameras positioned towards the hands. That way there’s no interference from moving your head or upper arms. You can see both confuse the tracker during the course of this video.

Another limitation is that although this tracks position well, it does nothing to recognize gestures. Gesture recognition could be implemented on top of point tracking as it exists now, although I know more specialized algorithms exist for that purpose. Either way, that’s beyond the scope of this.


This entry is crossposted from the Animating Fast blog that I updated as part of the Animating Fast class I took last semester at Hampshire College. The class covered experimental ways to speed up the production of 3D animation, and this entry details my experiences getting up and running with Python in Maya, and some of the work I did for the class.

0. Thoughts on Maya

It’s easy to find learning a piece of software like Maya daunting. At most schools, it takes two full semester-long classes in 3D animation before someone really understands how to create 3D animation from beginning to end, and that doesn’t even cover the advanced techniques and features buried inside of the program. Scripting something like Maya can seem an even more difficult task, as most of the time, we know Maya from its user interface, and not from the way it stores data internally.

What Maya does do for you is echo the API commands in its own scripting language, MEL. Maya’s UI is written in MEL, at the same level I was working when I started writing my scripts. That means that the source code to the Maya tools written in MEL would provide a valuable learning tool in my attempt to learn the ins and outs of the program. Autodesk don’t, however, provide step-by-step tutorials or even an explanation of its API by feature, rather, Maya developers are expected to learn the program through a combination of its command reference and the UI source code.

More after the jump…


So, I’ve decided to start updating this blog again. As you may notice, things have changed, as this blog will no longer serve to record my experience dealing with the RIAA, but instead be a place for me to show off what I’m working on. Whether or not anyone actually reads this doesn’t matter, since I’m mostly using it to organize what I’m doing and show off some of my projects to my friends. Hopefully, I’ll stay dedicated to updating this and it will grow to be a meaningful log of cool things I’ve made. If you’re still looking for the RIAA story, it’s not going anywhere.

Recently, I’ve been experimenting with drawing graphics using Javascript. I’ve been programming Flash for some time now, so I’ve experienced the fun of graphically intense apps in the web browser already, but I really wanted to break out of the plugin and see what you can do inside of a web page just in Javascript, and how fast you can make it.

I’ve made two demos, neither of which are finished, but I’ll show off links to them anyway.

The first will eventually become the new homepage of my site, that links to my blog, replacing the white vacuous page that currently is there. It’s Wolfram cellular automata generator, that picks random rules and random states and displays them, changing rules occasionally, or when the action gets repetitive. You can see a demo of it here. Eventually, I want to add support for the other kind of cellular automaton we studied, and maybe make it a little interactive with regards to which rule is displayed. This project was inspired by a program I developed with a classmate for an Unconventional Computing class at school. That’s over here.

The second still needs a lot more work, but still is pretty enough to show off. It’s a port of a program I originally wrote in Java and OpenGL (JoGL), for a class I took at Smith College called Programming for the Interactive Arts. This program loads a SVG font (easily converted from a TrueType font using a utility like batik) and displays the poem “Jabberwocky” by Lewis Carroll word for word, tweening between each word. It looks fairly cool, and I learned a lot coding it, unfortunately it can get a little slow on some of the bigger words. Check it out.

While Javascript will never beat OpenGL in terms of speed, I think there are still a lot of optimizations I can make. First and foremost, it’s important to say that the tweening function eats up most of the time. The program starts for me in a couple seconds, which is pretty impressive, considering it’s downloading an XML document, parsing it, and generating vector points for about a hundred glyphs. The tweening algorithm I’m using now results in much prettier tweening than my first, more obvious solution, but has the disadvantage of being slow. One idea I had is to sneakily prerender each frame throughout the tween during the pause during which the word isn’t animated. Right now this is about a half a second, which could be enough to render ten frames of tweening between two words. I still have to implement this, of course.

There are a lot of other cool things I can do here as well, such as give the user the choice of fonts, color, tweening algorithm, and text to display. Also, there’s the practical issue of big words not fitting on the screen. =x But otherwise, I think this is ready for a little showing off. If you come across this blog and try it, I’d love to hear how it runs for you.

I’ll also eventually release the font rendering API under an LGPL license, once it’s mature enough.

My eventual goal for both of these projects is to take some meaningful benchmarks, and compare them to the running time of the programs I ported them from. I wonder if we’ll be seeing more games or animations in the future done entirely in JS.


Something for “Nothing”

I’m sure anyone reading this has heard about Nine Inch Nails’ new release, Ghosts, which is at least partially available for free, released under a Creative Commons license. Regardless of what you think of his music, you have to congratulate Trent Reznor for being one of the few mainstream artists in music today who understands the digital music. This isn’t his first experiment with distribution methods of this kind, having released isolated tracks for previous albums for remixers, and even admitting to be a part of the late great music tracker, OiNK.

Of course, giving away music for free online is nothing new to independent and unsigned artists. Many unsigned artists used Jukebox to share the music they themselves wrote on their Facebook profile, and even some independent artists on major labels used Jukebox for their own personal profile. One recent release I’d like to draw your attention to comes from the Boston sludge band Disappearer. You can get their new EP here. Even if you don’t think you’d like this kind of music, allow me to suggest you give this a download. Artists like this need exposure, and after all, it costs you “nothing”, only a little bit of your listening time.


Jukebox, my music player application for Facebook, has over 44,000 songs in its database. It originated as a project I wrote just for fun this summer, mainly because I was bored and because I wanted to play with the Facebook API. I was pretty disappointed with the other popular music apps at the time, and thought I could contribute something. I guess it was a bit of an accidental success. I launched it in late June, and watched over 100,000 Facebook users add it, over 34,000 remain users, and 1,500 – 2,000 use it daily. This is all without selling any ad space, using any shady invite system (something now common among Facebook applications), or publishing any stories to a user’s news or mini feed. I’ve made no money off of it, nor do I plan to. Since Jukebox works by linking users to music hosted online, I feel that it would violate common sense to sell advertising on pages that only display content other people (i.e., musicians) have worked so hard to create.

With that said, it probably should have come as no surprise to me that my program would eventually gain the attention of the RIAA. At the time I started Jukebox, I already knew about the legal threats the RIAA regularly makes against individuals who download music. I also knew about their relentless pursuit of websites and peer to peer applications dedicated to sharing music, and their practice of holding the programmers themselves or their companies responsible, even though the authors of those programs designed them to be used in ways that are completely legal.

I knew this, but I also didn’t think it would happen to Jukebox. Jukebox isn’t Napster, after all, it’s only a Facebook application. All it knows about its songs are links to MP3 files hosted elsewhere. Some songs were hosted on Google Pages, others on various other free file hosts online, and some on users’ personal web space. At any rate, it seemed logical to me that linking to MP3s is not wrong, and even if it was, it seemed clear to me that it makes no sense to hold be accountable for links I did not put up. I thought the law would take my position as well, as Jukebox had many legitimate legal uses.

The letter came anyway. By e-mail, I received the two page legal threat, and a list of 1,495 links found in the Jukebox database that the RIAA claimed infringed on copyright. In case you don’t feel like reading it all, it basically said that the list was meant not to be “exhaustive”, but was provided so that I could “understand the scope of infringement”. Furthermore it was “evident that countless other RIAA member copyrighted sound recordings are being infringed through your service in an identical manner.” The next day, I received the same letter in a thick FedEx package, including the imposing list of 1,495 links. There was also a purple CD-R which contained an Excel database that was a “more extensive, yet still non-exhaustive, listing of sound recordings the rights to which are held by RIAA member companies.”

Is it any surprise that a corporation that proudly boasts that it represents “approximately 90% of all legitimate sound recordings produced and sold in the United States” can claim any music sharing service online infringes on their copyrights? Is it possible to avoid committing such infringement without sacrificing much of what users are looking for in an online music application? Of course, what’s right and wrong doesn’t matter, all that matters is what they can get away with doing or saying. So, before I did anything, I had to become certain as to the legality of what Jukebox does.

My housemate and good friend Rich connected me to a lawyer at Harvard Law School, Sam Bayard. I had a brief e-mail correspondence with him, and he gave me some very helpful (and encouraging) input. It’s important to underscore here that the fact that Mr.Bayard is not my lawyer, nor do we have an attorney-client relationship. He is a smart guy who knows about the law who chose to comment on this situation. Here’s what he had to say:

In general, linking to copyrighted material is not an infringement because no copy is made or hosted on the linking person’s site. So, it is pretty clear that Jukebox does not directly infringe anyone’s copyright.

However, services like Grokster, Napster, etc. that help others infringe copyright have been found liable under secondary copyright infringement theories — contributory infringement and vicarious infringement.

Contributory infringement exists where one (1) has knowledge of another’s infringement and (2) either materially contributes to or induces that infringement. Here, your friend would have a good argument that he/she has no knowledge of any specific infringing acts because the jukebox app is capable of significant non-infringing uses (this seems to be the case from my understanding of the facts — the program could enable users to play mp3s that are placed online by the copyright owner for public consumption for free, or mp3s that are licensed under a CC license; query: does it allow them to play files on their own computer?).

The Supreme Court complicated this issue a bit in the Grokster case by recognizing a form of secondary liability known as “intentional inducement.” The court said that where there is evidence of a device’s characteristics or knowledge of how it may be used (i.e., it was clear how Grokster could be used to infringe), combined with “statements or actions directed to promoting infringement,” then the person or entity distributing that device may be held liable despite lack of knowledge of specific infringing acts. The court said “one who distributes a device with the object of promoting its use to infringe copyright, as shown by clear expression or other affirmative steps taken to foster infringement, is liable for the resulting acts of infringement by third parties.”

Without looking too deeply into the specifics, it seems unlikely to me that the jukebox app fits into this category of intentional inducement.

Vicarious infringement exists where one (1) has the right and ability to supervise the allegedly infringing activity and (2) receives a direct financial benefit from the infringing activity. It doesn’t appear that your friend has any right and ability to supervise outside web pages or whomever may be posting infringing material accessible by the apps users, so this doesn’t seem like much of a problem.

So, it became clear that I could only be held accountable if I was inducing users to commit infringement. It seems clear that I’m not inducing anyone to break the law, as Jukebox’s music library is entirely submitted by users. I can’t “induce” my users to add links to copyrighted music (in fact, there is a little warning not to when music is added, not that it is realistic to expect all users to follow it). It’s entirely possible to use Jukebox without even being involved with the database, also. Many users just add links to their own personal music library hosted elsewhere, and many are even musicians who are using Jukebox to add their own tracks to their profile. So, it is entirely possible to use Jukebox without finding shared music through my service. And the users that do share links are the ones who have made the decision to infringe copyright, by uploading them offsite in the first place.

Unfortunately, knowing that you’re right and even that you’re not breaking the law is not enough in this situation. The RIAA could still sue me, and being a nineteen year old college student, a legal fight just is not feasible for me. They don’t have to worry about suing people like me, all they need to do is send a legal threat. This is known as the chilling effect, and more such letters (from the RIAA and other infamous groups such as the Church of Scientology) are available for viewing here.

Shortly after I made the decision to go along with the RIAA’s wishes, I received this e-mail from Facebook, which certainly complicated things further:

We have recently received the attached complaint regarding your “Jukebox” application from the RIAA. As you know, the Facebook Developer Terms of Service prohibit infringement of any other person’s intellectual property rights, and therefore if your application does contain infringing materials, you must remove them immediately.

As you are solely responsible for the operation of your application, we request that you resolve this issue directly with the complaining party, whom we have cc:ed on this e-mail at antipiracy@riaa.com. Facebook will disable your application if the complainant’s concerns are not addressed within 48 hours (by 11:30am PST on February 23rd). Please notify the complainant when the necessary steps have been taken. We reserve all rights in regard to this matter, including all of our rights under the Developer Terms of Service.

It was just not feasible for me to sort out this entire issue in 48 hours. I’d called Mr.McDevitt, the man from the RIAA who sent me the letter, several times only to get his voicemail and an extremely unhelpful secretary (interesting sidenote: the RIAA uses The Smiths as hold music). I wrote Facebook back explaining my situation to them and, to their credit, they offered me more time. Unfortunately, this couldn’t officially be over until I had the RIAA contact Facebook again and have them withdraw their compliant against Jukebox, which meant speaking to someone from the RIAA. While I contacted McDevitt over and over, hoping to get a few minutes on the phone with the guy, I deleted the 1,495 links enumerated in the RIAA’s letter to me from Jukebox’s database. It felt like admitting defeat, and I wasn’t happy about doing it, especially since I knew I was in the right. But as the letter itself stated, those 1,495 links were only the beginning, and I would have to remove many more in order for the RIAA to consider Jukebox’s infringement of their copyrights over.

While I was waiting on them, the founder of Pandora Radio, Tim Westergren gave a talk at my school about digital music and his experiences in starting Pandora. Specifically, he spoke about the difficulty he encountered in trying to negotiate with the music industry. Hearing that, I felt the need to talk to him after his presentation, to hear someone with actual industry experience weigh in on the situation I found myself in. Unsurprisingly, he felt my outlook was pretty bleak. He told me, point-blank, that I would not receive a call back from these people and that it just made the most sense to shut Jukebox down now before the RIAA actually came after me legally.

I had decided to give up after a week of silence on their part, but Mr.McDevitt did eventually get back to me on the phone. He was surprisingly courteous and professional, and after I told him I wanted to keep Jukebox working within the boundaries of the law, receptive to my ideas. What I am going to do is use the Excel spreadsheet sent to me by the RIAA as a list to check all of the songs in the database against. While not a complete list of RIAA-represented music, there’s no doubt that the spreadsheet contains the names of many of the 44,000 songs linked to in the Jukebox database. He was receptive to this idea, and said he wanted to see Jukebox once it was implemented. If the RIAA found it satisfactory, there would be no more risk of Facebook pulling my application or me getting sued.

Whether or not I choose to continue to update this blog is still up in the air, as it depends on how much and what kind of interest this entry generates. I never considered myself much of a blogger, but sometimes I have things to say, and this may be a useful place for me to say them. Even if I don’t, I hope this entry helps developers in the future who are curious about how something like this may happen to them, and generally keeps people informed about what they can expect and do.