This implementation uses OpenCV and Python. I used these bindings for Python, which I now realize are somewhat out of date. Still, I’m pretty satisfied with the results, given that it was developed over the course of a few weeks.

It works by first selecting “good features to track”, or points in the image where the largest changes in pixel values occur. So, this works best in front of a background that contrasts well with flesh tones. Then, it tracks the location of points as they change from frame to frame of video. Together, these two techniques are known as KLT tracking. You can read more about it, as well as read the original papers describing it here.

I put up my code on GitHub, though it is under commented and could use a great deal of optimization. Like a lot of my class projects, I think it represents a solid start, though. You’ll need the OpenCV libraries I linked before and Pygame in order to run it.

Here’s a video of me using it:

Computer Vision Tracking Final Project from Paul, who badly needs a haircut.

The left window shows the tracking. Here you can see me press “f” to find these “good features”, which are highlighted in green. Then, I press “r” to tell it that I intend on moving the object to track to the right. I move my hand to the right (my right, anyway), and then the most relevant points are highlighted in red. You can see that as I move, the tracker updates which points it finds the most relevant, discarding points that go against the grain of the average movement of my hand, and picking up new ones. The gray circle represents where the program thinks the center of my hand is.

The right window is a simple “game”. The object is to move the hollow blue circle on the green one, while avoiding the red one. I think this does well to demonstrate the level of precision the system has, although if I had more time, I would have done something more interesting.

It’s also possible to see some of the limitations of this approach. For one, you can only track one object per camera. That isn’t really a problem for what I was trying to do, as a final version of this should have two cameras positioned towards the hands. That way there’s no interference from moving your head or upper arms. You can see both confuse the tracker during the course of this video.

Another limitation is that although this tracks position well, it does nothing to recognize gestures. Gesture recognition could be implemented on top of point tracking as it exists now, although I know more specialized algorithms exist for that purpose. Either way, that’s beyond the scope of this.

0. Thoughts on Maya

It’s easy to find learning a piece of software like Maya daunting. At most schools, it takes two full semester-long classes in 3D animation before someone really understands how to create 3D animation from beginning to end, and that doesn’t even cover the advanced techniques and features buried inside of the program. Scripting something like Maya can seem an even more difficult task, as most of the time, we know Maya from its user interface, and not from the way it stores data internally.

What Maya does do for you is echo the API commands in its own scripting language, MEL. Maya’s UI is written in MEL, at the same level I was working when I started writing my scripts. That means that the source code to the Maya tools written in MEL would provide a valuable learning tool in my attempt to learn the ins and outs of the program. Autodesk don’t, however, provide step-by-step tutorials or even an explanation of its API by feature, rather, Maya developers are expected to learn the program through a combination of its command reference and the UI source code.

In addition to MEL, Autodesk provide a binding into Maya for the popular programming language Python. If you’ve already begun scripting for Maya in Python, you probably already know the weaknesses in the provided Python layer can make for some ugly code. The good news is that translating MEL into Python is fairly obvious. The bad news is that it results in code like this:

curQual = cmds.modelEditor (currentPanel, q=True, rnm = True)

if curQual == ‘base_OpenGL_Renderer’:

cmds.modelEditor (currentPanel, e=True,displayAppearance = “smoothShaded”, displayLights = “default”, displayTextures = True, rnm = “hwRender_OpenGL_Renderer”)

else:

cmds.modelEditor (currentPanel, e=True,displayAppearance = “smoothShaded”, displayLights = “default”, displayTextures = True, rnm = “base_OpenGL_Renderer”)

Calling one function for everything relating to, say, 3D Paint or a model view can get confusing quickly, especially since you’re either setting or getting flags that are explained only briefly in the documentation. In order to see how these functions are actually used, you need to dive into the Maya source code and pick out the relevant bits separate from the code that creates the user interface. In practice, this means turning on “Echo All Commands” inside of Maya, doing something you want to automate inside of Maya, and then grabbing the name of the UI call from the script editor and grepping through the hundreds of thousands of lines of MEL lurking in Maya’s /scripts folder. I think most Python hackers would agree that this development process and way of laying out the API is decidedly Unpythonic (http://en.wikipedia.org/wiki/Pythonic#Programming_philosophy). Some projects exist, such as PyMel (http://code.google.com/p/pymel/), that make Python syntax and MEL objects make more sense with each other, but unfortunately we didn’t have time to look into those for this class.

With all of that said, once you get the hang of development in Maya, you quickly realize that you have a powerful 3D package at your control that allows you to take advantage of some very advanced features without having to code a renderer for them yourself. In terms of time taken, using Maya’s particle engine or a 3D paint tool is undoubtably faster than writing your own, and this class is all about speed, so learning how to manipulate these tools with code was necessary to the mission of the class.

When we first decided talking about the kinds of tools a programmer might contribute to the class, a lot of my initial ideas centered around creating art programmatically. I thought about writing some code that laid out a virtual world based on a few basic objects that populate it, so that a program could be fed a few models of carnival booths and rides and then spit out an intelligently laid out scene. Another idea we talked about was taking a rigged model and generating walking animation for it based on the layout of its limbs, similar to what happens in the game Spore.

We eventually decided to pursue a different path, and focus on improving the interaction between the animator and Maya. We talked a lot about how the animator can get the ideas in his or her mind into Maya as quickly and efficiently as possible, while still having creative control over what results.

1. 3d Paint Tool

We decided that one group would start exploring the possibility of 3D matte painting, which would greatly reduct the amount of world building and modeling necessary to create the scene for our movie, without a sacrifice in visual quality (provided the camera stayed put). I was tasked with creating a 3D paint tool to streamline the painting process in Maya.

This tool would include a “Prime” button that would let an artist prepare a mesh for 3d painting a single click. Naturally, this requires a lot of steps to complete, and I ultimately decided to give the artist the option of turning off each step as needed. This saves the artist time in the beginning, but we realized that the real time saver would be minimizing the number of times the artist needs to go back to the Maya 3d Paint panel and change settings. For that reason, I added some hotkey management code that would swap out common single-button hotkeys for 3d Paint specific functions (e.g. C for choose color, G for grab color from screen). The animators really liked this, and said that specifically saved them a lot of time. Another feature we were interested in was moving the current 3d paint texture map to Photoshop. That eventually became the next tool in itself.

One final thing I should mention on the 3d paint tool is that there were some troubles getting it working at first. One peculiar thing about Maya 2008 and 8.5 is that it requires all named commands to be linked to MEL code, even in Python. This has apparently been fixed in 2009 with the sourceType flag on the nameCommand command, but most of the computers in Hampshire’s lab weren’t running 2009, so I used a quick fix using the MEL command python(), to effectively jump from Python to MEL and back to Python. A little unusual, but it worked…

Another problem with the 3d paint tool was its tendency to not restore the hotkeys once you were done using it. This was a nuisance that the animators had to go into the hotkey editor and fix themselves. The problem is that even though my script could restore the hotkeys, Maya didn’t seem to provide a way to trap the event of the script ending. Strangely enough, it does provide events for when your tool window is minimized and restored (minimizeCommand/restoreCommand flags on the window command), but even these weren’t called at the appropriate time. I ended up adding a restore all hotkeys button to the window for the artist to use before they finished with the tool.

2. Getting Renders out of Maya and Into Photoshop

This posed an interesting problem at first, as we weren’t sure how to divide up the painting duty between Photoshop and Maya’s 3D paint, and in which order they would be used. It seems like the painters were mostly outlining in Maya and finishing up all of the detail painting in Photoshop, so it seemed necessary to write a tool that could facilitate an export to Photoshop as quickly as my previous tool made it possible to manage a palette inside of Maya.

Since our artists were already dividing up the textures for 3D paint into categories and render layers in Maya, it seemed to make sense to develop a tool that could organize a camera, its materials, and its child render layers and textures all in one, while pushing renders of each layer out to Photoshop. I found the render command in Maya script, which I preferred over Maya’s external renderer because it let us have a scene file open as we rendered it and had built in flags for rendering specific render layers at a time. By default, it rendered to .iff with bad transparency. Maya provides a function called convertIffToPsd which I started using in the first version of my code, but Maya’s generated .psd files didn’t retain transparency as alpha (instead painting the empty void black), so we tried several different file formats. Maya’s export to .png preserved transparency perfectly, so I devised a system to include such a file in a .psd by means of a smart object (that’s Photoshop for an updatable reference to an image stored elsewhere) and a Photoshop actions file (.atn) to update it.

Getting Maya to spit out a .png as opposed to a .iff required some hunting through the MEL code as well. The trick is to call setAttr (‘defaultRenderGlobals.outf’,32), with 32 being Maya’s special identifier for .png files. You may want to setAttr the value back to 7 (.iff, the default) after your call to render.

I built an early version of this tool and demoed it for the class, but we ultimately decided we wanted to lose the organizational overhead and let the artists push out a single render reference image themselves that they could paint over, and “spray” the new paint over the old texture back in Maya.

Getting that single image out was surprisingly easy, and the script ended up being deployed as a windowless button that you could press and generate a quick render of the selected camera on your desktop.

3.Creating a Hypergraph Shader

Having created a series of paintings on top of a quick render, it became necessary to plan how to move the new paint back into Maya. Basically, we moved the organization task of fitting the painting images in place from the “before” Photoshop stage to the “after”.

The two animators, Taryn and Tatiana, who were painting in the class and provided commentary on the tools took two different approaches. One chose to use a quad shading switch in Maya, which is a special shadingNode that acts as kind of a multiplexer for pixel values within a shader. Because we had so many paintings that would have to fill a single surface, we thought it would be a good idea to have the shader make an intelligent decision as to which psdNode to source the image from, instead of building shaders for each image.

We did run out of time at this point in the semester, so I was never able to build a final version of this script, but I did have an early version to show off on the last day. In this final tool, a painter would first select a camera and lock it into place, and then create some materials associated with that camera’s shot. The painter could then assign each material (which already had a psdNode created for it in hypershade). Then, the painter would select a meshes they wanted to texture, and assign it to a material (it’s important to realize that the relationship between meshes and materials is many to one), which would then update the hypergraph shader. Finally, when they were done, they could mash a final button which would connect the whole mess together through the quad shading switch. Seeing what this looked like created manually made it seem like a process in dire need of speeding up.

The animators decided later in that class to move to multiple shaders for other reasons, which does eliminate the problem of creating the quad shading switch, although a tool to create multiple shaders could certainly have been just as handy, and that most likely is what I would have started on next if I had more time.

4.Conclusion

I hope that the code I’ve written as part of this class becomes helpful to CG students here at Hampshire and animators anywhere that might be reading this blog. Improving the interaction between technically minded artists and necessarily complicated software is a rich area to explore, and I wouldn’t be surprised if there were big gains in efficiency to be had in the process of streamlining the interface of a massive piece of software like Maya. As for me, I’m satisfied to have learned a lot more about the techniques and terminology surrounding computer graphics, and to have picked up Maya scripting skills along the way. I’m still impressed by the program’s size and capability, but no longer intimidated, and plan on continuing to play around with Maya this summer.

Recently, I’ve been experimenting with drawing graphics using Javascript. I’ve been programming Flash for some time now, so I’ve experienced the fun of graphically intense apps in the web browser already, but I really wanted to break out of the plugin and see what you can do inside of a web page just in Javascript, and how fast you can make it.

I’ve made two demos, neither of which are finished, but I’ll show off links to them anyway.

The first will eventually become the new homepage of my site, that links to my blog, replacing the white vacuous page that currently is there. It’s Wolfram cellular automata generator, that picks random rules and random states and displays them, changing rules occasionally, or when the action gets repetitive. You can see a demo of it here. Eventually, I want to add support for the other kind of cellular automaton we studied, and maybe make it a little interactive with regards to which rule is displayed. This project was inspired by a program I developed with a classmate for an Unconventional Computing class at school. That’s over here.

The second still needs a lot more work, but still is pretty enough to show off. It’s a port of a program I originally wrote in Java and OpenGL (JoGL), for a class I took at Smith College called Programming for the Interactive Arts. This program loads a SVG font (easily converted from a TrueType font using a utility like batik) and displays the poem “Jabberwocky” by Lewis Carroll word for word, tweening between each word. It looks fairly cool, and I learned a lot coding it, unfortunately it can get a little slow on some of the bigger words. Check it out.

While Javascript will never beat OpenGL in terms of speed, I think there are still a lot of optimizations I can make. First and foremost, it’s important to say that the tweening function eats up most of the time. The program starts for me in a couple seconds, which is pretty impressive, considering it’s downloading an XML document, parsing it, and generating vector points for about a hundred glyphs. The tweening algorithm I’m using now results in much prettier tweening than my first, more obvious solution, but has the disadvantage of being slow. One idea I had is to sneakily prerender each frame throughout the tween during the pause during which the word isn’t animated. Right now this is about a half a second, which could be enough to render ten frames of tweening between two words. I still have to implement this, of course.

There are a lot of other cool things I can do here as well, such as give the user the choice of fonts, color, tweening algorithm, and text to display. Also, there’s the practical issue of big words not fitting on the screen. =x But otherwise, I think this is ready for a little showing off. If you come across this blog and try it, I’d love to hear how it runs for you.

I’ll also eventually release the font rendering API under an LGPL license, once it’s mature enough.

My eventual goal for both of these projects is to take some meaningful benchmarks, and compare them to the running time of the programs I ported them from. I wonder if we’ll be seeing more games or animations in the future done entirely in JS.

Of course, giving away music for free online is nothing new to independent and unsigned artists. Many unsigned artists used Jukebox to share the music they themselves wrote on their Facebook profile, and even some independent artists on major labels used Jukebox for their own personal profile. One recent release I’d like to draw your attention to comes from the Boston sludge band Disappearer. You can get their new EP here. Even if you don’t think you’d like this kind of music, allow me to suggest you give this a download. Artists like this need exposure, and after all, it costs you “nothing”, only a little bit of your listening time.

With that said, it probably should have come as no surprise to me that my program would eventually gain the attention of the RIAA. At the time I started Jukebox, I already knew about the legal threats the RIAA regularly makes against individuals who download music. I also knew about their relentless pursuit of websites and peer to peer applications dedicated to sharing music, and their practice of holding the programmers themselves or their companies responsible, even though the authors of those programs designed them to be used in ways that are completely legal.

I knew this, but I also didn’t think it would happen to Jukebox. Jukebox isn’t Napster, after all, it’s only a Facebook application. All it knows about its songs are links to MP3 files hosted elsewhere. Some songs were hosted on Google Pages, others on various other free file hosts online, and some on users’ personal web space. At any rate, it seemed logical to me that linking to MP3s is not wrong, and even if it was, it seemed clear to me that it makes no sense to hold be accountable for links I did not put up. I thought the law would take my position as well, as Jukebox had many legitimate legal uses.

The letter came anyway. By e-mail, I received the two page legal threat, and a list of 1,495 links found in the Jukebox database that the RIAA claimed infringed on copyright. In case you don’t feel like reading it all, it basically said that the list was meant not to be “exhaustive”, but was provided so that I could “understand the scope of infringement”. Furthermore it was “evident that countless other RIAA member copyrighted sound recordings are being infringed through your service in an identical manner.” The next day, I received the same letter in a thick FedEx package, including the imposing list of 1,495 links. There was also a purple CD-R which contained an Excel database that was a “more extensive, yet still non-exhaustive, listing of sound recordings the rights to which are held by RIAA member companies.”

Is it any surprise that a corporation that proudly boasts that it represents “approximately 90% of all legitimate sound recordings produced and sold in the United States” can claim any music sharing service online infringes on their copyrights? Is it possible to avoid committing such infringement without sacrificing much of what users are looking for in an online music application? Of course, what’s right and wrong doesn’t matter, all that matters is what they can get away with doing or saying. So, before I did anything, I had to become certain as to the legality of what Jukebox does.

My housemate and good friend Rich connected me to a lawyer at Harvard Law School, Sam Bayard. I had a brief e-mail correspondence with him, and he gave me some very helpful (and encouraging) input. It’s important to underscore here that the fact that Mr.Bayard is not my lawyer, nor do we have an attorney-client relationship. He is a smart guy who knows about the law who chose to comment on this situation. Here’s what he had to say:

In general, linking to copyrighted material is not an infringement because no copy is made or hosted on the linking person’s site. So, it is pretty clear that Jukebox does not directly infringe anyone’s copyright.

However, services like Grokster, Napster, etc. that help others infringe copyright have been found liable under secondary copyright infringement theories — contributory infringement and vicarious infringement.

Contributory infringement exists where one (1) has knowledge of another’s infringement and (2) either materially contributes to or induces that infringement. Here, your friend would have a good argument that he/she has no knowledge of any specific infringing acts because the jukebox app is capable of significant non-infringing uses (this seems to be the case from my understanding of the facts — the program could enable users to play mp3s that are placed online by the copyright owner for public consumption for free, or mp3s that are licensed under a CC license; query: does it allow them to play files on their own computer?).

The Supreme Court complicated this issue a bit in the Grokster case by recognizing a form of secondary liability known as “intentional inducement.” The court said that where there is evidence of a device’s characteristics or knowledge of how it may be used (i.e., it was clear how Grokster could be used to infringe), combined with “statements or actions directed to promoting infringement,” then the person or entity distributing that device may be held liable despite lack of knowledge of specific infringing acts. The court said “one who distributes a device with the object of promoting its use to infringe copyright, as shown by clear expression or other affirmative steps taken to foster infringement, is liable for the resulting acts of infringement by third parties.”

Without looking too deeply into the specifics, it seems unlikely to me that the jukebox app fits into this category of intentional inducement.

Vicarious infringement exists where one (1) has the right and ability to supervise the allegedly infringing activity and (2) receives a direct financial benefit from the infringing activity. It doesn’t appear that your friend has any right and ability to supervise outside web pages or whomever may be posting infringing material accessible by the apps users, so this doesn’t seem like much of a problem.

So, it became clear that I could only be held accountable if I was inducing users to commit infringement. It seems clear that I’m not inducing anyone to break the law, as Jukebox’s music library is entirely submitted by users. I can’t “induce” my users to add links to copyrighted music (in fact, there is a little warning not to when music is added, not that it is realistic to expect all users to follow it). It’s entirely possible to use Jukebox without even being involved with the database, also. Many users just add links to their own personal music library hosted elsewhere, and many are even musicians who are using Jukebox to add their own tracks to their profile. So, it is entirely possible to use Jukebox without finding shared music through my service. And the users that do share links are the ones who have made the decision to infringe copyright, by uploading them offsite in the first place.

Unfortunately, knowing that you’re right and even that you’re not breaking the law is not enough in this situation. The RIAA could still sue me, and being a nineteen year old college student, a legal fight just is not feasible for me. They don’t have to worry about suing people like me, all they need to do is send a legal threat. This is known as the chilling effect, and more such letters (from the RIAA and other infamous groups such as the Church of Scientology) are available for viewing here.

Shortly after I made the decision to go along with the RIAA’s wishes, I received this e-mail from Facebook, which certainly complicated things further:

We have recently received the attached complaint regarding your “Jukebox” application from the RIAA. As you know, the Facebook Developer Terms of Service prohibit infringement of any other person’s intellectual property rights, and therefore if your application does contain infringing materials, you must remove them immediately.

As you are solely responsible for the operation of your application, we request that you resolve this issue directly with the complaining party, whom we have cc:ed on this e-mail at antipiracy@riaa.com. Facebook will disable your application if the complainant’s concerns are not addressed within 48 hours (by 11:30am PST on February 23rd). Please notify the complainant when the necessary steps have been taken. We reserve all rights in regard to this matter, including all of our rights under the Developer Terms of Service.

It was just not feasible for me to sort out this entire issue in 48 hours. I’d called Mr.McDevitt, the man from the RIAA who sent me the letter, several times only to get his voicemail and an extremely unhelpful secretary (interesting sidenote: the RIAA uses The Smiths as hold music). I wrote Facebook back explaining my situation to them and, to their credit, they offered me more time. Unfortunately, this couldn’t officially be over until I had the RIAA contact Facebook again and have them withdraw their compliant against Jukebox, which meant speaking to someone from the RIAA. While I contacted McDevitt over and over, hoping to get a few minutes on the phone with the guy, I deleted the 1,495 links enumerated in the RIAA’s letter to me from Jukebox’s database. It felt like admitting defeat, and I wasn’t happy about doing it, especially since I knew I was in the right. But as the letter itself stated, those 1,495 links were only the beginning, and I would have to remove many more in order for the RIAA to consider Jukebox’s infringement of their copyrights over.

While I was waiting on them, the founder of Pandora Radio, Tim Westergren gave a talk at my school about digital music and his experiences in starting Pandora. Specifically, he spoke about the difficulty he encountered in trying to negotiate with the music industry. Hearing that, I felt the need to talk to him after his presentation, to hear someone with actual industry experience weigh in on the situation I found myself in. Unsurprisingly, he felt my outlook was pretty bleak. He told me, point-blank, that I would not receive a call back from these people and that it just made the most sense to shut Jukebox down now before the RIAA actually came after me legally.

I had decided to give up after a week of silence on their part, but Mr.McDevitt did eventually get back to me on the phone. He was surprisingly courteous and professional, and after I told him I wanted to keep Jukebox working within the boundaries of the law, receptive to my ideas. What I am going to do is use the Excel spreadsheet sent to me by the RIAA as a list to check all of the songs in the database against. While not a complete list of RIAA-represented music, there’s no doubt that the spreadsheet contains the names of many of the 44,000 songs linked to in the Jukebox database. He was receptive to this idea, and said he wanted to see Jukebox once it was implemented. If the RIAA found it satisfactory, there would be no more risk of Facebook pulling my application or me getting sued.

Whether or not I choose to continue to update this blog is still up in the air, as it depends on how much and what kind of interest this entry generates. I never considered myself much of a blogger, but sometimes I have things to say, and this may be a useful place for me to say them. Even if I don’t, I hope this entry helps developers in the future who are curious about how something like this may happen to them, and generally keeps people informed about what they can expect and do.