Guitar Heronoid

Sergei Brin at GarageGeeks

2008-10-07T03:34:00.000-07:00

The GarageGeeks and Yossi Vardi hosted a special garage party event
"GarageGeeks hosting Google"
Sergey Brin, Co-founder and President of Technology at Google

GuitarHeroNoid at LeWeb3 France

2007-12-14T00:24:00.000-08:00

Kathy brooks from Six Apart and Loic Le Meur invited the GuitarHeroNoid to perform at LeWeb3 conference France.

Our man Tal Chalozin, the heronoid Puppeteer, went there and presented the garageGeeks activities.

Here is a video of the show

Construction - The Body

2007-03-20T11:13:00.000-07:00

Intro

The robot is divided into two basic parts, "brain" and "body".
The "brain" is in charge of detecting and analyzing the PS2 video signals.
The "brain" generates play-commands (which tone to press, when to do it and when to strum) for the "body".
The "body" reads the play-commands and controls the fingers movement by them.

Some basic features were required:

No hacking on the guitar nor the PS2.
The hand should have a "look-n-feel" of a real human hand.
(AKA, the fingers should fully move, just "like humans do")
It have to work fast enough, in order to be able to keep up with the song.
The "brain" and the "body" should be fully separated.
All the construction have to be detachable.

Parts (chronologically ordered):

Listener

As I said earlier, the "brain" generates the play-commands and sends them via TCP to the "body" listener. The listener translates the play-commands to a proper BYTE sent from the parallel to the body main control board.

Control Board

The control board is in charge of the getting the play-commands feed from the computer and delivering them to the fingers engines.

The feeds are getting directly from the parallel port, which generates not more then ~3.5volts. The hands solenoids needs 7VDC and around 800mA to operate. So, current amplifier was needed.

The control board is contains a seven-darlingron-array, where each input pin is connected to one of the data bits on the parallel post (pins 2-7). It's responsible for the current amplifying needed for the solenois which were connected to the darlington's output pins. Our mistake was the we haven't used the right darlington. The solenoids consumed 800mA but the darlington could manage only till 600mA. This caused to burn Refael's parallel port on his laptop….

The lack of time till the first robot live show, I built a new board using the same darlington but instead of connecting straight to the solenoids, I used relay to made the extra current amplifying.

Hands

The hand are the most important and complicated part of my job. I needed to build it to as close I can to a real hand (a robotic one). The main barrier was to be able create a real-like finger movement, and still pressing the guitar buttons.

It may look easy when a human holds the guitar, but the buttons are quite hard to press. You need to generate quite a lot of power in order to made them pressed.

The left hand is the "notes-hanler", I needed to make five finger hand.

The right hand is the "strummer", here I only needed one moving finger and four frozens.

The hands were made from aluminum "sculptured" by hand (w/o any use of heavy machinery). Each finger was connected to a solenoid that was hidden inside the palm.

(it's hard to see, but here are 5 solenoids,
connected by wires to the fingers)

Both hands are completely detachable.

Doll

We wanted to create the feeling of a full-scale humanoid (human-android) is playing the guitar. First I thought basing it on a skeleton (like the one form the biology labs) will be cool, but I encountered too many mechanical difficulties.

The other option was using a display-window doll. A cool friend (that by incidence is the son of the owner of one of ISRAEL bigger clothing company (Honigman) ) was kind enough to give us a brand new display-window doll free of charge.

The problem with the doll was that his hands weren't in the right position (the elbow's angle wasn't changeable), so I needed to make them a changeable ones…

After cutting the right elbow, I inserted pipes for the control wires going from the control board to the hands and casted whipped polyurethane to make it permanent.

The Brain

2007-03-19T01:07:00.000-07:00

The Brain of the Robot was developed by Rafael Mizrahi, a GarageGeeks made-man and research manager at Feng-GUI the artificial vision lab.

GuitarHeroNoid source code home page at Google Code service.

Each song is presented on a set of five columns, resembling a real guitar fret
board, that scroll constantly towards the player. The five columns correspond to
the five fret buttons and appropriately colored notes appear in these columns.

Notes detection area

Detecting the notes could be accomplished by using several approaches to detect
moving objects; most of them might not be fast enough to implement and use. So
we came up with this idea: Looking at the game you can quickly realize that the
important information, such as the plates is brighter than the rest of the
image.

HSV color space

Representing the color of the pixels in each trapezoid from RGB as HSV (Hue,
Saturation, Value), also known as HSB (Hue, Saturation, Brightness), and
defining a threshold (Brightness Threshold at the properties dialog),
somewhere in the middle of the Brightness value, gives you a binary
representation of the pixels inside the trapezoid. Now, when the trapezoid is
filled with enough white pixels, there is probably a plate over there.

Binary black and white image

We connecting the game video output using a capture device into a computer. A
live video streaming filter captures the video frames as images and sends each
image into the image processing part of the program that detects the notes.

one Robot Server and one Image Processing
Delays, Delays, , , Delays
To play a note, the player must hold the correct fret button and press the strum bar. After playing and watching the game, you find out that the PlayStation add another delay to the equation. Pressing the frets buttons is recognized by the PlayStation game within some 100 milliseconds or so. Having this delay along with the delay it takes for the strum solenoids to go up and down, I realized that sending fret notes together with the strum action is not possible. I divided the protocol into two main actions:
The first, at the area of detection, pressing one or more of the fret notes and lift up the strum. The fret notes information is added into a FIFO queue. The second, after about 250 milliseconds (Strum Delay at the properties dialog), pop the fret notes from the queue and send them to the guitar with a strum down.

A nice TODO is to detect the BPM of the song at the first seconds of the song, and adjust all those delays according to that.

Sending notes to Robot Server

Multiplayer is split-screen. In a "dueling guitars" fashion,
two players tackle segments of the selected song. Unlike other modes, it is not
possible to fail a song in multiplayer, but scoring dictates that one player
will generally win.
Just by moving the area of detection to the side of the screen, using smaller trapezoids, all parameters in a different configurations file (guitarhero.ini), and there you go, you can play with or against the GuitarHeroNoid.

Multi Player mode

Articles

2007-03-17T22:54:00.000-07:00

Nic Fulton from Reuters interview with the garagegeeks builders of the Guitar Heronoid

GuitarHeroNoid at Slashdot slashgear

joystiq website Man still beats machine at Guitar Hero

Source code explanation at GuitarHeroNoid at Code Project

KinnerNet 2007 at WMMNA

more images at flickr

Videos

2007-03-17T22:02:00.000-07:00

Guitar HeroNoid first live show at KinnerNet2007 dining room.

10 minutes before the first live show at KinnerNet2007 dining room.

Guitar HeroNoid 4th integration 5 AM
The first time it came to life.

Solenoid tests with an input file created by the robot's brain.

Video detection - single player

Video detection - multi player

Whats Next

2007-02-16T01:45:00.000-08:00

We are working to improve the GuitarHeroNoid, cause we can't have an 8 years old kid be better than a robot.
http://www.joystiq.com/2007/04/03/man-still-beats-machine-at-guitar-hero/
http://www.joystiq.com/2007/01/15/eight-year-old-is-better-than-you-at-guitar-hero/

GuitareHeroNoid V2 at VON Boston 2007

2007-02-12T06:28:00.000-08:00

Mr Jeff Pulver invited the human-size robot playing guitar hero aka the GuitarHeroNoid to preform in VON Boston 2007 (Video Over the Net) conference in Boston.

Tal Chalozin and Yuval Tal have improved the robot version 2, adding some more features like the Controller-Controller and the Penis Guitar Holder.

Pictures from VON