EDN Admin
Well-known member
Introduction In early January, we were tasked with creating a unique, interactive experience for the http://sxsw.com/interactive SXSW Interactive launch party with http://www.frogdesign.com/ Frog Design . We bounced around many ideas, and finally settled on a project that Rick suggested during our first meeting: boxing robots controlled via Kinect. The theme of the opening party was Retro Gaming, so we figured creating a life size version of a classic tabletop boxing game mashed up with a "Real Steel"-inspired Kinect experience would be a perfect fit. Most importantly, since this was going to be the first big project of the new Coding4Fun team, we wanted to push ourselves to create an experience that needed each of us to bring our unique blend of hardware, software, and interaction magic to the table under an aggressively tight deadline. Hardware The BoxingBots had to be fit a few requirements: <ol>They had to be fun They had to survive for 4 hours, the length of the SXSW opening party Each robot had to punch for 90 seconds at a time, the length of a round They had to be life-size They had to be Kinect-drivable They had to be built, shipped, and reassembled for SXSW </ol> Creating a robot that could be beaten up for 4 hours and still work proved to be an interesting problem. After doing some research on different configurations and styles, it was decided we should leverage a prior project to get a jump start to meet the deadline. We repurposed sections of our Kinect drivable lounge chair, http://channel9.msdn.com/coding4fun/articles/Jellybean-the-Kinect-Drivable-Lounge-Chair Jellybean ! This was an advantage because it contained many known items, such as the motors, motor controllers, and chassis material. Additionally, it was strong and fast, it was modular, and the code to drive it was already written. Jellybean would only get us part of the way there, however. We also had to do some retrofitting to get it to work for our new project. The footprint of the base needed to shrink from 32x50 inches to 32x35 inches, while still allowing space to contain all of the original batteries, wheels, motors, motor controllers, switches, voltage adapters. We also had to change how the motors were mounted with this new layout, as well as provide for a way to easily "hot swap" the batteries out during the event. Finally, we had to mount an upper body section that looked somewhat human, complete with a head and punching arms. http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0087%5B4%5D.jpg <img title="IMG_0087" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0087_thumb%5B1%5D.jpg" alt="IMG_0087" width="240" height="180" border="0 http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0089%5B9%5D.jpg <img title="IMG_0089" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0089_thumb%5B4%5D.jpg" alt="IMG_0089" width="240" height="180" border="0
Experimenting with possible layouts The upper body had its own challenges, as it had to support a ton of equipment, including: Punching arms Popping head Pneumatic valves Air manifold Air Tank(s) Laptop Phidget interface board Phidget relay boards Phidget LED board Xbox wireless controller PC transmitter / receiver Chest plate LEDs Sensors to detect a punch http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0113%5B5%5D.jpg <img title="IMG_0113" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0113_thumb%5B2%5D.jpg" alt="IMG_0113" width="500" height="667" border="0
Brian and Rick put together one of the upper frames <h3>Punching and Air Tanks</h3> We had to solve the problem of getting each robot to punch hard enough to register a hit on the opponent bot while not breaking the opponent bot (or itself). Bots also had to withstand a bit of side load in case the arms got tangled or took a side blow. Pneumatic actuators provided us with a lot of flexibility over hydraulics or an electrical solution since they are fast, come in tons of variations, wont break when met with resistance, and can fine tuned with a few onsite adjustments. To provide power to the actuators, the robots had two 2.5 gallon tanks pressurized to 150psi, with the actuators punching at ~70psi. We could punch for about five 90-second rounds before needing to re-pressurize the tanks. Pressurizing the onboard tanks was taken care of by a pair of off-the-shelf DeWalt air compressors. http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0134%5B4%5D.jpg <img title="IMG_0134" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0134_thumb%5B1%5D.jpg" alt="IMG_0134" width="240" height="180" border="0 http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0184%5B4%5D.jpg <img title="IMG_0184" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0184_thumb%5B1%5D.jpg" alt="IMG_0184" width="240" height="180" border="0 http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0279%5B4%5D.jpg <img title="IMG_0279" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0279_thumb%5B1%5D.jpg" alt="IMG_0279" width="240" height="180" border="0 http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0172%5B4%5D.jpg <img title="IMG_0172" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0172_thumb%5B1%5D.jpg" alt="IMG_0172" width="240" height="180" border="0 <h3>The Head</h3> It wouldn’t be a polished game if the head didn’t pop up on the losing bot, so we added another pneumatic actuator to raise and lower the head, and some extra red and blue LEDs. This pneumatic is housed in the chest of the robot and is triggered only when the game has ended. To create the head, we first prototyped a concept with cardboard and duct tape. A rotated welding mask just happened to provide the shape we were going for on the crown, and we crafted each custom jaw with a laser cutter. We considered using a mold and vacuum forming to create something a bit more custom, but had to scrap the idea due to time constraints. http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0169%5B4%5D.jpg <img title="IMG_0169" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0169_thumb%5B1%5D.jpg" alt="IMG_0169" width="240" height="180" border="0 http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0201%5B4%5D.jpg <img title="IMG_0201" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0201_thumb%5B1%5D.jpg" alt="IMG_0201" width="240" height="180" border="0 http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0268%5B4%5D.jpg <img title="IMG_0268" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0268_thumb%5B1%5D.jpg" alt="IMG_0268" width="240" height="180" border="0 http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0291%5B4%5D.jpg <img title="IMG_0291" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0291_thumb%5B1%5D.jpg" alt="IMG_0291" width="240" height="180" border="0 <h3>Sensors</h3> Our initial implementation for detecting punches failed due to far too many false positives. We thought using IR distance sensors would be a good solution since we could detect a “close” punch and tell the other robot to retract the arm before real contact. The test looked promising, but in practice, when the opposite sensors were close, we saw a lot of noise in the data. The backup and currently implemented solution was to install simple push switches in the chest and detect when those are clicked by the chest plate pressing against them. http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0200%5B4%5D.jpg <img title="IMG_0200" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0200_thumb%5B1%5D.jpg" alt="IMG_0200" width="240" height="180" border="0 http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0205%5B4%5D.jpg <img title="IMG_0205" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0205_thumb%5B1%5D.jpg" alt="IMG_0205" width="240" height="180" border="0 <h3>Power</h3> Different items required different voltages. The motors and pneumatic valves required 24V, the LEDs required 12V and the USB hub required 5V. We used Castle Pro BEC converters to step down the voltages. These devices are typically used in RC airplanes and helicopters. <h3>Shipping</h3> http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0278%5B4%5D.jpg <img title="IMG_0278" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0278_thumb%5B1%5D.jpg" alt="IMG_0278" width="240" height="180" border="0 http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0280%5B4%5D.jpg <img title="IMG_0280" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0280_thumb%5B1%5D.jpg" alt="IMG_0280" width="240" height="180" border="0 So how does someone ship two 700lb robots from Seattle to Austin? We did it in 8 crates. <img src=http://ecn.channel9.msdn.com/o9/content/images/emoticons/emotion-1.gif?v=c9 alt=Smiley />. The key thing to note is that the tops and bottoms of each robot were separated. Any wire that connected the two parts had to be able to be disconnected in some form. This affected the serial cords and the power cords (5V, 12V, 24V). Software The software and architecture went through a variety of iterations during development. The final architecture used 3 laptops, 2 desktops, an access point, and a router. Its important to note that the laptops of Robot 1 and Robot 2 are physically mounted on the backs of each Robot body, communicating through WiFi to the Admin console. The entire setup looks like the following diagram: http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/network%5B4%5D.png <img title="network" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/network_thumb%5B2%5D.png" alt="network" width="387" height="426" border="0 <h3> Admin Console </h3> The heart of the infrastructure is the Admin Console. Originally, this was also intended to be a scoreboard to show audience members the current stats of the match, but as we got further into the project, we realized this wouldnt be necessary. The robots are where the action is, and peoples eyes focus there. Additionally, the robots themselves display their current health status via LEDs, so duplicating this information isnt useful. However, the admin side of this app remains. <h3> Sockets </h3> The admin console is the master controller for the game state and utilizes socket communication between it, the robots, and the user consoles. A generic socket handler was written to span each computer in the setup. The SocketListener object allows for incoming connections to be received, while the SocketClient allows clients to connect to those SocketListeners . These are generic objects, which must specify objects of type GamePacket to send and receive: <pre class="brush: csharp
public class SocketListener<TSend, TReceive> where TSend : GamePacket
where TReceive : GamePacket, new()
[/code]
GamePacket is a base class from which specific packets inherit: <pre class="brush: csharp
public abstract class GamePacket
{
public byte[] ToByteArray()
{
MemoryStream ms = new MemoryStream();
BinaryWriter bw = new BinaryWriter(ms);
try
{
WritePacket(bw);
}
catch(IOException ex)
{
Debug.WriteLine("Error writing packet: " + ex);
}
return ms.ToArray();
}
public void FromBinaryReader(BinaryReader br)
{
try
{
ReadPacket(br);
}
catch(IOException ex)
{
Debug.WriteLine("Error reading packet: " + ex);
}
}
public abstract void WritePacket(BinaryWriter bw);
public abstract void ReadPacket(BinaryReader br);
}
[/code] For example, in communication between the robots and the admin console, GameStatePacket and MovementDescriptorPacket are sent and received. Each GamePacket must implement its own ReadPacket and WritePacket methods to serialize itself for sending across the socket. Packets are sent between machines every "frame". We need the absolute latest game state, robot movement, etc. at all times to ensure the game is functional and responsive. http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/image%5B20%5D-2.png <img title="image" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/image_thumb%5B12%5D-3.png" alt="image" width="640" height="278" border="0 As is quite obvious, absolutely no effort was put into making the console "pretty". This is never seen by the end users and just needs to be functional. Once the robot software and the user consoles are started, the admin console initiates connections to each of those four machines. Each machine runs the SocketListener side of the socket code, while the Admin console creates four SocketClient objects to connect to each those. Once connected, the admin has control of the game and can start, stop, pause, and reset a match by sending the appropriate packets to everyone that is connected. <h3> Robot </h3> The robot UI is also never intended to be seen by an end user, and therefore contains only diagnostic information. http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/image%5B19%5D-1.png <img title="image" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/image_thumb%5B11%5D-1.png" alt="image" width="640" height="361" border="0 Each robot has a wireless Xbox 360 controller connected to it so it can be manually controlled. The UI above reflects the positions of the controller sticks and buttons. During a match, its possible for a bot to get outside of our "safe zone". One bot might be pushing the other, or the user may be moving the bot toward the edge of the ring. To counter this, the players coach can either temporarily move the bot, turning off Kinect input, or force the game into "referee mode" which pauses the entire match and turns off Kinect control on both sides. In either case, the robots can be driven with the controllers and reset to safe positions. Once both coaches signal that the robots are reset, the admin can then resume the match. <h3>Controlling Hardware</h3> Phidget hardware controlled our LEDs, relays, and sensors. Getting data out of a Phidget along with actions, such as opening and closing a relay, is shockingly easy as they have pretty straightforward C# APIs and samples, which is why they typically are our go-to product for projects like this. Below are some code snippets for the LEDs, relays, and sensor. LEDs – from LedController.cs This is the code that actually updates the health LEDs in the robots chest. The LEDs were put on the board in a certain order to allow this style of iteration. We had a small issue of running out of one color of LEDs so we used some super bright ones and had to reduce the power levels to the non-super bright LEDs to prevent possible damage: <pre class="brush: csharp
private void UpdateLedsNonSuperBright(int amount, int offset, int brightness)
{
for (var i = offset; i < amount + offset; i++)
{
_phidgetLed.leds = brightness / 2;
}
}
private void UpdateLedsSuperBright(int amount, int offset, int brightness)
{
for (var i = offset; i < amount + offset; i++)
{
_phidgetLed.leds = brightness;
}
}
[/code] Sensor data – from SensorController.cs This code snippet shows how we obtain the digital and analog inputs from the Phidget 8/8/8 interface board: <pre class="brush: csharp
public SensorController(InterfaceKit phidgetInterfaceKit) : base(phidgetInterfaceKit)
{
PhidgetInterfaceKit.ratiometric = true;
}
public int PollAnalogInput(int index)
{
return PhidgetInterfaceKit.sensors[index].Value;
}
public bool PollDigitalInput(int index)
{
return PhidgetInterfaceKit.inputs[index];
}
[/code] Relays – from RelayController.cs Electrical relays fire our pneumatic valves. These control the head popping and the arms punching. For our application, we wanted the ability to reset the relay automatically. When the relay is opened, an event is triggered and we create an actively polled thread to validate whether we should close the relay. The reason why we actively poll is someone could be quickly toggling the relay. We wouldnt want to close it on accident. The polling and logic does result in a possible delay or early trigger for closing the relay, but for the BoxingBots the difference of 10ms for a relay closing is acceptable: <pre class="brush: csharp
public void Open(int index, int autoCloseDelay)
{
UseRelay(index, true, autoCloseDelay);
}
public void Close(int index)
{
UseRelay(index, false, 0);
}
private void UseRelay(int index, bool openRelay, int autoCloseDelay)
{
AlterTimeDelay(index, autoCloseDelay);
PhidgetInterfaceKit.outputs[index] = openRelay;
}
void _relayController_OutputChange(object sender, OutputChangeEventArgs e)
{
// closed
if (!e.Value)
return;
ThreadPool.QueueUserWorkItem(state =>
{
if (_timeDelays.ContainsKey(e.Index))
{
while (_timeDelays[e.Index] > 0)
{
Thread.Sleep(ThreadTick);
_timeDelays[e.Index] -= ThreadTick;
}
}
Close(e.Index);
});
}
public int GetTimeDelay(int index)
{
if (!_timeDelays.ContainsKey(index))
return 0;
return _timeDelays[index];
}
public void AlterTimeDelay(int index, int autoCloseDelay)
{
_timeDelays[index] = autoCloseDelay;
}
[/code] <h3> User Console </h3> http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0297%5B5%5D.jpg <img title="IMG_0297" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0297_thumb%5B2%5D.jpg" alt="IMG_0297" width="500" height="375" border="0 Since the theme of the party was Retro Gaming, we wanted to go for an early 80s Sci-fi style interface, complete with starscape background and solar flares! We wanted to create actual interactive elements, though, to maintain the green phosphor look of early monochrome monitors. Unlike traditional video games, however, the screens are designed not as the primary focus of attention, but rather to help calibrate the player before the round and provide secondary display data during the match. The player should primarily stay focused on the boxer during the match, so the interface is designed to sit under the players view line and serve as more of a dashboard during each match. However, during calibration before each round, it is important to have the player understand how their core body will be used to drive the Robot base during each round. To do this, we needed to track an average of the joints that make up each fighters body core. We handled the process by creating a list of core joints and a variable that normalizes the metric distances returned from the Kinect sensor into a human-acceptable range of motion: <pre class="brush: csharp
private static List<JointType> coreJoints = newList<JointType>(
newJointType[] {
JointType.AnkleLeft,
JointType.AnkleRight,
JointType.ShoulderCenter,
JointType.HipCenter
});
private const double RangeNormalizer = .22;
private const double NoiseClip = .05;
[/code] And then during each skeleton calculation called by the game loop, we average the core positions to determine the averages of the players as they relate to their playable ring boundary: <pre class="brush: csharp
public staticMovementDescriptorPacket AnalyzeSkeleton(Skeleton skeleton)
{
// ...
CoreAverageDelta.X = 0.0;
CoreAverageDelta.Z = 0.0;
foreach (JointType jt in CoreJoints)
{
CoreAverageDelta.X += skeleton.Joints[jt].Position.X - RingCenter.X;
CoreAverageDelta.Z += skeleton.Joints[jt].Position.Z - RingCenter.Z;
}
CoreAverageDelta.X /= CoreJoints.Count * RangeNormalizer;
CoreAverageDelta.Z /= CoreJoints.Count * RangeNormalizer;
// ...
if (CoreAverageDelta.Z > NoiseClip || CoreAverageDelta.Z < -NoiseClip)
{
packet.Move = -CoreAverageDelta.Z;
}
if (CoreAverageDelta.X > NoiseClip || CoreAverageDelta.X < -NoiseClip)
{
packet.Strafe = CoreAverageDelta.X;
}
}
[/code] In this way, we filter out insignificant data noise and allow the players average core body to serve as a joystick for driving the robot around. Allowing them to lean at any angle, the move and strafe values are accordingly set to allow for a full 360 degrees of movement freedom, while at the same time not allowing any one joint to unevenly influence their direction of motion. Another snippet of code that may be of interest is the WPF3D rendering we used to visualize the skeleton. Since the Kinect returns joint data based off of a center point, it is relatively easy to wire up a working 3D model in WPF3D off of the skeleton data, and we do this in the ringAvatar.xaml control. In the XAML, we simply need a basic Viewport3D with camera, lights, and an empty ModelVisual3D container to hold or squares. The empty container looks like this: <pre class="brush: xml
<ModelVisual3D x:Name="viewportModelsContainer2
<ModelVisual3D.Transform>
<Transform3DGroup>
<RotateTransform3D x:Name="bodyRotationCenter" CenterX="0" CenterY="0" CenterZ="0
<RotateTransform3D.Rotation>
<AxisAngleRotation3D x:Name="myAngleRotation" Axis="0,1,0" Angle="-40"/>
</RotateTransform3D.Rotation>
</RotateTransform3D>
</Transform3DGroup>
</ModelVisual3D.Transform>
</ModelVisual3D>
[/code] In the code, we created a generic WPF3DModel that inherits from UIElement3D and is used to store the basic positioning properties of each square. In the constructor of the object, though, we can pass a reference key to a XAML file that defines the 3D mesh to use: <pre class="brush: csharp
public WPF3DModel(string resourceKey)
{
this.Visual3DModel = Application.Current.Resources[resourceKey] as Model3DGroup;
}
[/code] This is a handy trick when you need to do a fast WPF3D demo and require a certain level of flexibility. To create a 3D cube for each joint when ringAvatar is initialized, we simply do this: <pre class="brush: csharp
private readonly List<WPF3DModel> _models = new List<WPF3DModel>();
private void CreateViewportModels()
{
for (int i = 0; i < 20; i++)
{
WPF3DModel model = new WPF3DModel("mesh_cube");
viewportModelsContainer2.Children.Add(model);
// ...
_models.Add(model);
}
// ...
}
[/code] And then each time we need to redraw the skeleton, we loop through the skeleton data and set the cube position like so: <pre class="brush: csharp
if (SkeletonProcessor.RawSkeleton.TrackingState == SkeletonTrackingState.Tracked)
{
int i = 0;
foreach (Joint joint in SkeletonProcessor.RawSkeleton.Joints)
{
if (joint.TrackingState == JointTrackingState.Tracked)
{
_models.Translate(
joint.Position.X * 8.0,
joint.Position.Y * 10.0,
joint.Position.Z * -10.0);
i++;
}
}
// ...
}
[/code] There are a few other areas in the User Console that you may want to further dig into, including the weighting for handling a punch as well dynamically generating arcs based on the position of the fist to the shoulder. However, for this experience, the User Console serves as a secondary display to support the playing experience and gives both the player and audience a visual anchor for the game. <h3>Making a 700lb Tank Drive like a First Person Shooter</h3> The character in a first person shooter (FPS) video game has an X position, a Y position, and a rotation vector. On an Xbox controller, the left stick controls the X,Y position. Y is the throttle (forward and backward), X is the strafing amount (left and right), and the right thumb stick moves the camera to change what youre looking at (rotation). When all three are combined, the character can do things such as run around someone while looking at them. In the prior project, we had existing code that worked for controlling all 4 motors at the same time, working much like a tank does, so we only had throttle (forward and back) and strafing (left and right). Accordingly, we can move the motors in all directions, but there are still scenarios in which the wheels fight one another and the base wont move. By moving to a FPS style, we eliminate the ability to move the wheels in an non-productive way and actually make it a lot easier to drive. Note that Clint had some wiring "quirks" with polarity and which motor was left VS right, he had to correct in these quirks in software <img src=http://ecn.channel9.msdn.com/o9/content/images/emoticons/emotion-1.gif?v=c9 alt=Smiley />: <pre class="brush: csharp
public Speed CalculateSpeed(double throttleVector, double strafeVector, double rotationAngle)
{
rotationAngle = VerifyLegalValues(rotationAngle);
rotationAngle = AdjustValueForDeadzone(rotationAngle, AllowedRotationAngle, _negatedAllowedRotationAngle);
// flipped wiring, easy fix is here
throttleVector *= -1;
rotationAngle *= -1;
// miss wired, had to flip throttle and straff for calc
return CalculateSpeed(strafeVector + rotationAngle, throttleVector, strafeVector - rotationAngle, throttleVector);
}
protected Speed CalculateSpeed(double leftSideThrottle, double leftSideVectorMultiplier, double rightSideThrottle, double rightSideVectorMultiplier)
{ /* code from Jellybean */ }
[/code] Conclusion The Boxing Bots project was one of the biggest things we have built to date. It was also one of our most successful projects. Though it was a rainy, cold day and night in Austin when the bots were revealed, and we had to move locations several times during setup to ensure the bots and computers wouldnt be fried by the rain, they ran flawlessly for the entire event and contestants seemed to have a lot of fun driving them. http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0302%5B5%5D.jpg <img title="IMG_0302" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0302_thumb%5B2%5D.jpg" alt="IMG_0302" width="500" height="375" border="0 <img src="http://m.webtrends.com/dcs1wotjh10000w0irc493s0e_6x1g/njs.gif?dcssip=channel9.msdn.com&dcsuri=http://channel9.msdn.com/Feeds/RSS&WT.dl=0&WT.entryid=Entry:RSSView:247dcab61560403d8e5da04e01521034
View the full article
Experimenting with possible layouts The upper body had its own challenges, as it had to support a ton of equipment, including: Punching arms Popping head Pneumatic valves Air manifold Air Tank(s) Laptop Phidget interface board Phidget relay boards Phidget LED board Xbox wireless controller PC transmitter / receiver Chest plate LEDs Sensors to detect a punch http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0113%5B5%5D.jpg <img title="IMG_0113" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0113_thumb%5B2%5D.jpg" alt="IMG_0113" width="500" height="667" border="0
Brian and Rick put together one of the upper frames <h3>Punching and Air Tanks</h3> We had to solve the problem of getting each robot to punch hard enough to register a hit on the opponent bot while not breaking the opponent bot (or itself). Bots also had to withstand a bit of side load in case the arms got tangled or took a side blow. Pneumatic actuators provided us with a lot of flexibility over hydraulics or an electrical solution since they are fast, come in tons of variations, wont break when met with resistance, and can fine tuned with a few onsite adjustments. To provide power to the actuators, the robots had two 2.5 gallon tanks pressurized to 150psi, with the actuators punching at ~70psi. We could punch for about five 90-second rounds before needing to re-pressurize the tanks. Pressurizing the onboard tanks was taken care of by a pair of off-the-shelf DeWalt air compressors. http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0134%5B4%5D.jpg <img title="IMG_0134" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0134_thumb%5B1%5D.jpg" alt="IMG_0134" width="240" height="180" border="0 http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0184%5B4%5D.jpg <img title="IMG_0184" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0184_thumb%5B1%5D.jpg" alt="IMG_0184" width="240" height="180" border="0 http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0279%5B4%5D.jpg <img title="IMG_0279" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0279_thumb%5B1%5D.jpg" alt="IMG_0279" width="240" height="180" border="0 http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0172%5B4%5D.jpg <img title="IMG_0172" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0172_thumb%5B1%5D.jpg" alt="IMG_0172" width="240" height="180" border="0 <h3>The Head</h3> It wouldn’t be a polished game if the head didn’t pop up on the losing bot, so we added another pneumatic actuator to raise and lower the head, and some extra red and blue LEDs. This pneumatic is housed in the chest of the robot and is triggered only when the game has ended. To create the head, we first prototyped a concept with cardboard and duct tape. A rotated welding mask just happened to provide the shape we were going for on the crown, and we crafted each custom jaw with a laser cutter. We considered using a mold and vacuum forming to create something a bit more custom, but had to scrap the idea due to time constraints. http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0169%5B4%5D.jpg <img title="IMG_0169" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0169_thumb%5B1%5D.jpg" alt="IMG_0169" width="240" height="180" border="0 http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0201%5B4%5D.jpg <img title="IMG_0201" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0201_thumb%5B1%5D.jpg" alt="IMG_0201" width="240" height="180" border="0 http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0268%5B4%5D.jpg <img title="IMG_0268" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0268_thumb%5B1%5D.jpg" alt="IMG_0268" width="240" height="180" border="0 http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0291%5B4%5D.jpg <img title="IMG_0291" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0291_thumb%5B1%5D.jpg" alt="IMG_0291" width="240" height="180" border="0 <h3>Sensors</h3> Our initial implementation for detecting punches failed due to far too many false positives. We thought using IR distance sensors would be a good solution since we could detect a “close” punch and tell the other robot to retract the arm before real contact. The test looked promising, but in practice, when the opposite sensors were close, we saw a lot of noise in the data. The backup and currently implemented solution was to install simple push switches in the chest and detect when those are clicked by the chest plate pressing against them. http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0200%5B4%5D.jpg <img title="IMG_0200" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0200_thumb%5B1%5D.jpg" alt="IMG_0200" width="240" height="180" border="0 http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0205%5B4%5D.jpg <img title="IMG_0205" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0205_thumb%5B1%5D.jpg" alt="IMG_0205" width="240" height="180" border="0 <h3>Power</h3> Different items required different voltages. The motors and pneumatic valves required 24V, the LEDs required 12V and the USB hub required 5V. We used Castle Pro BEC converters to step down the voltages. These devices are typically used in RC airplanes and helicopters. <h3>Shipping</h3> http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0278%5B4%5D.jpg <img title="IMG_0278" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0278_thumb%5B1%5D.jpg" alt="IMG_0278" width="240" height="180" border="0 http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0280%5B4%5D.jpg <img title="IMG_0280" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0280_thumb%5B1%5D.jpg" alt="IMG_0280" width="240" height="180" border="0 So how does someone ship two 700lb robots from Seattle to Austin? We did it in 8 crates. <img src=http://ecn.channel9.msdn.com/o9/content/images/emoticons/emotion-1.gif?v=c9 alt=Smiley />. The key thing to note is that the tops and bottoms of each robot were separated. Any wire that connected the two parts had to be able to be disconnected in some form. This affected the serial cords and the power cords (5V, 12V, 24V). Software The software and architecture went through a variety of iterations during development. The final architecture used 3 laptops, 2 desktops, an access point, and a router. Its important to note that the laptops of Robot 1 and Robot 2 are physically mounted on the backs of each Robot body, communicating through WiFi to the Admin console. The entire setup looks like the following diagram: http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/network%5B4%5D.png <img title="network" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/network_thumb%5B2%5D.png" alt="network" width="387" height="426" border="0 <h3> Admin Console </h3> The heart of the infrastructure is the Admin Console. Originally, this was also intended to be a scoreboard to show audience members the current stats of the match, but as we got further into the project, we realized this wouldnt be necessary. The robots are where the action is, and peoples eyes focus there. Additionally, the robots themselves display their current health status via LEDs, so duplicating this information isnt useful. However, the admin side of this app remains. <h3> Sockets </h3> The admin console is the master controller for the game state and utilizes socket communication between it, the robots, and the user consoles. A generic socket handler was written to span each computer in the setup. The SocketListener object allows for incoming connections to be received, while the SocketClient allows clients to connect to those SocketListeners . These are generic objects, which must specify objects of type GamePacket to send and receive: <pre class="brush: csharp
public class SocketListener<TSend, TReceive> where TSend : GamePacket
where TReceive : GamePacket, new()
[/code]
GamePacket is a base class from which specific packets inherit: <pre class="brush: csharp
public abstract class GamePacket
{
public byte[] ToByteArray()
{
MemoryStream ms = new MemoryStream();
BinaryWriter bw = new BinaryWriter(ms);
try
{
WritePacket(bw);
}
catch(IOException ex)
{
Debug.WriteLine("Error writing packet: " + ex);
}
return ms.ToArray();
}
public void FromBinaryReader(BinaryReader br)
{
try
{
ReadPacket(br);
}
catch(IOException ex)
{
Debug.WriteLine("Error reading packet: " + ex);
}
}
public abstract void WritePacket(BinaryWriter bw);
public abstract void ReadPacket(BinaryReader br);
}
[/code] For example, in communication between the robots and the admin console, GameStatePacket and MovementDescriptorPacket are sent and received. Each GamePacket must implement its own ReadPacket and WritePacket methods to serialize itself for sending across the socket. Packets are sent between machines every "frame". We need the absolute latest game state, robot movement, etc. at all times to ensure the game is functional and responsive. http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/image%5B20%5D-2.png <img title="image" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/image_thumb%5B12%5D-3.png" alt="image" width="640" height="278" border="0 As is quite obvious, absolutely no effort was put into making the console "pretty". This is never seen by the end users and just needs to be functional. Once the robot software and the user consoles are started, the admin console initiates connections to each of those four machines. Each machine runs the SocketListener side of the socket code, while the Admin console creates four SocketClient objects to connect to each those. Once connected, the admin has control of the game and can start, stop, pause, and reset a match by sending the appropriate packets to everyone that is connected. <h3> Robot </h3> The robot UI is also never intended to be seen by an end user, and therefore contains only diagnostic information. http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/image%5B19%5D-1.png <img title="image" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/image_thumb%5B11%5D-1.png" alt="image" width="640" height="361" border="0 Each robot has a wireless Xbox 360 controller connected to it so it can be manually controlled. The UI above reflects the positions of the controller sticks and buttons. During a match, its possible for a bot to get outside of our "safe zone". One bot might be pushing the other, or the user may be moving the bot toward the edge of the ring. To counter this, the players coach can either temporarily move the bot, turning off Kinect input, or force the game into "referee mode" which pauses the entire match and turns off Kinect control on both sides. In either case, the robots can be driven with the controllers and reset to safe positions. Once both coaches signal that the robots are reset, the admin can then resume the match. <h3>Controlling Hardware</h3> Phidget hardware controlled our LEDs, relays, and sensors. Getting data out of a Phidget along with actions, such as opening and closing a relay, is shockingly easy as they have pretty straightforward C# APIs and samples, which is why they typically are our go-to product for projects like this. Below are some code snippets for the LEDs, relays, and sensor. LEDs – from LedController.cs This is the code that actually updates the health LEDs in the robots chest. The LEDs were put on the board in a certain order to allow this style of iteration. We had a small issue of running out of one color of LEDs so we used some super bright ones and had to reduce the power levels to the non-super bright LEDs to prevent possible damage: <pre class="brush: csharp
private void UpdateLedsNonSuperBright(int amount, int offset, int brightness)
{
for (var i = offset; i < amount + offset; i++)
{
_phidgetLed.leds = brightness / 2;
}
}
private void UpdateLedsSuperBright(int amount, int offset, int brightness)
{
for (var i = offset; i < amount + offset; i++)
{
_phidgetLed.leds = brightness;
}
}
[/code] Sensor data – from SensorController.cs This code snippet shows how we obtain the digital and analog inputs from the Phidget 8/8/8 interface board: <pre class="brush: csharp
public SensorController(InterfaceKit phidgetInterfaceKit) : base(phidgetInterfaceKit)
{
PhidgetInterfaceKit.ratiometric = true;
}
public int PollAnalogInput(int index)
{
return PhidgetInterfaceKit.sensors[index].Value;
}
public bool PollDigitalInput(int index)
{
return PhidgetInterfaceKit.inputs[index];
}
[/code] Relays – from RelayController.cs Electrical relays fire our pneumatic valves. These control the head popping and the arms punching. For our application, we wanted the ability to reset the relay automatically. When the relay is opened, an event is triggered and we create an actively polled thread to validate whether we should close the relay. The reason why we actively poll is someone could be quickly toggling the relay. We wouldnt want to close it on accident. The polling and logic does result in a possible delay or early trigger for closing the relay, but for the BoxingBots the difference of 10ms for a relay closing is acceptable: <pre class="brush: csharp
public void Open(int index, int autoCloseDelay)
{
UseRelay(index, true, autoCloseDelay);
}
public void Close(int index)
{
UseRelay(index, false, 0);
}
private void UseRelay(int index, bool openRelay, int autoCloseDelay)
{
AlterTimeDelay(index, autoCloseDelay);
PhidgetInterfaceKit.outputs[index] = openRelay;
}
void _relayController_OutputChange(object sender, OutputChangeEventArgs e)
{
// closed
if (!e.Value)
return;
ThreadPool.QueueUserWorkItem(state =>
{
if (_timeDelays.ContainsKey(e.Index))
{
while (_timeDelays[e.Index] > 0)
{
Thread.Sleep(ThreadTick);
_timeDelays[e.Index] -= ThreadTick;
}
}
Close(e.Index);
});
}
public int GetTimeDelay(int index)
{
if (!_timeDelays.ContainsKey(index))
return 0;
return _timeDelays[index];
}
public void AlterTimeDelay(int index, int autoCloseDelay)
{
_timeDelays[index] = autoCloseDelay;
}
[/code] <h3> User Console </h3> http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0297%5B5%5D.jpg <img title="IMG_0297" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0297_thumb%5B2%5D.jpg" alt="IMG_0297" width="500" height="375" border="0 Since the theme of the party was Retro Gaming, we wanted to go for an early 80s Sci-fi style interface, complete with starscape background and solar flares! We wanted to create actual interactive elements, though, to maintain the green phosphor look of early monochrome monitors. Unlike traditional video games, however, the screens are designed not as the primary focus of attention, but rather to help calibrate the player before the round and provide secondary display data during the match. The player should primarily stay focused on the boxer during the match, so the interface is designed to sit under the players view line and serve as more of a dashboard during each match. However, during calibration before each round, it is important to have the player understand how their core body will be used to drive the Robot base during each round. To do this, we needed to track an average of the joints that make up each fighters body core. We handled the process by creating a list of core joints and a variable that normalizes the metric distances returned from the Kinect sensor into a human-acceptable range of motion: <pre class="brush: csharp
private static List<JointType> coreJoints = newList<JointType>(
newJointType[] {
JointType.AnkleLeft,
JointType.AnkleRight,
JointType.ShoulderCenter,
JointType.HipCenter
});
private const double RangeNormalizer = .22;
private const double NoiseClip = .05;
[/code] And then during each skeleton calculation called by the game loop, we average the core positions to determine the averages of the players as they relate to their playable ring boundary: <pre class="brush: csharp
public staticMovementDescriptorPacket AnalyzeSkeleton(Skeleton skeleton)
{
// ...
CoreAverageDelta.X = 0.0;
CoreAverageDelta.Z = 0.0;
foreach (JointType jt in CoreJoints)
{
CoreAverageDelta.X += skeleton.Joints[jt].Position.X - RingCenter.X;
CoreAverageDelta.Z += skeleton.Joints[jt].Position.Z - RingCenter.Z;
}
CoreAverageDelta.X /= CoreJoints.Count * RangeNormalizer;
CoreAverageDelta.Z /= CoreJoints.Count * RangeNormalizer;
// ...
if (CoreAverageDelta.Z > NoiseClip || CoreAverageDelta.Z < -NoiseClip)
{
packet.Move = -CoreAverageDelta.Z;
}
if (CoreAverageDelta.X > NoiseClip || CoreAverageDelta.X < -NoiseClip)
{
packet.Strafe = CoreAverageDelta.X;
}
}
[/code] In this way, we filter out insignificant data noise and allow the players average core body to serve as a joystick for driving the robot around. Allowing them to lean at any angle, the move and strafe values are accordingly set to allow for a full 360 degrees of movement freedom, while at the same time not allowing any one joint to unevenly influence their direction of motion. Another snippet of code that may be of interest is the WPF3D rendering we used to visualize the skeleton. Since the Kinect returns joint data based off of a center point, it is relatively easy to wire up a working 3D model in WPF3D off of the skeleton data, and we do this in the ringAvatar.xaml control. In the XAML, we simply need a basic Viewport3D with camera, lights, and an empty ModelVisual3D container to hold or squares. The empty container looks like this: <pre class="brush: xml
<ModelVisual3D x:Name="viewportModelsContainer2
<ModelVisual3D.Transform>
<Transform3DGroup>
<RotateTransform3D x:Name="bodyRotationCenter" CenterX="0" CenterY="0" CenterZ="0
<RotateTransform3D.Rotation>
<AxisAngleRotation3D x:Name="myAngleRotation" Axis="0,1,0" Angle="-40"/>
</RotateTransform3D.Rotation>
</RotateTransform3D>
</Transform3DGroup>
</ModelVisual3D.Transform>
</ModelVisual3D>
[/code] In the code, we created a generic WPF3DModel that inherits from UIElement3D and is used to store the basic positioning properties of each square. In the constructor of the object, though, we can pass a reference key to a XAML file that defines the 3D mesh to use: <pre class="brush: csharp
public WPF3DModel(string resourceKey)
{
this.Visual3DModel = Application.Current.Resources[resourceKey] as Model3DGroup;
}
[/code] This is a handy trick when you need to do a fast WPF3D demo and require a certain level of flexibility. To create a 3D cube for each joint when ringAvatar is initialized, we simply do this: <pre class="brush: csharp
private readonly List<WPF3DModel> _models = new List<WPF3DModel>();
private void CreateViewportModels()
{
for (int i = 0; i < 20; i++)
{
WPF3DModel model = new WPF3DModel("mesh_cube");
viewportModelsContainer2.Children.Add(model);
// ...
_models.Add(model);
}
// ...
}
[/code] And then each time we need to redraw the skeleton, we loop through the skeleton data and set the cube position like so: <pre class="brush: csharp
if (SkeletonProcessor.RawSkeleton.TrackingState == SkeletonTrackingState.Tracked)
{
int i = 0;
foreach (Joint joint in SkeletonProcessor.RawSkeleton.Joints)
{
if (joint.TrackingState == JointTrackingState.Tracked)
{
_models.Translate(
joint.Position.X * 8.0,
joint.Position.Y * 10.0,
joint.Position.Z * -10.0);
i++;
}
}
// ...
}
[/code] There are a few other areas in the User Console that you may want to further dig into, including the weighting for handling a punch as well dynamically generating arcs based on the position of the fist to the shoulder. However, for this experience, the User Console serves as a secondary display to support the playing experience and gives both the player and audience a visual anchor for the game. <h3>Making a 700lb Tank Drive like a First Person Shooter</h3> The character in a first person shooter (FPS) video game has an X position, a Y position, and a rotation vector. On an Xbox controller, the left stick controls the X,Y position. Y is the throttle (forward and backward), X is the strafing amount (left and right), and the right thumb stick moves the camera to change what youre looking at (rotation). When all three are combined, the character can do things such as run around someone while looking at them. In the prior project, we had existing code that worked for controlling all 4 motors at the same time, working much like a tank does, so we only had throttle (forward and back) and strafing (left and right). Accordingly, we can move the motors in all directions, but there are still scenarios in which the wheels fight one another and the base wont move. By moving to a FPS style, we eliminate the ability to move the wheels in an non-productive way and actually make it a lot easier to drive. Note that Clint had some wiring "quirks" with polarity and which motor was left VS right, he had to correct in these quirks in software <img src=http://ecn.channel9.msdn.com/o9/content/images/emoticons/emotion-1.gif?v=c9 alt=Smiley />: <pre class="brush: csharp
public Speed CalculateSpeed(double throttleVector, double strafeVector, double rotationAngle)
{
rotationAngle = VerifyLegalValues(rotationAngle);
rotationAngle = AdjustValueForDeadzone(rotationAngle, AllowedRotationAngle, _negatedAllowedRotationAngle);
// flipped wiring, easy fix is here
throttleVector *= -1;
rotationAngle *= -1;
// miss wired, had to flip throttle and straff for calc
return CalculateSpeed(strafeVector + rotationAngle, throttleVector, strafeVector - rotationAngle, throttleVector);
}
protected Speed CalculateSpeed(double leftSideThrottle, double leftSideVectorMultiplier, double rightSideThrottle, double rightSideVectorMultiplier)
{ /* code from Jellybean */ }
[/code] Conclusion The Boxing Bots project was one of the biggest things we have built to date. It was also one of our most successful projects. Though it was a rainy, cold day and night in Austin when the bots were revealed, and we had to move locations several times during setup to ensure the bots and computers wouldnt be fried by the rain, they ran flawlessly for the entire event and contestants seemed to have a lot of fun driving them. http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0302%5B5%5D.jpg <img title="IMG_0302" src="http://files.channel9.msdn.com/wlwimages/1932b237046e4743a4e79e6800c0220f/IMG_0302_thumb%5B2%5D.jpg" alt="IMG_0302" width="500" height="375" border="0 <img src="http://m.webtrends.com/dcs1wotjh10000w0irc493s0e_6x1g/njs.gif?dcssip=channel9.msdn.com&dcsuri=http://channel9.msdn.com/Feeds/RSS&WT.dl=0&WT.entryid=Entry:RSSView:247dcab61560403d8e5da04e01521034
View the full article