While I am not the first person to create a LEGO Robot that interacts with smart phones, I had fun building this Android Player. The most interesting work was done on the PC, I figured out how to grab the Android Screen using the Android SDK:

AndroidDebugBridge.init(true /* debugger support */);
AndroidDebugBridge.createBridge();
RawImage rawImage = AndroidDebugBridge.getBridge().getDevices()[0].getScreenshot();
rawImage = rawImage.getRotated();
BufferedImage image = new BufferedImage(
    rawImage.width, rawImage.height, BufferedImage.TYPE_INT_ARGB);
int index = 0;
int bytesPerPixel = rawImage.bpp >> 3;
for (int y = 0; y < rawImage.height; y++) {
    for (int x = 0; x < rawImage.width; x++) {
        image.setRGB(x, y, rawImage.getARGB(index) | 0xff000000);
        index += bytesPerPixel;
    }
}
ImageIO.write(bi, "png", outputfile);

Note that you already have to have the ADB server running for this code to work correctly.

From there I wrote some image processing code to interpret the various screens in the game and numbers on screen, this allowed me to build a smart macro that plays a clash of clans clone.

Check out the images and video to get a better idea of the build. I used LeJOS to remotely drive my EV3 from the PC, which worked great along with the Android code as a single app in my IDE.





The robot can support a 7 inch tablet. I had some difficulty with LeJOS driving the Z Pen motor, I ended up adding some gearing to reduce the stress on the motor. The touch sensor can detect when the Z is at its highest point.



I used medium motors simply due to the noise level. I run the robot over night while sleeping, and the larger motors were too loud.



I used the rubber axle connector to secure the device in place, I also made the top part of the robot detachable so I can quickly access the device.



At first I had this working on my Windows Laptop, but either due to the Android SDK or the LeJOS interface I was getting blue screens. I've switched over to my Mac and haven't had any system crashes. The ADB server occasionally crashes and requires restarting.

This has been a fun project, and I've earned 50M+ in gold over the past few days. I only wish I could transfer the gold to USD!