Arduino Bot Shop Control Harmony Remote with Amazon Echo

Controlling Harmony Remote with Amazon Echo

Back in the late 80’s I taught electronics technologies at a local community college in Portland, Oregon. Some of the classes I taught were centered around the Heathkit HERO robot series, and included building, programming and extending its capabilities. In those days, generating speech by computer was done by programming phonemes, such as AH, OOH, EM, and the like. You’d string a bunch of phonemes together, and by adding pauses (and lots of tweaking), you could approximate human speech. In those days I dreamed of being able to use electronics and programming recognize speech, and in so doing, program my robots by speaking to them and having them talk back. It simply wasn’t possible with the technologies available at the time. The best I could accomplish was teaching my students how to have their robots render a toneless and artificial rendition of “Mary had a little lamb” for laughs.

My how times have changed.

Today’s smart phones and mobile devices all have some kind of built in speech recognition capabilities, from Apple’s SIRI to Microsoft’s’ CORTANA to Android’s IRIS to OK GOOGLE, and thus speech recognition has become commonplace and (nearly) flawless.

Now enter ALEXA, Amazon’s entry into the speech game.

What is ALEXA?

Amazon EchoPrimarily used as an internet device, Amazon quite cleverly added some Home Automation capabilities into Alexa, then by releasing their SDK, the world suddenly opened up to my dreams of yesteryear.

Have WeMo or Phillips Hue devices in your home? Simply plug Alexa in and tell her to discover devices (“Alexa, discover my devices”), and within 20 seconds you are commanding Alexa to turn on and off lights, appliances, and whatever else you have in your connected home “Alexa, turn on kitchen lights”, and “Alexa, turn on living room fan” are all within 20 seconds of your initiating the device discovery. (Assuming of course, that you have Wemo or Phillips Hue installed and working already). Expanding Alexa is pretty easy, too! With the SDK, Alexa can be used to control devices you may not have though possible, such as “Alexa, turn on the TV” or “Alexa, open the garage door”, or even “Alexa, start some coffee”. Now it’s possible to have your home controlled entirely by speech, from literally anywhere Alexa can hear you.

I’m going to devote this blog to showing you how to have Alexa control your entertainment center via Harmony Hub. I’ll show you how to create the services, the software, identify the activities, and install everything so that you can say “Alexa, turn on the TV” and have her start your television, turn on the surround sound system, enable your cable box, and select a start channel. Commanding Alexa to turn it off will shut down each of your entertainment center devices in turn, too. Later, we’ll explore alternatives, such as “Alexa, turn on XBOX” and “Alexa, turn on NetFlix (or Stereo, or Playstation, or any of the devices in your entertainment center)”.

It all starts with Alexa. So lets get started, shall we?

No Native Support for Remote Controls, but we’ve got you covered.

Arduino Bot Shop Amazon Echo with Harmony ControlAt the time of this writing, Alexa didn’t have any onboard support for remote controlling things via Infrared or bluetooth. Doing so requires the addition of an out-board device that can issue the appropriate IR or bluetooth commands to turn on/off your devices, set volumes, etc.

Fortunately, there already exists such a device: the Logitech Harmony Hub. This Logitech device is designed to control your devices from your mobile device. Simply using your cell phone as a control surface the Harmony Hub accepts commands from your phone, and translates them into controls for your tv and other devices. Unfortunately, the Harmony Hub is a closed system (Logitec wants to protect it’s investment, after all) and we’d need to intercept the communications between your phone and the hub to determine what’s going on, and then to write some code to emulate that so that we can talk to the Hub ourselves. Fortunately, this has already been done, by way of pyHarmony.py: A Python script that allows you to talk to your harmony Hub. We’ll dig into this more later, but, unfortunately, pyHarmony only solves part of the problem. While it can talk to your Harmony Hub, it can’t talk to Alexa (or rather, Alexa cant talk to pyHarmony), so we need to find a way to do that, too.

This means that we also need to listen in on the conversations Alexa is having with the devices it CAN talk to so we can determine what’s going on there and emulate that too. Enter another piece of software, fauxmos.py, another python script. This script looks like a WeMo device and Alexa is quite happy to talk to it.

So now we have Alexa speaking to fauxmos.py as a Wemo device (supporting Turn ON and Turn OFF commands), and we have pyHarmony.py talking to the Harmony, (Also Supporting Turn ON and Turn OFF commands), but we need one more small piece.

We need something that FauxMo can talk to, which in turn can talk to pyHarmony which in turn can talk to your Harmony Hub. That is a small web page, which I call ha.php.

So our complete system looks like this:

Alexa->FauxMo.py->ha.php->pyHarmony->Harmony Hub

These are all the pieces we need for controlling our entertainment center. Oh… And one more thing. You need a Linux server on which to run all of these bits and pieces. The server can be a full on dedicated machine, a Raspberry PI, or a virtual box on your windows desktop. The choice is yours. Setting up a Linux box on your own is beyond the scope of this article, but fortunately there are many, many resources online you can use to help you no matter which server type you choose. My particular installation is on a Mac Mini running Ubuntu 14.04 LTS. Be sure to install Python, and the full LAMP Stack. And force a static IP, you’ll be glad you did. See Google for full details.

So go ahead – go build your Linux server now before proceeding. I’ll wait……

Welcome back! You didn’t think I’d wait, did you? OK – so you have your Linux server running, huh? Have you updated it to the latest versions of software? Can you connect to the webserver? Have you verified that PHP is installed? Is python working? Know your static IP?

OK then, now that your server is running, web pages are being served, and python is installed and running, lets start with first things first.

Installing and testing fauxmo.py

Arduino Bot Shop harmony remote with amazon echo

We’ll start with installing the WeMo emulator. You can find the file here: simply copy this file into your home directory on the Linux server. Typically, this will be /home/yourloginname/

After downloading the file, simply place it on your server. Use any method you like. I used SCP, like this:

c:\ SCP yourloginname@youserverIP:path/to/fauxmo.py/home/yourloginname/

Once done, you can run it as is on the server with the command:

$> cd /home/yourloginname/
$> python fauxmo.py –d

Before you do, though, we’ll need to take a look in the file at lines 352 – 368

class rest_api_handler(object):
    def __init__(self, on_cmd, off_cmd):
        self.on_cmd = on_cmd
        self.off_cmd = off_cmd

    def on(self):
        r = requests.get(self.on_cmd)
        return r.status_code == 200

    def off(self):
        r = requests.get(self.off_cmd)
        return r.status_code == 200

FAUXMOS = [
    ['office lights', rest_api_handler('http://192.168.5.4/ha-api?cmd=on&a=office', 'http://192.168.5.4/ha-api?cmd=off&a=office')],
    ['kitchen lights', rest_api_handler('http://192.168.5.4/ha-api?cmd=on&a=kitchen', 'http://192.168.5.4/ha-api?cmd=off&a=kitchen')],
]

The section “FAUXMOS=” describe the “Fake” WeMo switches you’ll want to create.
There’s one for each switch, and each switch contains two urls, one for turning the “Switch” ON and one for OFF. You can have quite a few of these, simply by adding lines for each “Switch” you want to define. Lets look at one of them:

</pre>

['kitchen lights', rest_api_handler('http://192.168.5.4/haapi?cmd=on&a=kitchen', 'http://192.168.5.4/haapi?cmd=off&a=kitchen')],]

This basically has a couple of things in it to keep in mind…. The “Fake Switch” is called “Kitchen Lights”, and it will go to the webpage at ‘http://192.168.5.4/haapi?cmd=on&a=kitchen to turn it on, and it will go to the webpage ‘http://192.168.5.4/haapi?cmd=off&a=kitchen’ to turn it off.

So you’ll need to modify these lines to reflect your servers IP, and to reflect the web page we’ll create in the next steps. For now, lets, change the name and urls so we can actually test…

Replace that whole FAUXMOS Section with this one, so we have something to test with.
Replace:

FAUXMOS = [
    ['office lights', rest_api_handler('http://192.168.5.4/ha-api?cmd=on&a=office', 'http://192.168.5.4/ha-api?cmd=off&a=office')],
    ['kitchen lights', rest_api_handler('http://192.168.5.4/ha-api?cmd=on&a=kitchen', 'http://192.168.5.4/ha-api?cmd=off&a=kitchen')],
]

With this:

FAUXMOS = [
    [‘T.V.’, rest_api_handler(‘http://www.google.com?cmd=on&a=office', 'http://www.google.com?cmd=off&a=office')],
]

We’re creating a single “Fake” switch, called T.V., and sending it to Google to turn it ON and OFF. Basically – it won’t actually DO anything setup like this, but we can test that Alexa sees and can talk to what will be the “T.V” control.

Save the file, and run it with this command:

$> python fauxmo.py –d

 this will start the WeMo Emulator. You should see some output similar to this:

Listening for UPnP broadcasts
got local address of YOUR.SERVER.IP.ADDRESS
UPnP broadcast listener: new device registered
FauxMo device ‘TV ‘ ready on :49297
Entering main loop

If you get that, you are ready to connect your Alexa to the Wemo Device. Simply command Alexa to find the “Fake” WeMo with “Alexa, discover devices” – she should report that she found a new device. If you browse to your echo.amazon.com page, you’ll find “TV” under the devices Alexa knows about. Again – she can’t actually DO anything with the TV yet, but she can see it and talk to it. Prove it telling Alexa to turn on the TV. “Alexa, turn on the T.V.” She should respond with “OK”. If she complains that the device isn’t responding at this time, don’t worry about it. It’s just because she’s upset that google didn’t do anything she expected.

Depending on the version of server you are running, you may need to tweak the python command a bit to get it to work. Try “python /home/yourloginname/fauxmo.py” or “/usr/bin/python fauxmo.py”.

My commands are set for Ubunto. Yours may vary. See documentation on python for your version of Linux if you need help getting it to run.

If you’ve gotten this far and all is working, good for you! You’re making great progress. If you’re having trouble, use the standard divide-and-conquer troubleshooting techniques. Is python running? Does the script fail with an error? Is the server working fine? Are you in the right directory? You’re going to have to get fauxmo.py working before you go on to the next steps.

Installing pyHarmony Software:

Arduino Bot Shop harmony remote with amazon echo

The next thing to install is the pyHarmony software. Fortunately, it’s also a simple copy process to get these files running, but they actually need to be in your Web Server directory structure to run correctly.

My webserver root is /var/www/html/ (yours may be different, but this one is pretty standard.) I created a new sub directory, HA (for home automation) and off of that a directory for the pyHarmony software. Use the following commands to create the directories:

$>mkdir /var/www/html/HA
$>mkdir /var/www/html/HA/harmony

Next, you’re going to have to insure that the webserver has access to these two directories. In Ubuntu, the webserver is granted access by user (and group) name www-data. To grant www-data rights to these directories, use the chown command, like this:

$>chown www-data:www-data /var/www/html/HA
$>chown www-data:www-data /var/www/html/HA/harmony

Remember, your particular installation of Linux may do this differently, but the result is the same. You have to give apache (www-data) rights to these directories so that files in them can be read by the web server.

Now we have to make these directories executable. Use these commands for Ububtu: (your actual commands may be a little different – such as chmod 777 /var/www/html/HA)

$>chmod +x /var/www/html/HA
$>chmod +x /var/www/html/HA/harmony

With these changes, apache can access and execute files in these directories.

Now we can copy the files for pyHarmony into the directory structure. There are 4 files:

  • __main__.py
  • __init__.py
  • auth.py
  • client.py

Copy each of them into the /var/www/html/HA/harmony directory. Once they are there, it’s time to get to work.

We’ll need to edit the __main__.py file to make some changes for your installation. Before we do that, lets discuss how the harmony hub works.

While the harmony hub sits in your home on your network, it doesn’t communicate using typical HTTP. It actually uses XMPP, a protocol designed for services and hardware. In addition, it uses some HTTP protocols to communicate with Logitec servers to validate credentials and access your devices and their controls. Why is all this stored on Logitec servers instead of the local hub? It makes sense really. Logitec actually determines the controls needed to access your devices. What actually happens when you press “Up Volume” or “Down Channel” is created and maintained by the folks at Logitech. This means that they need to be able to manage and update what happens when you click a button. You also setup “Activities” for your hub. This means that when you click on “Watch Netflix” Logitech actually has to determine what devices need to be turned on, what volume, channel, etc. need to be executed. The hub could literally issue hundreds of commands via Infra-red and bluetooth to many different devices in your home theater cabinet to turn them on and set them up to actually watch Netflix. These configurations and groupings are stored on their servers for transparency across their devices, so that your remotes all can be programmed the same way, now and in the future.

BUT – this means that you need to control your hub with a mix of commands that go out to Logitec, and go to the hub directly, to do a task like “Watch Netflix”. Fortunately the authors of pyHarmony took care of all the dirty work for us. We just have to teach the program what our particular requirements are for things that we want to execute.

In this case, I’m going to focus on a couple of Activities. In my home these are “Watch TV”, “Play XBOX” and “Watch Netflix”. Oh – and Turn Off…. There are many more, but showing you how to execute these four commands should give you the experience you need to get going programming your particular setup.

The Harmony refers to commands by a number. Each command has a number, and commands can be grouped together into Activities that are represented by a number as well. Since all commands and activities like “Watch TV” are represented by a number, we need a way to discover what that number is. Once we have it, we can issue a command to the hub, “Start Activity XXX”, so part of our challenge is determining what that activity number actually is.

Helpfully, pyHarmony reports the activity number when we shut the system down. If we start an activity, then have pyHarmony shut everything off, it will report the activity number back to us when it completes shutting the system down. We’ll make use of that feature to determine our activity numbers for each of our activities “Watch TV”, “Play Xbox”, and “Watch Netflix”.

Once you’ve copied the files to your server, we can start by testing pyHarmony. Type the following 2 commands:

$> cd /var/www/html/HA

$> PYTHONPATH=”.” python harmony –help

pyHarmony should respond with something similar to the following:

usage: harmony [-h] --email EMAIL --password PASSWORD --harmony_ip HARMONY_IP
[--harmony_port HARMONY_PORT]
[--loglevel {DEBUG,INFO,WARNING,CRITICAL,ERROR}]
[--activity ACTIVITY] [--action ACTION]
{show_config} ...

pyharmony utility script

positional arguments:
{show_config}
show_config Print the Harmony device configuration.

optional arguments:
-h, --help show this help message and exit
--harmony_port HARMONY_PORT
Network port that the Harmony is listening on.
(default: 5222)
--loglevel {DEBUG,INFO,WARNING,CRITICAL,ERROR}
Logging level to print to the console. (default: INFO)
--activity ACTIVITY activity number to start/stop (default: -1)
--action ACTION {start} or [stop] or [debug] the activity (default:
stop)

required arguments:
--email EMAIL Logitech username in the form of an email address.
(default: None)
--password PASSWORD Logitech password. (default: None)
--harmony_ip HARMONY_IP
IP Address of the Harmony device. (default: None)

If you get all that, you can be assured that pyHaromony is actually working.
Pat yourself on the back! You are almost done!!

To make actual use of pyHarmony, you need three critical pieces of information:
Your logitec ID (it’s an email address)
Your Logitech Password
Your Logitec Hub IP

Armed with those three pieces of information, you can run your first real test of the pyHarmony software.

Using your mobile device, start an activity on the harmony hub. Wait for it to fully start, and insure that all devices are on and set as they should be. Once done, we are going to use pyHarmony to “turn off” everything so that we can get the activity number. As part of the shutdown process, pyHarmony will tell us the activity ID of the activity we started.

Test 1: Communicating with the HUB:
Type the following 2 commands. Replace with your ID, Password and HUB IP where indicated. The second command is critical, be sure to type in in exactly right! It’s one long command, so don’t press enter till the end!

$> cd /var/www/html/HA

$> PYTHONPATH=”.” python harmony –email YOUR_HARMONY_ID –password YOUR_HARMONY_PASSWD –harmony_ip YOUR_HARMONY_IP –loglevel ERROR –activity -1 –action debug show_config

This test insures that pyHarmony can talk to Logitech, and to your hub. It will display tons of information about your devices, configurations and other settings.
If you get a bunch of information back, pat yourself on the back again! It’s working, and you are very nearly done!

Test 2 Shutdown:

If you have trouble, make sure your credentials and IP are correct, and make sure you didn’t make a mistake typing in the command. If all that is correct, you’re getting some other kind of error. Troubleshoot as before, making sure python is running, you are executing the script, the hub is on and responding, and your credentials are correct. When you have it right, it’s time for the next test. Double check to make sure your activity is up and running on your entertainment center, then execute the following command:

$> PYTHONPATH=”.” python harmony –email YOUR_HARMONY_ID –password YOUR_HARMONY_PASSWD –harmony_ip YOUR_HARMONY_IP –loglevel ERROR –activity -1 –action stop show_config

If all goes well, your screen should respond with something like the following:

</pre>
ERROR sleekxmpp.xmlstream.xmlstream Error reading from XML stream.
Action: stop Activity: -1
stopping activity
6805838
OFF

Additionally, your entertainment center should shut down!!

Note the number “6805838” – that is the activity ID. (yours will be different) – write it down! We’ll need that for the next test.

Test 3 – Starting an Actvity:
Type in the following command to start that activity:

$> PYTHONPATH=”.” python harmony –email YOUR_HARMONY_ID –password YOUR_HARMONY_PASSWD –harmony_ip YOUR_HARMONY_IP –loglevel ERROR –activity 6805838 –action start show_config

As before, type it in carefully. Don’t forget to replace the 6805838 activity ID with yours. If all goes well, your activity should start with all the presets you want! Congratulations! Just a little more to go!

By now, you can see the pattern:
-–activity gets set to the activity you want. There is a unique number for each activity you have defined. An activity ID of -1 is “shut down”
–action is set to either start, stop or debug

You execute pyHarmony to control your harmony hub, which controls your entertainment center. Alexa is happily talking to fauxMo.py thinking it’s a TV. We know FauxMo talks to a webserver (web page) and that webpage talks to pyHarmony, so that’s the last little piece. Lets create that now.

Building your Web Service: (ok, it’s not a real webservice, it’s just a page)

Navigate to your /var/www/html/HA directory and make a new file index.php with the following command:
touch /var/www/html/HA/index.php

then open it to edit it:

$> nano /var/www/html/HA/index.php

Copy the following lines into that file, and save it:

<?php   $param1 = "--email YOUR_HARMONY_LOGIN"; $param2 = "--password YOUR_HARMONY_PASSWD"; $param3 = "--harmony_ip YOUR_HARMONY_IP"; $param4 = "--loglevel ERROR"; $param5 = "--activity YOUR_ACTIVITY_ID"; $param6 = "--action start"; $param7 = "show_config"; $action = $_GET['action']; $device = $_GET['device'];   if ($action =="start") {         $param5 = "--activity YOUR_ACTIVITY_ID";         $param6 = "--action start"; } else {         $param5 = "--activity -1";         $param6 = "--action stop"; } //$command = 'PYTHONPATH="."'; $command = "PYTHONPATH=\".\" python harmony"; $command .= " $param1 $param2 $param3 $param4 $param5 $param6 $param7 2>&1";

echo "attepmting to turn ". $action . " " . $device . " with command\n";
echo $command;

header('Content-Type: text/html; charset=utf-8');
echo '<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />';
echo '<title>200 OK</title>';
echo "

<style type='text/css'>
 body{
 background:#000;
 color: #7FFF00;
 font-family:'Lucida Console',sans-serif !important;
 font-size: 12px;
 }
 </style>

";

$pid = popen( $command,"r");
echo "<body>

<pre>";
while( !feof( $pid ) )
{
 echo fread($pid, 256);
 flush();
 ob_flush();
 echo "<script>window.scrollTo(0,99999);</script>";
 usleep(100000);
}
pclose($pid);

echo "</pre>

<script>window.scrollTo(0,99999);</script>";
echo "

Script Complete

";
echo '
200 OK
';
?>

This file takes a url, with some special commands in it (called querystrings), and uses them to talk to pyHarmony and tell it what to do. By using this technique, we open the webpage with a couple of commands and it takes care of typing in those long and complicated pyHarmony commands for us. Lets test it:

Open your browser, and type in the following URL (replace YOUR_SERVER_IP with your actual server ip):
http://Your_server_ip/HA?action=start&device=TV

The webpage should respond like this:

AND – wait for it….. Your Activity should start!

You can use the web page to test the start/stop of your activity, by simply changing the action= from start to stop, and different devices by changing device= from TV to whatever you named your activities. You’ll need to modify the webpage to handle more than one device, I’ll explore that later. For now, if your device starts and stops according to plan, all that’s left is to tell fauxmo.py to use THIS page instead of google.

Finishing up:
We need to edit fauxmo.py to change the url. Remember this?

FAUXMOS = [
    [‘T.V.’, rest_api_handler(‘http://www.google.com?cmd=on&a=office', 'http://www.google.com?cmd=off&a=office')],
]

We just need to change it and we’re done.

FAUXMOS = [
    [‘T.V.’, rest_api_handler(‘http://YOUR_SERVER_IP/ha.php?device=TV&action=start’, ‘http://YOUR_SERVER_IP/ha.php?device=TV&action=stop)],
]

That’s it! You’re done!

Arduino Bot Shop harmony remote with amazon echo

Tell Alexa to “Turn on the TV, and she’ll talk to fauxmo.py, which will navigate to your page at http://YOUR_SERVER_IP/ha.php?device=TV&action=start. This page will fire up pyHarmony with the command
PYTHONPATH=”.” python harmony –email YOUR_HARMONY_ID –password YOUR_HARMONY_PASSWD –harmony_ip YOUR_HARMONY_IP –loglevel ERROR –activity YOUR_ACTIVITY_ID –action start show_config,
Which will tell your Harmony Hub to start YOUR_ACTIVITY_ID, and Presto!!!  Your TV turns on!!

Adding additional activities is pretty simple. Remember to start an activity with your mobile device, use pyHarmony to shut down, and it will report the activity number. Build a “Switch” in fauxmo.py, and send it to the web page. In the web page, call pyHarmony with the appropriate activityID and an action = Start, and you should be good to go!

The method I’ve outlined here isn’t all that efficient. It takes several dedicated systems, all working in concert to actually accomplish the task. There are lots of areas that could be improved. The web page could be converted to an actual service, by building an API that does the job more efficiently and faster. The fauxmo.py code could also be improved, so that it handles more than simple binary (on/off) actions. pyHarmony could be improved as well. In fact, fauxMo and pyHarmony could be combined into a single application to remove the web page completely from the process. That is actually on my to-do list for a future article. But, I chose this methodology for a particular reason. It interfaces quite well with other devices you might find out there in the “internet of things”. Internet ‘Tings” all have a web-presence and some form of control. Adding your internet device to be controlled by Alexa should be quite easy using the techniques illustrated in this how-to. Don’t get me wrong, though, I’m quite interested in combining these features into a single application to make it faster and give it more capabilities. Stay tuned for a future article illustrating that project too. Till then, “Alexa, save and shut down”

One comment to Controlling Harmony Remote with Amazon Echo

Leave a reply

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>