Edison

REST Easy – An Edison Project

This post represents the end (for now) of my series on the Intel Edison board (part 1 and part 2) wherein I describe my latest project. This was an app designed partially for use in my apartment, partially as a trial in programming for such a small device (storage and power-wise). In the end, I managed to squeeze out a minimum viable product with lots of room for growth. There was definitely a need for scaling back initially, however, since the grand design for the project would have overflowed the memory just on install. After swapping some tools for their smaller counterparts, trading uglier code for less space, and dropping of some top level design characteristics, the app was able to build and run on an Edison in almost any storage state, allowing this app to run quietly alongside whatever other tools you have decided to use on your Edison board.

What’s it Do?

For once, I have a project which needs no complicated justification for its use. This app deploys a very simple REST API (get the name now?) which can manipulate the many GPIO pins which make this device a real player in the Internet of Things. Without any need for understanding APIs, no knowledge of javascript, and very little need for understanding linux at all, this makes it possible to manipulate your Edison’s pins from anywhere with an internet connection.  The idea is simple. The Edison has a built in wifi adapter, and using that combined with DHCP reservations can give the Edison a permanent local IP address. Port forwarding port 3000 (the REST API port) to that address allows you to send commands through your router to the wireless device from anywhere in the world. Sure, you could ssh in and manually manipulate each pin from anywhere already. What this api REALLY leads to is extension. When you can reduce a series of complicated ssh commands to the press of a button, that’s progress. With REST Easy, the button need only send an http request to the API, and all changes will propagate.

How?

Very, very simplistically. The app is essentially a prebuilt nodejs application, which self installs and runs with the app’s own install script. Beyond that, the only other trick is a library of very simple python scripts which use JSON to communicate data between the linux representation of the GPIO pins and the API. These scripts are really aggregate read/writers of all the status files associated with each pin.

I have no idea who thought this was a good way to handle GPIO

This is a simple view of the file structure for each pin

 

When the request for the status of a pin is received, the script grabs the information about every aspect of that pin from the file system (/sys/class/gpio/gpio<pin#>/[direction, edge, value, etc.]) and compiles it to a JSON object which can be easily sent back over the API. The reverse is also true. When a POST command is sent to the API with a JSON object (or array) the python script will break down the object and set each field to the value in the received data.  The JSON object for each pin looks like this:

{
 "ID": 130,
 "active_low": 1,
 "direction": "out",
 "edge": "none",
 "value": 0,
 "power": {
     "async": "disabled",
     "control": "auto",
     "runtime_enabled": "disabled",
     "runtime_status": "unsupported",
     "runtime_active_kids": 0,
     "runtime_active_time": 0,
     "runtime_suspended_time": 0,
     "runtime_usage": 0
     }
}

This should be pretty easy to understand, even with no prior experience with JSON. The words before the colons are the keys or variables, and those after are the values. The format is actually how javascript saves objects in flat files, hence the name JavaScript Object Notation. This makes it doubly useful in a javascript based API, since upon the receipt of a JSON payload, the data is immediately converted into an object. This can be manipulated and altered using standard OO operations including dot notation, rather than messing with string manipulation and regexes.

What’s Left

By this point you should be expecting that this project has not fully realized its potential (how many projects ever do?).  There are several features on my list which will hopefully be added in the near future, and many which I plan to leave out until needed unless some willing soul wants to contribute. Some of my planned features:

  • API Keys/Auth
    • The app comes with a SQLite database and authentication system (endpoints /users and /login) but as of yet, they are not used for anything. The plan is to require a login every so often to release an API key. This would limit your board from attack by anyone who knows your IP.
  • Aliases
    • While the pins are currently accessed by passing their linux ID number, it would be even better to be able to assign an alias to a pin and use that to access the pin’s state.
  • Groups
    • Many processes using GPIO rely on the simultaneous change of more than one pin in order to change a multiplexer input or some other multi-bit operation. The app does not currently support the simultaneous changing of pins, however multiple pins can be accessed in the same request via passing an array of ID’s or JSON objects for GET and POST respectively.

I have no intention right now of working on a front end to this application. That task along with any other features not in the above list are up to you, the user. This code is 100% open source, and available in the usual spot on my github. Feel free to alter, use, adapt and destroy any of it to suit your needs. The install process is rather simple:

  1. Download the source code and package all but the “installer” directory into a tar.gz archive (this will be done for you when a release medium is linked).
  2. Send the installer folder and tar.gz package to your edison.
  3. SSH in and run the shell script included in the installer folder.
  4. assuming your Edison is already set up for wifi, you should be able to access the global GET endpoint at: <Edison’s IP>:3000/api/pins

Go forth and do wonderful things with this. If you use it for something cool, I only ask you mention me in the documentation, and of course share a link to what you’ve built so I can admire your work! Next week I will be writing up a very short summary of some project currently on my plate, so stay tuned!

Advertisements

Crowd Computing

Last year, I was a part of the inaugural HackGT. This is an annual hackathon sponsored by Georgia Tech, which seeks to gather programmers from all around the country for one weekend to develop the best app they can. The grand prize is $60,000. The prize drew a lot of interest, but what compelled me to participate was the presence of a variety of big companies with new technologies. One such pre-announced presence was Intel, with an early look at the Edison board I wrote about last week. The board fascinated me, and the ability to hack on one for a weekend before it was even available for purchase ensured my name would be on the HackGT signup list.

Hackathons

If this word is unfamiliar to you, its time to learn. Hackathons are spreading, becoming more frequent at software companies, schools, clubs, even cities (see HackATL) because of their tendency to produce minimum viable product prototypes in a short amount of time. Essentially, a hackathon is just a gathering of programmers with the resources needed for extended programming times. Often these hackathons feature diversions and entertainment to allow for breaks, food and drink so you never need to leave, and caffeine and/or alcohol for those late night coding sessions. At the end of the 24-72 hour span, apps created by the participating teams and individuals are presented to judges in order to determine winners. These winners could be awarded prizes, or have their idea produced, or may even be offered a job.

Crowd Computing

Crowd computing was my HackGT project, done over a 48 hour period with 2 teammates. (See how much more sense that makes after the intro?)  The idea was to create a big data platform on a tiny board. These Edison boards were great, but they lacked space and computational power compared to traditional computers. In theory however, their price meant that there would be many of them. The number of boards combined with a tendency to be used for passive computation made them ripe for use in cloud computing. Essentially, jobs that couldn’t be run on one board could be run on LOTS of boards working together. A simple website would allow for you to enroll your board in the program by installing a tiny script. This script reports to the webserver every couple minutes to verify the availability of resources on the board. When a job submitted to the website needed boards, yours could be chosen, and the un-utilized resources would be used to compute a portion of the job. When a few dozen boards contribute as such, the resultant power is pretty astounding.

Our app leverages the Map Reduce framework common in big data applications, with a tiny twist. since the boards are hardly big enough to function as nodes, we had to use something with a little more power as the master node. The webserver played that role, allowing for mapper scripts to be run on it that distribute data and a reducer script to the Edisons. From there, the boards would each execute the reducer script on their small portion of data, then return the output to the webserver along with an id which denotes which board the data belonged to. In our proof-of-concept demo we used a very simple example. A single Edison would first attempt to sort the entire text of  War and Peace alphabetically in a very short python script. Simply allocating the space for the novel alone was a struggle, and once the sort process began, the ram began to overflow and the board rebooted. This was expected. This task is simply too large for the memory and computational capabilities of the device. For contrast, we uploaded the same task to our webservice, to which we had registered 6  boards. A mapper script was created along the following lines:

def map(text):
words = text.split(' ')
letters = dict()
for word in words:
#map each word to a list by its first letter
letters[word.lower()[0]] .append(word)
return letters

This split the book into 26 arrays by the starting letter (plus a few for symbols) for every word in the book. Now, we had smaller chunks we could work with. The webserver sent a single array of data to each device, along with the index of the array. Since “A” comes first, a machine would receive all the words beginning with “A”, plus an ID of 0. The device also received a short python script, which told it to sort the list, then communicate the results and original ID back to the webserver. This process repeated until all the arrays of words had been sorted and returned. At that point, the web server would run it’s handler, which sorts the lists by ID. Since “A” had an ID of 0, “B” was ID 1, and so on, the result was a completely sorted novel in a short period of time. In our example it took around 15 seconds to sort the entire book. When some of the devices are in use it may take longer to lobby for access to CPU time and memory, but the idea remains the same.

Where are we now?

The code is on my github. It was just recently open-sourced, and there’s a reason it took this long. The code is VERY sloppy. One of the downsides to hackathons is that programming competence tends to decrease with tiredness. After 36 straight hours of working on the code, we began to make VERY bad mistakes. compound that with a teammate leaving in the middle of the night and frustration with new technologies and poor internet connection, and you get a mess. I’m not entirely sure that what is on github will actually work anymore, and I know that what was on the webserver no longer works. However, over the course of the next few weeks, I intend to revisit it and clean up large sections of the code, hopefully producing a live website soon enough. Please feel free to contribute and fork, or just stay tuned for a beta invite if you own an Edison board (and if you don’t you totally should).

Visit the code HERE

That’s all for this week. Next week I will wrap up my discussion on the Edison for now with my latest and current project: “Rest Easy”. Until then, raise a glass and code on.