I make good on my promises. I may be 3-6 months late, but I promised you guys that I was open-sourcing one of my work projects that I was super excited about, and the day has finally come. This delay was worth it however. The timing coincides with Careerbuilder officially accepting an open-source policy which allows developers to open non-business-specific code, as well as use and contribute to open-source libraries. I am proud to say that my project, CloudSeed, is the first to follow this policy and become a publicly viewable open-source repo!
What is it?
Magic in a bottle. Or rather, a server I guess, since no bottles are involved. In essence, CloudSeed provides an easy way to build and maintain “stacks” of resources in the Amazon Cloud. It leverages Amazon’s CloudFormation framework, but strives to remove human error (to which that framework is incredibly prone) and increase maintainability. If you have ever used CloudFormation and gotten lost in a sea of JSON needed for a large template, this tool will help you. For my readers unfamiliar with the Amazon Cloud and its tools, let me give a little background.
The first acronym in the coming alphabet soup here is “AWS”. This is Amazon Web Services, and it refers to the application which provides business with an externally hosted cloud environment. This means we can move expensive datacenters in to AWS, and cut the maintenance costs, location issues, and uptime concerns.
AWS has a lot of of services
The next big guys are probably the services. There’s a BUNCH of these guys, and Amazon adds more every day, but the big ones I’m going to mention are RDS, EC2, and S3. In order, those are Relational Database Service, Elastic Cloud Computing, and Simple Storage Service. RDS is a service providing hosted databases, so you do not need to spin up a server and install an instance. EC2 is a service offering complete control of a private cloud and servers therein. One can create complex networks populated with servers in different regions and subnets, just like local datacenters. S3 is a hosted storage system as the name implies. It allows for easy storage and retrieval of data via an API.
There are many more acronyms, but for the sake of not turning this post in to a glossary, I will define the others as we need them.
What about CloudFormation?
All of these services in the Amazon cloud make it a very useful tool. This means that it gets used for some very important business functions. Things that important always need a backup. So you could feasibly keep a network map and record every change you make, so that if anything happens you can rebuild it according to your notes. And that would totally work, until it doesn’t. It doesn;t take too long for a network to get far too complicated to rebuild by hand. That’s why Amazon created CloudFormation!
CloudFormation is yet another Amazon service. This one accepts JSON templates representing all the services, networks, and servers needed to make a stack, and builds it according to the configurations you set! When you need to add a server, you add it in the JSON template, then rebuild. It keeps a record of all your changes, and makes sure your stack is always consistent and rebuildable at a moment’s notice. Clearly, the smartest way to use AWS is to ditch their web interface and maintain your entire infrastructure through CloudFormation!
At least, that’s the theory…
If it ain’t broke…
The reality is that CloudFormation has its own problems. Small stacks are easy to build, configure, and maintain in JSON. Large ones get considerably less easy. Couple that with some questionable choices on Amazon’s side (Really, ACL rules are their own part, but security group rules aren’t?) and you end up crippling yourself if you vow to only use CloudFormation.
Our team made an honest go of it. We did pretty well too, excepting the 10-20 minutes needed to debug the new template each time we created a stack. However, it soon got out of control. We added a VPN tunnel to one of our subnets, and it created a hellscape of complicate network interactions. Now, the template for creating a subnet (which was previously only about 100 lines) needed the possibility to include VPC peering and routes to the VPN tunnel. Security concerns meant those new routes needed security groups and access control lists, and before you knew it, the template titled “Subnet.json” made a LOT more than just a subnet. When we needed to change a single ACL rule for one of our subnets, we now had to dig through ~2000 lines of JSON to get to the relevant portion. Its important to note that this was the case whether that specific subnet needed those other features or not.
WAY too much JSON for 3 subnets…
Our workflow was broken, and I was assigned to fix it.
I proposed a new workflow using unique templates per stack, which solved the complicated stacks issue, but ended with a ton of reused code. I was confident this was the correct approach though, so I made a simple python script to illustrate my point. By keeping a record of the common code (mostly the actual resources needed to build a stack) I could prompt for parts and their required parameters. This allowed us to assemble unique stacks without having to copy and paste common blocks. It worked so well, I spent quite a bit of time on adding to those common blocks of code which I began referring to as “Parts”. I was so proud, I showed my team and watched their dissatisfied faces. They didn’t like it. A script was still too error prone and much too hard to visualize.
Bunch of whiners if you ask me…
Well, nobody asked me. Instead, I decided to help them visualize it with a simple UI. Side note, UIs are freaking hard. I made an agile card titled “Create CloudSeed UI”, which I estimated at a 2. Here I am 3 months later, still working on it.
I set my sights on the MEAN stack, mostly because I already knew Node/Express, and my closest coworker had spent the week preaching the word of MongoDB. It was only a day or so before I had a nice looking API set up to handle stack saves and loads, serve all of the parts files, and push stacks to AWS. Then came Angular. I struggled my way to a mostly working angular app by the end of the week, littered with a poor understanding of the framework, misused bootstrap classes, and a non-functioning set of Material Design classes. Regardless of its hideous face, I was proud. The app now presented the user with a collection of parts which could be independently configured and assembled in to a working single-file template.
Buttons > !Buttons
I had replaced the arduous task of typing out a new stack with a few simple clicks. I could click “VPC”, “Subnet”, “Subnet”, “Subnet”, “EC2Instance” and instantly have a stack ready for configuration. More importantly, Existing stacks could now be updated as easily as using the web portal for AWS, only this way they were tracked for future rebuilds. The stacks were auto-committed to a git repo for reliable change tracking, and loading them up filled the info into user-friendly forms for easy editing.
Where are we now?
Its mobile friendly too!
Since CloudSeed’s first showing, I have been pulled in many different directions. The app has been dramatically improved from its first state, but there is still much to be done. I have learned a lot of web apps since I started, and would love to bring that knowledge back to CloudSeed. However, the nature of my job has made it such that I can only really afford to work on CloudSeed when it directly solves a problem we are having within our team. It is primarily for that reason that I sought to open source this project. I want to see what other teams can do with it, and look forward to making it a truly useful tool for all AWS users.
If you or your team use CloudFormation and think this could be a good fit for you, please give it a try! The code is hosted at Careerbuilder’s new Open Source github account and I would be happy to help set you up if you experience any problems! A big thank you to the Careerbuilder Open Source Development Council for allowing me to see this through, and as always, code on.