Here you will find ideas and code straight from the Software Development Team at SportsEngine. Our focus is on building great software products for the world of youth and amateur sports. We are fortunate to be able to combine our love of sports with our passion for writing code.
The SportsEngine application originated in 2006 as a single Ruby on Rails 1.2 application. Today the SportsEngine Platform is composed of more than 20 applications built on Rails and Node.js, forming a service oriented architecture that is poised to scale for the future.
Over the last two years the Sport Ngin Development team has been progressively moving towards a Continuous Delivery style of Software Development. Two years ago we averaged two deploys a week to production. One year ago we averaged 8 deploys a week. We now average 30 deploys a week to production. As our deploys have increased each individual deploy has gotten smaller.
A core component of Continuous Delivery is the Development Pipeline, which is the set of steps that get a chunk of code from a developer's desk to the live production servers. A Continuous Integration test server and a comprehensive test suite are critical to making this work.
The Sport Ngin Development Pipeline key checkpoints are:
The checkpoints in the Pipeline above are requirements for a chunk of code to be deployed to Sport Ngin production servers. Typically the developer doing the code review also does the manual testing associated with the Quality Assurance sign-off. This includes manually exercising the feature in a web browser or on a mobile device against our staging servers.
What I like most about the ideas behind Continuous Delivery is that it encourages developers to Code Smaller by slicing their work down into smaller chunks. This has huge benefits and is not a new idea. Jeff Atwood wrote a 2007 article on Coding Horror with the same title: Code Smaller.
Jeff Atwood points out that "So many aspects of software development can be summarized as small is beautiful". I absolutely agree. This rings true when considering all the benefits that come with a Service Oriented Architecture versus a monolithic application.
Small is beautiful is spot on in regards to Continuous Delivery. A core part of Continuous Delivery is slicing large tasks down into small pieces and deploying these small pieces of progress individually to production servers.
From Atwood's Code Smaller article:
The odds of failure for a software project are directly proportional to the size of the project. Slicing a large project into several smaller subprojects is the single most direct way to increase your project's chances of success.
Slicing a multiple month project into two week projects is definitely valuable. The two week sprint popularized by Scrum is a basic example of this. This is slicing on a macro level, which is typically done by Project Managers as they organize the overall development effort behind a product.
With ideas from Continuous Delivery this concept of slicing work down into smaller chunks applies to a much smaller scale - a daily personal scale. This is slicing on a micro level. When all the individual developers on a team are able to slice their personal work into small deployable chunks representing two days of work or less, then several benefits began to be realized. This is what we like to call Code Smaller and this is the essence behind Continuous Delivery.
Arguably the greatest value with Code Smaller is all about feedback. Have you ever tried to do code review on several weeks worth of development a few days before the deadline? Basically pointless as there is no time to be concerned about the quality of the code. It is crunch time and every last minute must be spent making sure the feature works as intended. The feature inevitably is deployed and the several thousand lines of newly introduced code becomes the whole team's problem to maintain.
When a team Codes Smaller the feedback loop shrinks. It is easy to code review a small chunk of work. Immediate feedback regarding code quality, bugs or performance can be given and acted upon. Architectural related concerns are addressed earlier, which can produce better results and shorten the time to completion.
The feedback loop goes beyond code. Code Smaller techniques force conversations on requirements between developers and product managers to occur earlier than they would otherwise. Maybe the requirements need to change or maybe the developer is not implementing the requirements correctly. The earlier these conversations occur the better. Nothing makes these things happen sooner than when a developer tries to actually get their work through the Pipeline and deployed to production.
Those big ugly integration problems go away when a team Codes Smaller. The kind caused by working for several weeks in a different branch and then trying to merge the code into the mainline branch only to find out that someone else has modified the same code in a major way. This can leave the developer with very complicated conflicts to fix before the merge can be completed and the feature deployed. Martin Fowler in his influential Continuous Integration article speaks of crazy long integration efforts:
I was told that this project had been in development for a couple of years and was currently integrating, and had been integrating for several months.
In contrast, when the Sport Ngin Development team has integration issues or conflicts they are typically so small in nature that they are fixed in one minute. We do use Github branches, but we make a conscious effort to keep them short-lived, personal, and local, which Randy Defauw points out is consistent with the Mainline model preferred by Continuous Delivery.
By combining Continuous Delivery with Code Smaller the work being deployed is forced through the Development Pipeline on a frequent basis. This is like climbing stairs one step at a time. As time goes on we are confident that real progress is being made.
With Code Smaller progress is clear and steady.
In contrast, deploying several weeks worth of code all at once is like running in place for awhile and then trying to jump to the top of a building. As time goes on the amount of real progress towards the end goal is questionable as the likelihood of a serious integration, requirements, or architectural problem is quite high.
Progress is questionable when deploying once every 10 days.
How can this be done when a new feature may take 4 weeks or more to develop? There are several techniques that make this possible. Feature toggles enable progress on a feature to be deployed while being confident the feature is not available for users until it is ready. While extremely useful, feature toggles are not always needed. Backend work that doesn't touch the user interface can be deployed to production in small chunks without concern about users seeing a half-finished feature. Are migrations in place to create new tables for this feature? Deploy it. Models partially built out for these tables with corresponding tests in place? Deploy it.
Manually testing small changes is fast and focused as it is clear from the small code diff what needs to be tested. Often changes are so small they may not need any manual testing, especially when the automated test coverage is high and the risk is deemed low during the code review. As progress is being made and deployed in small chunks for a feature, manual testing should be focused on finding regressions instead of solely on the feature itself. Prior to releasing the new feature to customers, a larger QA effort should be done against the feature as a whole to ensure it is ready to go.
Code Smaller works best when each developer on a team is able to slice their personal work down into small deployable chunks. A project manager can attempt to outline how to break this work down, but ultimately this level of detail is too fine grained to warrant the attention of most project managers. It is the developer who knows best how to slice their own work down into small deployable chunks, which may or may not match one-to-one with the stories the developer is tackling for the sprint.
We have found it important to recognize Code Smaller as a skill. A small amount of planning prior to starting each new task is needed to do it well. It is very easy to get caught up in ones work, to find yourself four days later with a chunk of code changes growing in size and complexity that has yet to be reviewed by anyone else on the team. Before writing any code it is always a good idea to think about how to slice your work into small deployable chunks. Slice-First, Test-Second Development?
The Sport Ngin Development team has had great success leveraging Code Smaller techniques along with Continuous Delivery. While we are doing 30 deploys a week, there is room for improvement to reach the level that Etsy's Development team operates at with their 25 deploys a day. Being capable of doing 25 deploys a day successfully is a direct indication of the maturity of the software development team and its processes and deployment pipeline. The end result, 25 deploys a day, is less important than what it takes to get there: a comprehensive test suite, a high level of automation, fast one button deploys, and a development team that actively Codes Smaller.
Tag(s): Home Continuous Delivery