Introduction
On August 17, 2015, GitHub experienced a small issue. They phrased it as: “affecting a small number of repositories.” Unfortunately, one of Sport Ngin’s biggest repositories was one of the “small number.” And for one hour, while GitHub was solving their own issue, the Development team at Sport Ngin struggled to work normally. GitHub is a central part of Sport Ngin development. Sport Ngin takes advantage of GitHub’s organization feature, which means it uses GitHub to manage all of its repositories. Also, the use of GitHub enables Sport Ngin to use a development workflow that includes pull requests, code review, quality assurance, and merging into master. Because Sport Ngin practices continuous delivery, these small code changes are quick; Sport Ngin merges approximately fifteen pull requests a day.
For the most part, Sport Ngin’s Development team likes almost everything about GitHub. However, Sport Ngin has found that GitHub is lacking some additional statistics, sorting, and lists that the Development team would find useful. Hubstats began as a Platform Operations project in the summer of 2014. Its intention was to provide Development with an easy list of pull requests ready for deployment, a total number of pull requests, and a number of contributions to any specific repository. Elliot Hursh was the original developer who began this project. Hubstats, an open source Ruby on Rails engine, connects via GitHub webhooks in order to accurately monitor, update, and show data from the Sport Ngin GitHub organization, sportngin. The webhooks receive information whenever a pull request is made, closed, commented on, or has a label updated. The use of a specific “deployable” label is used to filter down an entire list of pull requests to show only what a developer deems ready for production.
After the summer of 2014, Hubstats development was put on hold for a small time; it was doing everything Platform Operations had originally intended. But as time went on, the team found that there could be even more functionality added to Hubstats, making it even more useful to the entire Development team.
My Work with Hubstats
When I came on as a summer 2015 intern on the Platform Operations team, I had no idea what Hubstats was or what it would become. When I took my first look at the repository, it seemed massive. There were so many files, and I didn’t know what the purpose was of half of them. I also was completely new to Ruby on Rails, so I had a lot to learn there as well. But slowly, I began to catch on. Each model is associated with a table in the database. Each directory in the views folder (which will each contain an index file and a show file) corresponds to a controller file, which will contain an index method and a show method. Slowly, mostly through working on the project and doing bug fixes, I got more and more familiar with Hubstats and with RoR.
Deployments Page
The first project I was assigned involved tracking deployments. This project involved adding a deployment table to the database, creating a method that would receive API calls about a deployment, assigning pull requests, users, and repositories to a deployment, and then creating both an index page (for viewing multiple deployments) and a show page (for showing individual deployments). The entire first project took me just under a month. The final deployments index page looks something like this:
This page shows a list of all of the deployments, who they were by, what repositories they were in, the Git revision sha, and all of the pull requests that were merged in to make this deployment. If there are more deployments than 15, then the list is paginated, so users of Hubstats can flip through the history of deployments.
This page was a great first step for me to take with Hubstats. I got to reuse a lot of previously written code from the pull requests page, and I had a lot of help making the API calls come together.
Datepicker
The next Hubstats assignment I received was for the datepicker seen at the top right corner of the previous image. Before the datepicker, there was a drop down menu, where users of Hubstats could choose between “1 day,” “1 week,” “2 weeks,” “1 month,” “3 months,” “6 months,” and “All Time.” “All Time” wasn’t even all time; it was only set for 520 weeks. These values were not only vague, inflexible, and ambiguous, but they also prevented sorting data that was strictly in the past. What if someone wanted to see all of the pull requests from three weeks ago until one week ago? They could not. What if they wanted to be able to see deployments from the past two months up until now? They could not. The idea behind a datepicker was to create more flexibility, the ability to view past data, and the chance to be more clear about which dates were being included when showing data.
The Ruby gem I selected to use was bootstrap-datepicker-rails. With currently 684,538 downloads from RubyGems, it is by far the most popular when it comes to a datepicker. And it worked out really well for us as well. However, understanding how to connect the UI datepicker with the application.js was difficult. Luckily, there was a decent API written about bootstrap-datepicker-rails describing how to read from the datepicker, update the datepicker, etc. What we ended up doing was simply elaborating on the previous method of setting the date: using a cookie. With this new method, we store one cookie with two dates separated by "~~". When refreshing the page, we simply read from this cookie and write the dates to the datepicker so the Hubstats user can visually see what dates they’re looking at. When someone presses the Apply button, then we read what dates are currently in the datepicker, set the cookie, and automatically refresh the page, hence updating both the data shown and the cookie. The picture below would show how someone would set the date range to be only one week in the past:
Teams Page
The last main project I worked on within Hubstats was a teams page. All of the developers at Sport Ngin are sorted into GitHub teams, such as Competition Management, Platform Development, Mobile, etc. We wanted to show these teams on Hubstats and be able to record statistics based specifically on teams. After working on Hubstats for almost two months, I could easily navigate around the files, and had a much deeper understanding of how all of the pieces of code connected with each other.
Making a teams page required making both an index page and show page, making a teams table in the database, connecting the teams table with the users table, and storing the teams. After assigning users to teams, we had to figure out how we wanted to portray the new data, and what statistics would be useful to show. We elected to show the number of pull requests and number of comments per team, as well as showing the number of pull requests per developer and comments per reviewer.
There were two trickier parts to this assignment. The first was creating rake commands that would automatically update the users on a team. This command, when run, would call another method which would need to connect to an Octokit client and use the GitHub API to collect both the teams in an organization and all of the members in a team. After getting all of the data from GitHub, then it would call other methods to add the users and teams to the database. The final outcome of this large method looks like this:
The other challenging part of the teams assignment was processing a webhook from GitHub. Hubstats’ teams should be automatically updated when Sport Ngin’s GitHub teams are updated. The first main part of this was initializing the webhook. Not extremely difficult, just a little bit of googling. The other main part was handling the webhook when it was received. This was also not extremely challenging... until we discovered that what we initially wrote didn’t work. Hubstats was correctly receiving webhooks, but we weren’t processing them correctly. This led to two days of fiddling around with error messages, trying to mimic our error locally, and trying to solve the problem using new creative solutions. Unfortunately, what we found was that the error we got using Hubstats couldn’t be easily replicated locally. However, we did manage to discover that RoR was taking one of GitHub’s webhook api parameters, action should be either "removed" or "added", and replacing "removed" or "added" with the processor the webhook would route to, making it impossible for us to decide what to do with the team’s user. Eventually, we just had to try something to gather the original value of action. It was a stroke of luck we found something that works, but somehow, it handles all of our webhooks correctly. The solution was two lines:
The final teams index page looks like this:
The third column from the left, “Merged Pull Requests,” is the amount of pull requests that were merged in by each team. This number directly correlates to the deployments page; each pull request was merged in a deployment. Many pull requests means many deployments, leading to continuous delivery.
The final teams show page gives even more data about the individual team:
What Sport Ngin Uses Hubstats For
Hubstats has become an invaluable way for Sport Ngin’s Development team to interact with GitHub. By tracking the progress of our developers’ pull requests and our deployments, we are able to measure statistics on how much code has been written, how many pull requests our developers have created, and how much code has been shipped to production. It becomes easy for our Development team to see the impact of our code changes when using Hubstats. The repositories page helps us track how much each repository is being modified; some of our repositories are constantly edited and deployed to, while others haven’t been touched for months. We can also see exactly how many developers work on each repository as well. It is fascinating to see our repositories slowly changing. The teams and users page allow us to get more insight into which roles each developer holds. Some of our developers write many pull requests, while others make edits and comments on pull requests. These pages allow us to see who does what, which teams do what types of work, and how many members on each team code and review.
Why is Hubstats a Useful Tool?
Hubstats gives GitHub users new perspectives of their development and their interaction with GitHub. Pull request label sorting, a feature that is available in GitHub, is much easier with Hubstats. Sorting an organization’s most active repositories is simple on Hubstats, while it is not easily done on GitHub. Calculating what Sport Ngin calls the “smallest coder” (developer with the most pull requests) is possible with Hubstats, even though it is not possible through GitHub. Using a calendar view to choose the specific data to be seen is simple with Hubstats, while it is much more challenging with GitHub. Hubstats takes the same data that GitHub uses, and reorganizes, sorts, and shows new GitHub statistics that are not immediately provided by GitHub.
How to Use and Contribute to Hubstats
Since Hubstats is open source, Sport Ngin welcomes other organizations and individuals to use and contribute to Hubstats. The source code for Hubstats can be found here. The README.md has some basic information about setting up Hubstats; however, the Wiki on the right sidebar has even more information about other details of Hubstats and can help simplify the Hubstats setup process. Feel free to fork Hubstats, make changes, and make a new pull request if you have any further ideas. And in fact, if you make a pull request on the Hubstats source code, you’ll show up as a user on our Hubstats.
Happy coding!
