To celebrate Halloween season, the Bitbucket team put out a call to the Atlassian Community to share their spookiest software development stories. And Community delivered, sharing their tales of zombie issues, lurking semi-colons that snuck in right before a code freeze, and bugs that just wouldn’t die.
Below is the winning story from Jimmy, a Bitbucket user and Developer for Igloo Software in Canada.
When we first started using Git and Bitbucket Server (which was Stash at the time). I accidentally reverted a merge commit in our master branch the day before we were supposed to deploy to production.
In my inexperience at the time, my first plan to fix this was to reset the head commit to an earlier point in history. Instead, I ended up wiping out about two months of development changes.
Let’s give some context. This all happened 5 years ago when our company did a shift from SVN to Git and no one was experienced with Git’s commands or structure. We were getting ready for our current release, and I was asked if we could make three versions of our “Release” branch that contained different sets of features.
We were just finishing testing a couple of features and the project manager weren’t sure if we were ready to release feature “A”, feature “B” or both. In hindsight, the feature release should never have been dictated by the project manager. If testing wasn’t finished, we shouldn’t have been trying to ship either of them!
Still, I went to work to setup these branches. I managed to merge things out of order and ended up getting the features mashed together in the wrong branches. My first response was to revert the merge commit that matched the feature I didn’t want to merge. Unfortunately, this also reverted a number of bug fixes required for our release. So, my next action was to simply reset to a point in history before I made any of the merges.
No big deal right?
I didn’t realize what I was doing at the time, but the specific options I was passing in actually deleted all the history to that point. So now I’m in a state where my local and the remote have a bunch of history deleted, which anyone who uses Git can explain is no good. I decide that I now need to raise some red flags to the development managers and explain what I did. They were very understanding, since no one was very experienced with Git and thankfully, I wasn’t blamed for doing the wrong thing.
At this point, we talked with the rest of the development team to see if anyone had not pulled from the remote since I pushed those “bad” changes. Most people had already pulled the bad changes, but we managed to find a few people who had not. Still, their versions were missing some of the other changes we needed.
We were able to combine their changes with some cherry-picking from one of my branches into a single branch and we forced that branch on top of the “bad” release branch on the remote.
We made sure that everyone fetched this copy of the branch so that everyone would have this new consistent history. From then on, we started looking closer at understanding the Git commands and their options. And the features in Bitbucket that we can use to help prevent these types of mistakes in the future, such as preventing rewriting history, became key to our workflow.
While that incident was a long and stressful night as we tried to recover in time for our production deployment, I did learn three big lessons out of the experience that’s stuck with me since
- Always double check everything locally before pushing to the remote.
- Creating backup branches locally “just in case” never hurts.
- Using serious branch permissions to prevent deleting branches and rewriting history can save you from stupid mistakes.
