[This is a cross-post from the Atlassian Developer’s Blog. This post is written by Steve Streeting.]
Need to store large media files in Git? We’re making major contributions to the Git LFS open source project to help make this happen! Want to know how this came about? What follows is a true story…
The challenge
Git’s great. When it comes to keeping a handle on your source code, there’s nothing quite as flexible, and developers are adopting it in droves. But there are a lot of teams whose needs haven’t been particularly well met by Git in the past, whose projects consist of not just code, but media files or other large assets. People like game developers and web design studios are common examples, and in many cases Git’s inability to elegantly handle this issue has meant they’ve had to remain on older source control systems.
I’m well aware of this problem myself, because before I came to work at Atlassian via creating SourceTree, I ran an open source graphics engine called Ogre3d for 10 years. I worked with a lot of teams with large textures, models, and so on in their repositories using pre-DVCS tools. In the years after, when I started making Git & Mercurial tools, I kept in contact with many of these people and witnessed frequent problems adopting Git because of large files clogging up repositories, slowing everything down, and making clone sizes intolerable. A few teams used Mercurial’s large- files extension, but many of them still wished they had the option of using Git as well.
Atlassian has focused on serving the needs of professional software teams for the last 13 years and we’ve heard from many of our customers that they’ve struggled with the transition from legacy source code management systems to Git because of the lack of a good solution for the large files problem. Software teams really like using Git but this one problem is a real spanner in the works for their adoption plans. What could we do?
The quest begins
So, in late 2014, we started seriously looking at this issue. We tried everything that was out there already, and reluctantly concluded that the best way to solve this properly for the long term was to create a new tool. Creating a brand new tool wasn’t our first preference, but we felt that existing tools were either too complicated to configure for a team environment (once all the features you really needed like pruning were factored in) or were not fully developed enough and used technology we didn’t think would scale, so extending them wasn’t attractive.
We chose to write this new tool in Go, a modern language that was both good at producing stand-alone, fast, native binaries for all platforms, but was also fairly easy for most developers to learn; important because we intended to make it open source.
We initially called it Git LOB, and after working on it for a few months we attended Git Merge this May to announce it.
The twist
What neither Atlassian nor GitHub realised is that we’d both been working on the same problem! Atlassians and GitHubbers met in the bar the night before our talks, to discover that we’d both:
– made the decision to write a new tool for large files in git
– chosen to write it in Go
– designed the tools in very similar ways
– planned to announce the work on the same day at Git Merge.
Crazy right? Was it really all a coincidence? Turns out that yeah, it absolutely was, we’d both done this completely independently. I guess great minds really do think alike 🙂
The resolution
We decided it made no sense to fragment the community when Git LOB and Git LFS were clearly so similar in their approach. It wasn’t a complete overlap, there were things that Git LOB did that Git LFS didn’t and vice versa – the best solution would be to have the best of both. So we open sourced our version as a reference, then switched our efforts to contribute to Git LFS instead, starting with porting across features we’d already developed that were useful for Git LFS. We at Atlassian plan to continue collaborating on the Git LFS project for the foreseeable future.
How it’s been going
I’ve really enjoyed working with the community around Git LFS, it’s turning out to be a really productive team effort. I’ve contributed 36 pull requests so far, making me the biggest non-GitHub contributor to the project right now:
If you’re using Git LFS, my name crops up quite a lot in the new features for v0.6 🙂 Many of these features were ported across from Git LOB, but the best thing is, I actually managed to make them better as I did the port, since you can always find improvements when you look at a problem the second time.
While it was a hard decision for me personally to stop working on our own solution, a few months on I’m really happy with the outcome and it was absolutely the right thing to do for the community. I firmly believe we’ll create something even more awesome by working with the open source community, and that I’ll be able to contribute positively to that effort. We at Atlassian feel it makes perfect sense for us to work to a common standard just like we do with Git itself and concentrate on creating great solutions around it, especially those that fit the needs of professional teams.
What’s Next…
I’m working on a number of features for future Git LFS versions, including support for pruning, extensions to the SSH support, and more. I’ll also be just generally around the community commenting on stuff and trying to help out. I’ll be at GitHub Universe too, talking about our collaboration as part of the “Using Git LFS” panel on 1st October.
You’ll also be hearing much more about Git LFS support in Atlassian products, which of course includes Bitbucket and SourceTree. Watch this space! In any case, thanks for reading and I hope my little story of random coincidences and community collaboration was interesting. 🙂
