In case you haven’t noticed, Bitbucket’s been suffering from slowdowns. This is mainly due to poor I/O performance we get from our Amazon EBS mounts.
After much planning and benchmarking, we’ve decided to move our data over to an 8-disk RAID 0 setup instead. The past few days, we’ve been synchronizing data to the new mountpoint, and earlier this evening, we made the switch. This resulted in about 2 minutes downtime.
Now, seeing as causing more I/O load on the live drives would result in Bitbucket being completely unreachable, we did our best to get the new RAID disks into action with as-fresh-as-possible data before we made the switch. This allows us to have minimal downtime, but it also means that the live data will now be a few hours old.
If you’re seeing old data in your repositories (or in the worst case, missing repositories that were created within the last few hours before the switch), that’s to be expected.
We’re currently synchronizing everything back from the now-legacy drives to the live drives. Synchronizing data can be a somewhat lengthy operation, but it’s going pretty fast.
In the end, this should result in a much faster Bitbucket. We’ve been under I/O strains for some time now, and this is the first (large) step in resolving these issues for good.
