Challenges when Scaling: Continued adventures in Swift's sharding

Presented by Matthew Oliver
Wednesday 3:40 p.m.–4:25 p.m.
Target audience: Developer


Background: Swift is an open-source, highly available, distributed, eventually-consistent object storage system. As the project has matured, so has the need to store more and more data. Some sites have moved to using expensive solid state drives (SSDs) for storage, but there has to be a better way we can solve this in software, right?

This presentation continues our Swift sharding adventure from where we left it last year - but now addressing a new set of challenges we've come across in our proof of concept container sharding implementation. During testing of even larger data sets we've come across a number of limitations of the underlying SQLite database, meaning we've had to develop work arounds so that we are able to shard even larger containers. Come and see how we performed large scale testing, how we narrowed down where the limitations were, how we developed our solutions to these problems, all the while trying to maintain backwards comparability.

Presented by

Matthew Oliver

Matthew is a software engineer working at Rackspace Australia, more specifically Rackspace Private Cloud where he primarily works on upstream OpenStack as a Swift core. Based on the south coast of NSW, he spends his time hacking on Swift. Matthew was the co-founder of the Kororaa Linux distribution which has given him careers in both Linux system administration and software development. During his time with Rackspace he has worked both as an Enterprise Linux Sysadmin based in the UK and is now back at his roots as software developer in Australia.

©2016 Linux Australia and 2017. Linux is a registered trademark of Linus Torvalds. Site design by Takeflight. Image credits can be found on our Colophon.