400,000 ephemeral containers: testing entire ecosystems with Docker
Presented by
Daniel Axtens
Wednesday
1:20 p.m.–2:05 p.m.
Target audience:
Developer
Abstract
The value of a high-level language isn't just in the language itself, but rather in the ecosystem surrounding it. Python is hugely enriched by PyPI, Ruby by gems, and Node.js by npm. These ecosystems have hundreds of thousands of packages, evolve rapidly, and vary wildly in quality.
What if you want to test how all the packages work in a given situation? This is the question we set out to answer in this talk.
Previously, testing each and every one of the 458,100+ packages came with an enormous infrastructure cost - for each package you need a VM with a full OS, and the associated memory, CPU and disk requirements. This makes exhaustive testing infeasible for most teams. However, we found that by using Docker, we could test the entire Python, Ruby or Node.js ecosystem over a weekend with a single VM.
In this talk, we look at:
- how we leveraged Docker to rapidly test the entire Python, Ruby and Node.js package sets on OpenPOWER,
- how the characteristics of containers vs VMs made an otherwise infeasible task both feasible and simple,
- how we evolved this into a continuous-integration process,
- other interesting applications of testing at this scale,
- what we found, and some of the interesting quirks and bugs we encountered and lessons we learned.
Presented by
Daniel Axtens
Daniel is an OpenPOWER performance engineer at IBM OzLabs. He works with containers, continuous integration, and performance optimisation - often all at the same time! He has a particular focus on the challenges of interpreted language behaviour and performance, as well as a growing interest in developing scalable continuous integration systems for the Power architecture.
He has also worked as a Linux kernel developer, where he worked on kernel support for IBM CAPI accelerator cards.
Before joining IBM he studied Engineering at ANU, and worked in web development. He also dabbled in embedded systems.
When not in front of a screen of some description, he enjoys walking around Canberra and spending time with his wife.