Published: Wed 25 April 2018
Updated: Wed 25 April 2018
hpc openstack infrastructure deployment baremetal
Kayobe is an OpenStack deployment
tool I started at the beginning of 2017. Things have come a long way since
then, and this article aims to provide an update on where the project is
currently, recent changes, and where we expect it to go next.
Testing & Continuous Integration (CI)
Over time we built up a set of CI jobs for Kayobe in TravisCI. It's an
impressive service - they provide free CI for any public repository on Github,
with pull request integration. A full control plane deployment test remained
elusive however. TravisCI provides only Ubuntu 12.04 and 14.04 as test
instance images, and these releases are no longer supported by OpenStack. We
tried installing Kayobe in a Docker container, but continued to hit issues.
While moving to OpenStack's infrastructure, I started reading up on Zuul v3.
The latest release of the project uses a simple in-repo job definition syntax
based on YAML. Jobs are implemented using Ansible, and a form of inheritence
is supported that allows jobs to be built up in layers and reused. Zuul is
currently going through the process of becoming a project in its own right -
still under the OpenStack Foundation umbrella, but separate from OpenStack.
Hopefully this will increase adoption of this great tool.
At the time of writing, Kayobe has 10 Zuul jobs, including one for deploying a
single node control plane, and one for deploying a seed host. We still have a
way to go to improve test coverage, but these jobs have already shown their
worth and should allow us to move forward with greater confidence.
Here are some of the work items in the pipeline:
A feature that I'm particularly excited about is
supporting extension points
for custom behaviour.
This will allow users to integrate their own Ansible playbooks and roles with
Kayobe, and configure them to run at specific points during existing workflows.
This will allow the core of Kayobe to remain small without limiting the
flexibility of the tool.
I attended the snowy Project Teams Gathering (PTG) in Dublin along with other
folks from StackHPC. During the Kolla sessions
I presented Kayobe to the team. There
was interest from a number of parties, from wanting to try out Kayobe, to
considering how the two projects could collaborate in future.
I also gave a video update on the project while in Dublin. Thanks to Rich
Bowen from Red Hat for putting these interviews together.
The following demo runs through deployment of an OpenStack control plane in a
Vagrant VM, using the Kayobe development environment tooling. The commands
were taken from the
Kayobe development guide, and it
should be possible to replicate the demo on a modern laptop.
The asciicast has been edited to limit idle time to 0.5 seconds, and playback
is in double time. The real time video is available
here, and the single time, idle time limited
video is available here.
Manchester OpenStack Meetup
Thanks to the Manchester OpenStack meetup for allowing me to
speak on Kayobe
recently. Always interesting to meet new folks in the community, and great to
see OpenStack remains strong in Manchester. The talk was not recorded, but the
slides are available.
Kayobe's History: In the Beginning
Like many projects, Kayobe started life as a pile of shell scripts. These
provided automation of
Kolla Ansible to deploy the control
plane for the Square Kilometre Array's (SKA) Performance Prototype Platform
(P3). It soon became apparent that a more capable automation tool than bash
would be required, and I switched to my old friend Ansible.
Before long, the core of Kayobe as it currently stands began to emerge, filling
several gaps in Kolla Ansible:
control plane server provisioning via
Bifrost (similar to TripleO's
undercloud) configuration of the control plane hosts
improved support for bare metal compute node provisioning
Widening the Audience
P3 is a High Performance Computing (HPC) cluster, and Kayobe inevitably took on
a HPC focus initially. We quickly realised that the project would have wider
applicability, and made an effort not to restrict its use to HPC.
Kayobe's next opportunity came from
Verne Global, who use it to deploy OpenStack for hpcDIRECT, their HPC-as-a-service
offering powered by green Icelandic energy. Thanks to the team at Verne for
their early adoption of Kayobe, allowing the project to really thrive.
I'd like to thank Kevin Tibi (IRC: ktibi) and the folks at
Devoteam for seeing Kayobe's potential early on while
building their CI/CD platform. This lead to the addition of support for
virtualised compute, greatly widening the potential audience for Kayobe.
A Viable Alternative to TripleO
We feel that Kayobe has become a viable alternative to TripleO for users
wanting to deploy an OpenStack private cloud. It provides an end-to-end
solution for provisioning, reconfiguring, and upgrading the control plane.
It also provides several advantages over TripleO, including:
Ansible for everything vs. many technologies in TripleO
standalone Ironic vs. full undercloud in TripleO
benefits of containers without using kubernetes (which is on the TripleO
roadmap) - one less turtle to worry about!
separation of concerns - Kayobe for provisioning, Kolla Ansible for
containerised OpenStack deployment
make targeted changes, e.g. reconfigure or upgrade a specific service vs.
full overcloud Heat stack update in TripleO
set any OpenStack config option via Kolla Ansible
run custom Ansible playbooks & roles (coming soon!)
There are many ways to get involved with Kayobe:
That's all for now, thanks for reading.