I talk a lot about multi-tenant container security, that is, the security that prevents accidental or unintentional escapes from a container or incursions into another. For most, the immaturity of the Linux kernel container implementation with respect to secure isolation disqualifies it from running containers from multiple, untrusted tenants.

Most cloud infrastructure providers use virtual machines to isolate customers (tenants) from each other, but Joyent takes a different approach. At Joyent, we run our containers on a purpose-built container hypervisor. That's why we can run containers on bare metal, and we've been using this technology to power our cloud for almost ten years¹.

But "why does this matter," I've been asked. "Is running a container on Triton really more secure than running a single container in a virtual machine," they press. A container on Triton may be more secure than a container in the virtual machine, but that's not the point.

The point is that by providing container security that's at least as good as that provided by a virtual machine², we can eliminate the virtual machine entirely. The VM doesn't just sap performance, it makes running the container more complex and defeats many of the advantages that application containers offer in the first place.

Tying the lifecycle of a container to the lifecycle of a VM is the first mistake. It means VMs have to be provisioned before the containers can be, and scaling down means deprovisioning the VMs and consolidating the remaining containers into the smaller VM surface area. I shared how that almost doubled costs for one company recently.

Container data persistence is far more complicated inside VMs as well. Properly managing the containers with persistent data needs is far easier when you don't have to worry about the lifecycle of a VM around that data. Eliminating the layering of the VM management also makes it easier to keep track of that data as well.

Perversely, the presence of a VM actually encourages data management anti-patterns in the cloud--patterns that make it more difficult to achieve the immutable infrastructure modern apps and operations require. Dropping a disk turd here or there is OK, since you "own" the VM. Again, the problem manifests itself in its worst form when you need to shut down the VM (or when you leave it running and doing nothing just to preserve the data on it).

Ironically, all this additional cost and complexity also comes with reduced performance. The hardware hypervisor tax slows performance, especially I/O, compared to containers running "naked" on bare metal. Implementation details can make this even worse: combine a VM's poor network I/O with a container's NAT and latency and throughput both suffer. This can add up to real and inescapable delays as apps grow to include more services and network interactions.

And, if the containers aren't secure enough to run on bare metal, are your containers secure from each other in the VM? Among many orgs, security concerns require that tasks and data be thoroughly isolated from each other. It's comparatively easy to secure the interfaces we intend to expose to others, but securing the internals of an application is far more difficult or perhaps impossible if you can't trust the security of the operating environment for it. If other containers can sneak into yours, either accidentally or intentionally, there's little that can be done to prevent mayhem. Organizations covered under any of a number of data security regulations and others that are aware of the risks insist on secure isolation that can prevent accidental or intentional crossover between tasks.

Bare metal containers avoid these frustrations, but they're only possible if we can run them securely on multi-tenant hardware. Single tenant hardware is like a fast VM: you have all the costs of infrastructure management and none of the elasticity of the cloud. Running containers in any context where they're not secure is a non-starter for many organizations. Running containers as second class citizens should be a non-starter for all.

So how important is multi-tenant bare metal security? I'd say the future of our apps depends on it. Containerization in VMs multiplies the worst of both, but containerization without security may be worse. Tested and uncomplicated security that works by default, security that isn't dependent on configuration and can't be overridden, is an absolute requirement for our apps. The good news is that it's available now: it's Joyent Triton, and it's open source. What you build next is up to you.