Triton DataCenter, the open source orchestration software that powers Joyent's public cloud and private clouds worldwide is largely built on microservices written in Node.js, but it didn't start out that way. That architecture is a result of Joyent's focus on innovation, continuous improvement, and maximized reliability. Joyent achieves those goals not just with its software architecture, but with a suite of tools it has developed to support rapid fault detection, debugging, and post-mortem analysis for Node.js.

Read on to learn how Joyent came to embrace Node.js and microservices, as well as develop that tooling.

The monolith

To help manage its public cloud infrastructure, in 2009, Joyent built an API that was accessible over HTTP. This API was written using Ruby in a codebase that continued to expand. As new features became necessary, they were added to the existing API project. The project was named MAPI (Master Control Program API), and all of the code resided in a single git repository. Very quickly, the team had built a monolith, which brought with it many difficulties.

Working inside of a monstrously sized codebase brings with it equally sized struggles. A common issue that engineers face is a result of the tight-coupling that often exists in a monolith, which is fixing one problem only to cause a dozen more. In addition to the cascading bugs, there were sometimes cascading disruptions whenever a method in the API changed. The reason for these disruptions was a result of people relying on a method that they shouldn’t be or not informing the right parties that they were utilizing their functionality. Therefore, whenever a method would change, methods that were previously unknown were negatively impacted.

Fixing issues were made harder by the monolithic nature because being a monolith meant that code was shared all over the place — even where it shouldn't have been. - Josh Wilsdon

On top of the issues with maintaining a monolith, Joyent also required tools to make interacting with the cloud API simple. However, Ruby wasn't the prime candidate to use to create the CLI tools that were necessary. Furthermore, the components that comprised the monolith didn't scale and didn’t perform as well as they needed to. As a result of all of the above, Joyent decided to abandon Ruby and the monolith, and instead to adopt Node.js and a microservices architecture.

Adopting Node.js

The choice to adopt Node.js at Joyent was a matter of choosing the right tool for the job. Node.js was a technology selection. With that choice and the criticality of these systems to the business, the decision was made to invest in every way necessary to ensure the success of their efforts. Fortunately, Joyent and the Node.js community were in alignment regarding the goal to provide a free and open Node.js to everyone. As a result of the alignment and commitment to Node.js, Joyent was able to hire the Node.js creator, Ryan Dahl, and purchase both the copyright and trademark — really every way one can invest in an open source technology aside from hiring additional resources to make contributions and implement best practices, which Joyent did as well. With this massive investment, Joyent became the corporate steward of Node.js, and so began Node.js’s rocketship ascent in popularity and dramatic improvements in the stability of Node.js itself and all of Joyent’s product and services built on top of it.

Choosing to bring in a new technology into an organization also helps to attract people who love the new technology. Therefore, alongside Ryan Dahl, Joyent was able to attract and hire additional developers to help with Node.js, including Isaac Schlueter, the creator of npm, and Bryan Cantrill, Joyent’s CTO. At the time, Bryan published on his personal blog some of the reasons why he was excited to join Joyent; the following is one of the reasons that stands out, which relates to Node.js.

While I am excited about Node.js, it’s more than just a great idea that’s well executed — it also represents Joyent’s vision for itself as an innovator up and down the stack. One can view Node.js as being to Joyent what Java was to Sun: transforming the company from one confined to a certain layer into a true systems company that innovates up and down the stack. - Bryan Cantrill

With the decision made and the people in place, Joyent was ready to embark on the journey to rewrite their cloud services and to further utilize Node.js in new and exciting ways.

Decomposition of the monolith

The engineers at Joyent began breaking apart the Ruby monolith into smaller and smaller Node.js services. With Node.js, they were able to develop high-performance services for instrumenting the public cloud. At the time, the term microservices wasn't in use, but that is exactly the kinds of services constructed.

Joyent engineers chose to implement a microservices architecture, in part because it’s the type of system they were already used to building. Engineers at Joyent have enjoyed a fruitful history of designing and building modular systems that embrace the Unix Philosophy. As such, they are accustomed to creating micro-purpose services, and much prefer that type of system over the monolith.

Node.js lends itself well to microservices given its modular structure and small footprint. The smaller codebase for each service that resulted from the shift to Node.js and microservices brought with it numerous benefits. One new benefit was that the resulting services were more easily controlled and maintained. In contrast to the monolith, having fine-grained control over each service meant increased visibility for the consumers of the services, which resulted in few, if any, disruptions. As a result, the overall reliability of the system improved significantly.

Alongside having fewer disruptions, services are now easier to maintain without causing cascading issues. The microservices that Joyent created are better architected and more loosely coupled than their predecessors. As a result, changes to one service are easier to grok and are less likely to impact other services. Much of this behavior is due to the services being small in size and the boundaries between the services being clear.

Lessons learned from the transformation

At the time when Joyent chose to adopt Node.js, there was a lack of proper tooling that existed to support post-mortem debugging and observability of a Node.js process. This lack of observability meant that troubleshooting production issues were difficult when using Node.js. Therefore, in 2011, Bryan Cantrill added DTrace support to Node.js. Further, other Joyent employees created Node.js modules and documentation to contribute to making applications more observable. Similarly, Joyent built mdb_v8 to aid with post-mortem debugging of Node.js applications using MDB. Having mdb_v8 helped Joyent find and fix a memory leak in Node.js itself that was impacting Walmart's preparedness for the paramount Black Friday and Cyber Monday online and mobile shopping days. It’s widely speculated that had Walmart fallen back to Java before or during these peak traffic periods, it would have been the end of their Node.js adoption, which has been a flagship use case in today’s widespread Enterprise adoption.

Beyond Node.js itself, there were additional issues that Joyent had to overcome as a result of the nature of microservices. A major hurdle is how to trace and observe service requests that are spread out across multiple systems. To help with this difficulty, Joyent adhered to the OpenTracing specification and added better tracing into services.

Along with the move to microservices and Node.js, there has also been an increased adoption of Linux containers to deliver these services. Unfortunately, there was a lack of post-mortem support for Node.js running on Linux. Therefore, Joyent added better post-mortem debugging support for Node.js executing on Linux.

Benefits of adopting Node.js and microservices

Switching directions and adopting both Node.js and microservices brings numerous benefits to Joyent. Engineers can iterate and deliver new services faster than they previously could. As a result of being able to deliver more than before, Joyent was able to rapidly construct and ship two new complex systems, Manta and Triton. These software products and services were only possible because of the investments Joyent made to improve Node.js and the support for it.

Joyent was able to add further value to their microservices by creating CLI tools with Node.js to make interacting with Triton and Manta from the command-line straightforward.

CLI tools and HTTP services are typical use-cases for Node.js. Joyent didn’t stop there; they pushed the use-cases for Node.js and built numerous solutions using it in areas that are uncommon. The following list is some of the unusual places where Joyent chose to use Node.js. Each solution uses Node.js for various reasons, most notably for how well it works with JSON configuration, API services over HTTP, and how lightweight it is.

• NFS server
• DHCP server
• Compute node NIC agent
• Container Name Service
• Bootstrapping agent
• Log and data archival system
• Telemetry agent,
• Key/value store.

In addition to the above solutions, Node.js is in use throughout Triton. Below is an architecture diagram that represents the components that comprise Triton and marks those components that rely on Node.js using the color green.

There are many new and exciting places that Node.js fits perfectly. After the migration to microservices using Node.js had gone so well, it was easy for engineers at Joyent to see other problems that Node.js can solve. Because so many of the products and services that Joyent relies on incorporate Node.js, they are furthering their investments in Node.js and microservices by hiring additional Node.js experts.

At Joyent, there is a love for open source software and open-source communities. Therefore, the tools that are created to help with Node.js support are open-source and freely available. Being open-source is a direct result of Joyent's desire to empower the Node.js community to be able to write better software and debug issues when they occur. Similarly, Joyent produces and publishes many videos and best practices to help engineers who are building software with Node.js.

Joyent continues to evolve and innovate in areas that are beneficial to software engineers who enjoy the use of Node.js and microservices. Therefore, expect to see even more valuable open-source tools and products in the future!