Editor's note: Richard Kiene built his career developing software and running ops in telecomms and startups before joining Joyent in mid 2015. Like a number of our recent hires, Richard got to know Joyent from our open source data center automation software. Richard previously built a component of Triton Container Name Service (screencast) and is now building Triton Container Monitor.

I've been toying around with the idea of writing a post about the cultural and mental shift I've undergone in the last year and a half at Joyent for quite some time. The biggest transition being that I've gone from "ship ship ship, even if it is shit, ship ship!" to "get it right, preferably the first time."

In my opinion, the emphasis on upfront planning, architectural review, academic-ish rigor, and fully understanding each problem set is what most people coming out of a university with a Computer Science degree would like to be doing right off the bat. Unfortunately, most people aren't lucky enough to fall into a place like Joyent immediately and are subsequently subjected to cultures which don't value these things at all. Consequently, incentives shift from what brought a lot of people into Computer Science (e.g. solving hard problems, designing, making a difference, etc.) to simply getting the next feature out the door without regard for long-term quality. The ship ship ship culture breeds people who work unnecessarily long hours and deal with a perpetual mess.

In my first year at Joyent, I had to battle a monkey on my back telling me I wasn't doing a good job because I wasn't putting something in production every day. I had to re-train myself to value getting it right the first time over ship ship ship. The result is a better quality of life, better quality output, one feature in production for about a year without incident, and a monitoring and metrics architecture that is in early beta.

I have to admit, shaking the ship ship ship mentality was difficult. I frequently found myself getting frustrated with conversations that felt like a waste of time. These were conversations that previously would have been deferred until the project was minimally complete. However, perhaps due to lots of growing up, I realized that the culture at Joyent necessitated these hard conversations about scale, future usage, debuggability, observability, de-risking, and failure modes. So I kept an open mind (after all I took this job in part to learn from the best and to grow) and embraced the culture.

What I found was that the conversations, rigor, design process, etc. all resulted in solving problems during the design process that normally would have been solved after realizing the shortcomings in production. The biggest difference is that I, the engineer, pay the price up-front for my decisions rather than foist them on to the operations and support teams while I "figure the rest out." To be clear, I'm not saying that I previously put no effort into designing and scaling my solutions. I'm saying that much of that effort was done informally, often after the first cut was shipped. This was in large part caused by the ship ship ship culture pressuring me to get my work out the door as fast as possible.

At Joyent we have a process called "requests for discussion," or RFD for short, that prevents such informal design processes. This process is modeled after the IETF RFC process but scoped to our product. Anything which might take a significant amount of engineering effort (two weeks is generally considered the threshold) requires an RFD to be written and some form of consensus within the team before any earnest code-based engineering takes place. Additionally, we use Gerrit to gate commits from going directly to the shipping branch. Code must pass an automated make check and receive a +1 for code review and integration approval from one or more members of the engineering team before it can reach the shipping branch.

This might seem like a ton of red tape to prevent progress, but I assure you it is not. Throughout my journey, I've continually been surprised by how many problems have been solved upfront because of these processes. Many times I've reached what I believed to be an impasse while writing my code, only to look back at my RFD and realize I'd actually solved the problem and documented it. These processes allow our team to help each other ship the best code we can and agree upon what we are going to do before any code reaches production.

For me, one of the things that make these processes work is that I'm not dealing with a bunch of political infighting. It's not that we're without politics and people with a competitive nature. It's that I'm able to trust that my coworkers' critiques come from a genuine desire to make others better and to produce the best product we can. This makes it far easier to take criticism via code review or the RFD process, knowing that the other person isn't simply against you.

Furthermore, it is quite helpful that I'm working with a team of very accomplished engineers. This aids me in checking my ego at the door. I don't need to feel like I'm conceding ground by admitting I don't know something. Instead, I can ask the dumb questions, learn, and improve myself and the product.

Overall my quality of life, quality of work, and the amount of learning has increased substantially since joining Joyent. I feel like this is exactly the kind of job I wanted to do after I earned my degree, but for whatever reason, I didn't find my way here until much later on.

There is a ton more to this topic, including mean time to failure vs. mean time to repair, arguments based on merit vs. arguments based on politics, etc. This is not meant to refute CI/CD which are completely appropriate tools in certain products and organizations. Perhaps in follow-up posts, I'll address the different production burdens that can necessitate or benefit from a different shipping cadence, but the overall rigor that we exercise is still valuable either way.

My hope is that this post will inspire others caught up in the ship ship ship world to challenge themselves and look for opportunities that let them be proud of what they create.

Editor's note: We're hiring. Apply to join the team.