Building high-quality, native mobile apps can require significant timeand cost, and require skill sets that enterprises don’t have in house.The resulting apps can be difficult to manage, update, and secure.

While many organizations have deployed their “hero” customer-facingmobile app, very few have made progress migrating their in-houseenterprise and line-of-business apps to mobile.

Microservices built with Node.js were already being used as primarymobile backend APIs. However seamless integration between nativefrontend and backend apis has been challenging and time consuming. Onthe flip side, full stack JS solutions (like MEAN) lacked the nativeoptimization needed for mobile apps.

In this case study, we showcase a path breaking solution fromSynchro Labs for building enterprise readynative mobile apps, leveraging JavaScript, Node.js and Containers as theleading technologies and deployed on Joyent’sTriton Container Native Cloud forhigh performance and cost optimization.

The Synchro platform allows enterprisedevelopers to create high quality, high performance, cross-platformnative mobile applications using simple and familiar tools - JavaScriptand the Node.js framework. Apps created with the Synchro platforminstall and run natively on mobile devices, but all application code,including client interaction logic, runs in the cloud under Node.js.

The Synchro Solution

Synchro offers a solution that makes it easyfor either front-end web developers or back-end Node.js developers touse their existing JavaScript/Node.js skills to build mobile client appsthat present as native apps to end users, but are deployed and managedlike web apps.

With Synchro, you get:

• High quality native mobile client apps
• Very short learning curve
• Significantly less code to write/manage
• Applications can be updated in real time
• Security is baked-in/automatic

What Makes Synchro Unique?

There are many options in the mobile app development platform space,including HTML5 based solutions, Phonegap/Cordova solutions,Appcelerator, Xamarin, React Native, Ionic, and others.

The main difference between Synchro and all of these technologies, isthat with Synchro your mobile application code lives and runs on theserver. That gives Synchro tremendous advantages in ease of integration,developer productivity, manageability, and security.

Technology stack

Synchro solutions consist of a Synchro Native Client app or apps, and aset of server technologies referred to collectively as Synchro Server.The Synchro Server core technology stack consists of Node.js, theSynchro Server Node.js app and related services, and a set of Node.jspackages used by Synchro. Please read more about our technology choiceshere

Solution Architecture

The diagram below represents the major components of the technologystack and their relationships to each other:

While this diagram shows the major architectural elements, it does notaddress operational or deployment considerations.

For example, the Synchro Server App running under Node.js will typicallyneed to scale to more than one instance, meaning that it will need tosit behind a load balancer of some kind, and be controllable by anorchestration/scheduling solution of some kind.

As a solution vendor, it is of paramount importance that our solution beable to run in any deployment environment or solution architecturerequired by our customers. The internal support for a variety of sessionstate and storage solutions is a good example of the kind of flexibilitythat has been engineered into Synchro. We show those generally as“Session State” and “Storage” in the diagram above, but those willtypically connect to either platform services provided by your managedservice platform, or to containers that you provide in a container-basedsolution.

Deployment considerations and reference architectures for deploymentwill be discussed below.

Managed Services Deployment

We originally deployed our own production Synchro server on MicrosoftAzure as an Azure Web Service, and we relied on various managed servicesavailable within the Azure environment for everything else. Here is whatour production server deployment looked like on Microsoft Azure:

While this solution was workable, it had some downsides. Among otherthings, we had a lot of DevOps issues, and it was very difficult toestablish and maintain staging environments for test or dev (it could bedone, but it wasn’t fun).

We knew that a container-based solution would be attractive tocustomers, and we wanted it for ourselves so that we could have someplatform mobility for our own production Synchro server.

Containers 1.0 - Basic Container Support

Back in February we blogged about Deploying Synchro usingDocker. Atthat time we created a Dockerfile which we began bunding in ourinstallation to make it easy to package an installed, configured Synchroinstallation into a Docker image.

That Dockerfile looked like this:

FROM node:argon## Create app directoryRUN mkdir -p /usr/src/appWORKDIR /usr/src/app## Bundle app sourceCOPY . /usr/src/app## Install depsRUN npm installRUN cd synchro-apps && npm installENV SYNCHRO__PORT 80EXPOSE $SYNCHRO__PORTCMD [ "node", "app.js" ]

While this was a good start and got us experimenting with runningSynchro in a container, it was a long way from a complete solution. Wedecided the best way to arrive at a true container-based solution was toeat our own dog food by deploying our own production Synchro API serverin containers.

Containers 2.0 - Self-Orchestrating Microservices

When we started the process of containerizing our production API server,we quickly learned that there is a lot more to containerizing a solutionthan just wrapping your app in a Docker image. Our app, like many apps,runs as one part of a system of interconnected services. When running ina pure managed environment (like Azure), those services are provided foryou, and your interface to/from those services abstracts you from theirimplementation/operations. When running an orchestration ofmicro-services in containers, you are responsible for the entiresolution (creating, configuring, and running all of the micro-servicecontainers and connecting them to each other).

Orchestration solutions, like Docker Compose, Mesosphere Marathon,Google Kubernetes, and others, provide tools to support these complexcontainer deployments and interactions, and guidance for modifying yourapplication to support that tooling. But as a solution providerourselves, we were looking for a solution that supported whicheverorchestration platform our customers had chosen, with little or nofurther integration required.

We had two main goals in undertaking orchestration support:

• Provide a true container-based reference architecture for our customers/users that works with any orchestration platform
• Use that support ourselves to migrate our own production API server from a proprietary, vendor-specific managed service deployment to a open standards, portable, container-based deployment

The AutoPilot Pattern using ContainerPilot

Our friends at Joyent made us aware of asolution that they promote called The AutoPilotPattern usingContainerPilot, which iswell documented here. Notethat while Joyent actively promotes and supports this solution and itscomponents, it is not Joyent-specific (everything involved isopen-source and will work on any platform that can run containers).

This solution delivers "app-centric micro-orchestration", wherecontainers collaborate to adjust to each other automatically (both ascontainers scale, and as they become healthy/unhealthy), without needingany help from your orchestration system. These containers run on"auto-pilot".

All the orchestration solution / scheduler has to do is run yourcontainers, and the containers themselves will self-organize. As asolution provider ourselves, this features was a must-have. Ourcustomers can deploy our AutoPilot containers, from publicly publishedimages, with no changes, regardless of the orchestration system theyuse, and everything will just work. And if a customer changed from oneorchestration system / scheduler to another, no changes would berequired to any Synchro micro-service Docker images.

Synchro AutoPilot Reference Architecture

The Synchro AutoPilot reference architecture is based on Docker andContainerPilot:

Docker: All servicesrunning in the reference architecture run in Docker containers

ContainerPilot:Allows the system to scale automatically regardless of orchestrationsolution (AutoPilot pattern)

In a container-based solution we have to specify and provide allservices required to run the solution. The only solution element that werequire and for which we do not provide a container is storage (allsupported platforms provide storage as a service).

The reference architecture includes the following Docker containers:

Node.js: The Synchro Server itselfruns as a Node.js service

Redis: Application session management(end-user app state)

Consul: Service registry

Nginx: Load balancing, SSLtermination, static file caching

StashBox:Configuration support (env, file system, or storage)

Implementing Micro-Orchestration

The first step in our process was to build a Docker container for ourapp, and to either create or find Docker containers for the otherelements of our solution.

We then built a deployment blueprint that described all of thecontainers and their relationships to each other. We chosedocker-compose to do this, but any scheduler would work here.

Finally, we identified the containers that needed to scale and addedContainerPilot to those containers. In our case, the Nginx proxy and ourSynchro Node.js containers were the only containers that needed to scalein order for our solution to scale.

Our fully containerized solution looks like this:

Note that we will always run two or more Nginx and Synchro containers,and we will run Consul in a raft of three containers, and Redis in acluster of three containers, so a minimal deployment will consist of tencontainers, all working together (automatically). And of course we canscale out Nginx or Synchro as needed to handle load and the system willhandle that automatically.

We aren't going to cover the implementation details of the AutoPilotPattern using ContainerPilot, as Joyent has excellent documentationand many implementationsamples. If you would like tosee our implementation, we have published it onGitHub and you arefree to reference it and use it as desired.

Container Best Practices

We learned some lessons when implementing our apps and services incontainers. If you undertake a similar effort, you should consider thefollowing...

Flexible Configuration

Getting configuration into micro-services is one of the significantchallenges in these types of solutions. With Docker containers, we havethe option to build configuration into the container itself usingenvironment variables or bundled files, or we can inject configurationat runtime using environment variables or Docker volumes. There are prosand cons to each of these mechanisms. As a solution provider, we wantour own solution to be non-opinionated, which is a fancy way of sayingthat we want to support any of these mechanisms (includingcombinations).

To support the broadest range of configuration options, we recommend thefollowing:

• Support configuration via environment variables where possible - In order to support flexible configuration, support environment variables as broadly as possible.
• Any file-based configuration should have a configurable local path with a reasonable default - This is particularly important if you intend to inject configuration using Docker volumes. Having a configurable local file path allows you to map volumes to locations where they will not interfere with other configuration elements or potentially leak other local files from the container to the volume unintentionally.
• Any file-based configuration should be able to get its contents from a base64 encoded environment variable - It can be tricky to pass binary contents via environment variables, and even for multi-line text files, the differences between the support of Docker command line, the Docker env file, and Docker Compose make this more complex than it needs to be. Supporting base64 encoded environment variables is easy to do and works in all modalities of environment configuration (whether build-time or run-time).

ContainerPilot makes is very easy to do the above, as well as supportingflexible configuration in general, with its preStart entrypoint.Following is an example of the ContainerPilot preStart in our Nginxcontainer:

#!/bin/shpreStart(){  : ${SSL_CERTS_PATH:="/etc/ssl/certs/ssl.crt"}  if [ -n "$SSL_CERTS_BASE64" ]; then    echo $SSL_CERTS_BASE64 | base64 -d > $SSL_CERTS_PATH  fi;  # >>> Process ssl.key, any other files as above  # Rest of preStart logic (consul-template, etc)}

With this kind of configuration, we can do any of the following:

• Copy the ssl.crt file into our image at the default location with no environment variables (build-time)
• Populate the ssl.crt file from an environment variable (build-time or run-time)
• Supply the ssl.crt file from a Docker volume to a custom location (run-time)

Embrace the Proxy

Solutions consisting of micro-services will typically require a proxyserver. We happen to use Nginx as our proxy.It is not unusual for comparable implementations to useHAProxy.

While managed services solutions provided much of this functionality foryou, micro-service solutions often require you to do some or all of thebelow:

Load distribution

The primary function of the proxy is to distribute traffic among yourservices. ContainerPilot, through its use of Consul and consul-template,more or less handles this part for you.

If you have session affinity requirements (as Synchro does), you want tomake sure your proxy configuration accommodates those. With nginx we useip_hash for session affinity.

SSL termination

Another significant function of the proxy in micro-containerorchestrations is to terminate SSL connections. This requires basicSSL/TLS configuration, as well as injection of SSL credentials into theproxy container. While this can be a cumbersome process, it does allowyou to tune your SSL implementation for performance and security.

When you have configured SSL successfully, you should run an externalvalidation test like the one from SSLLabs.

You will also likely want to forward non-SSL traffic to your SSLendpoint.

Static file caching

In our managed services deployment, we served all static files from aCDN (Content Distribution Network). Because we did not want to rely ontarget environments having CDN support, we instead implemented staticfile caching in Nginx (with support from Synchro). We found theperformance to be comparable to a CDN, and the deployment/maintenancecomplexity to be much lower with this approach.

Websocket support

Not all proxies that support http traffic will support WebSockets out ofthe box. Nginx in particular requires specific configuration to supportWebSockets.

Proxy Config in Practice

Our Nginx configuration template does all of the above. See: SynchroAutoPilot nginx.conf.ctmpl onGitHub

Prepare Your App

You application may require some modification in order to bewell-behaved in a ContainerPilot deployment:

Health check

You should implement a health check if your app does not already haveone. We recommend that the health check endpoint be configurable in yourapp. We also recommend that you take care not to expose the health checkendpoint unintentionally (your app can check to see if the connectinghost is the local machine, or you can have your proxy deny access to thehealth check from external clients, as appropriate).

Signal handling

Your app should implement signal handling in order to be well behaved(and responsive) in an orchestrated/scheduled environment. The Node.jsdefault handlers, in particular, are not well-behaved. You shouldhandle:

• SIGINT - Shutdown
• SIGHUP - Reconfiguration (if your app can reconfigure itself, such as may be required by ContainerPilot if your app uses resources that may change over its execution)

Static file cache support

If your app serves static files, it probably already has a cachingsupport implementation in place. Since our app previously relied on aCDN, we had to add static file caching support (our proxy now handlesthe cache, but our app still had to write the appropriate cacheheaders).

Our New Reference Architecture: Synchro AutoPilot

The Synchro AutoPilot solution discussed above is now our recommendedreference architecture. As previously noted, the implementation isavailable onGitHub. You are freeto review, use, and modify it as desired. We have also published theimages from our solution onDockerHub. Note that our ownproduction API servers use these exact images. If you are deployingSynchro in a container environment, we recommend that you use these sameimages.

Deploying On Joyent

Once we came up with our new container-based reference architecture, weneeded to migrate our own production Synchro API servers to acontainer-friendly cloud service provider. We chose Joyent.

Why Joyent?

Joyent is a very natural fit for Synchro, both as a home for our publicAPI server, and as a hosting solution for customers deploying Synchro tosupport their own mobile apps. While Joyent excels in many areas, theyare uniquely ahead of the pack in:

• Node.js leadership
• Docker integration
• Native containers (best price/performance)
• Thought leadership (ContainerPilot is a great example)

Joyent Deployment: How To

If you have Docker installed on your local machine and you have a Joyentaccount, it’s very easy to deploy containers to the Joyent cloud.

First, install the Triton CLItool, thenconfigure Docker forTriton.Once you’ve done that, you can use the standard Docker CLI commands,including docker-compose, to manage containers on the Joyent publiccloud exactly like you were running them locally. To deploy ourproduction Synchro API server solution, we simply did:

docker-compose up

And our service was up and running on the Joyent cloud. You can then useeither Docker commands or the Joyent public cloud portal to monitor andmanage your containers.

The final step in our case was to enable Triton Container NameService (at theJoyent account level, via the portal), then configure our Nginx proxyinstances with the appropriate tags/labels.

Our deployment on the Joyent cloud looks very similar to our standardreference deployment:

Synchro Labs Production Deployment on Joyent

The Synchro public API server is currently deployed on the Joyent cloudvia Triton andManta, using the referencearchitecture discussed above. This deployment supports the SynchroCivics application (accessible from the Synchro Civics mobile appavailable in all app stores). It also serves Synchro Samples, whichallows customers to evaluate various aspects of the Synchro platformusing the Synchro Explorer mobile app (also available in all appstores).

Comparing Solutions: Azure Managed Service vs Triton Orchestrated Containers

Cost

Note: Our own Synchro API server doesn’t operate at a large scale orrack up a lot of cost (it only needs to support a few thousandsimultaneous users). That being said, our cost calculations are providedhere.

We were initially concerned with the cost of running all of thecontainers that our new solution required. Below we show the cost of ourprevious managed solution versus the cost to run our new orchestratedcontainer solution on Joyent Triton:

You can see that even with running 12 containers in our baseline/minimalcontainerized configuration, our Triton cost is less than 50% of ourprevious Azure solution that only had a single managed app container.

That being said, the pricing is not apples-to-apples. With Azure, thereare many arbitrary limitations put on your instances that force you tobuy capacity that you don't need. For example, we did not need nearly asmuch memory or processor as the "S1 Standard" image costs on Azure, butthe smaller images that would have been more appropriate had arbitrarylimitations that made them unusable (poor networking, maximum number ofinstances that preventing scaling, etc).

So part of the cost improvement is related to being able to pay forexactly the right size container on Triton (they provide a very widevariety of container profiles with no arbitrary limitations). But partof it is that Triton is just less expensive (the Triton"g4-general-4G" is comparable to the Azure "S1 Standard" except that ithas more than twice as much RAM, and it's only 65% of the cost).

Performance

We have not done any detailed end-to-end performance testing. We did dosome initial smoke testing, and we profiled some specific areas ofconcern (our app is very dependent on low-latency, and on fastintra-solution transactions between the app and Redis). For everythingthat we did test, we saw marked improvement in performance. And ourapp "feels" faster, for whatever that's worth.

We can say with some confidence that the performance didn't get worse,and particularly with the cost improvement, we would have been happywith that.

Service and Support

In using Azure for the past year plus, we have suffered from being asmall customer dealing with a giant service provider. We didn't run intotoo many issues, but when we did, support was non-existent. The model isessentially that to get reasonable support you either have to be a verylarge customer, or you have to subscribe and pay a non-trivial amountfor the right to access support. This was not much different with ouruse of Amazon AWS services in the past (except that Amazon had muchbetter self-support and peer-support, so it was easier to solve your ownproblems if you were willing to put in the work).

Our experience with Joyent has been a breath of fresh air. We had acouple of operational issues with our account provisioning and thesupport tickets were handled almost immediately with a high level ofprofessionalism/competence. We opened a couple of GitHub issues againstContainerPilot and got thoughtful feedback from the maintainer (a Joyentemployee). We even asked in advance if we could get a technical backstopin case we ran into porting issues, and an SE was assigned to us (andthey knew in advance that we weren't going to be spending a lot of moneywith them in the near term). Joyent just seemed top-to-bottom to becommitted to making sure we were successful.

DevOps Benefits

Environmental: A major, unexpected benefit of moving to aorchestrated container solution is that we can now simulate ourproduction environment in development with a high degree of fidelity,and we can actually directly use our production configuration and statefrom staging environments before deployment. For example, we can set upa deployment environment that points at the Joyent Docker daemon forproduction, and set up the exact same environment (same directory, sameenvironment variables, etc) pointing to a local Docker daemon, allowingus to create the exact same Docker environment for local/dev orremote/prod deployment. This is something we couldn't ever really dowith any confidence on Azure.

Portability: Because ContainerPilot is open source and will runanywhere containers will run, and because all of the elements of oursolution now run in containers, we could port this solution to anotherprovider very quickly and easily (it took two weeks to port from Azureto Joyent, but we're confident we could redeploy our current solution onanother provider in matter of hours at most). We are currently usingDocker Compose as our scheduler with Triton, but again, we could changeto a different orchestration solution / scheduler, even on a differentcloud provider, very easily and with a high degree of confidence.

Summary

We are very happy with our new AutoPilot implementation of Synchroand its components. It has helped us consolidate our application stateand move to a production environment that is a much better fit for us(in terms of cost, performance, tooling, environmental fidelity, andother factors).

We are confident that our AutoPilot support will make it easier forcustomers deploying Synchro to integrate it into their operatingenvironments, regardless of their cloud provider or orchestrationsolution. And it means that we are now able to offer a referencearchitecture that is a true turn-key solution, instead of just an"integrate it yourself" Docker container.

Appendix

Synchro Technology stack

Synchro solutions consist of a Synchro Native Client app or apps, and aset of server technologies referred to collectively as Synchro Server.The Synchro Server core technology stack consists of Node.js, theSynchro Server Node.js app and related services, and a set of Node.jspackages used by Synchro.

Synchro Native Client

Synchro has native clients on the iOS, Android, Windows Mobile, andWindows platforms. There is a general purposed version of the Synchronative client called Synchro Explorer, which will run a Synchro apprunning at any provided endpoint.

When a customer is ready to deploy their own branded native mobile appto end users, they supply their app name, icon, and server endpoint tothe Synchro App Builder. The resulting app can then be distributed viaapp stores or any other desired mechanism. Note that this app containsno customer application code (only a reference to the server endpointwhere that code is running).

Node.js

Synchro Server and its components run under Node.js. In addition, appswritten using Synchro also run under Node.js. This allows Synchro appdevelopers to build mobile apps with full access to all of the Node.jspackages they know and love.

Synchro Server App

A top-level Node.js server application that distributes requests to oneof the following support services:

• Synchro API - Core Synchro app server for Synchro apps
• Synchro Web - Web client for Synchro apps
• Synchro Studio - Web-based IDE for Synchro apps

Note: Synchro Web and Synchro Studio are both optional.

Additional Services

Synchro also relies on Session State and Storage support, typically fromexternal services (whether provided by the environment, or deployed asseparate servers/services as part of the Synchro deployment).

Key Supporting Node.js Packages

Asynchronous Operations: co and async - provide support forasynchronous operations, both of Synchro itself, and of Synchro apps

Storage: pkgcloud (access to storage on Amazon S3, Azure, Google,HP, OpenStack - including IBM BlueMix, and RackSpace) and manta (accessto Joyent Manta storage)

Session State: redis - access to Redis for application sessionmanagement

Logging: log4js - provides logging services from all Synchrocomponents and services, highly configurable