Production Practices

Node.js debugging best practices

Debugging Performance Issues

Performance problems are best resolved with a methodical approach. Except in cases of clear regressions and other simple situations, resolving poor performance starts as a system-wide debugging affair. It’s important to be able to narrow scope quickly and definitively.

So while strictly not a specific Node.js practice, the Utilization, Saturation, Error (USE) Method is recommended. The USE Method is a simple strategy you can use to perform a complete a check of system health, identifying common bottlenecks and errors. It can be deployed early in the investigation and quickly identify problem areas, which then can be studied in more detail other methodologies, if need be. The strength of USE is its speed and visibility: by considering all resources, you are unlikely to overlook any issues. It is developed by Joyent Performance Engineer Brendan Gregg.

Performance Methods and Checklists

The checklists below break down common systems into components (CPU, Memory, Network, Storage) and specify what tools are available for, say system-wide CPU utilization as a command-line tool.

• USE Method Utilization, Saturation, Error
• SmartOS

               Comes with DTrace by default

                mpstat, kstat, sm-cpuinfo, zonememstat, dladm, etc

• Mac OS X

               Comes with DTrace by default

               iostat, top, uptime, vm_stat, netstat, iostat, iosnoop, etc

• Linux

              vmstat, dstat, mpstat, sar, iostat, etc

• System Performance, the book, for the deepest debugging.

Debug a Running Process (DTrace)

DTrace is a dynamic tracing framework with the ability to trace from the kernel all the way through your application’s code. It’s extremely powerful, but most importantly, safe to use in production.

You instrument your application by enabling predefined probes in the kernel (for instance for syscalls like read() and write()), with probes you defined at compile time for your application, or with probes defined at runtime for your application. This visibility into your applciations performance is a crucial tool to effectively practice the USE method.

Node comes with a series of built-in probes to instrument common events throughout your application.

• gc-start, gc-done
• net-server-connection, net-stream-end, net-socket-read, net-socket-write
• http-client-request, http-client-response
• http-server-request, http-server-respones

By instrumenting these probes you can gain insight in the patterns of your application by observing the latencies of the common tasks you perform. For an example of using these static probes look at nhttpsnoop, which allows you to see accounting of HTTP requests and responses coming into and out of your application.

By using the dtrace-provider module you are able to define DTrace probes at runtime for your Node application. These probes come with little to no overhead for your application, and allow you to customize very quickly what information you will want to observe about your application while it’s running. See bunyan and restify for examples of how these runtime probes can be leveraged.

You can also use other probes built into the system for more sophisticated analysis. For example, you can trace malloc or operator new and get a count of allocations by stacktrace.

Cloud Analytics

Cloud Analytics is a tool for real-time performance analysis of production systems and applications deployed in the Triton Cloud. Built on DTrace, Cloud Analytics provides extraordinary visualizations into how your application is performing, as well as being able to correlate that information with your other related services.

Using Cloud Analytics you’re able to observe latencies throughout many processes running in your environments, from filesystem and network I/O through the static probes defined for Node. You’re able to visualize your performance characteristics through traditional graph formats, as well as powerful heatmaps which helps identify outliers instead of only viewing the average latency of your application.

Read More

Profiling your application

Being able to determine where your application is spending its time is critical to successful operation. Using DTrace, its profile probes, and the ustack helper you can get that information.

Read more about profiling Node.js

Profiling your add-on modules

Your application may use binary add-on modules for connecting to things like databases, external resources, or integration with existing business logic. Under those circumstances you may have need to quantify the latencies for execution of those interactions. Using just a little bit of DTrace know-how you can see the distribution of latencies for arbitrary methods that queue work in the Node event queue.

Read more about tracing latency in add-ons.

Stuck Process

DTrace can be used for more than just data collection, including debugging non-fatal buggy program behavior. DTrace can instrument very specific events on the system (like “the close syscall in a Node process returned -1 with EBADF” or any application-level event), and you can use that to take action like stopping the process to take a core dump and then resuming it again. Putting these pieces together makes it possible to gather all kinds of debugging information from a program that’s behaving badly.

Read more about this technique.

Postmortem

The technique of determining the root cause of a programming error after a process has died is known as postmortem debugging. This is an important technique to practice regularly, as being able to quickly and accurately diagnose the cause and not the symptom of a failure will mean fewer iterations of these procedures. Fewer iterations of debugging will mean faster development cycles and thus having confidence in your application logic sooner.

To ability to successfully diagnose the root cause of a software problem after a crash depends on the amount and quality of information you have at your disposal. Often quality of information will trump quantity. Depending on the design of your application you may want to be able to garner the state of as many inter-related processes at the time of the crash. Then you’ll need to be able to correlate that information into something actionable.

Aside from quality logs, it’s important to be able to capture as much of the state of the process that died as possible. A stack trace is a good start, but is probably not enough information to be able to act quickly to solve a problem. So along with the stack trace of where the process was executing, you want a dump of all the memory that was wired at the time of the process’s death. This is functionality is often provided by the operating system, and is referred to as a core file.

Not all environments enable core file generation by default, so make sure that when deploying you’ve enabled those features, a simple ulimit -c unlimited in the shell where the process will be running can be sufficient. On SmartOS-based environments you are also able to generate a core file even if the process hasn’t died, you can do this with gcore.

Postmortem Debugging with mdb

mdb is the modular debugger, it can be used to debug running processes or with core files. Core files can be generated in one of two ways, your process abort()d, or you used gcore to generate a core file for a running process.

Generating core files for a running process is a very powerful technique, doing so allows you to inspect your applications state before something has gone wrong. Also, if you’re trying to debug a memory leak having multiple core files allows you to track the changes in the counts and kinds of memory being kept around.

Your Node process may abort() for a variety of reasons, from a programming error inside Node itself, or by programmatically invoking abort directly to ensure a core is generated. There are two ways you can programmatically induce your Node process to abort, the first is process.abort which you can call at anytime in your code. The other is to run your Node process with --abort-on-uncaught-exception which tells V8 that if your process encounters an uncaught exception it should generate a core file.

mdb is the modular debugger, it can be used to debug running processes or with core files. Core files can be generated in one of two ways, your process abort()d, or you used gcore to generate a core file for a running process.

Generating core files for a running process is a very powerful technique, doing so allows you to inspect your applications state before something has gone wrong. Also, if you’re trying to debug a memory leak having multiple core files allows you to track the changes in the counts and kinds of memory being kept around.

Your Node process may abort() for a variety of reasons, from a programming error inside Node itself, or by programmatically invoking abort directly to ensure a core is generated. There are two ways you can programmatically induce your Node process to abort, the first is process.abort which you can call at anytime in your code. The other is to run your Node process with --abort-on-uncaught-exception which tells V8 that if your process encounters an uncaught exception it should generate a core file.

::findjsobjects

The ::findjsobjects command in mdb is a crucial utility for determining the current state of active objects found in the V8 heap. Running without any parameters will give you a list of representative object addresses, their counts, the number of properties, the constructor name, and the shape (the enumerated order of the properties) for the objects.

While that in and of itself is helpful for tracking down memory leaks, you can also iterate over those objects and find who is keeping a reference to them.

Read more about debugging memory leaks.

::jsstack

While operating in mdb on a core file from Node, you can use the ::jsstack command to determine from the entry point through JavaScript frames where the process was when the core file was created.

That can be quite powerful information, but used in combination with ::jsframe you are also able to inspect the arguments that were passed to the frame as well.

::findleaks

In addition to encountering memory leaks from closures in JavaScript you may find you have to tackle traditional memory leaks of malloc()s without corresponding free()s. To diagnose these sorts of issues run your process with UMEM_DEBUG=default set in the environment. This environment variable will (among other things) begin storing audit information about allocations happening in your process, that information includes the stack trace for the allocation.

When you have a core that was run with UMEM_DEBUG set, you can then in mdb use the command ::findleaks which will give you a list of any leaked memory aggregated by their callstacks.