In addition to our role as corporate steward of Node.js, Joyent operates one of thelargest deployments of Node. Because Node is so crucial to ourinfrastructure, we developed state of the art tooling to debug it. These toolsinclude our v8 module for mdb and DTraceustack helpers, which together provide unprecedented observability into your Nodeapplication. They allow you to see all of your application's state withoutmodifying your application, which makes them especially useful in production, whereyou often won't know which piece of application state to expose beforehand.

Until now these tools were available only for debugging Node applications running on ourinfrastructure (SmartOS and other illumos based installs), but we're excited to announcethat you're now able to debug core files from your Linux environments as well.

This is the result of a hackathon that Joyent hosted for all of its engineers thispast November. Everyone spitballed ideas and voted on which projects they wanted to work onfor the next week. The intent was to have a working prototype by the end of thework week. Max Bruning and I decided we wantedto be able to load a core file from a Node.js Linux process and be able to run::findjsobjects on it in mdb. As a result of this project, Joyent was able tolaunch a support offering forthose running Node.js on 64-bit Linux.

These examples all use Joyent's Mantaservice, so if you haven't already, signup and start yourfree trial today. Also make sure to install andconfigure theManta client utilities. You can also run this demo on your own SmartOS or illumos basedinstall provided you have the appropriate libproc,but I find Manta the easiest way to bootstrap.

You should be running the latest stable version of Node.js (or at leastv0.10.24), make sure it's 64-bit, that you obtained the binaries directly fromhttp://nodejs.org/ and that your environment is configured to produce core files.(ulimit -c unlimited will direct your current shell to write out a file thatis as large as required.)

There are a few ways to get a core file from your Node application.

• Run Node as normal and it aborts on its own.
  • This is the worst case scenario. Either Node, one of its dependencies, or a binary module has a programmingerror that resulted in an abort. In those cases please report the problem to eitherthe core team (if it was Node) or to themodule author.
• Run Node with --abort-on-uncaught-exception.
  • This directs v8 that anytime you fail to catch anexception to exit uncleanly and produce a core file. This is how Joyent runs all of its Node processes inproduction.
• Trigger a core file at runtime by calling process.abort().
  • Be advised this will cause your Node your process to exit.
• Use a platform tool to generate a core file.
  • On SmartOS and related systems there's a utility gcore which will generate a core file without causing the process to exit.
  • On some Linux installs you may also find a gcore script, it works by attaching gdb and running generate-core. Be advised that gdb will pause your process until you tell it to continue.

Demo Time

For our purposes here's the script I will use to generate a core file:

var obj = {  myproperty: "Hello World",  count: 0,};function increment() {  obj.count++;  if (obj.count === 1000)    throw new Error("sad trombone");  setImmediate(increment);}setImmediate(increment);

Here's how I ran it:

$ uname -aLinux 7527bd77-ab3e-474b-ace7-eed6053931e7 3.1.10joyent-ubuntu-10-opt #1 SMP Fri Jan 20 09:55:31 PST 2012 x86_64 GNU/Linux$ ulimit -c unlimited$ node --abort-on-uncaught-exception t.jsUncaught Error: sad tromboneFROMObject.increment [as _onImmediate] (/data/test/t.js:10:5)processImmediate [as _immediateCallback] (timers.js:330:15)Trace/breakpoint trap (core dumped)$ ls -alh core-rw------- 1 root root 12M 2014-01-28 18:37 core

Once you have your core file, we'll need to get the core file and the actualnode binary uploaded into Manta.

$ mkdir debug$ cp $(which node) debug/$ cp core debug/$ tar cz debug | mput ~~/stor/debug.tar.gz

You can use this little wrapper script to launch an mlogin session, designedto drop you immediately into a debugging session.

$ cat << EOF > mmdb.sh#!/bin/bashmlogin \  -s /NodeCore/public/linux-cores/mdb-linux.sh \  -s /NodeCore/public/linux-cores/libproc64.so \  -c 'echo "::load v8" > ~/.mdbrc &&      tar xzf \$MANTA_INPUT_FILE --strip-components=1 &&      bash /assets/NodeCore/public/linux-cores/mdb-linux.sh node core' \  \$@EOF$ chmod +x mmdb.sh

This script expects you to be using a tarball that has the node binaryliterally named "node" and the core file literally named "core" like we builtabove. It's using a prebuilt asset of libproc64.so which has been built withLinux support, as well as an asset mdb-linux.sh here is itssourcewhich merely launches mdb with the patched libproc64.so preloaded.

Start your interactive debugging session by running mmdb.sh with the locationin Manta you uploaded the tarball to.

$ ./mmdb.sh ~~/stor/debug.tar.gz * created interactive job -- 97b0b830-bafe-edb6-a271-b6a308a1a947 * waiting for session... / establishedmdb: warning: librtld_db failed to initialize; shared library information will not be availableV8 version: 3.14.5.9Autoconfigured V8 support from targetC++ symbol demangling enabled>

We're in a debugging session, have the v8 module loaded, it's recognized version 3.14.5.9 of v8and we can now start performing some of our normal debugging techniques.

> ::jsstack -v7fffa26a02b0 v8::internal::OS::Abort+0xe7fffa26a0380 v8::internal::Isolate::DoThrow+0x2c37fffa26a0390 v8::internal::Isolate::Throw+97fffa26a03d0 v8::internal::Runtime_Throw+0x447fffa26a03f8 0x344a26e06362 internal (Code: 344a26e062c1)7fffa26a0428 0x344a26e76411 increment (328d77f0c679)    file: /data/test/t.js    posn: position out of range7fffa26a0460 0x344a26e743ee processImmediate (328d77f0c7a1)    file: timers.js    posn: position out of range7fffa26a0498 0x344a26e0d507 7fffa26a0520 0x344a26e06116 7fffa26a05b0 _ZN2v88internalL6InvokeEbNS0_6HandleINS0_10JSFunctionEEENS1_INS0_6ObjectEEEiPS5_Pb+0xf27fffa26a0600 v8::internal::Execution::Call+0x1067fffa26a0670 v8::Function::Call+0x1127fffa26a06e0 node::MakeCallback+0x607fffa26a0740 node::MakeCallback+0x597fffa26a0780 _ZN4nodeL14CheckImmediateEP10uv_check_si+0x4c7fffa26a07b0 uv__run_check+0x427fffa26a0810 uv_run+0xe07fffa26a08a0 node::Start+0x152

There's our stack, now let's find our sentinel object we defined

> ::findjsobjects -p myproperty137289672551> 137289672551::jsprint{    myproperty: "Hello World",    count: 1000,}

And there you have it, you're now able to debug your Node.js applicationsthat run on Linux with our existing tooling.

Consider the possibilities:

• grab core files at regular intervals and perform differential v8 heap analysis
• process aborted from too much memory you can see exactly what was in the object space
• after a missed an exception you can see the precise state at that moment
• get the stack from a Node process stuck on CPU

It's important for us to be able debug our Node.js applications, so as we buildand improve the tooling we use, we will continue to make sure you are reapingthose same benefits. Your success with Node is important to us.

You can learn moreabout debugging Node.js applications with mdb.Also check out our other pages about designing, deploying, and debuggingyour Node.js applications. You can find the patch for libproc that enabled debugging Linux core files in this gist.