I am currently working on an online course that is aimed at Linuxsystem administrators and developers who want to use SmartOS.One of the many differences between Linux and Solaris-based systemsis the /proc file system. On Linux, this is documentedproc(5), and for SmartOS, proc(4).

On Linux, /proc includes information about processes, aswell as many other pieces of information about the system. Many ofthe files are readable (i.e., you can cat them), and someare writeable allowing you to modify some aspects of the system. OnSmartOS, /proc only contains information aboutprocesses. Most (all?) of the data in the files are accessed viaioctl(2) system calls, as explained in the man pagementioned above.

In this blog, I'll just talk about a set of tools available on SmartOSfor accessing data in/proc. Some of these tools can alsobe used on core files to get information about the environment of aprocess at the time it generated a core file. Most of the tools I amreferring to are documented in proc(1).People familiar with Linux will find that most of the information thetools give you on SmartOS are available in some form by cat-ing theappropriate file in /proc.

First, a list of the tools. Afterwords, we'll take a look at some examples.

• pargs(1) - Display process arguments, environment variables, and auxillary information. Also works on core files.
• pgrep(1) - Find process(es) by name and other attributes
• pkill(1) - Kill a process with name or attribute. Note that on Linux, the killall(1) command can be used instead of pkill, and on SmartOS, the killall(1M) command kills everything. Not too great when porting shell scripts containing killall from Linux to SmartOS.
• pcred(1) - Print or set credentials (user/group ids) of a process. Usable on core files.
• pldd(1) - Show dynamic shared libraries used by a process. Includes libraries linked via dlopen(3C). Usable on core files.
• psig(1) - Show disposition of signals for a process.
• pstack(1) - Show stack backtraces for all threads (or a specific thread) within a process. This can also be used to examine stack traces from core files.
• pfiles(1) - Show information about open files/sockets of a process. For network endpoints, shows local (and remote, if connected) addresses.
• pmap(1) - Display process address space.
• pwdx(1) - Print current working directory of a process.
• pstop(1) - Stop process or thread.
• prun(1) - Inverse of pstop
• pwait(1) - Wait for process(es) to terminate.
• ptime(1) - Time process using microstate information. Like time(1), but does not count time spent by child processes.
• ptree(1) - Print process tree (parent/child relationships). Can be used for specific pid, zone, or user.

Now we'll go through some examples.

# pgrep sh35668911149111391117#

Note that this will match, for instance, sh, bash, and sshd.

# pgrep a.out91145# pkill a.out# [1]+  Terminated              ./a.out#

Here is pargs(1) output showing command line andenvironment variables.

# pargs -ea $$91117: -bashargv[0]: -bashenvp[0]: MANPATH=/opt/local/gcc47/man:/opt/local/java/sun6/man:/opt/local/lib/perl5/man:/opt/local/lib/perl5/vendor_perl/man:/opt/local/gnu/man:/opt/local/man:/usr/share/manenvp[1]: TERM=dumbenvp[2]: SHELL=/usr/bin/bashenvp[3]: SSH_CLIENT=10.88.88.162 43182 22envp[4]: SSH_TTY=/dev/pts/3envp[5]: USER=rootenvp[6]: COLUMNS=80envp[7]: PAGER=lessenvp[8]: MAIL=/usr/mail/rootenvp[9]: PATH=/opt/local/gnu/bin:/opt/local/bin:/opt/local/sbin:/usr/bin:/usr/sbinenvp[10]: PWD=/var/tmpenvp[11]: LANG=Cenvp[12]: TZ=UTCenvp[13]: LINES=24envp[14]: SHLVL=1envp[15]: HOME=/rootenvp[16]: LOGNAME=rootenvp[17]: SSH_CONNECTION=10.88.88.162 43182 8.17.87.25 22envp[18]: FTPMODE=autoenvp[19]: _=/usr/bin/pargsenvp[20]: OLDPWD=/root#

The same for a core file.

# pargs -ea core.91145core 'core.91145' of 91145:       ./a.outargv[0]: ./a.outenvp[0]: MANPATH=/opt/local/gcc47/man:/opt/local/java/sun6/man:/opt/local/lib/perl5/man:/opt/local/lib/perl5/vendor_perl/man:/opt/local/gnu/man:/opt/local/man:/usr/share/manenvp[1]: TERM=dumbenvp[2]: SHELL=/usr/bin/bashenvp[3]: SSH_CLIENT=24.6.92.162 43182 22envp[4]: SSH_TTY=/dev/pts/3envp[5]: USER=rootenvp[6]: COLUMNS=80envp[7]: PAGER=lessenvp[8]: MAIL=/usr/mail/rootenvp[9]: PATH=/opt/local/gnu/bin:/opt/local/bin:/opt/local/sbin:/usr/bin:/usr/sbinenvp[10]: PWD=/var/tmpenvp[11]: LANG=Cenvp[12]: TZ=UTCenvp[13]: LINES=24envp[14]: SHLVL=1envp[15]: HOME=/rootenvp[16]: LOGNAME=rootenvp[17]: SSH_CONNECTION=24.6.92.162 43182 8.17.87.25 22envp[18]: FTPMODE=autoenvp[19]: _=./a.outenvp[20]: OLDPWD=/root#

Some pldd(1) output.

# pldd $$91117:       -bash/usr/bin/bash/lib/libcurses.so.1/lib/libsocket.so.1/lib/libnsl.so.1/lib/libdl.so.1/lib/libc.so.1## pldd core.91145core 'core.91145' of 91145:  ./a.out/lib/amd64/libc.so.1#

And psig(1).

# psig $$91117: -bashHUP        caught  termsig_sighandler      0       HUP,INT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,VTALRM,XCPU,XFSZ,LOSTINT    caught  sigint_sighandler       0QUIT   ignoredILL      caught  termsig_sighandler      0       HUP,INT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,VTALRM,XCPU,XFSZ,LOSTTRAP   caught  termsig_sighandler      0       HUP,INT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,VTALRM,XCPU,XFSZ,LOSTABRT   caught  termsig_sighandler      0       HUP,INT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,VTALRM,XCPU,XFSZ,LOSTEMT    caught  termsig_sighandler      0       HUP,INT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,VTALRM,XCPU,XFSZ,LOSTFPE    caught  termsig_sighandler      0       HUP,INT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,VTALRM,XCPU,XFSZ,LOSTKILL   defaultBUS      caught  termsig_sighandler      0       HUP,INT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,VTALRM,XCPU,XFSZ,LOSTSEGV   caught  termsig_sighandler      0       HUP,INT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,VTALRM,XCPU,XFSZ,LOSTSYS    caught  termsig_sighandler      0       HUP,INT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,VTALRM,XCPU,XFSZ,LOSTPIPE   caught  termsig_sighandler      0       HUP,INT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,VTALRM,XCPU,XFSZ,LOSTALRM   caught  termsig_sighandler      0       HUP,INT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,VTALRM,XCPU,XFSZ,LOSTTERM   ignoredUSR1     caught  termsig_sighandler      0       HUP,INT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,VTALRM,XCPU,XFSZ,LOSTUSR2   caught  termsig_sighandler      0       HUP,INT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,VTALRM,XCPU,XFSZ,LOSTCLD    blocked,caught  sigchld_handler 0PWR    defaultWINCH    caught  sigwinch_sighandler     0URG    defaultPOLL     defaultSTOP     defaultTSTP     ignoredCONT     defaultTTIN     ignoredTTOU     ignoredVTALRM   caught  termsig_sighandler      0       HUP,INT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,VTALRM,XCPU,XFSZ,LOSTPROF   defaultXCPU     caught  termsig_sighandler      0       HUP,INT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,VTALRM,XCPU,XFSZ,LOSTXFSZ   caught  termsig_sighandler      0       HUP,INT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,VTALRM,XCPU,XFSZ,LOSTWAITING        defaultLWP      defaultFREEZE   defaultTHAW     defaultCANCEL   defaultLOST     caught  termsig_sighandler      0       HUP,INT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,VTALRM,XCPU,XFSZ,LOSTXRES   default...#

And pstack(1). Note this is very useful to helpdetermine where/if a process is hung, and, on core files, where theprocess died.

# pstack 2896328963:       /opt/local/sbin/mysqld --user=mysql --basedir=/opt/local --datadir=/va-----------------  lwp# 1 / thread# 1  -------------------- fffffd7fff2601da pollsys  (fffffd7fffdff480, 2, 0, 0) fffffd7fff1e704a poll () + 62 0000000000613276 _Z26handle_connections_socketsv () + b6 0000000000615535 _Z11mysqld_mainiPPc () + e45 000000000060d64c _start () + 6c-----------------  lwp# 2 / thread# 2  -------------------- fffffd7fff2575e7 lwp_park (0, 0, 0) fffffd7fff250960 cond_wait_queue () + 68 fffffd7fff250f33 __cond_wait () + 7b fffffd7fff250f8b cond_wait () + 23 fffffd7fff250fc1 pthread_cond_wait () + 9 000000000099e7f6 os_event_wait_low () + 46 000000000099d3a6 os_aio_simulated_handle () + 6d6 000000000095f375 fil_aio_wait () + 45 00000000008ed758 io_handler_thread () + 58 fffffd7fff2572d4 _thrp_setup () + bc fffffd7fff2575a0 _lwp_start ()...#

pfiles(1)

# pfiles 9336293362:       /usr/lib/ssh/sshd  Current rlimit: 1024 file descriptors   0: S_IFCHR mode:0666 dev:532,0 ino:19922952 uid:0 gid:3 rdev:38,2      O_RDWR      /devices/pseudo/mm@0:null      offset:0   1: S_IFCHR mode:0666 dev:532,0 ino:19922952 uid:0 gid:3 rdev:38,2      O_RDWR      /devices/pseudo/mm@0:null      offset:0   2: S_IFCHR mode:0666 dev:532,0 ino:19922952 uid:0 gid:3 rdev:38,2      O_RDWR      /devices/pseudo/mm@0:null      offset:0   3: S_IFDOOR mode:0444 dev:542,0 ino:52 uid:0 gid:0 size:0      O_RDONLY|O_LARGEFILE FD_CLOEXEC  door to nscd[1614]      /var/run/name_service_door   4: S_IFSOCK mode:0666 dev:540,0 ino:5398 uid:0 gid:0 size:0      O_RDWR|O_NONBLOCK       SOCK_STREAM     SO_REUSEADDR,SO_KEEPALIVE,SO_SNDBUF(49152),SO_RCVBUF(128872)    sockname: AF_INET6 ::ffff:10.0.110.25  port: 22 peername: AF_INET6 ::ffff:10.0.110.7  port: 54326   5: S_IFIFO mode:0000 dev:530,0 ino:5480480 uid:0 gid:0 size:0      O_RDWR|O_NONBLOCK FD_CLOEXEC...#

pmap(1) output can give you a good idea of how theaddress space of a process is being used, also on core files.

# pmap -x $$91117:        -bash Address  Kbytes     RSS    Anon  Locked Mode   Mapped File08045000      12      12       4       - rw---    [ stack ]08050000     724     668       -       - r-x--  bash08114000      24      24       8       - rwx--  bash0811A000     160     160      44       - rwx--    [ heap ]FEAF0000     456     456       -       - r-x--  libnsl.so.1FEB72000       8       8       4       - rw---  libnsl.so.1FEB74000      20      12       -       - rw---  libnsl.so.1FED50000      64      16       -       - rwx--    [ anon ]FED6D000       4       4       -       - rwxs-    [ anon ]FED70000      24      12       4       - rwx--    [ anon ]FED80000       4       4       -       - rwx--    [ anon ]FED90000    1216    1216       -       - r-x--  libc.so.1FEEC0000      36      36      16       - rwx--  libc.so.1FEEC9000       8       8       -       - rwx--  libc.so.1FEED0000       4       4       -       - r-x--  libdl.so.1FEEE0000       4       4       4       - rwx--    [ anon ]FEEF0000      56      56       -       - r-x--  libsocket.so.1FEF0E000       4       4       -       - rw---  libsocket.so.1FEF10000       4       4       4       - rwx--    [ anon ]FEF20000     180     180       -       - r-x--  libcurses.so.1FEF5D000      28      28       -       - rw---  libcurses.so.1FEF64000       8       4       -       - rw---  libcurses.so.1FEF70000       4       4       -       - r--s-  ld.configFEF80000       4       4       4       - rw---    [ anon ]FEF90000       4       4       -       - rw---    [ anon ]FEFA0000       4       4       4       - rwx--    [ anon ]FEFB0000       4       4       -       - rwx--    [ anon ]FEFB7000     208     208       -       - r-x--  ld.so.1FEFFB000       8       8       4       - rwx--  ld.so.1FEFFD000       4       4       -       - rwx--  ld.so.1-------- ------- ------- ------- -------total Kb    3288    3160     100       -### pmap core.91145core 'core.91145' of 91145:     ./a.out0000000000400000          4K r-x--  /var/tmp/a.out0000000000410000          8K rw---    [ heap ]FFFFFD7FFF170000         24K rwx--    [ anon ]FFFFFD7FFF180000          4K rwx--    [ anon ]FFFFFD7FFF190000       1584K r-x--  /lib/amd64/libc.so.1FFFFFD7FFF32C000         52K rw---  /lib/amd64/libc.so.1FFFFFD7FFF339000          8K rw---  /lib/amd64/libc.so.1FFFFFD7FFF350000          4K rwx--    [ anon ]FFFFFD7FFF360000          4K rwx--    [ anon ]FFFFFD7FFF370000          4K rw---    [ anon ]FFFFFD7FFF380000          4K rw---    [ anon ]FFFFFD7FFF390000          4K rwx--    [ anon ]FFFFFD7FFF393000        348K r-x--  /lib/amd64/ld.so.1FFFFFD7FFF3FA000         12K rwx--  /lib/amd64/ld.so.1FFFFFD7FFF3FD000          8K rwx--  /lib/amd64/ld.so.1FFFFFD7FFFDFE000          8K rw---    [ stack ]         total         2080K#

Some ptime(1) output.

# ptime dd if=/dev/zero of=/dev/null bs=8k count=10241024+0 records in1024+0 records out8388608 bytes (8.4 MB) copied, 0.00246212 s, 3.4 GB/sreal        0.010517277user        0.002187369sys         0.007724275#

And here is ptree.

# ptree $$17701 zsched  35668 /usr/lib/ssh/sshd    91113 /usr/lib/ssh/sshd      91114 /usr/lib/ssh/sshd        91117 -bash          93577 ptree 91117#

There are other tools using /proc, including ps(1), truss(1), and sotruss(1).

For those interested, I am teaching a DTrace course in San Francisco,May 14 through May 16. Details can be found here. Hope to see you there!