Getting errors while using systemtap

rvsw · 2014-05-21T01:57:43+00:00

Hello! On Tue, May 20, 2014 at 7:36 PM, bin al wrote: > Thank you for your answer. > I am interested in the cookies which would be used for the latter...

Getting errors while using systemtap

rvsw

Hello agentzh

I was hoping to learn and start using the systemtap facility for nginx. I (think) I installed systemtap properly on my Ubuntu machine. As explaiend at http://goo.gl/nRkLbU. I ran the following command successfully to test if the installation is valid

But when I run the systemtap as mentioned at http://goo.gl/5prKkS, none of the scripts except ngx-body-filters succeeds

e.g. I get the following messages. I tried to see if I can fix but I don't understand systemtap now well enough

1. sudo ./ngx-leaked-pools -p 6791

semantic error: while resolving probe point: identifier 'process' at <input>:9:7

source: probe process("/opt/wapp/nginx/sbin/nginx").provider("nginx").mark("create-pool-done")

2. sudo ./ngx-lua-bt -p 6791 --lua51

semantic error: while resolving probe point: identifier 'process' at <input>:182:1

source: process("/opt/wapp/nginx/sbin/nginx").function("str_find"),

Can you please advise on how to troubleshoot this? The versions of the systemtap and perl are valid and are appended.

For tcp receive queue, I got another set of error but that maybe kernel problem

Mentioning it just in case

sudo ./tcp-accept-queue --port=80

stap --version
Systemtap translator/driver (version 2.3/0.152, Debian version 2.3-1ubuntu1 (trusty))
Copyright (C) 2005-2013 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
enabled features: AVAHI LIBSQLITE3 NSS TR1_UNORDERED_MAP NLS
$ perl --version

This is perl 5, version 14, subversion 2 (v5.14.2) built for x86_64-linux-gnu-thread-multi
(with 56 registered patches, see perl -V for more detail)

Copyright 1987-2011, Larry Wall

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using "man perl" or "perldoc perl". If you have access to the
Internet, point your browser at http://www.perl.org/, the Perl Home Page.

agentzh

Hello!

On Tue, May 20, 2014 at 10:57 AM, rvsw wrote:
> 1.  sudo ./ngx-leaked-pools -p 6791
> semantic error: while resolving probe point: identifier 'process' at
> <input>:9:7
>         source: probe
> process("/opt/wapp/nginx/sbin/nginx").provider("nginx").mark("create-pool-done")
>

You need to enable the dtrace static probes in your nginx build. To
quite the documentation for that ngx-leaked-pools tool:

"The bundle ngx_openresty 1.2.3.3+ includes the right dtrace patch by
default. And you just need to build it with the --with-dtrace-probes
configure option."

See https://github.com/openresty/nginx-systemtap-toolkit#ngx-leaked-pools
for more details.

The latest openresty 1.5.12.1 release should work, for example.

> 2. sudo ./ngx-lua-bt -p 6791 --lua51
> semantic error: while resolving probe point: identifier 'process' at
> <input>:182:1
>         source: process("/opt/wapp/nginx/sbin/nginx").function("str_find"),
>

You specify the --lua51 option in your command line but are you sure
your nginx is linked with the standard Lua 5.1 interpreter? If yes,
are you sure your standard Lua 5.1 interpreter was built with the C
compiler option -g? Your nginx may link with LuaJIT or your standard
Lua 5.1 interpreter was built without enabling debug symbols.

Also if you're using LuaJIT 2.1 (enabled by recent releases of
OpenResty by default), then you should use the following tool instead:

     https://github.com/openresty/nginx-systemtap-toolkit#ngx-leaked-pools

> For tcp receive queue, I got another set of error but that maybe kernel
> problem
> Mentioning it just in case
>
> sudo ./tcp-accept-queue --port=80
> semantic error: not accessible at this address [man error::dwarf]
> (0xffffffff8163b220, dieoffset: 0x59b0fbc): identifier '$skb' at
> <input>:8:31
>         source:     tcphdr = __get_skb_tcphdr($skb)
>

Ensure your kernel was built without the CONFIG_HAVE_FENTRY=y option
(which was the default for many Linux distributions), otherwise gcc
tends to emit bad debug symbols which is an known issue in gcc. You
can check out your kernel's .config file.

Also, the version of the kernel is important here but you did not
provide the kernel version (try the uname -r option, for example).

Regards,
-agentzh

agentzh

Hello!

On Tue, May 20, 2014 at 5:05 PM, Yichun Zhang (agentzh) wrote:
> Also if you're using LuaJIT 2.1 (enabled by recent releases of
> OpenResty by default), then you should use the following tool instead:
>
>      https://github.com/openresty/nginx-systemtap-toolkit#ngx-leaked-pools
>

Sorry, copy&paste error. The link should be
https://github.com/openresty/stapxx#lj-lua-stacks instead.

Regards,
-agentzh

brendanoreilly99

Hi Agentzh,

I'm also having some issues getting systemtap to work with openresty (version 1.5.12.1-1.x86_64) and hoping you can point me in the right direction. I've recompiled openresty including the --with-debug parameter and installed the latest systemtap and perl packages as outlined on the ngx-systemtap-toolkit github site. When I run any of the tools I get the following messages:

$ sudo ./ngx-leaked-pools -p 16479

semantic error: while resolving probe point: identifier 'process' at <input>:9:7

source: probe process("/usr/sbin/nginx").provider("nginx").mark("create-pool-done")

semantic error: no match

WARNING: cannot find module /usr/sbin/nginx debuginfo: No DWARF information found [man warning::debuginfo]

semantic error: while resolving probe point: identifier 'process' at :25:7

source: probe process("/usr/sbin/nginx").function("ngx_destroy_pool")

I then ran the check-debug-info tool which gives me the following output:

$ sudo ./check-debug-info -p 16479

File /lib64/ld-2.12.so has no debug info embedded.

File /lib64/libc-2.12.so has no debug info embedded.

File /lib64/libcrypt-2.12.so has no debug info embedded.

File /lib64/libdl-2.12.so has no debug info embedded.

File /lib64/libfreebl3.so has no debug info embedded.

File /lib64/libm-2.12.so has no debug info embedded.

File /lib64/libnss_files-2.12.so has no debug info embedded.

File /lib64/libpthread-2.12.so has no debug info embedded.

File /lib64/libresolv-2.12.so has no debug info embedded.

File /lib64/librt-2.12.so has no debug info embedded.

File /usr/sbin/nginx has no debug info embedded.

File /usr/share/openresty/lualib/posix_c.so has no debug info embedded.

File /usr/share/openresty/lualib/socket/core.so has no debug info embedded.

I've tried re-installing some of the listed libraries with the debug versions but they still appear when I re-run check-debug-info. I also noticed that nginx is also listed even though i compiled openresty using --with-debug. I've tried a lot of different things to get this to work but at this point I feel like I'm chasing my own tail. Any suggestions you have is much appreciated.

Thanks,

Brendan

agentzh

Hello!

On Thu, Aug 21, 2014 at 2:25 PM,  brendanoreilly99 wrote:
>> Hi Agentzh,
>

Please do not capitialize my nick.

> I've recompiled openresty including the --with-debug parameter

No, the --with-debug option is not required for any of my systemtap
tools. It makes things slower and better turn it off for production
use.

> $ sudo ./ngx-leaked-pools -p 16479
> semantic error: while resolving probe point: identifier 'process' at
> <input>:9:7
>         source: probe
> process("/usr/sbin/nginx").provider("nginx").mark("create-pool-done")
>                       ^
>
> semantic error: no match

https://github.com/openresty/nginx-systemtap-toolkit#ngx-leaked-pools

"The bundle ngx_openresty 1.2.3.3+ includes the right dtrace patch by
default. And you just need to build it with the --with-dtrace-probes
configure option."

So you've missed the --with-dtrace-probes option?

> WARNING: cannot find module /usr/sbin/nginx debuginfo: No DWARF information
> found [man warning::debuginfo]

BTW, the path mentioned above, /usr/sbin/nginx, does not look like
OpenResty's nginx (which should be
/usr/local/openresty/nginx/sbin/nginx by default). Also, OpenResty's
nginx should always contain DWARF information by default (even without
--with-debug).

BTW, it is required to subscribe to this list before posting. See
http://openresty.org/#Community

Regards,
-agentzh

brendanoreilly99

Thank-you for the response agentzh. Apologies in advance if some of these questions seem silly, I'm still learning openresty. The reason i'm attempting to use your openresty systemtap tools is so I can do some memory profiling. We've noticed that the nginx worker processes slowly consume more and more memory and then eventually die one by one which releases the memory. During some performance testing it appears that the worker processes are dying off gracefully in that no http response errors were captured during the perf testing. However, I'm still interested in what is happening so we can catch any issues before becoming a real problem. I appreciate you guidance.

I've now removed the --with-debug parameter and added the --with-dtrace-probes parameter however I'm getting the following error when I compile and run gmake:

gmake[2]: Entering directory `/mint-proxy/build/nginx-1.5.12'

cat src/dtrace/nginx_provider.d \

../ngx_lua-0.9.7/dtrace/ngx_lua_provider.d > objs/dtrace_providers.d

dtrace -xnolibs -h -o objs/ngx_dtrace_provider.h -s objs/dtrace_providers.d

gmake[2]: dtrace: Command not found

gmake[2]: *** [objs/ngx_dtrace_provider.h] Error 127

gmake[2]: Leaving directory `/mint-proxy/build/nginx-1.5.12'

gmake[1]: *** [build] Error 2

gmake[1]: Leaving directory `/mint-proxy/build/nginx-1.5.12'

gmake: *** [all] Error 2

I thought maybe I had to install dtrace separately but when I do an rpm search it appears to be part of systemtap package which is installed on my build machine.

Also, the reason I'm using the /usr/sbin/nginx path is because that is how the package has been compiled in the past. Hopefully that doesn't make a difference, nginx seems to startup fine when I start the service.

Thanks,

Brendan

p.s. I'll also subscribe to the community.

On Fri, Aug 22, 2014 at 3:47 PM, Yichun Zhang (agentzh) <age...@gmail.com> wrote:

Hello!

On Thu, Aug 21, 2014 at 2:25 PM, brendanoreilly99 wrote:
>> Hi Agentzh,
>

Please do not capitialize my nick.

> I've recompiled openresty including the --with-debug parameter

No, the --with-debug option is not required for any of my systemtap
tools. It makes things slower and better turn it off for production
use.

> $ sudo ./ngx-leaked-pools -p 16479
> semantic error: while resolving probe point: identifier 'process' at
> <input>:9:7
> source: probe
> process("/usr/sbin/nginx").provider("nginx").mark("create-pool-done")
> ^
>
> semantic error: no match

https://github.com/openresty/nginx-systemtap-toolkit#ngx-leaked-pools

"The bundle ngx_openresty 1.2.3.3+ includes the right dtrace patch by
default. And you just need to build it with the --with-dtrace-probes
configure option."

So you've missed the --with-dtrace-probes option?

> WARNING: cannot find module /usr/sbin/nginx debuginfo: No DWARF information
> found [man warning::debuginfo]

BTW, the path mentioned above, /usr/sbin/nginx, does not look like
OpenResty's nginx (which should be
/usr/local/openresty/nginx/sbin/nginx by default). Also, OpenResty's
nginx should always contain DWARF information by default (even without
--with-debug).

BTW, it is required to subscribe to this list before posting. See
http://openresty.org/#Community

Regards,
-agentzh

.

agentzh

Hello!

On Tue, Aug 26, 2014 at 11:44 AM, brendan oreilly wrote:
> The reason i'm
> attempting to use your openresty systemtap tools is so I can do some memory
> profiling.

The following tools can be more useful than ngx-leaked-pools:

    https://github.com/openresty/stapxx#sample-bt-leaks

    https://github.com/openresty/stapxx#lj-gc

    https://github.com/openresty/stapxx#lj-gc-objs

> gmake[2]: Entering directory `/mint-proxy/build/nginx-1.5.12'
> cat src/dtrace/nginx_provider.d \
> ../ngx_lua-0.9.7/dtrace/ngx_lua_provider.d > objs/dtrace_providers.d
> dtrace -xnolibs -h -o objs/ngx_dtrace_provider.h -s objs/dtrace_providers.d
> gmake[2]: dtrace: Command not found

The "dtrace" utility is provided by systemtap. For example, on my
system, it is located at

    /opt/systemtap/bin/dtrace

since I built systemtap from source and configured the prefix to be
/opt/systemtap.

I'm not sure about your systemtap installation. If its dtrace utility
is not in the default search paths, then you should add its path to
your PATH environment.

Regards,
-agentzh