systemtap problems

lordnynex · 2015-03-28T08:57:34+00:00

hi, I just ported cloudwu's lua-mongo to lua_ngx:work with lua_ngx's cosocketcompatible with luajit-2.0 via https://github.com/keplerproject/lua-compat-...

systemtap problems

lordnynex

Hello,

I'm attempting to debug high CPU usage in my app and I'm hoping to generate a flame graph of whats going on.

When I attempt to run any of the tools from https://github.com/openresty/nginx-systemtap-toolkit nginx hangs and does not respond to requests. Did I compile openresty incorrectly?

I am using

- ubuntu 14.04

- running linux-image-3.13.0-36-generic-dbgsym kernel

- systemtap 2.3.

- Openresty version 1.7.10.1 compiled with --with-debug and -O2 -gdwarf-2'

configure arguments: --prefix=/usr/local/openresty/nginx --with-debug --with-cc-opt='-I/home/ubuntu/build/openresty/ngx_openresty-1.7.10.1/build/luajit-root/usr/local/openresty/luajit/include/luajit-2.1 -DNGX_LUA_USE_ASSERT -DNGX_LUA_ABORT_AT_PANIC -O2 -gdwarf-2' --add-module=../ngx_devel_kit-0.2.19 --add-module=../echo-nginx-module-0.57 --add-module=../xss-nginx-module-0.04 --add-module=../ngx_coolkit-0.2rc2 --add-module=../set-misc-nginx-module-0.28 --add-module=../form-input-nginx-module-0.10 --add-module=../encrypted-session-nginx-module-0.03 --add-module=../srcache-nginx-module-0.29 --add-module=../ngx_lua-0.9.15 --add-module=../ngx_lua_upstream-0.02 --add-module=../headers-more-nginx-module-0.25 --add-module=../array-var-nginx-module-0.03 --add-module=../memc-nginx-module-0.15 --add-module=../redis2-nginx-module-0.11 --add-module=../redis-nginx-module-0.3.7 --add-module=../rds-json-nginx-module-0.13 --add-module=../rds-csv-nginx-module-0.05 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -L/home/ubuntu/build/openresty/ngx_openresty-1.7.10.1/build/luajit-root/usr/local/openresty/luajit/lib' --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/www/nginx//log/error.log --http-client-body-temp-path=/www/nginx//lib/nginx/body --http-fastcgi-temp-path=/www/nginx//lib/nginx/fastcgi --http-log-path=/www/nginx//log/access.log --http-proxy-temp-path=/www/nginx//lib/nginx/proxy --http-scgi-temp-path=/www/nginx//lib/nginx/scgi --http-uwsgi-temp-path=/www/nginx//lib/nginx/uwsgi --lock-path=/var/lock/nginx.lock --pid-path=/var/run/nginx.pid --with-http_realip_module --with-http_stub_status_module --with-http_ssl_module --with-http_sub_module --with-sha1=/usr/include/openssl --with-md5=/usr/include/openssl

agentzh

Hello!

On Fri, Mar 27, 2015 at 5:57 PM, Lord Nynex wrote:
> When I attempt to run any of the tools from
> https://github.com/openresty/nginx-systemtap-toolkit nginx hangs and does
> not respond to requests.

This is weird. Will you provide as much details as possible? Like the
output of your systemtap tool command and your nginx worker process's
detailed state (via system tools like top, ps, pstack, strace and
etc). Is there any core dumps involved? I've never seen such things
before.

What specific tools are you using?

> Did I compile openresty incorrectly?
>

You do not need any special OpenResty building flags to make it work.
The defaults should be perfect.

> I am using
> - ubuntu 14.04
> - running linux-image-3.13.0-36-generic-dbgsym kernel

There has been a lot of important bug fixes in Linux kernel's uprobes
mechanism since 3.13. Is it feasible for you to upgrade the kernel to
the latest stable?

> - systemtap 2.3.

Try upgrading to the latest systemtap :) You can check out this page:

   http://openresty.org/#BuildSystemtap

The latest is 2.7 though.

> - Openresty version 1.7.10.1 compiled with --with-debug and -O2 -gdwarf-2'
>

Do not specify --with-debug while profiling. It slows everything down
significantly.

Regards,
-agentzh

lordnynex

What specific tools are you using?

I tried a few but I figured the easiest one to start was ngx-active-req -p <master pid>. All tools yield the same result. The tool prints that it's tracing the pid (and worker pids) yet never outputs anything. While the tool is running, I am testing localhost via curl. It seems the socket opens and hangs.

When I try the same tools/environment with openresty/1.7.2.1, after I stop the tool (CTRL-C), all workers segfault. I do not receive the same segfault for 1.7.10.1.

You do not need any special OpenResty building flags to make it work.
The defaults should be perfect.

I am receiving the same results with or without these flags. I added the DWARF cc-opt (and in another build dtrace) thinking I was doing something wrong.

There has been a lot of important bug fixes in Linux kernel's uprobes
mechanism since 3.13. Is it feasible for you to upgrade the kernel to
the latest stable?

I think latest stable means 3.19? I'm not sure how smoothly this will go because ubuntu has no mainline build for this kernel. I did find http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.19-vivid/, however, I can't find a build with debug symbols. I have found a mainline release for 3.16 and am trying that now. I am nervous to compile from source because this is an AWS EC2 box, and I don't know if there are any special considerations at compile time.

Try upgrading to the latest systemtap :) You can check out this page:

http://openresty.org/#BuildSystemtap

The latest is 2.7 though.

I am building 2.7 now.

Do not specify --with-debug while profiling. It slows everything down
significantly.

I added this flag just hoping to get it to work. I pulled the debug log output that is helpful and will roll a new build without this flag.

lordnynex

I am able to get data now without hangups or coredumps.

New Stack is:

- Ubuntu 14.04

- running linux-image-3.16.0-31-generic kernel (https://wiki.ubuntu.com/Kernel/Systemtap#How_do_I_build_a_debuginfo_kernel_if_one_isn.27t_available.3F)

- Systemtap 2.7

- Openresty 1.7.10.1 (no debug no dwarf no dtrace)

Sorry for the false alarm.

On Sat, Mar 28, 2015 at 1:50 PM, Lord Nynex <lord...@gmail.com> wrote:

What specific tools are you using?

I tried a few but I figured the easiest one to start was ngx-active-req -p <master pid>. All tools yield the same result. The tool prints that it's tracing the pid (and worker pids) yet never outputs anything. While the tool is running, I am testing localhost via curl. It seems the socket opens and hangs.

When I try the same tools/environment with openresty/1.7.2.1, after I stop the tool (CTRL-C), all workers segfault. I do not receive the same segfault for 1.7.10.1.

You do not need any special OpenResty building flags to make it work.
The defaults should be perfect.

I am receiving the same results with or without these flags. I added the DWARF cc-opt (and in another build dtrace) thinking I was doing something wrong.

There has been a lot of important bug fixes in Linux kernel's uprobes
mechanism since 3.13. Is it feasible for you to upgrade the kernel to
the latest stable?

I think latest stable means 3.19? I'm not sure how smoothly this will go because ubuntu has no mainline build for this kernel. I did find http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.19-vivid/, however, I can't find a build with debug symbols. I have found a mainline release for 3.16 and am trying that now. I am nervous to compile from source because this is an AWS EC2 box, and I don't know if there are any special considerations at compile time.

Try upgrading to the latest systemtap :) You can check out this page:

http://openresty.org/#BuildSystemtap

The latest is 2.7 though.

I am building 2.7 now.

Do not specify --with-debug while profiling. It slows everything down
significantly.

I added this flag just hoping to get it to work. I pulled the debug log output that is helpful and will roll a new build without this flag.

lordnynex

How can I resolve this?

/home/ubuntu/stapxx/samples/lj-lua-stacks.sxx: line 3: cannot find @use library nginx.lua

On Sat, Mar 28, 2015 at 4:07 PM, Lord Nynex <lord...@gmail.com> wrote:

I am able to get data now without hangups or coredumps.

New Stack is:
- Ubuntu 14.04
- running linux-image-3.16.0-31-generic kernel (https://wiki.ubuntu.com/Kernel/Systemtap#How_do_I_build_a_debuginfo_kernel_if_one_isn.27t_available.3F)
- Systemtap 2.7
- Openresty 1.7.10.1 (no debug no dwarf no dtrace)

Sorry for the false alarm.

On Sat, Mar 28, 2015 at 1:50 PM, Lord Nynex <lord...@gmail.com> wrote:

What specific tools are you using?

I tried a few but I figured the easiest one to start was ngx-active-req -p <master pid>. All tools yield the same result. The tool prints that it's tracing the pid (and worker pids) yet never outputs anything. While the tool is running, I am testing localhost via curl. It seems the socket opens and hangs.

When I try the same tools/environment with openresty/1.7.2.1, after I stop the tool (CTRL-C), all workers segfault. I do not receive the same segfault for 1.7.10.1.

You do not need any special OpenResty building flags to make it work.
The defaults should be perfect.

I am receiving the same results with or without these flags. I added the DWARF cc-opt (and in another build dtrace) thinking I was doing something wrong.

There has been a lot of important bug fixes in Linux kernel's uprobes
mechanism since 3.13. Is it feasible for you to upgrade the kernel to
the latest stable?

I think latest stable means 3.19? I'm not sure how smoothly this will go because ubuntu has no mainline build for this kernel. I did find http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.19-vivid/, however, I can't find a build with debug symbols. I have found a mainline release for 3.16 and am trying that now. I am nervous to compile from source because this is an AWS EC2 box, and I don't know if there are any special considerations at compile time.

Try upgrading to the latest systemtap :) You can check out this page:

http://openresty.org/#BuildSystemtap

The latest is 2.7 though.

I am building 2.7 now.

Do not specify --with-debug while profiling. It slows everything down
significantly.

I added this flag just hoping to get it to work. I pulled the debug log output that is helpful and will roll a new build without this flag.

agentzh

Hello!

On Sat, Mar 28, 2015 at 4:07 PM, Lord Nynex wrote:
> I am able to get data now without hangups or coredumps.
>

Glad you've got it working :)

> New Stack is:
[...]
> - Systemtap 2.7
> - Openresty  1.7.10.1 (no debug no dwarf no dtrace)

I believe you do need dwarf here though. The default build options of
OpenResty enables DWARF debuginfo, otherwise none of those systemtap
tools will work :)

> How can I resolve this?
>
> /home/ubuntu/stapxx/samples/lj-lua-stacks.sxx: line 3: cannot find @use library nginx.lua

It means it fails to find the stap++ tapset module nginx.lua in the
stapxx source tree.

Getting this error usually means you've manually moved stap++ out of
its original stapxx source tree. Better set your PATH environment
properly without moving stap++ out of its tree.

Regards,
-agentzh