CPU 100%，应该如何调试

moonbingbing

春哥好，有一个线上系统，偶尔出现nginx 所有worker 持续cpu 100%的情况，不响应新进来的其他请求。需要重启解决。
代码里面没有死循环的逻辑。应该用什么工具去调试呢？sysyemtap可以调试吗？还是需要gdb抓core dump？

发自我的 iPhone

agentzh

Hello!

2014-11-19 4:25 GMT-08:00 Ming:
> 春哥好，有一个线上系统，偶尔出现nginx 所有worker 持续cpu 100%的情况，不响应新进来的其他请求。需要重启解决。
> 代码里面没有死循环的逻辑。应该用什么工具去调试呢？

这种情形应当使用 on-CPU 火焰图进行分析：

https://github.com/openresty/nginx-systemtap-toolkit#sample-bt  （C 级别）

https://github.com/openresty/stapxx#lj-lua-stacks  （Lua 级别）

一看图就一目了然了。

从你描述的情况看，应当是你的某个 Lua 代码路径不正确地陷入了死循环（或者过热），因此 Lua 级别的火焰图最有用，特别是指定 --arg
detailed=1 命令行选项的时候。

Regards,
-agentzh

agentzh

Hello!

2014-11-19 11:34 GMT-08:00 Yichun Zhang (agentzh):
> 从你描述的情况看，应当是你的某个 Lua 代码路径不正确地陷入了死循环（或者过热），因此 Lua 级别的火焰图最有用，特别是指定 --arg
> detailed=1 命令行选项的时候。
>

当然，也可以使用 gdb. 配合 nginx-gdb-utils 项目的 lbt 命令：

    https://github.com/openresty/nginx-gdb-utils#lbt

你可以使用 gdb 的 batch 模式以避免长时间阻塞 nginx worker（不过既然 nginx worker
已无法再响应新的请求了，阻塞不阻塞也无所谓了）。

值得一的是，即使使用 gdb 的 batch 模式，其对目标进程产生的影响也远大于 systemtap 工具 :)

Regards,
-agentzh

vincent.mc.li

On Wednesday, November 19, 2014 11:39:16 AM UTC-8, agentzh wrote:

Hello!

2014-11-19 11:34 GMT-08:00 Yichun Zhang (agentzh):
> 从你描述的情况看，应当是你的某个 Lua 代码路径不正确地陷入了死循环（或者过热），因此 Lua 级别的火焰图最有用，特别是指定 --arg
> detailed=1 命令行选项的时候。
>

当然，也可以使用 gdb. 配合 nginx-gdb-utils 项目的 lbt 命令：

https://github.com/openresty/nginx-gdb-utils#lbt

你可以使用 gdb 的 batch 模式以避免长时间阻塞 nginx worker（不过既然 nginx worker
已无法再响应新的请求了，阻塞不阻塞也无所谓了）。

值得一的是，即使使用 gdb 的 batch 模式，其对目标进程产生的影响也远大于 systemtap 工具 :)

we have used gcc flag -fno-omit-frame-pointer –ggdb to recompile our userland process and use https://perf.wiki.kernel.org/index.php/Main_Page to profile the process, it is very intuitive and can run perf like top command to profile the userland process, it will show the top hot symbol consuming CPU cycles right in front of your face :)

#perf top

agentzh

Hello!

2014-11-21 14:28 GMT-08:00  vincent.mc.li:
>
> we have used gcc flag  -fno-omit-frame-pointer –ggdb to recompile our
> userland process and use https://perf.wiki.kernel.org/index.php/Main_Page to
> profile the process,

Does perf still need -fno-omit-frame-pointer to get proper backtrace?
It could have used the debuginfo instead of using the frame pointers
like systemtap.

> it is very intuitive and can run perf like top command
> to profile the userland process, it will show the top hot symbol consuming
> CPU cycles right in front of your face :)

Yes, perf can be used to generate C-land flame graphs as well though
I've found flame graphs are more intuitive and more informative than
the default "perf top" output format.

Alas, it's not easy to get perf to understand Lua-land backtraces (or
Perl-land, you name it) while it's trivial for systemtap. In many
cases they are way more useful than the C-land stats (like you're
stuck in a tight Lua loop or in a crazily backtracking regex run by
the dynamic machine code generated by PCRE JIT where the Lua-land
flame graph is the only viable choice).

Regards,
-agentzh

vincent.mc.li

On Friday, November 21, 2014 4:47:26 PM UTC-8, agentzh wrote:

Hello!

2014-11-21 14:28 GMT-08:00 vincent.mc.li:
>
> we have used gcc flag -fno-omit-frame-pointer –ggdb to recompile our
> userland process and use https://perf.wiki.kernel.org/index.php/Main_Page to
> profile the process,

Does perf still need -fno-omit-frame-pointer to get proper backtrace?
It could have used the debuginfo instead of using the frame pointers
like systemtap.

last time we tested, no debuginfo or unstripped binary needed, just the -fno-omit-frame-pointer -ggdb is good to
go, but I am not clear how it works internally, so I might be incorrect.

> it is very intuitive and can run perf like top command
> to profile the userland process, it will show the top hot symbol consuming
> CPU cycles right in front of your face :)

Yes, perf can be used to generate C-land flame graphs as well though
I've found flame graphs are more intuitive and more informative than
the default "perf top" output format.

Alas, it's not easy to get perf to understand Lua-land backtraces (or
Perl-land, you name it) while it's trivial for systemtap. In many
cases they are way more useful than the C-land stats (like you're
stuck in a tight Lua loop or in a crazily backtracking regex run by
the dynamic machine code generated by PCRE JIT where the Lua-land
flame graph is the only viable choice).

Regards,
-agentzh

agentzh

Hello!

On Mon, Nov 24, 2014 at 2:30 PM,  vincent.mc.li wrote:
> last time we tested, no debuginfo or unstripped binary needed, just the
> -fno-omit-frame-pointer -ggdb is good to
> go, but I am not clear how it works internally, so I might be incorrect.
>

If you want to strip debuginfo, then I don't see why -ggdb is useful here :)

Generally debuginfo can provide many useful information, including the
(virtual) inlined function frames :) See
https://github.com/openresty/nginx-systemtap-toolkit#resolve-inlines
for example.

Regards,
-agentzh