about no-pool-nginx

yang.yang · 2015-09-22T17:06:39+00:00

Hello! On Tue, Sep 22, 2015 at 11:04 AM, Martin Gee wrote: > I've written a C shared lib (which I load via Lua FFI) that has its own > internal cache...

about no-pool-nginx

yang.yang

Hi, agentzh

now I want to check the lua code using lua-nginx-module whether has memory leak and now first I install no-pool-patch, how to check nginx has install no-pool-nginx patch ? besides, how to check lua code whether has memory leak? thanks.

agentzh

Hello!

On Tue, Sep 22, 2015 at 5:06 PM, 杨阳 wrote:
> now I want to check the lua code using lua-nginx-module whether has memory
> leak and now first I install no-pool-patch, how to check nginx has install
> no-pool-nginx patch ? besides, how to check lua code whether has memory
> leak? thanks.
>

If you just want a "yes" or "no" answer to the memory leak problem,
then you can just keep loading your server with one request or a bunch
of typical requests or even random requests for hours or days or even
weeks. In the meantime, you check the memory footprint of the
processes and observe its curve trend. This is exactly how the "check
leaks" testing mode of my Test::Nginx::Socket test scaffold [1] works
while testing various nginx C modules and lua-resty-* libraries. For
example,

    http://qa.openresty.org/linux_x86_64_nginx_1_9_3__no_pool__ngx_rds_csv_l.html

If you just want to observe the GC-managed memory footprint in the
LuaJIT VM, then you can just use this systemtap-based tool, lj-gc [2],
or the gdb-based tool, lgc [3]. You can check out other tools in their
repositories which can give many more details about the GC objects
(live ones and/or live + dead ones).

Alternatively if you are quite sure that the leaks do exist, and you
actually want to locate the leaking place and eventually fix it, then
the answer can be quite long. To keep it extremely short: you just
need to use Valgrind memcheck and/or the "memory leak flamegraph" tool
[4].

Good luck!

Best regards,
-agentzh

[1] https://github.com/openresty/test-nginx
[2] https://github.com/openresty/stapxx#lj-gc
[3] https://github.com/openresty/nginx-gdb-utils#lgc
[4] https://github.com/openresty/stapxx#sample-bt-leaks

yang.yang

Is it necessary to just use one worker process to test or just turn master_process off ? If not, is it mean I just to capture one worker process's memory footprint to check the memory leak?

在 2015年9月22日星期二 UTC+8下午5:06:39，杨阳写道：

Hi, agentzh

now I want to check the lua code using lua-nginx-module whether has memory leak and now first I install no-pool-patch, how to check nginx has install no-pool-nginx patch ? besides, how to check lua code whether has memory leak? thanks.

agentzh

Hello!

On Wed, Sep 23, 2015 at 10:27 AM, 杨阳 wrote:
> Is it necessary to just use one worker process to test or just turn
> master_process off ?

For Valgrind, yes. For leak flame graphs and other things, no.
Valgrind just needs to run in the foreground.

>  If not,  is it mean I just to capture one worker
> process's memory footprint to check the memory leak?
>

Yes, and it's easy to identify a single worker that leaks.

Regards,
-agentzh

yang.yang

thanks for replying in time. and I have more questions,

a you say valgrind run in foreground mode means If I use valgrind (now I use the newest release version in linux), maybe I should not first make nginx start and just run valgrind xxxxx ./nginx/sbin/nginx to check memory

b I find valgrind has print some error with luajit because my nginx process compile with luajit, so should I add the patch ?

在 2015年9月22日星期二 UTC+8下午5:06:39，杨阳写道：

Hi, agentzh

now I want to check the lua code using lua-nginx-module whether has memory leak and now first I install no-pool-patch, how to check nginx has install no-pool-nginx patch ? besides, how to check lua code whether has memory leak? thanks.

agentzh

Hello!

On Wed, Sep 23, 2015 at 10:58 AM, 杨阳 wrote:
> a  you say valgrind run in foreground mode means If I use valgrind (now I
> use the newest release version in linux), maybe I should not first make
> nginx start and just run  valgrind xxxxx  ./nginx/sbin/nginx to check memory
>

Valgrind is a virtual machine emulating a CPU and various system
runtime libraries. So you cannot run a program natively first then
"attach" valgrind to it (like gdb or systemtap).

Please refer to the Valgrind user manual for more details.

> b  I find valgrind has print some error with luajit because my nginx process
> compile with luajit, so should I add the patch ?
>

Please note that I've been constantly running valgrind to test all the
OpenResty components for several years already:

    http://qa.openresty.org/

If you find any hidden issues via valgrind, please start a dedicated
thread here or a new ticket on GitHub issues.

Regards,
-agentzh

yang.yang

after I generate flame graph using lj-lua-stacks , I find some lines filled with hex code like '0xffffxxxxxxx 45%', how can I analysis what it is?

在 2015年9月22日星期二 UTC+8下午5:06:39，杨阳写道：

Hi, agentzh

now I want to check the lua code using lua-nginx-module whether has memory leak and now first I install no-pool-patch, how to check nginx has install no-pool-nginx patch ? besides, how to check lua code whether has memory leak? thanks.

agentzh

Hello!

On Wed, Sep 30, 2015 at 5:15 PM, 杨阳 wrote:
> after I generate flame graph using lj-lua-stacks , I find some lines filled
> with hex code like '0xffffxxxxxxx 45%', how can I analysis what it is?
>

Such binary frames are usually dynamic machine code generated by a JIT
compiler or static machine code without debug symbols or unwind
tables. It's hard to tell without actually seeing the graph. Mind to
share your SVG flame graph?

Regards,
-agentzh

yang.yang

pleasure to share with you. should I run debug nginx when generate the flame graph?

在 2015年10月1日星期四 UTC+8上午10:38:13，agentzh写道：

Hello!

On Wed, Sep 30, 2015 at 5:15 PM, 杨阳 wrote:
> after I generate flame graph using lj-lua-stacks , I find some lines filled
> with hex code like '0xffffxxxxxxx 45%', how can I analysis what it is?
>

Such binary frames are usually dynamic machine code generated by a JIT
compiler or static machine code without debug symbols or unwind
tables. It's hard to tell without actually seeing the graph. Mind to
share your SVG flame graph?

Regards,
-agentzh

Attachment: bf_range.svg
Description: image/svg

agentzh

Hello!

On Thu, Oct 8, 2015 at 10:14 AM, 杨阳 wrote:
>
> pleasure to share with you.  should I run debug nginx when generate the
> flame graph?
>

Seems like your nginx or luajit builds lack debug symbols. Those
binary frames do not look like JITted code to me.

Did you succeed in getting perfect C-land flame graphs?

Regards,
-agentzh

yang.yang

maybe I should add --with-debug when compile the nginx process

cd $(LUAJIT_DIR) && $(MAKE) CCDEBUG=-g

在 2015年10月8日星期四 UTC+8上午11:55:26，agentzh写道：

Hello!

On Thu, Oct 8, 2015 at 10:14 AM, 杨阳 wrote:
>
> pleasure to share with you. should I run debug nginx when generate the
> flame graph?
>

Seems like your nginx or luajit builds lack debug symbols. Those
binary frames do not look like JITted code to me.

Did you succeed in getting perfect C-land flame graphs?

Regards,
-agentzh

agentzh

Hello!

On Thu, Oct 8, 2015 at 3:08 PM, 杨阳 wrote:
> maybe I should add --with-debug when compile the nginx process
>

No. nginx enables debug symbols by default. So if you do not
specifically strip the nginx binary, you should be cool.

Another possibility is that you lack debug symbols in your system's
libc library.

Usually systemtap can give you hints on the console output while you
are running my tools.

>
> cd $(LUAJIT_DIR) && $(MAKE) CCDEBUG=-g
>

Yeah, this is the right way to enable debug symbols in custom LuaJIT
build. BTW, if you use the OpenResty bundle, then it ensures the
default LuaJIT library it builds contain debug symbols properly.

Regards,
-agentzh

yang.yang

pid=`ps aux | grep nginx | grep worker | awk '{print $2}'`

sudo ./stap++ -I ./tapset --arg time=60 --skip-badvars -x ${pid} samples/lj-lua-stacks.sxx > a.bt

../nginx-systemtap-toolkit/fix-lua-bt a.bt > lua.bt

../FlameGraph/stackcollapse-stap.pl a.bt > lua.cbt

../FlameGraph/flamegraph.pl --encoding="ISO-8859-1" --title="Lua-land on-CPU flamegraph" lua.cbt > lua.svg

after I turn on debug mode of nginx, I also find some hex code in the flame graph, but a little more clear than before.

在 2015年9月22日星期二 UTC+8下午5:06:39，杨阳写道：

Hi, agentzh

now I want to check the lua code using lua-nginx-module whether has memory leak and now first I install no-pool-patch, how to check nginx has install no-pool-nginx patch ? besides, how to check lua code whether has memory leak? thanks.

Attachment: bf_whole_debug.svg
Description: image/svg

agentzh

Hello!

On Thu, Oct 8, 2015 at 4:35 PM, 杨阳 wrote:
> sudo ./stap++ -I ./tapset --arg time=60 --skip-badvars -x ${pid}

What's the complete console output of this command?

> after I turn on debug mode of nginx, I also find some hex code in the flame
> graph, but a little more clear than before.
>

Will you share the C-land flame graph produced by the sample-bt tool instead?

Regards,
-agentzh

yang.yang

how to check whether libc has debug symbols?

在 2015年9月22日星期二 UTC+8下午5:06:39，杨阳写道：

Hi, agentzh

now I want to check the lua code using lua-nginx-module whether has memory leak and now first I install no-pool-patch, how to check nginx has install no-pool-nginx patch ? besides, how to check lua code whether has memory leak? thanks.

Attachment: a.bt
Description: Binary data

agentzh

Hello!

On Thu, Oct 8, 2015 at 4:45 PM, 杨阳 wrote:
> how to check whether libc has debug symbols?
>

Please just answer the two questions asked in my previous email. Thank
you. That info can give answers to your questions like above.

Regards,
-agentzh

yang.yang

when use userspace, it seems ok, but all hex code when use kernel space

在 2015年9月22日星期二 UTC+8下午5:06:39，杨阳写道：

Hi, agentzh

now I want to check the lua code using lua-nginx-module whether has memory leak and now first I install no-pool-patch, how to check nginx has install no-pool-nginx patch ? besides, how to check lua code whether has memory leak? thanks.

Attachment: bf_whole_user.svg
Description: image/svg

Attachment: bf_whole_kernel.svg
Description: image/svg

agentzh

Hello!

On Thu, Oct 8, 2015 at 5:23 PM, 杨阳 <yang...@upai.com> wrote:
> when use userspace, it seems ok,

Yeah, the user space C land flame graphs look perfect in that no sign
of the lack of debug symbols in nginx, luajit, or libc.

As you can see, however, your nginx is spending way too much CPU time
on flushing error logs. Maybe it's due to your use of --with-debug
option?

> but all hex code when use kernel space
>

Well, it's just because you don't have kernel debuginfo.

Regards,
-agentzh