why will nginx's map directive eat all ram under only OpenResty?

paynechu · 2013-08-03T13:54:54+00:00

Thanks for clarification :). and I will open another thread about the new issue thanksOn Saturday, August 3, 2013 1:43:22 PM UTC+8, agentzh wrote:Hello! On...

why will nginx's map directive eat all ram under only OpenResty?

paynechu

I have another problem, please see this post
http://forum.nginx.org/read.php?2,241478,241478#msg-241478
Before, I thought it's nginx problem. But after compare with the default
nginx setup I cannot reproduce the same issue. seems this is OpenResty's
nginx problem. any idea?

agentzh

Hello!

On Fri, Aug 2, 2013 at 10:54 PM, Payne Chu wrote:
> I have another problem, please see this post
> http://forum.nginx.org/read.php?2,241478,241478#msg-241478
> Before, I thought it's nginx problem. But after compare with the default
> nginx setup I cannot reproduce the same issue. seems this is OpenResty's
> nginx problem. any idea?
>

Several questions:

1. For your homebrew installation of nginx, is it also of the version
number 1.4.1?

2. Also, can you reproduce the memory leak with plain http protocol
(that is, without all the SSL configurations)?

3. And What is your ab command line?

4. Can you provide the detailed memory usage? Like detailed VIRT and RES values.

Best regards,
-agentzh

paynechu

1.) yes I tried both 1.4.1 and 1.4.2 in Homebrew both work fine even with or without SSL enabled.

2.) yes even without SSL the issue still there
3.) ab -k -c 100 -n 100000 http://localhost/

4.) please see attached file. the memory grow to 133MB after the ab run. if without use map directive the memory will only maintain around 19MB. and if continue run another times `ab` test. the memory will just continue grow more and I don't see it will drop back. I think this considerable is a leak.

On Saturday, August 3, 2013 2:00:20 PM UTC+8, agentzh wrote:

Hello!

On Fri, Aug 2, 2013 at 10:54 PM, Payne Chu wrote:
> I have another problem, please see this post
> http://forum.nginx.org/read.php?2,241478,241478#msg-241478
> Before, I thought it's nginx problem. But after compare with the default
> nginx setup I cannot reproduce the same issue. seems this is OpenResty's
> nginx problem. any idea?
>

Several questions:

1. For your homebrew installation of nginx, is it also of the version
number 1.4.1?

2. Also, can you reproduce the memory leak with plain http protocol
(that is, without all the SSL configurations)?

3. And What is your ab command line?

4. Can you provide the detailed memory usage? Like detailed VIRT and RES values.

Best regards,
-agentzh

Attachment: mem.log
Description: Binary data

agentzh

Hello!

On Sat, Aug 3, 2013 at 12:19 AM, Payne Chu wrote:
> 1.) yes I tried both 1.4.1 and 1.4.2 in Homebrew both work fine even with or
> without SSL enabled.

I've just built an nginx 1.4.1 from the official source tarball
(http://nginx.org/download/nginx-1.4.1.tar.gz ) and I can reproduce
the leak with it (with or without SSL). And I'm getting the leak on
two different Linux x86_64 boxes.

By using a simple systemtap script to trace the leak, and I've quickly
found the following place in the nginx 1.4.1 core is indeed leaking
most of the memory in the global ngx_cycle->pool memory pool:

    ??:0  posix_memalign
    .../nginx-1.4.1/src/os/unix/ngx_alloc.c:59  ngx_memalign
    .../nginx-1.4.1/src/core/ngx_palloc.c:185  ngx_palloc_block
    .../nginx-1.4.1/src/core/ngx_palloc.c:169  ngx_pnalloc
    .../nginx-1.4.1/src/core/ngx_conf_file.c:826  ngx_conf_full_name
    .../nginx-1.4.1/src/http/ngx_http_core_module.c:2019
ngx_http_map_uri_to_path
    .../nginx-1.4.1/src/http/modules/ngx_http_index_module.c:164
ngx_http_index_handler
    .../nginx-1.4.1/src/http/ngx_http_core_module.c:1417
ngx_http_core_content_phase
    .../nginx-1.4.1/src/http/ngx_http_core_module.c:890
ngx_http_core_run_phases
    .../nginx-1.4.1/src/http/ngx_http_core_module.c:872  ngx_http_handler

So we need to fix the leak in the nginx core.

BTW, I'm using the following systemtap script to trace this leak. One
can also easily port it over to dtrace (for Mac OS X, Solaris, or
FreeBSD):

    http://agentzh.org/misc/leaks.stp

I'm not sure why you cannot reproduce it with homebrew installations
of nginx. Could you please re-check your configurations? Because a
leak is definitely there.

Best regards,
-agentzh

paynechu

haa~ you are right Agentzh, the leak is there.~

I found out it will only leak when I use relative path. and all version also can reproduce. b4 I test Homebrew version with full path not relative path that's the different....XD~

map $pid $public_root { default public; } <-- this one relative path leak happened

map $pid $public_root { default /usr/local/Cellar/nginx/1.4.1/public; } <- this one full path without leak..

anyways I will report back to Nginx

On Saturday, August 3, 2013 3:37:54 PM UTC+8, agentzh wrote:

Hello!

On Sat, Aug 3, 2013 at 12:19 AM, Payne Chu wrote:
> 1.) yes I tried both 1.4.1 and 1.4.2 in Homebrew both work fine even with or
> without SSL enabled.

I've just built an nginx 1.4.1 from the official source tarball
(http://nginx.org/download/nginx-1.4.1.tar.gz ) and I can reproduce
the leak with it (with or without SSL). And I'm getting the leak on
two different Linux x86_64 boxes.

By using a simple systemtap script to trace the leak, and I've quickly
found the following place in the nginx 1.4.1 core is indeed leaking
most of the memory in the global ngx_cycle->pool memory pool:

??:0 posix_memalign
.../nginx-1.4.1/src/os/unix/ngx_alloc.c:59 ngx_memalign
.../nginx-1.4.1/src/core/ngx_palloc.c:185 ngx_palloc_block
.../nginx-1.4.1/src/core/ngx_palloc.c:169 ngx_pnalloc
.../nginx-1.4.1/src/core/ngx_conf_file.c:826 ngx_conf_full_name
.../nginx-1.4.1/src/http/ngx_http_core_module.c:2019
ngx_http_map_uri_to_path
.../nginx-1.4.1/src/http/modules/ngx_http_index_module.c:164
ngx_http_index_handler
.../nginx-1.4.1/src/http/ngx_http_core_module.c:1417
ngx_http_core_content_phase
.../nginx-1.4.1/src/http/ngx_http_core_module.c:890
ngx_http_core_run_phases
.../nginx-1.4.1/src/http/ngx_http_core_module.c:872 ngx_http_handler

So we need to fix the leak in the nginx core.

BTW, I'm using the following systemtap script to trace this leak. One
can also easily port it over to dtrace (for Mac OS X, Solaris, or
FreeBSD):

http://agentzh.org/misc/leaks.stp

I'm not sure why you cannot reproduce it with homebrew installations
of nginx. Could you please re-check your configurations? Because a
leak is definitely there.

Best regards,
-agentzh

paynechu

http://forum.nginx.org/read.php?2,241478,241523#msg-241523

This is the patch for fix this issue :)~

On Saturday, August 3, 2013 3:37:54 PM UTC+8, agentzh wrote:

Hello!

On Sat, Aug 3, 2013 at 12:19 AM, Payne Chu wrote:
> 1.) yes I tried both 1.4.1 and 1.4.2 in Homebrew both work fine even with or
> without SSL enabled.

I've just built an nginx 1.4.1 from the official source tarball
(http://nginx.org/download/nginx-1.4.1.tar.gz ) and I can reproduce
the leak with it (with or without SSL). And I'm getting the leak on
two different Linux x86_64 boxes.

By using a simple systemtap script to trace the leak, and I've quickly
found the following place in the nginx 1.4.1 core is indeed leaking
most of the memory in the global ngx_cycle->pool memory pool:

??:0 posix_memalign
.../nginx-1.4.1/src/os/unix/ngx_alloc.c:59 ngx_memalign
.../nginx-1.4.1/src/core/ngx_palloc.c:185 ngx_palloc_block
.../nginx-1.4.1/src/core/ngx_palloc.c:169 ngx_pnalloc
.../nginx-1.4.1/src/core/ngx_conf_file.c:826 ngx_conf_full_name
.../nginx-1.4.1/src/http/ngx_http_core_module.c:2019
ngx_http_map_uri_to_path
.../nginx-1.4.1/src/http/modules/ngx_http_index_module.c:164
ngx_http_index_handler
.../nginx-1.4.1/src/http/ngx_http_core_module.c:1417
ngx_http_core_content_phase
.../nginx-1.4.1/src/http/ngx_http_core_module.c:890
ngx_http_core_run_phases
.../nginx-1.4.1/src/http/ngx_http_core_module.c:872 ngx_http_handler

So we need to fix the leak in the nginx core.

BTW, I'm using the following systemtap script to trace this leak. One
can also easily port it over to dtrace (for Mac OS X, Solaris, or
FreeBSD):

http://agentzh.org/misc/leaks.stp

I'm not sure why you cannot reproduce it with homebrew installations
of nginx. Could you please re-check your configurations? Because a
leak is definitely there.

Best regards,
-agentzh

agentzh

Hello!

On Sat, Aug 3, 2013 at 12:37 AM, Yichun Zhang (agentzh) wrote:
> BTW, I'm using the following systemtap script to trace this leak. One
> can also easily port it over to dtrace (for Mac OS X, Solaris, or
> FreeBSD):
>
>     http://agentzh.org/misc/leaks.stp
>

It's worth mentioning that the output of the this simply systemtap
tool can also be used to generate a "Memory Leak Flame Graph". For
example, for this very leak in the Nginx core, we can get the
following "leak flamegraph":

    http://agentzh.org/misc/flamegraph/nginx-memleak-2013-08-05.svg

This tool is very general and can be used to analyze *any* C processes
with DWARF debug info enabled.

I'll ask Guanlan Dai to add this tool to my Nginx Systemtap Toolkit in
the near future :)

BTW, glad to see that the official Nginx team has fixed this this leak
:) The fix is straightforward as expected, that is, just allocating in
the request memory pool instead of the global cycle one :)

Best regards,
-agentzh