OpenResty 1.7.10.1 crash FreeBSD 10.1 amd64

sparvu_71a1 · 2015-03-31T21:54:11+00:00

Hello! On Sat, Mar 28, 2015 at 4:07 PM, Lord Nynex wrote: > I am able to get data now without hangups or coredumps. > Glad you've got it working :)...

OpenResty 1.7.10.1 crash FreeBSD 10.1 amd64

sparvu_71a1

Hi,

We are seeing lots of crashes from our OpenResty instance. Details below:

OS: FreeBSD 10.1 amd64

NGINX: nginx version: openresty/1.7.10.1
built by clang 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
TLS SNI support enabled
configure arguments: --prefix=/opt/kronometrix/kernel/nginx 
--with-cc-opt='-I/usr/home/krmx/workspace/kronometrix/kernel/openresty/ngx_openresty-1.7.10.1/build/luajit-root/opt/kronometrix/kernel/luajit/include/luajit-2.1 -O2' 
--add-module=../ngx_devel_kit-0.2.19 --add-module=../echo-nginx-module-0.57 --add-module=../xss-nginx-module-0.04 
--add-module=../ngx_coolkit-0.2rc2 --add-module=../set-misc-nginx-module-0.28 --add-module=../form-input-nginx-module-0.10 
--add-module=../encrypted-session-nginx-module-0.03 --add-module=../srcache-nginx-module-0.29 --add-module=../ngx_lua-0.9.15 
--add-module=../ngx_lua_upstream-0.02 --add-module=../headers-more-nginx-module-0.25 --add-module=../array-var-nginx-module-0.03 
--add-module=../memc-nginx-module-0.15 --add-module=../redis2-nginx-module-0.11 --add-module=../redis-nginx-module-0.3.7 
--add-module=../rds-json-nginx-module-0.13 --add-module=../rds-csv-nginx-module-0.05 
--with-ld-opt='-Wl,-rpath,/opt/kronometrix/kernel/luajit/lib -L/usr/home/krmx/workspace/kronometrix/kernel/openresty/ngx_openresty-1.7.10.1/build/luajit-root/opt/kronometrix/kernel/luajit/lib' --with-cc=/usr/bin/cc --with-http_ssl_module

Under nginx error log:
2015/03/31 16:36:43 [alert] 18337#0: worker process 18498 exited on signal 11 (core dumped)
2015/03/31 16:37:13 [alert] 18337#0: worker process 18431 exited on signal 11 (core dumped)


Trying to use gdb on one of these return always this stack trace:

Core was generated by `nginx'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /lib/libcrypt.so.5...done.
Loaded symbols for /lib/libcrypt.so.5
Reading symbols from /lib/libm.so.5...done.
Loaded symbols for /lib/libm.so.5
Reading symbols from /usr/local/lib/libpcre.so.3...done.
Loaded symbols for /usr/local/lib/libpcre.so.3
Reading symbols from /usr/lib/libssl.so.7...done.
Loaded symbols for /usr/lib/libssl.so.7
Reading symbols from /lib/libcrypto.so.7...done.
Loaded symbols for /lib/libcrypto.so.7
Reading symbols from /lib/libz.so.6...done.
Loaded symbols for /lib/libz.so.6
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libc.so.7...done.
Loaded symbols for /lib/libc.so.7
Reading symbols from /lib/libthr.so.3...done.
Loaded symbols for /lib/libthr.so.3
Reading symbols from /opt/kronometrix/kernel/lualib/cjson.so...done.
Loaded symbols for /opt/kronometrix/kernel/lualib/cjson.so
Reading symbols from /opt/kronometrix/kernel/lualib/lfs.so...done.
Loaded symbols for /opt/kronometrix/kernel/lualib/lfs.so
Reading symbols from /libexec/ld-elf.so.1...done.
Loaded symbols for /libexec/ld-elf.so.1
#0  0x0000000802835618 in ?? ()
[New Thread 802806400 (LWP 100477/<unknown>)]
[New LWP 100423]
(gdb) bt

#0  0x0000000802835618 in ?? ()
#1  0x00000000004acc02 in ngx_http_lua_wev_handler (r=0x802879050) at ../ngx_lua-0.9.15/src/ngx_http_lua_util.c:1548
#2  0x000000000044f824 in ngx_http_request_handler (ev=<value optimized out>) at src/http/ngx_http_request.c:2184
#3  0x0000000000434b63 in ngx_event_process_posted (cycle=<value optimized out>, posted=<value optimized out>)
    at src/event/ngx_event_posted.c:33
#4  0x000000000043bcea in ngx_worker_process_cycle (cycle=0x802834050, data=<value optimized out>)
    at src/os/unix/ngx_process_cycle.c:824
#5  0x00000000004397ae in ngx_spawn_process (cycle=0x802834050, proc=0x43bc20 <ngx_worker_process_cycle>, data=0x2, 
    name=0x53fdfd "worker process", respawn=2) at src/os/unix/ngx_process.c:198
#6  0x000000000043a37f in ngx_master_process_cycle (cycle=<value optimized out>) at src/os/unix/ngx_process_cycle.c:631
#7  0x000000000041cb1a in main (argc=<value optimized out>, argv=<value optimized out>) at src/core/nginx.c:407

The nginx.conf is defined:
https://github.com/kronometrix/kernel/blob/master/config/freebsd/nginx.conf.maas


Anyone any ideas ? Can we get any sense whats going on without building nginx with debug
information ? 

thanks,
-- 
Stefan Parvu <sp...@kronometrix.org>

manchev

Seems related to this: https://github.com/openresty/lua-nginx-module/issues/478

I guess you're getting this under load?

Probably should wait for a fix for the issue pointed out above and then test again.

Best,

Vladislav

On Tue, Mar 31, 2015 at 4:54 PM, Stefan Parvu <sp...@kronometrix.org> wrote:

Hi,

We are seeing lots of crashes from our OpenResty instance. Details below:

OS: FreeBSD 10.1 amd64

NGINX: nginx version: openresty/1.7.10.1
built by clang 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
TLS SNI support enabled
configure arguments: --prefix=/opt/kronometrix/kernel/nginx
--with-cc-opt='-I/usr/home/krmx/workspace/kronometrix/kernel/openresty/ngx_openresty-1.7.10.1/build/luajit-root/opt/kronometrix/kernel/luajit/include/luajit-2.1 -O2'
--add-module=../ngx_devel_kit-0.2.19 --add-module=../echo-nginx-module-0.57 --add-module=../xss-nginx-module-0.04
--add-module=../ngx_coolkit-0.2rc2 --add-module=../set-misc-nginx-module-0.28 --add-module=../form-input-nginx-module-0.10
--add-module=../encrypted-session-nginx-module-0.03 --add-module=../srcache-nginx-module-0.29 --add-module=../ngx_lua-0.9.15
--add-module=../ngx_lua_upstream-0.02 --add-module=../headers-more-nginx-module-0.25 --add-module=../array-var-nginx-module-0.03
--add-module=../memc-nginx-module-0.15 --add-module=../redis2-nginx-module-0.11 --add-module=../redis-nginx-module-0.3.7
--add-module=../rds-json-nginx-module-0.13 --add-module=../rds-csv-nginx-module-0.05
--with-ld-opt='-Wl,-rpath,/opt/kronometrix/kernel/luajit/lib -L/usr/home/krmx/workspace/kronometrix/kernel/openresty/ngx_openresty-1.7.10.1/build/luajit-root/opt/kronometrix/kernel/luajit/lib' --with-cc=/usr/bin/cc --with-http_ssl_module

Under nginx error log:
2015/03/31 16:36:43 [alert] 18337#0: worker process 18498 exited on signal 11 (core dumped)
2015/03/31 16:37:13 [alert] 18337#0: worker process 18431 exited on signal 11 (core dumped)

Trying to use gdb on one of these return always this stack trace:

Core was generated by `nginx'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /lib/libcrypt.so.5...done.
Loaded symbols for /lib/libcrypt.so.5
Reading symbols from /lib/libm.so.5...done.
Loaded symbols for /lib/libm.so.5
Reading symbols from /usr/local/lib/libpcre.so.3...done.
Loaded symbols for /usr/local/lib/libpcre.so.3
Reading symbols from /usr/lib/libssl.so.7...done.
Loaded symbols for /usr/lib/libssl.so.7
Reading symbols from /lib/libcrypto.so.7...done.
Loaded symbols for /lib/libcrypto.so.7
Reading symbols from /lib/libz.so.6...done.
Loaded symbols for /lib/libz.so.6
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libc.so.7...done.
Loaded symbols for /lib/libc.so.7
Reading symbols from /lib/libthr.so.3...done.
Loaded symbols for /lib/libthr.so.3
Reading symbols from /opt/kronometrix/kernel/lualib/cjson.so...done.
Loaded symbols for /opt/kronometrix/kernel/lualib/cjson.so
Reading symbols from /opt/kronometrix/kernel/lualib/lfs.so...done.
Loaded symbols for /opt/kronometrix/kernel/lualib/lfs.so
Reading symbols from /libexec/ld-elf.so.1...done.
Loaded symbols for /libexec/ld-elf.so.1
#0 0x0000000802835618 in ?? ()
[New Thread 802806400 (LWP 100477/<unknown>)]
[New LWP 100423]
(gdb) bt

#0 0x0000000802835618 in ?? ()
#1 0x00000000004acc02 in ngx_http_lua_wev_handler (r=0x802879050) at ../ngx_lua-0.9.15/src/ngx_http_lua_util.c:1548
#2 0x000000000044f824 in ngx_http_request_handler (ev=<value optimized out>) at src/http/ngx_http_request.c:2184
#3 0x0000000000434b63 in ngx_event_process_posted (cycle=<value optimized out>, posted=<value optimized out>)
at src/event/ngx_event_posted.c:33
#4 0x000000000043bcea in ngx_worker_process_cycle (cycle=0x802834050, data="" optimized out>)
at src/os/unix/ngx_process_cycle.c:824
#5 0x00000000004397ae in ngx_spawn_process (cycle=0x802834050, proc=0x43bc20 <ngx_worker_process_cycle>, data=""> name=0x53fdfd "worker process", respawn=2) at src/os/unix/ngx_process.c:198
#6 0x000000000043a37f in ngx_master_process_cycle (cycle=<value optimized out>) at src/os/unix/ngx_process_cycle.c:631
#7 0x000000000041cb1a in main (argc=<value optimized out>, argv=<value optimized out>) at src/core/nginx.c:407

The nginx.conf is defined:
https://github.com/kronometrix/kernel/blob/master/config/freebsd/nginx.conf.maas

Anyone any ideas ? Can we get any sense whats going on without building nginx with debug
information ?

thanks,
--
Stefan Parvu <sp...@kronometrix.org.

sparvu_71a1

> Seems related to this:
> https://github.com/openresty/lua-nginx-module/issues/478

ok, thanks for hint. 

> I guess you're getting this under load?

Thats correct. We are testing our software simulating load from different clients.

Strange: If I use 4k as number of files and worker connections under nginx.conf I dont see that often the 
crash. If I increase the config like:

worker_rlimit_nofile 32768;
events {
    use kqueue;
    worker_connections 32768;
}

I can get more often the worker crashes. Im still testing ...  Im not 100% sure about this.

The OS has been configured to have:

# sysctl kern.maxfiles
kern.maxfiles: 204800
# sysctl kern.maxfilesperproc
kern.maxfilesperproc: 184320


-- 
Stefan Parvu <sp...@kronometrix.org>

agentzh

Hello!

On Tue, Mar 31, 2015 at 6:54 AM, Stefan Parvu wrote:
> #0  0x0000000802835618 in ?? ()
> #1  0x00000000004acc02 in ngx_http_lua_wev_handler (r=0x802879050) at ../ngx_lua-0.9.15/src/ngx_http_lua_util.c:1548
> #2  0x000000000044f824 in ngx_http_request_handler (ev=<value optimized out>) at src/http/ngx_http_request.c:2184
> #3  0x0000000000434b63 in ngx_event_process_posted (cycle=<value optimized out>, posted=<value optimized out>)
>     at src/event/ngx_event_posted.c:33

Do you have more than one light threads working on the same downstream
cosocket object? If yes, then it's very likely to be related to
https://github.com/openresty/lua-nginx-module/issues/478 Aviram will
post a patch soon according to him :)

Best regards,
-agentzh

sparvu_71a1

> Do you have more than one light threads working on the same downstream
> cosocket object? If yes, then it's very likely to be related to
> https://github.com/openresty/lua-nginx-module/issues/478 Aviram will
> post a patch soon according to him :)

yes, we do. I see. Many thanks indeed.


-- 
Stefan Parvu <sp...@kronometrix.org>