ssl_certificate_by_lua - nginx signal 11

petr · 2017-04-25T06:04:18+00:00

Whups, I mis-remember the implementation here. Thank Thibault for the correction!On Fri, Apr 28, 2017 at 3:42 PM, Thibault Charbonnier <thiba...@fastmail....

ssl_certificate_by_lua - nginx signal 11

petr

Hi,

I have rather highload server and trying to implement dynamic certificates base on server_name.

nginx 1.10.3 with lua-nginx-module-0.10.7

After all I manged to simplify lua/ssl related config like following:

server {

listen 1.2.3.4:80;

listen 1.2.3.5:80;

listen 1.2.3.4:443 ssl;

listen 1.2.3.5:443 ssl;

server_name server_name;

ssl_certificate /usr/local/nginx/conf/cert/default_cert.crt;

ssl_certificate_key /usr/local/nginx/conf/cert/default_cert.key;

ssl_certificate_by_lua_block {

return

}

... (lots of locations, proxy-passes, etc) ...
}

The issue is in nginx error log:

2017/04/25 00:11:38 [alert] 3305#0: worker process 25921 exited on signal 11 (core dumped)

2017/04/25 00:11:39 [alert] 3305#0: worker process 25916 exited on signal 11 (core dumped)

2017/04/25 00:11:40 [alert] 3305#0: worker process 27662 exited on signal 11 (core dumped)

Nginx worker does not fails on every request, it has SIGSEV on every 100th request or something like this. Yes, I have more than 100 rps.

gsb output:

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Core was generated by `nginx: worker process '.

Program terminated with signal SIGSEGV, Segmentation fault.

#0 0x0000000000000000 in ?? ()

(gdb) bt

#0 0x0000000000000000 in ?? ()

#1 0x000000000054f5cd in ngx_http_lua_ssl_cert_handler (ssl_conn=0x7f3847c681e0, data="" at ../lua-nginx-module-0.10.7/src/ngx_http_lua_ssl_certby.c:301

#2 0x00007f38b733f7d8 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2

#3 0x00007f38b73444df in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2

#4 0x00007f38b7353a64 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2

#5 0x000000000045b646 in ngx_ssl_handshake (c=0x7f3848896390) at src/event/ngx_event_openssl.c:1125

#6 0x0000000000476051 in ngx_http_ssl_handshake (rev=0x7f3848b4d2a0) at src/http/ngx_http_request.c:718

#7 0x0000000000449cbe in ngx_event_process_posted (cycle=0x7f384826b090, posted=0x7cf210 <ngx_posted_events>) at src/event/ngx_event_posted.c:33

#8 0x00000000004474f2 in ngx_process_events_and_timers (cycle=0x7f384826b090) at src/event/ngx_event.c:259

#9 0x00000000004556dc in ngx_worker_process_cycle (cycle=0x7f384826b090, data="" at src/os/unix/ngx_process_cycle.c:753

#10 0x0000000000451efe in ngx_spawn_process (cycle=0x7f384826b090, proc=0x4555e7 <ngx_worker_process_cycle>, data="" name=0x566013 "worker process", respawn=-4) at src/os/unix/ngx_process.c:198

#11 0x0000000000454518 in ngx_start_worker_processes (cycle=0x7f384826b090, n=8, type=-4) at src/os/unix/ngx_process_cycle.c:358

#12 0x0000000000454058 in ngx_master_process_cycle (cycle=0x7f384826b090) at src/os/unix/ngx_process_cycle.c:243

#13 0x00000000004186e4 in main (argc=1, argv=0x7ffffc1c2f28) at src/core/nginx.c:367

# /usr/local/nginx/sbin/nginx -V

nginx version: nginx/1.10.3

built by gcc 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.3)

built with OpenSSL 1.0.2k 26 Jan 2017

TLS SNI support enabled

configure arguments: --prefix=/usr/local/nginx --with-debug --with-cc-opt='-O0 -g' --with-ld-opt=-Wl,-rpath,-L/usr/local/lib,-luuid --with-http_stub_status_module --with-http_gzip_static_module --with-http_ssl_module --with-file-aio --with-http_realip_module --with-http_dav_module --with-http_geoip_module --with-http_secure_link_module --with-http_v2_module --add-module=../ngx_devel_kit-0.3.0 --add-module=../lua-nginx-module-0.10.7 --add-module=/opt/workspace/infrastructure/server/nginx/nginx-x-rid-header --add-module=../set-misc-nginx-module-0.31

I've tried to use lua 5.2.3 from ubuntu 14.04 packages as well as self-compiled LuaJIT 2.1.0-beta2 - and there is no change, I have the same issue.

What else info can I provide? It seems I can play with coredump to gather more info.

I would appreciate any help you can provide.

petr

вторник, 25 апреля 2017 г., 1:04:19 UTC+3 пользователь Serge Negodyuck написал:

Hi,
I have rather highload server and trying to implement dynamic certificates base on server_name.

nginx 1.10.3 with lua-nginx-module-0.10.7

I've just tried latest openresty/1.11.2.3, and I have the same issue with the same stacktrace.

petr

вторник, 25 апреля 2017 г., 12:36:03 UTC+3 пользователь Serge Negodyuck написал:

вторник, 25 апреля 2017 г., 1:04:19 UTC+3 пользователь Serge Negodyuck написал:
Hi,
I have rather highload server and trying to implement dynamic certificates base on server_name.

nginx 1.10.3 with lua-nginx-module-0.10.7

I've just tried latest openresty/1.11.2.3, and I have the same issue with the same stacktrace.

Seems lscf->srv is null

Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000000000000 in ?? ()(gdb) up

#1 0x000000000056bc36 in ngx_http_lua_ssl_cert_handler (ssl_conn=0x4450770, data="" at ../ngx_lua-0.10.8/src/ngx_http_lua_ssl_certby.c:301
301 rc = lscf->srv.ssl_cert_handler(r, lscf, L);
(gdb) print lscf
$1 = (ngx_http_lua_srv_conf_t *) 0x540da20
(gdb) print lscf->srv
$2 = {ssl_cert_handler = 0x0, ssl_cert_src = {len = 0, data = "" ssl_cert_src_key = 0x0, ssl_sess_store_handler = 0x0, ssl_sess_store_src = {len = 0, data = ""
ssl_sess_store_src_key = 0x0, ssl_sess_fetch_handler = 0x0, ssl_sess_fetch_src = {len = 0, data = "" ssl_sess_fetch_src_key = 0x0}
(gdb) print lscf->balancer
$3 = {src = "" = 0, data = "" src_key = 0x0, handler = 0x0}

petr

Seems lscf->srv is null

Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000000000000 in ?? ()(gdb) up
#1 0x000000000056bc36 in ngx_http_lua_ssl_cert_handler (ssl_conn=0x4450770, data="" at ../ngx_lua-0.10.8/src/ngx_http_lua_ssl_certby.c:301
301 rc = lscf->srv.ssl_cert_handler(r, lscf, L);
(gdb) print lscf
$1 = (ngx_http_lua_srv_conf_t *) 0x540da20
(gdb) print lscf->srv
$2 = {ssl_cert_handler = 0x0, ssl_cert_src = {len = 0, data = "" ssl_cert_src_key = 0x0, ssl_sess_store_handler = 0x0, ssl_sess_store_src = {len = 0, data = ""
ssl_sess_store_src_key = 0x0, ssl_sess_fetch_handler = 0x0, ssl_sess_fetch_src = {len = 0, data = "" ssl_sess_fetch_src_key = 0x0}
(gdb) print lscf->balancer
$3 = {src = "" = 0, data = "" src_key = 0x0, handler = 0x0}

This patch helped me to prevent segmentation fault:

--- ngx_http_lua_ssl_certby.c.orig      2017-04-25 14:21:02.773608318 +0300
+++ ngx_http_lua_ssl_certby.c   2017-04-25 14:21:56.693539874 +0300
@@ -298,7 +298,12 @@

    c->log->action = "" SSL certificate by lua";

-    rc = lscf->srv.ssl_cert_handler(r, lscf, L);
+    if (lscf->srv.ssl_cert_handler) {
+        rc = lscf->srv.ssl_cert_handler(r, lscf, L);
+    } else {
+        ngx_ssl_error(NGX_LOG_ALERT, c->log, 0, "srv.ssl_cert_handler is Null");
+        goto failed;
+    }

    if (rc >= NGX_OK || rc == NGX_ERROR) {
        cctx->done = 1;

But I stiil need help to find out the reason of null pointer.