大压力测试下，ssl_certificate_by_lua概率性发生coredump

anymaybe

环境信息：

nginx监听有1057、1059两个端口。1057是http业务；1059是https业务，使用ssl_certificate_by_lua设置证书

openresty信息：

nginx version: openresty/1.9.15.1

built with OpenSSL 1.0.2k 26 Jan 2017

TLS SNI support enabled

configure arguments: --prefix=/usr/local/NSP/nginx --with-debug --with-cc-opt='-DNGX_LUA_USE_ASSERT -DNGX_LUA_ABORT_AT_PANIC -O2 -I/usr/local/include -I/home/nginx/elb_openresty_package/osscrypto/include' --add-module=../ngx_devel_kit-0.3.0 --add-module=../iconv-nginx-module-0.14 --add-module=../echo-nginx-module-0.59 --add-module=../xss-nginx-module-0.05 --add-module=../ngx_coolkit-0.2rc3 --add-module=../set-misc-nginx-module-0.30 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.05 --add-module=../srcache-nginx-module-0.31 --add-module=../ngx_lua-0.10.5 --add-module=../ngx_lua_upstream-0.05 --add-module=../headers-more-nginx-module-0.30 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.17 --add-module=../redis2-nginx-module-0.13 --add-module=../redis-nginx-module-0.3.7 --add-module=../rds-json-nginx-module-0.14 --add-module=../rds-csv-nginx-module-0.07 --with-ld-opt='-Wl,-rpath,/usr/local/NSP/luajit/lib -L/home/nginx/elb_openresty_package/osscrypto/lib -losscrypto' --user=elb --group=elb --sbin-path=/usr/local/NSP/sbin --conf-path=/usr/local/NSP/etc/nginx/nginx.conf --http-log-path=/usr/local/NSP/var/log/nginx/access.log --error-log-path=/usr/local/NSP/var/log/nginx/error.log --pid-path=/usr/local/NSP/var/run/nginx.pid --lock-path=/usr/local/NSP/var/run/nginx.lock --http-client-body-temp-path=/usr/local/NSP/var/tmp/nginx_client_body --http-proxy-temp-path=/usr/local/NSP/var/tmp/nginx_proxy_temp --http-fastcgi-temp-path=/usr/local/NSP/var/tmp/nginx_fastcgi_temp --http-uwsgi-temp-path=/usr/local/NSP/var/tmp/uwsgi_temp --http-scgi-temp-path=/usr/local/NSP/var/tmp/scgi_temp --with-openssl=/home/nginx/elb_openresty_package/openresty-1.9.15.1/../3rdparty/openssl-1.0.2k --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-mail --with-mail_ssl_module --with-cpp_test_module --with-stream --with-stream_ssl_module --add-module=/home/nginx/elb_openresty_package/openresty-1.9.15.1/../3rdparty/nginx_upstream_check_module-0.2.0 --add-module=/home/nginx/elb_openresty_package/openresty-1.9.15.1/../3rdparty/stream-lua-nginx-module-0.0.1 --add-module=/home/nginx/elb_openresty_package/openresty-1.9.15.1/../3rdparty/nginx-goodies-nginx-sticky-module-ng-c78b7dd79d0d --with-pcre=/home/nginx/elb_openresty_package/openresty-1.9.15.1/../3rdparty/pcre-8.39 --with-zlib=/home/nginx/elb_openresty_package/openresty-1.9.15.1/../3rdparty/zlib-1.2.8 --with-pcre-jit --with-openssl-opt=enable-tlsext

https监听配置：

server {

listen 30.0.0.120:1059 ssl backlog=10000 reuseport;

server_name 50.0.0.18;

root /usr/local/NSP/nginx/html/;

ssl_certificate /usr/local/NSP/etc/nginx/template.crt;

ssl_certificate_key /usr/local/NSP/etc/nginx/template_nopwd.key;

ssl_session_cache shared:SSL:20m;

ssl_session_timeout 10m;

ssl_protocols TLSv1.2 TLSv1.1 TLSv1;

ssl_ciphers ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:AES128-SHA256:AES256-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:AES128-SHA:AES256-SHA:DHE-DSS-AES128-SHA:CAMELLIA128-SHA:EDH-RSA-DES-CBC3-SHA:DES-CBC3-SHA:ECDHE-RSA-RC4-SHA:RC4-SHA:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DHE-RSA-CAMELLIA256-SHA:DHE-DSS-CAMELLIA256-SHA:CAMELLIA256-SHA:EDH-DSS-DES-CBC3-SHA:DHE-RSA-CAMELLIA128-SHA:DHE-DSS-CAMELLIA128-SHA;

ssl_prefer_server_ciphers on;

ssl_certificate_by_lua_block {

local certificate_util = require "utils.certificate_util"

certificate_util.load_certificate_file("01dba61d434449d78b3ada88f3bec4ac")

}

access_log /usr/local/NSP/nginx/logs/access_01dba61d434449d78b3ada88f3bec4ac.log main;

keepalive_timeout 300;

include syshosts/proxy_location_ext;

proxy_set_header X-Forwarded-Port 7000;

}

http监听配置：

server {

listen 30.0.0.120:1057 backlog=10000 reuseport;

server_name 100.0.0.3;

root /usr/local/NSP/nginx/html/;

access_log /usr/local/NSP/nginx/logs/access_e63e2eff0bfd422b9de82fc7b5cda1cc.log main;

keepalive_timeout 300;

include syshosts/proxy_location_ext;

proxy_set_header X-Forwarded-Port 8000;

}

proxy配置：

location / {

set $router_request_uri $uri;

set $router_proxy_scheme 'http';

access_by_lua_file '/usr/local/NSP/etc/router/router_ext.luac';

proxy_set_header Host $http_host;

proxy_set_header Upgrade $http_upgrade;

proxy_set_header Connection $connection_upgrade;

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header X-Forwarded-Host $http_host;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

proxy_set_header X-Forwarded-Proto $scheme;

proxy_http_version 1.1;

proxy_pass $router_proxy_scheme://balancer_backend;

header_filter_by_lua_file '/usr/local/NSP/etc/router/router_header_filter_ext.luac';

log_by_lua_file '/usr/local/NSP/etc/router/router_log_ext.luac';

}

测试方法：

使用硬件测试仪对http端口1057进行压力测试，每秒7万新建，总并发连接保持在96万（nginx的worker_connections * worker数）

在压测过程中，每隔5秒访问一次https 1059服务。

跑的过程中内核日志出现：

[278527.697161] nginx[43879]: segfault at 0 ip 00000000004f38aa sp 00007ffce92cbea0 error 4 in nginx[400000+216000]

coredump堆栈如下：

[root@localhost NSP]# gdb ./sbin/nginx coredumps/core.3286

GNU gdb (GDB)

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law. Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-Huawei-linux-gnu".

For bug reporting instructions, please see:

<http://www.gnu.org/software/gdb/bugs/>...

Reading symbols from /usr/local/NSP/sbin/nginx...done.

[New LWP 3286]

Cannot access memory at address 0x7f283fa121e8

Cannot access memory at address 0x7f283fa121e0

(gdb) bt

#0 0x00000000005082c3 in ngx_http_lua_ssl_cert_handler (ssl_conn=<optimized out>, data="" out>)

at ../ngx_lua-0.10.5/src/ngx_http_lua_ssl_certby.c:269

#1 0x00000000005b04b8 in ssl3_get_client_hello ()

#2 0x00000000005b512f in ssl3_accept ()

#3 0x0000000000591184 in ssl23_accept ()

#4 0x0000000000453346 in ngx_ssl_handshake (c=c@entry=0x7f27db2158b0) at src/event/ngx_event_openssl.c:1199

#5 0x000000000046753c in ngx_http_ssl_handshake (rev=0x7f27dad40f30) at src/http/ngx_http_request.c:719

#6 0x00000000004503d9 in ngx_epoll_process_events (cycle=0x7f27dbd97060, timer=<optimized out>, flags=<optimized out>)

at src/event/modules/ngx_epoll_module.c:822

#7 0x0000000000446b67 in ngx_process_events_and_timers (cycle=cycle@entry=0x7f27dbd97060) at src/event/ngx_event.c:242

#8 0x000000000044ddb5 in ngx_worker_process_cycle (cycle=cycle@entry=0x7f27dbd97060, data="">

at src/os/unix/ngx_process_cycle.c:753

#9 0x000000000044c8c0 in ngx_spawn_process (cycle=cycle@entry=0x7f27dbd97060,

proc=proc@entry=0x44dd70 <ngx_worker_process_cycle>, data="" name=name@entry=0x5dd9fd "worker process",

respawn=respawn@entry=-4) at src/os/unix/ngx_process.c:198

#10 0x000000000044e024 in ngx_start_worker_processes (cycle=cycle@entry=0x7f27dbd97060, n=24, type=type@entry=-4)

at src/os/unix/ngx_process_cycle.c:358

#11 0x000000000044efbc in ngx_master_process_cycle (cycle=0x7f27dbd97060, cycle@entry=0x7f283f91f3d0)

at src/os/unix/ngx_process_cycle.c:243

#12 0x0000000000429faf in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:367

(gdb)

对应代码是：

if (SSL_set_ex_data(c->ssl->connection, ngx_http_lua_ssl_ctx_index, cctx)

== 0)

{

ngx_ssl_error(NGX_LOG_ALERT, c->log, 0, "SSL_set_ex_data() failed");

goto failed;

}

大家有遇到过么？

@agentzh

agentzh

Hello!

对此我有如下的建议：

1. 请使用最新的 OpenResty 1.11.2.5 发布。OpenResty 1.9.15.1 是一年多以前的老版本了。这期间有很多 bug 修复。

2. 请尽量使用 OpenResty 官方的 Linux 二进制包，你的编译过程可能有问题，尤其是你使用的 OpenSSL 的
build。请见 https://openresty.org/en/linux-packages.html

3. 请去除你使用的非 OpenResty 官方维护的第三方 nginx C 模块或第三方 Lua 库。这些第三方组件可能会引入
memory corruption，所以你的崩溃位置看起来并不像是第一现场，而只是堆被大量损坏之后的后果。最好是用一个最小化的例子来测试。

Regards,
Yichun

anymaybe

首先，感谢章老师的回复。

在 2017年9月24日星期日 UTC+8上午2:03:19，agentzh写道：

Hello!

对此我有如下的建议：

1. 请使用最新的 OpenResty 1.11.2.5 发布。OpenResty 1.9.15.1 是一年多以前的老版本了。这期间有很多 bug 修复。

升级最新的openresty在计划之中，我会先升级验证下。

2. 请尽量使用 OpenResty 官方的 Linux 二进制包，你的编译过程可能有问题，尤其是你使用的 OpenSSL 的
build。请见 https://openresty.org/en/linux-packages.html

我的环境无法使用预编译的二进制包，所以用源码重新执行的编译。openssl是直接使用的官方的1.0.2k源码一起编译的。

不知道章老师说的OpenSSL可能有问题是具体指哪方面，我好做下排查，谢谢。

3. 请去除你使用的非 OpenResty 官方维护的第三方 nginx C 模块或第三方 Lua 库。这些第三方组件可能会引入
memory corruption，所以你的崩溃位置看起来并不像是第一现场，而只是堆被大量损坏之后的后果。最好是用一个最小化的例子来测试。

是的，我有使用nginx_upstream_check_module和nginx-sticky-module一起编译，第三方lua模块就没有了。都是基于openresty自带的进行的lua开发。

进行了多次测试，截止目前一共出现了4次崩溃，位置都是一样的地方。下面是我的几点猜测和接口使用，请大家斧正，感谢。

1，我在ssl_certificate_by_lua阶段，使用lua-resty-lrucache永久缓存了cert_pem_to_der和priv_key_pem_to_der两个接口的执行结果。

2，我最近在配置中新增加了ss_session_cache的配置，在nginx社区中有看到关于ssl session ctx缓存的问题。不知道是否是这个原因。

我会将ssl_session_cache配置移除做下对比测试验证。

Regards.

anymaybe

改为单worker，使用gdb attach住，捕获到segment fault。这个时候是可以看到一些堆栈信息的，如下：

(gdb) c

Continuing.

Program received signal SIGSEGV, Segmentation fault.

0x00000000005082c3 in ngx_http_lua_ssl_cert_handler (ssl_conn=<optimized out>, data="" out>)

at ../ngx_lua-0.10.5/src/ngx_http_lua_ssl_certby.c:269

269 ../ngx_lua-0.10.5/src/ngx_http_lua_ssl_certby.c: No such file or directory.

(gdb) bt

#0 0x00000000005082c3 in ngx_http_lua_ssl_cert_handler (ssl_conn=<optimized out>, data="" out>)

at ../ngx_lua-0.10.5/src/ngx_http_lua_ssl_certby.c:269

#1 0x00000000005b04b8 in ssl3_get_client_hello ()

#2 0x00000000005b512f in ssl3_accept ()

#3 0x0000000000591184 in ssl23_accept ()

#4 0x0000000000453346 in ngx_ssl_handshake (c=c@entry=0x7fc7677e1f70) at src/event/ngx_event_openssl.c:1199

#5 0x000000000046753c in ngx_http_ssl_handshake (rev=0x7fc766ff88f0) at src/http/ngx_http_request.c:719

#6 0x00000000004503d9 in ngx_epoll_process_events (cycle=0x7fc7cb9873d0, timer=<optimized out>, flags=<optimized out>)

at src/event/modules/ngx_epoll_module.c:822

#7 0x0000000000446b67 in ngx_process_events_and_timers (cycle=cycle@entry=0x7fc7cb9873d0) at src/event/ngx_event.c:242

#8 0x000000000044ddb5 in ngx_worker_process_cycle (cycle=cycle@entry=0x7fc7cb9873d0, data="">

at src/os/unix/ngx_process_cycle.c:753

#9 0x000000000044c8c0 in ngx_spawn_process (cycle=cycle@entry=0x7fc7cb9873d0,

proc=proc@entry=0x44dd70 <ngx_worker_process_cycle>, data="" name=name@entry=0x5dd9fd "worker process",

respawn=respawn@entry=-4) at src/os/unix/ngx_process.c:198

#10 0x000000000044e024 in ngx_start_worker_processes (cycle=cycle@entry=0x7fc7cb9873d0, n=1, type=type@entry=-4)

at src/os/unix/ngx_process_cycle.c:358

#11 0x000000000044efbc in ngx_master_process_cycle (cycle=cycle@entry=0x7fc7cb9873d0) at src/os/unix/ngx_process_cycle.c:243

#12 0x0000000000429faf in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:367

(gdb) info locals

rc = <optimized out>

c = 0x7fc7677e1f70

fc = 0x7fc767312d70

r = 0x7fc7568f80c0

cln = <optimized out>

hc = <optimized out>

lscf = <optimized out>

cctx = 0x7fc7c9788240

(gdb) p *c

$1 = {data = "" read = 0x7fc766ff88f0, write = 0x7fc766c4e8f0, fd = -1, recv = 0x44b170 <ngx_unix_recv>,

send = 0x44b5e0 <ngx_unix_send>, recv_chain = 0x44b2b0 <ngx_readv_chain>, send_chain = 0x450790 <ngx_linux_sendfile_chain>,

listening = 0x7fc7c99d4fc0, sent = 0, log = 0x7fc75f753430, pool = 0x7fc75f7533d0, type = 1, sockaddr = 0x7fc75f753420,

socklen = 16, addr_text = {len = 9, data = "" "30.0.0.3157g\307\177"}, proxy_protocol_addr = {len = 0, data = "">

ssl = 0x0, local_sockaddr = 0x7fc7c9aebfc0, local_socklen = 16, buffer = 0x0, queue = {prev = 0x0, next = 0x0},

number = 10527762, requests = 0, buffered = 0, log_error = 2, unexpected_eof = 1, timedout = 0, error = 0, destroyed = 1,

idle = 0, reusable = 0, close = 1, shared = 0, sendfile = 0, sndlowat = 0, tcp_nodelay = 0, tcp_nopush = 0, need_last_buf = 0}

(gdb) p * f

No symbol "f" in current context.

(gdb) p * fc

$2 = {data = "" read = 0x7fc766de8ef0, write = 0x7fc766a3eef0, fd = -1, recv = 0x0, send = 0x0, recv_chain = 0x0,

send_chain = 0x0, listening = 0x7fc7c99d4fc0, sent = 0, log = 0x7fc736fbf910, pool = 0x7fc756647b80, type = 0, sockaddr = 0x0,

socklen = 0, addr_text = {len = 9, data = "" "30.0.0.3157g\307\177"}, proxy_protocol_addr = {len = 0, data = "">

ssl = 0x0, local_sockaddr = 0x0, local_socklen = 0, buffer = 0x0, queue = {prev = 0x0, next = 0x0}, number = 0, requests = 1,

buffered = 0, log_error = 2, unexpected_eof = 0, timedout = 0, error = 1, destroyed = 0, idle = 0, reusable = 0, close = 0,

shared = 0, sendfile = 0, sndlowat = 0, tcp_nodelay = 0, tcp_nopush = 0, need_last_buf = 0}

(gdb) p *r

$3 = {signature = 1347703880, connection = 0x7fc767312d70, ctx = 0x7fc7c99b5620, main_conf = 0x7fc7cb98a8b8,

srv_conf = 0x7fc7c99feec0, loc_conf = 0x7fc7c99ff0f0, read_event_handler = 0x0, write_event_handler = 0x0, cache = 0x0,

upstream = 0x0, upstream_states = 0x0, pool = 0x7fc756647b80, header_in = 0x0, headers_in = {headers = {last = 0x0, part = {

elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, host = 0x0, connection = 0x0,

if_modified_since = 0x0, if_unmodified_since = 0x0, if_match = 0x0, if_none_match = 0x0, user_agent = 0x0, referer = 0x0,

content_length = 0x0, content_type = 0x0, range = 0x0, if_range = 0x0, transfer_encoding = 0x0, expect = 0x0, upgrade = 0x0,

accept_encoding = 0x0, via = 0x0, authorization = 0x0, keep_alive = 0x0, x_forwarded_for = {elts = 0x0, nelts = 0, size = 0,

nalloc = 0, pool = 0x0}, x_real_ip = 0x0, depth = 0x0, destination = 0x0, overwrite = 0x0, date = 0x0, user = {len = 0,

data = "" passwd = {len = 0, data = "" cookies = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, server = {

len = 0, data = "" content_length_n = 0, keep_alive_n = -1, connection_type = 0, chunked = 0, msie = 0, msie6 = 0,

opera = 0, gecko = 0, chrome = 0, safari = 0, konqueror = 0}, headers_out = {headers = {last = 0x0, part = {elts = 0x0,

nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, status = 0, status_line = {len = 0, data = "" server = 0x0,

date = 0x0, content_length = 0x0, content_encoding = 0x0, location = 0x0, refresh = 0x0, last_modified = 0x0,

content_range = 0x0, accept_ranges = 0x0, www_authenticate = 0x0, expires = 0x0, etag = 0x0, override_charset = 0x0,

content_type_len = 0, content_type = {len = 0, data = "" charset = {len = 0, data = "" content_type_lowcase = 0x0,

content_type_hash = 0, cache_control = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, content_length_n = 0,

content_offset = 0, date_time = 0, last_modified_time = 0}, request_body = 0x0, lingering_time = 0, start_sec = 0,

start_msec = 0, method = 1, http_version = 0, request_line = {len = 0, data = "" uri = {len = 0, data = "" args = {

len = 0, data = "" exten = {len = 0, data = "" unparsed_uri = {len = 0, data = "" method_name = {len = 0,

data = "" http_protocol = {len = 0, data = "" out = 0x0, main = 0x7fc7568f80c0, parent = 0x0, postponed = 0x0,

post_subrequest = 0x0, posted_requests = 0x0, phase_handler = 0, content_handler = 0x0, access_code = 0, variables = 0x0,

ncaptures = 0, captures = 0x0, captures_data = 0x0, limit_rate = 0, limit_rate_after = 0, header_size = 0, request_length = 0,

err_status = 0, http_connection = 0x0, log_handler = 0x0, cleanup = 0x0, count = 1, subrequests = 51, blocked = 0, aio = 0,

---Type <return> to continue, or q <return> to quit---

http_state = 2, complex_uri = 0, quoted_uri = 0, plus_in_uri = 0, space_in_uri = 0, invalid_header = 0, add_uri_to_alias = 0,

valid_location = 0, valid_unparsed_uri = 0, uri_changed = 0, uri_changes = 11, request_body_in_single_buf = 0,

request_body_in_file_only = 0, request_body_in_persistent_file = 0, request_body_in_clean_file = 0,

request_body_file_group_access = 0, request_body_file_log_level = 0, request_body_no_buffering = 0, subrequest_in_memory = 0,

waited = 0, cached = 0, gzip_tested = 0, gzip_ok = 0, gzip_vary = 0, proxy = 0, bypass_cache = 0, no_cache = 0,

limit_conn_set = 0, limit_req_set = 0, pipeline = 0, chunked = 0, header_only = 0, keepalive = 0, lingering_close = 0,

discard_body = 1, reading_body = 0, internal = 0, error_page = 0, filter_finalize = 0, post_action = 0, request_complete = 0,

request_output = 0, header_sent = 0, expect_tested = 0, root_tested = 0, done = 0, logged = 0, buffered = 0,

main_filter_need_in_memory = 0, filter_need_in_memory = 0, filter_need_temporary = 0, allow_ranges = 0, subrequest_ranges = 0,

single_range = 0, disable_not_modified = 0, stat_reading = 0, stat_writing = 0, state = 0, header_hash = 0, lowcase_index = 0,

lowcase_header = '\000' <repeats 31 times>, header_name_start = 0x0, header_name_end = 0x0, header_start = 0x0,

header_end = 0x0, uri_start = 0x0, uri_end = 0x0, uri_ext = 0x0, args_start = 0x0, request_start = 0x0, request_end = 0x0,

method_end = 0x0, schema_start = 0x0, schema_end = 0x0, host_start = 0x0, host_end = 0x0, port_start = 0x0, port_end = 0x0,

http_minor = 0, http_major = 0}

(gdb) p *cctx

$4 = {connection = 0x7fc7677e1f70, request = 0x7fc7568f80c0, exit_code = 1, done = 0, aborted = 0}

(gdb)

从堆栈看上去，在callback执行的过程中，连接被释放了。

anymaybe

(gdb) f 0

#0 0x00000000005082c3 in ngx_http_lua_ssl_cert_handler (ssl_conn=<optimized out>, data="" out>)

at ../ngx_lua-0.10.5/src/ngx_http_lua_ssl_certby.c:269

269 ../ngx_lua-0.10.5/src/ngx_http_lua_ssl_certby.c: No such file or directory.

(gdb) p *fc

$23 = {data = "" read = 0x7fc766de8ef0, write = 0x7fc766a3eef0, fd = -1, recv = 0x0, send = 0x0, recv_chain = 0x0,

send_chain = 0x0, listening = 0x7fc7c99d4fc0, sent = 0, log = 0x7fc736fbf910, pool = 0x7fc756647b80, type = 0, sockaddr = 0x0,

socklen = 0, addr_text = {len = 9, data = "" "30.0.0.3157g\307\177"}, proxy_protocol_addr = {len = 0, data = "">

ssl = 0x0, local_sockaddr = 0x0, local_socklen = 0, buffer = 0x0, queue = {prev = 0x0, next = 0x0}, number = 0, requests = 1,

buffered = 0, log_error = 2, unexpected_eof = 0, timedout = 0, error = 1, destroyed = 0, idle = 0, reusable = 0, close = 0,

shared = 0, sendfile = 0, sndlowat = 0, tcp_nodelay = 0, tcp_nopush = 0, need_last_buf = 0}

(gdb) p *c

$24 = {data = "" read = 0x7fc766ff88f0, write = 0x7fc766c4e8f0, fd = -1, recv = 0x44b170 <ngx_unix_recv>,

send = 0x44b5e0 <ngx_unix_send>, recv_chain = 0x44b2b0 <ngx_readv_chain>, send_chain = 0x450790 <ngx_linux_sendfile_chain>,

listening = 0x7fc7c99d4fc0, sent = 0, log = 0x7fc75f753430, pool = 0x7fc75f7533d0, type = 1, sockaddr = 0x7fc75f753420,

socklen = 16, addr_text = {len = 9, data = "" "30.0.0.3157g\307\177"}, proxy_protocol_addr = {len = 0, data = "">

ssl = 0x0, local_sockaddr = 0x7fc7c9aebfc0, local_socklen = 16, buffer = 0x0, queue = {prev = 0x0, next = 0x0},

number = 10527762, requests = 0, buffered = 0, log_error = 2, unexpected_eof = 1, timedout = 0, error = 0, destroyed = 1,

idle = 0, reusable = 0, close = 1, shared = 0, sendfile = 0, sndlowat = 0, tcp_nodelay = 0, tcp_nopush = 0, need_last_buf = 0}

(gdb) p ngx_http_lua_ssl_ctx_index

$25 = 1

(gdb) p ngx_ssl_connection_index

$26 = 0

(gdb) p *cctx

$27 = {connection = 0x7fc7677e1f70, request = 0x7fc7568f80c0, exit_code = 1, done = 0, aborted = 0}

(gdb) p *c->read

$28 = {data = "" write = 0, accept = 0, instance = 1, active = 0, disabled = 0, ready = 1, _oneshot_ = 0, complete = 0,

eof = 0, error = 0, timedout = 0, timer_set = 0, delayed = 0, deferred_accept = 0, pending_eof = 0, posted = 0, closed = 1,

channel = 0, resolver = 0, cancelable = 0, available = 0, handler = 0x467350 <ngx_http_ssl_handshake>, index = 3503345872,

log = 0x7fc75f753430, timer = {key = 0, left = 0x0, right = 0x0, parent = 0x0, color = 0 '\000', data = 0 '\000'}, queue = {

prev = 0x0, next = 0x0}}

(gdb) p *c->write

$29 = {data = "" write = 1, accept = 0, instance = 1, active = 0, disabled = 0, ready = 1, _oneshot_ = 0, complete = 0,

eof = 0, error = 0, timedout = 0, timer_set = 0, delayed = 0, deferred_accept = 0, pending_eof = 0, posted = 0, closed = 1,

channel = 0, resolver = 0, cancelable = 0, available = 0, handler = 0x4625d0 <ngx_http_empty_handler>, index = 3503345872,

log = 0x7fc75f753430, timer = {key = 0, left = 0x0, right = 0x0, parent = 0x0, color = 0 '\000', data = 0 '\000'}, queue = {

prev = 0x0, next = 0x0}}

(gdb) f 5

#5 0x000000000046753c in ngx_http_ssl_handshake (rev=0x7fc766ff88f0) at src/http/ngx_http_request.c:719

719 src/http/ngx_http_request.c: No such file or directory.

(gdb) p *hc

$30 = {addr_conf = 0x7fc7c9aebff8, conf_ctx = 0x7fc7c99feea8, ssl_servername = 0x0, ssl_servername_regex = 0x0, busy = 0x0,

nbusy = 0, free = 0x0, nfree = 0, ssl = 1, proxy_protocol = 0}

(gdb) p *hc->addr_conf

$31 = {default_server = 0x7fc7c99ff320, virtual_names = 0x0, ssl = 1, proxy_protocol = 0}

(gdb) p *hc->conf_ctx

$32 = {main_conf = 0x7fc7cb98a8b8, srv_conf = 0x7fc7c99feec0, loc_conf = 0x7fc7c99ff0f0}

(gdb)

想不明白，为啥ngx_ssl_get_connection获取出来的连接对象在ngx_http_lua_ssl_cert_handler执行过程中会被释放了。

毕竟在调用SSL_set_ex_data之前，c->ssl和c->data都是有使用的。而且回调函数应该是同步执行的吧。

anymaybe

单worker，使用gdb脚本做了信息收集：

set height 0
handle SIGPIPE nostop

b ngx_http_lua_ssl_certby.c:213
comm $bpnum
p c
p *c
p *c->ssl
c
end

最终刚进入ngx_http_lua_ssl_cert_handler函数的信息打印和出现崩溃时的信息如下：

Breakpoint 1, ngx_http_lua_ssl_cert_handler (ssl_conn=<optimized out>, data="" out>)

at ../ngx_lua-0.10.5/src/ngx_http_lua_ssl_certby.c:213

213 in ../ngx_lua-0.10.5/src/ngx_http_lua_ssl_certby.c

$1168 = (ngx_connection_t *) 0x7f6fee3b50d0

$1169 = {data = "" read = 0x7f6fedde0bd0, write = 0x7f6feda36bd0, fd = 2116, recv = 0x44b170 <ngx_unix_recv>,

send = 0x44b5e0 <ngx_unix_send>, recv_chain = 0x44b2b0 <ngx_readv_chain>, send_chain = 0x450790 <ngx_linux_sendfile_chain>,

listening = 0x7f705094d920, sent = 0, log = 0x7f6fe7cccee0, pool = 0x7f6fe7ccce80, type = 1, sockaddr = 0x7f6fe7ccced0,

socklen = 16, addr_text = {len = 9, data = "" "30.0.0.31"}, proxy_protocol_addr = {len = 0, data = "">

ssl = 0x7f6fe7cccfa0, local_sockaddr = 0x7f7050a642e0, local_socklen = 16, buffer = 0x0, queue = {prev = 0x7f6fee628868,

next = 0x7f6fee81de68}, number = 476516, requests = 0, buffered = 0, log_error = 2, unexpected_eof = 1, timedout = 0,

error = 0, destroyed = 0, idle = 0, reusable = 1, close = 0, shared = 0, sendfile = 0, sndlowat = 0, tcp_nodelay = 0,

tcp_nopush = 0, need_last_buf = 0}

$1170 = {connection = 0x7f6fe1f7d120, initial_ctx = 0x7f705089a5f0, last = 0, buf = 0x0, buffer_size = 16384, handler = 0x0,

saved_read_handler = 0x0, saved_write_handler = 0x0, handshaked = 0, renegotiation = 0, buffer = 1, no_wait_shutdown = 0,

no_send_shutdown = 0, handshake_buffer_set = 0}

Program received signal SIGSEGV, Segmentation fault.

0x00000000005082c3 in ngx_http_lua_ssl_cert_handler (ssl_conn=<optimized out>, data="" out>)

at ../ngx_lua-0.10.5/src/ngx_http_lua_ssl_certby.c:269

269 in ../ngx_lua-0.10.5/src/ngx_http_lua_ssl_certby.c

(gdb) p c

$1171 = (ngx_connection_t *) 0x7f6fee3b50d0

(gdb) p *c

$1172 = {data = "" read = 0x7f6fedde0bd0, write = 0x7f6feda36bd0, fd = -1, recv = 0x44b170 <ngx_unix_recv>,

send = 0x44b5e0 <ngx_unix_send>, recv_chain = 0x44b2b0 <ngx_readv_chain>, send_chain = 0x450790 <ngx_linux_sendfile_chain>,

listening = 0x7f705094d920, sent = 0, log = 0x7f6fe7cccee0, pool = 0x7f6fe7ccce80, type = 1, sockaddr = 0x7f6fe7ccced0,

socklen = 16, addr_text = {len = 9, data = "" "30.0.0.31"}, proxy_protocol_addr = {len = 0, data = "" color="#ff0000"> ssl = 0x0,

local_sockaddr = 0x7f7050a642e0, local_socklen = 16, buffer = 0x0, queue = {prev = 0x0, next = 0x0}, number = 476516,

requests = 0, buffered = 0, log_error = 2, unexpected_eof = 1, timedout = 0, error = 0, destroyed = 1, idle = 0, reusable = 0,

close = 1, shared = 0, sendfile = 0, sndlowat = 0, tcp_nodelay = 0, tcp_nopush = 0, need_last_buf = 0}

(gdb) info locals

rc = <optimized out>

c = 0x7f6fee3b50d0

fc = 0x7f6fee432c70

r = 0x7f6fed49b050

cln = <optimized out>

hc = <optimized out>

lscf = <optimized out>

cctx = 0x7f6fe71d10a0

(gdb) p *fc

$1173 = {data = "" read = 0x7f6fede169f0, write = 0x7f6feda6c9f0, fd = -1, recv = 0x0, send = 0x0, recv_chain = 0x0,

send_chain = 0x0, listening = 0x7f705094d920, sent = 0, log = 0x7f6fed6c70d0, pool = 0x7f6fee934d10, type = 0, sockaddr = 0x0,

socklen = 0, addr_text = {len = 9, data = "" "30.0.0.31"}, proxy_protocol_addr = {len = 0, data = "" ssl = 0x0,

local_sockaddr = 0x0, local_socklen = 0, buffer = 0x0, queue = {prev = 0x0, next = 0x0}, number = 0, requests = 1, buffered = 0,

log_error = 2, unexpected_eof = 0, timedout = 0, error = 1, destroyed = 0, idle = 0, reusable = 0, close = 0, shared = 0,

sendfile = 0, sndlowat = 0, tcp_nodelay = 0, tcp_nopush = 0, need_last_buf = 0}

(gdb) p *(SSL*)0x7f6fe1f7d120

$1174 = {version = -535403168, type = 32623, method = 0x7f70514a3ac8, rbio = 0x7f6fee9d3a90, wbio = 0x7f6fee9d3a90, bbio = 0x0,

rwstate = 1, in_handshake = 2, handshake_func = 0x5b4a70 <ssl3_accept>, server = 1, new_session = 0, quiet_shutdown = 0,

shutdown = 0, state = 8466, rstate = 240, init_buf = 0x7f6fed1637b0, init_msg = 0x7f6fe90482b4, init_num = 238, init_off = 0,

packet = 0x7f6feeac30c3 "", packet_length = 0, s2 = 0x0, s3 = 0x0, d1 = 0x0, read_ahead = 1, msg_callback = 0x0,

msg_callback_arg = 0x0, hit = 0, param = 0x7f6fec0218d0, cipher_list = 0x0, cipher_list_by_id = 0x0, mac_flags = 0,

enc_read_ctx = 0x0, read_hash = 0x0, expand = 0x0, enc_write_ctx = 0x0, write_hash = 0x0, compress = 0x0, cert = 0x7f7050a78a50,

sid_ctx_length = 20, sid_ctx = "ɑvx\333\017\213[nM\321\307@e?\240\246y\313\301", '\000' <repeats 11 times>,

session = 0x7f6fe7360fb0, generate_session_id = 0x0, verify_mode = 0, verify_callback = 0x0, info_callback = 0x0, error = 0,

error_code = 0, psk_client_callback = 0x0, psk_server_callback = 0x0, ctx = 0x7f705089a5f0, debug = 0, verify_result = 0,

ex_data = {sk = 0x0, dummy = 0}, client_CA = 0x0, references = 0, options = 56232871, mode = 24, max_cert_list = 102400,

first_packet = 0, client_version = 771, max_send_fragment = 16384, tlsext_debug_cb = 0x0, tlsext_debug_arg = 0x0,

tlsext_hostname = 0x0, servername_done = 0, tlsext_status_type = -1, tlsext_status_expected = 0, tlsext_ocsp_ids = 0x0,

tlsext_ocsp_exts = 0x0, tlsext_ocsp_resp = 0x0, tlsext_ocsp_resplen = -1, tlsext_ticket_expected = 1,

tlsext_ecpointformatlist_length = 0, tlsext_ecpointformatlist = 0x0, tlsext_ellipticcurvelist_length = 0,

tlsext_ellipticcurvelist = 0x0, tlsext_opaque_prf_input = 0x0, tlsext_opaque_prf_input_len = 0, tlsext_session_ticket = 0x0,

tls_session_ticket_ext_cb = 0x0, tls_session_ticket_ext_cb_arg = 0x0, tls_session_secret_cb = 0x0,

tls_session_secret_cb_arg = 0x0, initial_ctx = 0x7f705089a5f0, next_proto_negotiated = 0x0,

next_proto_negotiated_len = 0 '\000', srtp_profiles = 0x0, srtp_profile = 0x0, tlsext_heartbeat = 1, tlsext_hb_pending = 0,

tlsext_hb_seq = 0, renegotiate = 0, srp_ctx = {SRP_cb_arg = 0x0, TLS_ext_srp_username_callback = 0x0,

SRP_verify_param_callback = 0x0, SRP_give_srp_client_pwd_callback = 0x0, login = 0x0, N = 0x0, g = 0x0, s = 0x0, B = 0x0,

A = 0x0, a = 0x0, b = 0x0, v = 0x0, info = 0x0, strength = 1024, srp_Mask = 0}, alpn_client_proto_list = 0x0,

alpn_client_proto_list_len = 832}

(gdb) p *(SSL_CTX*)0x7f70509ce1b0

$1175 = {method = 0x609780 <SSLv23_method_data.16072>, cipher_list = 0x7f70509ce000, cipher_list_by_id = 0x7f70509ceec0,

cert_store = 0x7f70509cddd0, sessions = 0x7f70509ce7e0, session_cache_size = 20480, session_cache_head = 0x0,

session_cache_tail = 0x0, session_cache_mode = 770, session_timeout = 600, new_session_cb = 0x4514d0 <ngx_ssl_new_session>,

remove_session_cb = 0x451780 <ngx_ssl_remove_session>, get_session_cb = 0x450f80 <ngx_ssl_get_cached_session>, stats = {

sess_connect = 0, sess_connect_renegotiate = 0, sess_connect_good = 0, sess_accept = 27, sess_accept_renegotiate = 0,

sess_accept_good = 25, sess_miss = 0, sess_timeout = 0, sess_cache_full = 0, sess_hit = 0, sess_cb_hit = 0}, references = 1,

app_verify_callback = 0x0, app_verify_arg = 0x0, default_passwd_callback = 0x0, default_passwd_callback_userdata = 0x0,

client_cert_cb = 0x0, app_gen_cookie_cb = 0x0, app_verify_cookie_cb = 0x0, ex_data = {sk = 0x7f70509cc8b0, dummy = 0},

rsa_md5 = 0x7f70527b8680 <md5_md>, md5 = 0x7f70527b8680 <md5_md>, sha1 = 0x7f70527b8980 <sha1_md>, extra_certs = 0x0,

comp_methods = 0x7f70528e0b60, info_callback = 0x4513f0 <ngx_ssl_info_callback>, client_CA = 0x7f70509cc160, options = 56232871,

mode = 24, max_cert_list = 102400, cert = 0x7f70509ce4e0, read_ahead = 1, msg_callback = 0x0, msg_callback_arg = 0x0,

verify_mode = 0, sid_ctx_length = 20, sid_ctx = "ɑvx\333\017\213[nM\321\307@e?\240\246y\313\301", '\000' <repeats 11 times>,

default_verify_callback = 0x0, generate_session_id = 0x0, param = 0x7f70509ccbd0, quiet_shutdown = 0, max_send_fragment = 16384,

client_cert_engine = 0x0, tlsext_servername_callback = 0x463a60 <ngx_http_ssl_servername>, tlsext_servername_arg = 0x0,

tlsext_tick_key_name = "\315\035\272\\\370\352\215\a\250(I\345\241#\347", <incomplete sequence \362>,

tlsext_tick_hmac_key = "\222\250\321E\212@M\200\b\212V+\236 \375\063",

tlsext_tick_aes_key = "\207\277\345\315\070\320`\226\341ʣ\324\061t", <incomplete sequence \344>, tlsext_ticket_key_cb = 0x0,

tlsext_status_cb = 0x0, tlsext_status_arg = 0x0, tlsext_opaque_prf_input_callback = 0x0,

tlsext_opaque_prf_input_callback_arg = 0x0, psk_identity_hint = 0x0, psk_client_callback = 0x0, psk_server_callback = 0x0,

freelist_max_len = 32, wbuf_freelist = 0x7f7052923f40, rbuf_freelist = 0x7f6feeda2500, srp_ctx = {SRP_cb_arg = 0x0,

TLS_ext_srp_username_callback = 0x0, SRP_verify_param_callback = 0x0, SRP_give_srp_client_pwd_callback = 0x0, login = 0x0,

N = 0x0, g = 0x0, s = 0x0, B = 0x0, A = 0x0, a = 0x0, b = 0x0, v = 0x0, info = 0x0, strength = 1024, srp_Mask = 0},

next_protos_advertised_cb = 0x49cdf0 <ngx_http_ssl_npn_advertised>, next_protos_advertised_cb_arg = 0x0,

next_proto_select_cb = 0x0, next_proto_select_cb_arg = 0x0, srtp_profiles = 0x0,

alpn_select_cb = 0x49ce40 <ngx_http_ssl_alpn_select>, alpn_select_cb_arg = 0x0, alpn_client_proto_list = 0x0,

alpn_client_proto_list_len = 0, tlsext_ecpointformatlist_length = 0, tlsext_ecpointformatlist = 0x0,

tlsext_ellipticcurvelist_length = 0, tlsext_ellipticcurvelist = 0x0}

(gdb)

可以看到SSL对象仍在，只是ngx_connection_t对象被提前回收了。

求指导下一步分析方向？谢谢。

anymaybe

上面的SSL_CTX给错了，应该是：

(gdb) p *(SSL_CTX*)0x7f705089a5f0

$1177 = {method = 0x609780 <SSLv23_method_data.16072>, cipher_list = 0x7f705089d2a0, cipher_list_by_id = 0x7f705089cd10,

cert_store = 0x7f7050898790, sessions = 0x7f705089af10, session_cache_size = 20480, session_cache_head = 0x0,

session_cache_tail = 0x0, session_cache_mode = 770, session_timeout = 600, new_session_cb = 0x4514d0 <ngx_ssl_new_session>,

remove_session_cb = 0x451780 <ngx_ssl_remove_session>, get_session_cb = 0x450f80 <ngx_ssl_get_cached_session>, stats = {

sess_connect = 0, sess_connect_renegotiate = 0, sess_connect_good = 0, sess_accept = 26, sess_accept_renegotiate = 0,

sess_accept_good = 23, sess_miss = 0, sess_timeout = 0, sess_cache_full = 0, sess_hit = 0, sess_cb_hit = 0}, references = 1,

app_verify_callback = 0x0, app_verify_arg = 0x0, default_passwd_callback = 0x0, default_passwd_callback_userdata = 0x0,

client_cert_cb = 0x0, app_gen_cookie_cb = 0x0, app_verify_cookie_cb = 0x0, ex_data = {sk = 0x7f705089aab0, dummy = 0},

rsa_md5 = 0x7f70527b8680 <md5_md>, md5 = 0x7f70527b8680 <md5_md>, sha1 = 0x7f70527b8980 <sha1_md>, extra_certs = 0x0,

comp_methods = 0x7f70528e0b60, info_callback = 0x4513f0 <ngx_ssl_info_callback>, client_CA = 0x7f7050898850, options = 56232871,

mode = 24, max_cert_list = 102400, cert = 0x7f705089ac10, read_ahead = 1, msg_callback = 0x0, msg_callback_arg = 0x0,

verify_mode = 0, sid_ctx_length = 20, sid_ctx = "ɑvx\333\017\213[nM\321\307@e?\240\246y\313\301", '\000' <repeats 11 times>,

default_verify_callback = 0x0, generate_session_id = 0x0, param = 0x7f7050897d00, quiet_shutdown = 0, max_send_fragment = 16384,

client_cert_engine = 0x0, tlsext_servername_callback = 0x463a60 <ngx_http_ssl_servername>, tlsext_servername_arg = 0x0,

tlsext_tick_key_name = "\326P\265tȽoh0'\332m;\237", <incomplete sequence \305>,

tlsext_tick_hmac_key = "\223\004\365\242<\361\326\071\332\376*\317(\027j\224",

tlsext_tick_aes_key = "\004h\216HV\255N\366\022#\346\t\245!<)", tlsext_ticket_key_cb = 0x0, tlsext_status_cb = 0x0,

tlsext_status_arg = 0x0, tlsext_opaque_prf_input_callback = 0x0, tlsext_opaque_prf_input_callback_arg = 0x0,

psk_identity_hint = 0x0, psk_client_callback = 0x0, psk_server_callback = 0x0, freelist_max_len = 32,

wbuf_freelist = 0x7f7050897d90, rbuf_freelist = 0x7f7050898a80, srp_ctx = {SRP_cb_arg = 0x0,

TLS_ext_srp_username_callback = 0x0, SRP_verify_param_callback = 0x0, SRP_give_srp_client_pwd_callback = 0x0, login = 0x0,

N = 0x0, g = 0x0, s = 0x0, B = 0x0, A = 0x0, a = 0x0, b = 0x0, v = 0x0, info = 0x0, strength = 1024, srp_Mask = 0},

next_protos_advertised_cb = 0x49cdf0 <ngx_http_ssl_npn_advertised>, next_protos_advertised_cb_arg = 0x0,

next_proto_select_cb = 0x0, next_proto_select_cb_arg = 0x0, srtp_profiles = 0x0,

alpn_select_cb = 0x49ce40 <ngx_http_ssl_alpn_select>, alpn_select_cb_arg = 0x0, alpn_client_proto_list = 0x0,

alpn_client_proto_list_len = 0, tlsext_ecpointformatlist_length = 0, tlsext_ecpointformatlist = 0x0,

tlsext_ellipticcurvelist_length = 0, tlsext_ellipticcurvelist = 0x0}

anymaybe

ngx_event_accept里accept连接后，调用ls->handler（ngx_http_init_connection）初始化连接。

对于https监听，会注册读事件钩子c->read->handler为ngx_http_ssl_handshake。并将连接放入reusable队列，添加事件到epoll。

读事件到来时：ngx_epoll_process_events->ngx_http_ssl_handshake->ngx_ssl_handshake->SSL_do_handshake。

ssl握手过程中会调用设置的证书回调ngx_http_lua_ssl_cert_handler，回调函数里ngx_http_lua_create_fake_connection->ngx_get_connection。

此时如果没有可用连接（free_connections），就会调用ngx_drain_connections去释放32个reusable队列的连接。就会把当前ssl握手的连接给释放掉。

以上是我根据堆栈和跟读nginx代码得出的定位结论，@agentzh，看下是否是nginx的一个bug。