部分nginx worker占用CPU 100%，openresty-1.15.8.1rc1版本

lizj3624

hi，春哥

我这边遇到问题，某个nginx worker占用CPU 100%，是偶发的，

版本信息：

内核：2.6.32-431.17.1.el6.x86_64

OpenResty版本：openresty-1.15.8.1rc1

Openssl版本： openssl-1.0.2j

LuaJIT版本： LuaJIT-2.1-20190228

附件中是perf top的信息

我gdb一下，信息如下，麻烦帮忙分析一下，非常感谢

(gdb) bt

#0 lj_tab_getstr (L=0x7ffc3477df88, t=0x7ffc35c56bd0, key=0x7ffc34618ec0) at lj_tab.c:423

#1 lj_tab_get (L=0x7ffc3477df88, t=0x7ffc35c56bd0, key=0x7ffc34618ec0) at lj_tab.c:430

#2 0x00007ffc372498b6 in lua_rawget (L=0x7ffc3477df88, idx=<value optimized out>) at lj_api.c:816

#3 0x00000000005020f9 in ngx_http_lua_socket_tcp_connect (L=0x7ffc3477df88) at ../ngx_lua-0.10.14/src/ngx_http_lua_socket_tcp.c:1099

#4 0x00007ffc372383ea in lj_BC_FUNCC () from /export/servers/nginx/luajit/lib/libluajit-5.1.so.2

#5 0x00000000004f30e0 in ngx_http_lua_run_thread (L=0x7ffc35c54378, r=0x15e145c0, ctx=0x7a53070, nrets=0) at ../ngx_lua-0.10.14/src/ngx_http_lua_util.c:1084

#6 0x000000000051300f in ngx_http_lua_ssl_sess_fetch_by_chunk (L=0x7ffc35c54378, r=0x15e145c0) at ../ngx_lua-0.10.14/src/ngx_http_lua_ssl_session_fetchby.c:540

#7 0x0000000000512a7b in ngx_http_lua_ssl_sess_fetch_handler (ssl_conn=<value optimized out>, id=0xf79d27 "3&\325", <incomplete sequence \314>, len=32, copy=<value optimized out>)

at ../ngx_lua-0.10.14/src/ngx_http_lua_ssl_session_fetchby.c:318

#8 0x00007ffc37001e7b in ssl_get_prev_session (s=0x2fea8330, session_id=0xf79d27 "3&\325", <incomplete sequence \314>, len=32, limit=<value optimized out>) at ssl_sess.c:638

#9 0x00007ffc36fd55ff in ssl3_get_client_hello (s=0x2fea8330) at s3_srvr.c:1048

#10 0x00007ffc36fd681e in ssl3_accept (s=0x2fea8330) at s3_srvr.c:364

#11 0x00007ffc36fe6464 in ssl23_get_client_hello (s=0x2fea8330) at s23_srvr.c:647

#12 0x00007ffc36fe6d31 in ssl23_accept (s=0x2fea8330) at s23_srvr.c:209

#13 0x0000000000450cbf in ngx_ssl_handshake (c=0x7ffc07061840) at src/event/ngx_event_openssl.c:1371

#14 0x0000000000463793 in ngx_http_ssl_handshake (rev=0x7ffc066d85d0) at src/http/ngx_http_request.c:736

#15 0x000000000044bbd5 in ngx_epoll_process_events (cycle=<value optimized out>, timer=<value optimized out>, flags=<value optimized out>) at src/event/modules/ngx_epoll_module.c:902

#16 0x0000000000442392 in ngx_process_events_and_timers (cycle=0x157fe8b0) at src/event/ngx_event.c:252

#17 0x0000000000449df8 in ngx_worker_process_cycle (cycle=0x157fe8b0, data="" optimized out>) at src/os/unix/ngx_process_cycle.c:816

#18 0x0000000000448127 in ngx_spawn_process (cycle=0x157fe8b0, proc=0x449dc0 <ngx_worker_process_cycle>, data="" name=0x591e87 "worker process", respawn=-4) at src/os/unix/ngx_process.c:199

#19 0x000000000044915c in ngx_start_worker_processes (cycle=0x157fe8b0, n=10, type=-4) at src/os/unix/ngx_process_cycle.c:397

#20 0x000000000044a764 in ngx_master_process_cycle (cycle=0x157fe8b0) at src/os/unix/ngx_process_cycle.c:251

#21 0x00000000004235ed in main (argc=<value optimized out>, argv=<value optimized out>) at src/core/nginx.c:382

Attachment: 截图20190516171512.png
Description: PNG image

zhaoyyuu

会不会是用了lua原生的方法？

lizj3624 lizj3624 <liz...@gmail.com> 于2019年5月16日周四下午6:35写道：

hi，春哥
我这边遇到问题，某个nginx worker占用CPU 100%，是偶发的，
版本信息：
内核：2.6.32-431.17.1.el6.x86_64
OpenResty版本：openresty-1.15.8.1rc1
Openssl版本： openssl-1.0.2j
LuaJIT版本： LuaJIT-2.1-20190228

附件中是perf top的信息
我gdb一下，信息如下，麻烦帮忙分析一下，非常感谢
(gdb) bt
#0 lj_tab_getstr (L=0x7ffc3477df88, t=0x7ffc35c56bd0, key=0x7ffc34618ec0) at lj_tab.c:423
#1 lj_tab_get (L=0x7ffc3477df88, t=0x7ffc35c56bd0, key=0x7ffc34618ec0) at lj_tab.c:430
#2 0x00007ffc372498b6 in lua_rawget (L=0x7ffc3477df88, idx=<value optimized out>) at lj_api.c:816
#3 0x00000000005020f9 in ngx_http_lua_socket_tcp_connect (L=0x7ffc3477df88) at ../ngx_lua-0.10.14/src/ngx_http_lua_socket_tcp.c:1099
#4 0x00007ffc372383ea in lj_BC_FUNCC () from /export/servers/nginx/luajit/lib/libluajit-5.1.so.2
#5 0x00000000004f30e0 in ngx_http_lua_run_thread (L=0x7ffc35c54378, r=0x15e145c0, ctx=0x7a53070, nrets=0) at ../ngx_lua-0.10.14/src/ngx_http_lua_util.c:1084
#6 0x000000000051300f in ngx_http_lua_ssl_sess_fetch_by_chunk (L=0x7ffc35c54378, r=0x15e145c0) at ../ngx_lua-0.10.14/src/ngx_http_lua_ssl_session_fetchby.c:540
#7 0x0000000000512a7b in ngx_http_lua_ssl_sess_fetch_handler (ssl_conn=<value optimized out>, id=0xf79d27 "3&\325", <incomplete sequence \314>, len=32, copy=<value optimized out>)
at ../ngx_lua-0.10.14/src/ngx_http_lua_ssl_session_fetchby.c:318
#8 0x00007ffc37001e7b in ssl_get_prev_session (s=0x2fea8330, session_id=0xf79d27 "3&\325", <incomplete sequence \314>, len=32, limit=<value optimized out>) at ssl_sess.c:638
#9 0x00007ffc36fd55ff in ssl3_get_client_hello (s=0x2fea8330) at s3_srvr.c:1048
#10 0x00007ffc36fd681e in ssl3_accept (s=0x2fea8330) at s3_srvr.c:364
#11 0x00007ffc36fe6464 in ssl23_get_client_hello (s=0x2fea8330) at s23_srvr.c:647
#12 0x00007ffc36fe6d31 in ssl23_accept (s=0x2fea8330) at s23_srvr.c:209
#13 0x0000000000450cbf in ngx_ssl_handshake (c=0x7ffc07061840) at src/event/ngx_event_openssl.c:1371
#14 0x0000000000463793 in ngx_http_ssl_handshake (rev=0x7ffc066d85d0) at src/http/ngx_http_request.c:736
#15 0x000000000044bbd5 in ngx_epoll_process_events (cycle=<value optimized out>, timer=<value optimized out>, flags=<value optimized out>) at src/event/modules/ngx_epoll_module.c:902
#16 0x0000000000442392 in ngx_process_events_and_timers (cycle=0x157fe8b0) at src/event/ngx_event.c:252
#17 0x0000000000449df8 in ngx_worker_process_cycle (cycle=0x157fe8b0, data="" optimized out>) at src/os/unix/ngx_process_cycle.c:816
#18 0x0000000000448127 in ngx_spawn_process (cycle=0x157fe8b0, proc=0x449dc0 <ngx_worker_process_cycle>, data="" name=0x591e87 "worker process", respawn=-4) at src/os/unix/ngx_process.c:199
#19 0x000000000044915c in ngx_start_worker_processes (cycle=0x157fe8b0, n=10, type=-4) at src/os/unix/ngx_process_cycle.c:397
#20 0x000000000044a764 in ngx_master_process_cycle (cycle=0x157fe8b0) at src/os/unix/ngx_process_cycle.c:251
#21 0x00000000004235ed in main (argc=<value optimized out>, argv=<value optimized out>) at src/core/nginx.c:382

zchao1995

有用 strace 看过吗？是否有系统调用的交互？