莫名奇妙的master退出，求大神帮忙看下

zhaohang200801

升级版本时发现自动化脚本部署时出现master退出现象，手动安装无问题，跟踪发现段错误，但具体哪里出的错误实难定位。求大神帮忙看下，谢谢。

coredump 如下：

strace 跟踪master部分结果如下：

05:08:00 open("/home/qboxserver/edge_openresty/_package/resty/lock.lua", O_RDONLY) = -1 ENOENT (No such file or directory)

05:08:00 open("/home/qboxserver/edge_openresty/_package/Lua/resty/lock.lua", O_RDONLY) = -1 ENOENT (No such file or directory)

05:08:00 open("/usr/share/nginx/site/lualib/resty/lock.ljbc", O_RDONLY) = -1 ENOENT (No such file or directory)

05:08:00 open("/usr/share/nginx/site/lualib/resty/lock/init.ljbc", O_RDONLY) = -1 ENOENT (No such file or directory)

05:08:00 open("/usr/share/nginx/lualib/resty/lock.ljbc", O_RDONLY) = -1 ENOENT (No such file or directory)

05:08:00 open("/usr/share/nginx/lualib/resty/lock/init.ljbc", O_RDONLY) = -1 ENOENT (No such file or directory)

05:08:00 open("/usr/share/nginx/site/lualib/resty/lock.lua", O_RDONLY) = -1 ENOENT (No such file or directory)

05:08:00 open("/usr/share/nginx/site/lualib/resty/lock/init.lua", O_RDONLY) = -1 ENOENT (No such file or directory)

05:08:00 open("/usr/share/nginx/lualib/resty/lock.lua", O_RDONLY) = 101

05:08:00 close(101) = 0

05:08:00 open("/usr/share/nginx/lualib/resty/lock.lua", O_RDONLY) = 101

05:08:00 fstat(101, {st_mode=S_IFREG|0644, st_size=4616, ...}) = 0

05:08:00 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5cb1fe9000

05:08:00 read(101, "-- Copyright (C) Yichun Zhang (a"..., 8192) = 4616

05:08:00 read(101, "", 4096) = 0

05:08:00 close(101) = 0

05:08:00 munmap(0x7f5cb1fe9000, 4096) = 0

05:08:00 setsockopt(39, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0

05:08:00 setsockopt(39, SOL_TCP, TCP_KEEPIDLE, [2], 4) = 0

05:08:00 setsockopt(39, SOL_TCP, TCP_KEEPINTVL, [2], 4) = 0

05:08:00 setsockopt(39, SOL_TCP, TCP_KEEPCNT, [8], 4) = 0

05:08:00 setsockopt(42, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0

05:08:00 setsockopt(42, SOL_TCP, TCP_KEEPIDLE, [2], 4) = 0

05:08:00 setsockopt(42, SOL_TCP, TCP_KEEPINTVL, [2], 4) = 0

05:08:00 setsockopt(42, SOL_TCP, TCP_KEEPCNT, [8], 4) = 0

05:08:00 dup2(66, 2) = 2

05:08:00 getrlimit(RLIMIT_NOFILE, {rlim_cur=65535, rlim_max=65535}) = 0

05:08:00 close(4) = 0

05:08:00 close(5) = 0

05:08:00 close(6) = 0

05:08:00 close(7) = 0

05:08:00 close(8) = 0

05:08:00 close(9) = 0

05:08:00 close(10) = 0

05:08:00 close(11) = 0

05:08:00 close(12) = 0

05:08:00 close(13) = 0

05:08:00 close(14) = 0

05:08:00 close(15) = 0

05:08:00 close(16) = 0

05:08:00 close(17) = 0

05:08:00 close(18) = 0

05:08:00 close(19) = 0

05:08:00 close(20) = 0

05:08:00 close(21) = 0

05:08:00 close(22) = 0

05:08:00 close(23) = 0

05:08:00 close(24) = 0

05:08:00 close(25) = 0

05:08:00 close(26) = 0

05:08:00 close(27) = 0

05:08:00 close(28) = 0

05:08:00 close(29) = 0

05:08:00 close(30) = 0

05:08:00 close(31) = 0

05:08:00 close(32) = 0

05:08:00 close(33) = 0

05:08:00 close(34) = 0

05:08:00 close(35) = 0

05:08:00 close(36) = 0

05:08:00 close(37) = 0

05:08:00 close(38) = 0

05:08:00 --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x7f0} ---

05:08:12 +++ killed by SIGSEGV (core dumped) +++

zhaohang200801

主要情况时，线上正在运行的正常nginx，包括master、worker都正常。脚本发布后，ps 查看进程就发现master不存在了，而worker依旧在。

zchao1995

Hello!

请问使用的 OpenResty 版本是多少？

spacewanderlzx

看调用栈，像是对 ffi.load 加载的库（或者 ffi.C) 做 GC 时，崩溃在 dlclose 里面了。你排查下这方面的可能性？

2018-06-07 5:16 GMT+08:00 Tom <zhaoha...@gmail.com>:

升级版本时发现自动化脚本部署时出现master退出现象，手动安装无问题，跟踪发现段错误，但具体哪里出的错误实难定位。求大神帮忙看下，谢谢。

coredump 如下：

strace 跟踪master部分结果如下：
05:08:00 open("/home/qboxserver/edge_openresty/_package/resty/lock.lua", O_RDONLY) = -1 ENOENT (No such file or directory)
05:08:00 open("/home/qboxserver/edge_openresty/_package/Lua/resty/lock.lua", O_RDONLY) = -1 ENOENT (No such file or directory)
05:08:00 open("/usr/share/nginx/site/lualib/resty/lock.ljbc", O_RDONLY) = -1 ENOENT (No such file or directory)
05:08:00 open("/usr/share/nginx/site/lualib/resty/lock/init.ljbc", O_RDONLY) = -1 ENOENT (No such file or directory)
05:08:00 open("/usr/share/nginx/lualib/resty/lock.ljbc", O_RDONLY) = -1 ENOENT (No such file or directory)
05:08:00 open("/usr/share/nginx/lualib/resty/lock/init.ljbc", O_RDONLY) = -1 ENOENT (No such file or directory)
05:08:00 open("/usr/share/nginx/site/lualib/resty/lock.lua", O_RDONLY) = -1 ENOENT (No such file or directory)
05:08:00 open("/usr/share/nginx/site/lualib/resty/lock/init.lua", O_RDONLY) = -1 ENOENT (No such file or directory)
05:08:00 open("/usr/share/nginx/lualib/resty/lock.lua", O_RDONLY) = 101
05:08:00 close(101) = 0
05:08:00 open("/usr/share/nginx/lualib/resty/lock.lua", O_RDONLY) = 101
05:08:00 fstat(101, {st_mode=S_IFREG|0644, st_size=4616, ...}) = 0
05:08:00 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5cb1fe9000
05:08:00 read(101, "-- Copyright (C) Yichun Zhang (a"..., 8192) = 4616
05:08:00 read(101, "", 4096) = 0
05:08:00 close(101) = 0
05:08:00 munmap(0x7f5cb1fe9000, 4096) = 0
05:08:00 setsockopt(39, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
05:08:00 setsockopt(39, SOL_TCP, TCP_KEEPIDLE, [2], 4) = 0
05:08:00 setsockopt(39, SOL_TCP, TCP_KEEPINTVL, [2], 4) = 0
05:08:00 setsockopt(39, SOL_TCP, TCP_KEEPCNT, [8], 4) = 0
05:08:00 setsockopt(42, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
05:08:00 setsockopt(42, SOL_TCP, TCP_KEEPIDLE, [2], 4) = 0
05:08:00 setsockopt(42, SOL_TCP, TCP_KEEPINTVL, [2], 4) = 0
05:08:00 setsockopt(42, SOL_TCP, TCP_KEEPCNT, [8], 4) = 0
05:08:00 dup2(66, 2) = 2
05:08:00 getrlimit(RLIMIT_NOFILE, {rlim_cur=65535, rlim_max=65535}) = 0
05:08:00 close(4) = 0
05:08:00 close(5) = 0
05:08:00 close(6) = 0
05:08:00 close(7) = 0
05:08:00 close(8) = 0
05:08:00 close(9) = 0
05:08:00 close(10) = 0
05:08:00 close(11) = 0
05:08:00 close(12) = 0
05:08:00 close(13) = 0
05:08:00 close(14) = 0
05:08:00 close(15) = 0
05:08:00 close(16) = 0
05:08:00 close(17) = 0
05:08:00 close(18) = 0
05:08:00 close(19) = 0
05:08:00 close(20) = 0
05:08:00 close(21) = 0
05:08:00 close(22) = 0
05:08:00 close(23) = 0
05:08:00 close(24) = 0
05:08:00 close(25) = 0
05:08:00 close(26) = 0
05:08:00 close(27) = 0
05:08:00 close(28) = 0
05:08:00 close(29) = 0
05:08:00 close(30) = 0
05:08:00 close(31) = 0
05:08:00 close(32) = 0
05:08:00 close(33) = 0
05:08:00 close(34) = 0
05:08:00 close(35) = 0
05:08:00 close(36) = 0
05:08:00 close(37) = 0
05:08:00 close(38) = 0
05:08:00 --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x7f0} ---
05:08:12 +++ killed by SIGSEGV (core dumped) +++

--

zhaohang200801

1.11.2，跟版本号没关系，定位是动态库的问题。

在 2018年6月7日星期四 UTC+8上午9:58:47，tokers写道：

Hello!

请问使用的 OpenResty 版本是多少？

zhaohang200801

嗯，动态库调用引起的。

在 2018年6月7日星期四 UTC+8上午10:32:11，Zexuan Luo写道：

看调用栈，像是对 ffi.load 加载的库（或者 ffi.C) 做 GC 时，崩溃在 dlclose 里面了。你排查下这方面的可能性？

2018-06-07 5:16 GMT+08:00 Tom <zhaohan...@gmail.com>:
升级版本时发现自动化脚本部署时出现master退出现象，手动安装无问题，跟踪发现段错误，但具体哪里出的错误实难定位。求大神帮忙看下，谢谢。

coredump 如下：

strace 跟踪master部分结果如下：
05:08:00 open("/home/qboxserver/edge_openresty/_package/resty/lock.lua", O_RDONLY) = -1 ENOENT (No such file or directory)
05:08:00 open("/home/qboxserver/edge_openresty/_package/Lua/resty/lock.lua", O_RDONLY) = -1 ENOENT (No such file or directory)
05:08:00 open("/usr/share/nginx/site/lualib/resty/lock.ljbc", O_RDONLY) = -1 ENOENT (No such file or directory)
05:08:00 open("/usr/share/nginx/site/lualib/resty/lock/init.ljbc", O_RDONLY) = -1 ENOENT (No such file or directory)
05:08:00 open("/usr/share/nginx/lualib/resty/lock.ljbc", O_RDONLY) = -1 ENOENT (No such file or directory)
05:08:00 open("/usr/share/nginx/lualib/resty/lock/init.ljbc", O_RDONLY) = -1 ENOENT (No such file or directory)
05:08:00 open("/usr/share/nginx/site/lualib/resty/lock.lua", O_RDONLY) = -1 ENOENT (No such file or directory)
05:08:00 open("/usr/share/nginx/site/lualib/resty/lock/init.lua", O_RDONLY) = -1 ENOENT (No such file or directory)
05:08:00 open("/usr/share/nginx/lualib/resty/lock.lua", O_RDONLY) = 101
05:08:00 close(101) = 0
05:08:00 open("/usr/share/nginx/lualib/resty/lock.lua", O_RDONLY) = 101
05:08:00 fstat(101, {st_mode=S_IFREG|0644, st_size=4616, ...}) = 0
05:08:00 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5cb1fe9000
05:08:00 read(101, "-- Copyright (C) Yichun Zhang (a"..., 8192) = 4616
05:08:00 read(101, "", 4096) = 0
05:08:00 close(101) = 0
05:08:00 munmap(0x7f5cb1fe9000, 4096) = 0
05:08:00 setsockopt(39, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
05:08:00 setsockopt(39, SOL_TCP, TCP_KEEPIDLE, [2], 4) = 0
05:08:00 setsockopt(39, SOL_TCP, TCP_KEEPINTVL, [2], 4) = 0
05:08:00 setsockopt(39, SOL_TCP, TCP_KEEPCNT, [8], 4) = 0
05:08:00 setsockopt(42, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
05:08:00 setsockopt(42, SOL_TCP, TCP_KEEPIDLE, [2], 4) = 0
05:08:00 setsockopt(42, SOL_TCP, TCP_KEEPINTVL, [2], 4) = 0
05:08:00 setsockopt(42, SOL_TCP, TCP_KEEPCNT, [8], 4) = 0
05:08:00 dup2(66, 2) = 2
05:08:00 getrlimit(RLIMIT_NOFILE, {rlim_cur=65535, rlim_max=65535}) = 0
05:08:00 close(4) = 0
05:08:00 close(5) = 0
05:08:00 close(6) = 0
05:08:00 close(7) = 0
05:08:00 close(8) = 0
05:08:00 close(9) = 0
05:08:00 close(10) = 0
05:08:00 close(11) = 0
05:08:00 close(12) = 0
05:08:00 close(13) = 0
05:08:00 close(14) = 0
05:08:00 close(15) = 0
05:08:00 close(16) = 0
05:08:00 close(17) = 0
05:08:00 close(18) = 0
05:08:00 close(19) = 0
05:08:00 close(20) = 0
05:08:00 close(21) = 0
05:08:00 close(22) = 0
05:08:00 close(23) = 0
05:08:00 close(24) = 0
05:08:00 close(25) = 0
05:08:00 close(26) = 0
05:08:00 close(27) = 0
05:08:00 close(28) = 0
05:08:00 close(29) = 0
05:08:00 close(30) = 0
05:08:00 close(31) = 0
05:08:00 close(32) = 0
05:08:00 close(33) = 0
05:08:00 close(34) = 0
05:08:00 close(35) = 0
05:08:00 close(36) = 0
05:08:00 close(37) = 0
05:08:00 close(38) = 0
05:08:00 --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x7f0} ---
05:08:12 +++ killed by SIGSEGV (core dumped) +++

--