Segfault - ngx_http_lua_socket_tcp_finalize

rosstimson · 2014-03-18T19:40:05+00:00

Hello Yichun,Both of these modules looks like still not production ready.Anyway, currently we suspend our research in this way.But thanks a lot for resp...

Segfault - ngx_http_lua_socket_tcp_finalize

rosstimson

I am using OpenResty to act as an API proxy and I'm experiencing seemingly random segfaults which causes the Nginx worker process to crash with 'signal 17 (SIGCHLD) received'. This seems to happen both when retrieving data direct from the app and from Memcached.

I've been seeing this behaviour both with OpenResty Stable and Mainline, my current version is:

nginx version: openresty/1.5.8.1
TLS SNI support enabled
configure arguments: --prefix=/usr/local/nginx
--with-debug
--add-module=../ngx_devel_kit-0.2.19
--add-module=../echo-nginx-module-0.51
--add-module=../xss-nginx-module-0.04
--add-module=../ngx_coolkit-0.2rc1
--add-module=../set-misc-nginx-module-0.24 --add-module=../form-input-nginx-module-0.07 --add-module=../encrypted-session-nginx-module-0.03 --add-module=../srcache-nginx-module-0.25
--add-module=../ngx_lua-0.9.4
--add-module=../headers-more-nginx-module-0.25 --add-module=../array-var-nginx-module-0.03 --add-module=../memc-nginx-module-0.14 
--add-module=../redis2-nginx-module-0.10 --add-module=../redis-nginx-module-0.3.7 --add-module=../auth-request-nginx-module-0.2 --add-module=../rds-json-nginx-module-0.13 --add-module=../rds-csv-nginx-module-0.05 --with-ld-opt=-Wl,-rpath,/usr/local/luajit/lib --with-pcre=/home/ec2-user/pcre-8.34 
--with-http_ssl_module

OS: CentOS release 6.4 (Final) x86_64

I've created a Gist with configs and backtrace here: https://gist.github.com/rosstimson/9618197 I hope that this has all the details you need to help me.

The segfault always happen just after the get_user_details function but it does not matter if the user details come from the app or from the cache (Memcached). I've also tried adding an extra function in cache.lua (referenced in Gist) like so:

local function memcache_disconnect()
  local ok, err = memc:close()
  if not ok then
    ngx.log(ngx.ERR, "failed to close connection to memcached: ", err)
    return ngx.exit(500)
  end
end

And then calling this function at the end of all the functions that use memcache_connect() but this did not improve matters.

Any help/pointers as to what I'm doing wrong would be greatly appreciated as I've been tearing my hair out over this for the past week or so.

Ross

agentzh

Hello!

On Tue, Mar 18, 2014 at 4:40 AM, Ross Timson wrote:
> I am using OpenResty to act as an API proxy and I'm experiencing seemingly
> random segfaults which causes the Nginx worker process to crash with 'signal
> 17 (SIGCHLD) received'.  This seems to happen both when retrieving data
> direct from the app and from Memcached.
>

I see one obvious mistake in your Lua code that could lead to such
crashes. That is, you incorrectly cache the resty.memcached object
(and thus the underlying cosocket object) on the Lua module level
(i.e., the "memc"  variable in your cache.lua module file). This will
make multiple requests access the same cosocket object, which is not
allowed at all. See

    https://github.com/agentzh/lua-resty-memcached#limitations

You should only store the resty.memcached object in function local
variables, function parameters, or per-request storage like the
ngx.ctx table.

And yeah, crashing when the Lua programmer makes such mistakes is not
an option. I'll try adding checks to the cosocket API to explicitly
test whether the owner request and the current request do match.

Regards,
-agentzh

agentzh

Hello!

On Tue, Mar 18, 2014 at 12:45 PM, Yichun Zhang (agentzh) wrote:
> And yeah, crashing when the Lua programmer makes such mistakes is not
> an option. I'll try adding checks to the cosocket API to explicitly
> test whether the owner request and the current request do match.
>

I've just committed a patch to ngx_lua's git master branch to do such checks:

    https://github.com/chaoslawful/lua-nginx-module/commit/414db1a

Thank you for bringing this bad use case up!

Best regards,
-agentzh

rosstimson

@agentzh

Thank you very much for this, at least I now know where to focus my attention and your patch should prevent me from making the same mistake twice.

Glad my mistake has at least served some purpose :)

Regards,

Ross