Thanks for your answer!
I've have some more information on this problem.
I'm using apache banch (ab) for load testing in our testing environment and every time i call the openresty server with more than 100 concurrent connections (ab -n 2000 -c 100 http://our_server) i get "lua tcp socket read timed out" on red:eval_sha(....) command.
I've checked the redis slowlog and it seems the redis opearation takes around 0.01 seconds.
Before i connect to the redis i set the timeout to 3 seconds (red:set_timeout(3000))
I also use keep_alive at the end of the redis operation (ok, err = red:set_keepalive(10000, 100))
After raising the connection timeout to 10 seconds, (red:set_timeout(10000)), it appears that there are no "lua tcp socket read time out" errors with 100 concurrent connections, but when i try with 200 concurrent connections, they are back.
*** Any idea what can i do to increase the number of concurrent connections to our system?
*** If i use the unix domain socket to connect to the redis, i will be able to support more concurrent connections?
Thanks,
Ron
On Tuesday, December 10, 2013 5:40:46 PM UTC+2, agentzh wrote:
Hello!
On Tue, Dec 10, 2013 at 1:23 AM, Ron Schwartz wrote:
> We are running OpenResty with redis client for quite a long in production.
> In the last days we started getting "lua tcp socket read timed out" errors.
>
If only the Lua code snippet you provided uses ngx_lua's cosocket API,
then only evalsha() can yield this "read timed out" error. Neither
connect() nor set_keepalive() would.
> From inspecting the error log, it doesn't seem it happens on the call
> red:connect(), red:evalsha or red:set_keep_alive. (Because i dont see the
> custom error messages as described above)
>
This is weird. Are you sure you don't have any other Lua code doing
cosockets? Also you can emulate redis read timeout in your redis Lua
script and test if your Lua-land error handling code can indeed work.
One way to debug this further is to write a systemtap or dtrace based
tool to trace this "read timed out" error in production and dump out
the Lua-land backtraces at those positions on the fly.
One probing point candidate is the C function
ngx_http_lua_socket_tcp_receive_retval_handler in nginx. Check out the
stap++ project for some similar examples:
https://github.com/agentzh/stapxx
Regards,
-agentzh
ab -n 2000 -c 100 'http://54.214.48.186/reads