Best practice handling of EAGAIN in lua call outs to redis.

dkierans · 2013-04-03T08:57:13+00:00

Hello! On Mon, Apr 1, 2013 at 10:47 AM, wrote: > I was wondering if anyone could recommend for me the best way to consume an > API response from an u...

Best practice handling of EAGAIN in lua call outs to redis.

dkierans

I've got a quick question about openresty, Lua and EAGAIN error/return codes.

When stress testing a server with many many connected clients after a ramp up EAGAIN will appear in the nginx logs as the stack manages it's resources.

Now, I am thinking that it's not so much an error as a status indicator caused by attempted reads of sockets that may not have data available on them yet for example

In the context of openresty is it typically the responsibility of the client code, lua in this case, to catch this error then retry? The alternative being that somewhere in openresty it magically does this for us?

EXAMPLE

Taking something like this...

local redis =require "resty.redis"

connectOk = redis:connect(config.host, config.port)

... if the above fails with something like [connect() to unix:/tmp/redis_27002.sock failed (11: Resource temporarily unavailable)] and from Lua [redis:connect() resource temporarily unavailable]

Dave.

agentzh

Hello!

On Tue, Apr 2, 2013 at 5:57 PM, Dave Kierans wrote:
> I've got a quick question about openresty, Lua and EAGAIN error/return
> codes.
>
> When stress testing a server with many many connected clients after a ramp
> up EAGAIN will appear in the nginx logs as the stack manages it's resources.
>
> Now, I am thinking that it's not so much an error as a status indicator
> caused by attempted reads of sockets that may not have data available on
> them yet for example
>
> In the context of openresty is it typically the responsibility of the client
> code, lua in this case, to catch this error then retry?

Unlike recv() and write(), connect() call returns EAGAIN for real
problems (in the Linux man page's words, "no more free local ports or
insufficient entries in the routing cache"), or in other words, the
connect() call completes but fails.

Because you're using the stream unix domain socket, this EAGAIN error
returned by connect() often means that your accept() queue on your
redis server side is overflowing. It happens when your single-threaded
Redis server is too busy and just cannot catch up with the incoming
connections with the CPU resources available. Please consider

1. increasing the backlog setting in your Redis sever's configuration; and/or
2. scaling your Redis backend with multiple nodes to take advantage of
more CPU cores.

Also, enabling the redis connection pool can also be helpful here. See
https://github.com/agentzh/lua-resty-redis#set_keepalive for details.

FWIW, both the Nginx core and ngx_lua automatically handle the
EINPROGRESS error returned by connect(), which indicates that "socket
is nonblocking and the connection cannot be completed immediately",
similar to the EAGAIN error returned by recv() and write(). You can
check out the "ngx_event_connect_peer" function in the Nginx core to
confirm this.

You're encouraged to check out the man page of connect() for more details.

>  The alternative
> being that somewhere in openresty it magically does this for us?
>

ngx_lua's cosockets use nginx core API to do sockets and
I don't think Nginx should automatically retry connect() when EAGAIN
is returned, because unlike the EINPROGRESS error which is common for
nonblocking sockets, EAGAIN for connect() indicates a real problem.

Best regards,
-agentzh