keepalive connections for resty redis and postgresql

gg.lafon · 2013-11-25T20:28:29+00:00

On Fri, Nov 22, 2013 at 11:23 AM, Yichun Zhang (agentzh) <age...@gmail.com> wrote: Hello! On Fri, Nov 22, 2013 at 11:15 AM, Roberto Ostinelli wro...

keepalive connections for resty redis and postgresql

gg.lafon

Hello,

why using a pool of keepalived connections for resty redis and postgresql is less efficient than open/close on each request?

Postgresql and redis are installed on localhost.

Jérôme

agentzh

Hello!

On Mon, Nov 25, 2013 at 4:28 AM, jerome lafon wrote:
> why using a pool of keepalived connections for resty redis and postgresql is
> less efficient than open/close on each request?
> Postgresql and redis are installed on localhost.
>

Because keepalive connections can save you expensive open() and
close() syscalls upon every request served by Nginx. And it can also
save expensive session creation and destruction work on the Redis/Pg
server side (which is particularly expensive for Pg).

Regards,
-agentzh

agentzh

Hello!

On Mon, Nov 25, 2013 at 11:04 AM, Yichun Zhang (agentzh) wrote:
>
> Because keepalive connections can save you expensive open() and
> close() syscalls upon every request served by Nginx.

Sorry, typo here. I meant socket() and connect() syscalls here instead
of open().

Regards,
-agentzh

gg.lafon

Thank you for your reply, I was expecting that behavior but after some benchmarks with weighttp on my side it's less efficient to use keepalive connections instead of open/close on each request.

I don't understand why.

does someone could explain me?

Le lundi 25 novembre 2013 20:05:13 UTC+1, agentzh a écrit :

Hello!

On Mon, Nov 25, 2013 at 11:04 AM, Yichun Zhang (agentzh) wrote:
>
> Because keepalive connections can save you expensive open() and
> close() syscalls upon every request served by Nginx.

Sorry, typo here. I meant socket() and connect() syscalls here instead
of open().

Regards,
-agentzh

agentzh

Hello!

On Mon, Nov 25, 2013 at 2:59 PM, jerome lafon wrote:
> Thank you for your reply, I was expecting that behavior but after some
> benchmarks with weighttp on my side it's less efficient to use keepalive
> connections instead of open/close on each request.
> I don't understand why.
>

The simplest way to validate your benchmark results is to do on-CPU
and off-CPU flamegraphs for one of your Nginx workers being loaded if
you are on Linux:

    https://github.com/agentzh/nginx-systemtap-toolkit#sample-bt
    https://github.com/agentzh/nginx-systemtap-toolkit#sample-bt-off-cpu

It's easy to make mistakes when doing benchmarks :)

Also,

1. always keep an eye on your nginx's error log file, and
2. use tools like netstat and ngx-lua-conn-pools
(https://github.com/agentzh/nginx-systemtap-toolkit#ngx-lua-conn-pools
) to ensure that your connection pools are indeed functioning.

Best regards,
-agentzh

gg.lafon

Thank you, I will use these scripts as soon as I will be able to install systemtap.

I think I found my mistake:

I'm using postgresql and redis in a LUA module. I was doing only one time a new() and on each function call I was doing a connect().

Now I do a new() and a connect on each function call and I use keepalive pool. It seems to boost the number of requests per second.

Best regards

Jerome

Le mardi 26 novembre 2013 00:12:49 UTC+1, agentzh a écrit :

Hello!

On Mon, Nov 25, 2013 at 2:59 PM, jerome lafon wrote:
> Thank you for your reply, I was expecting that behavior but after some
> benchmarks with weighttp on my side it's less efficient to use keepalive
> connections instead of open/close on each request.
> I don't understand why.
>

The simplest way to validate your benchmark results is to do on-CPU
and off-CPU flamegraphs for one of your Nginx workers being loaded if
you are on Linux:

https://github.com/agentzh/nginx-systemtap-toolkit#sample-bt
https://github.com/agentzh/nginx-systemtap-toolkit#sample-bt-off-cpu

It's easy to make mistakes when doing benchmarks :)

Also,

1. always keep an eye on your nginx's error log file, and
2. use tools like netstat and ngx-lua-conn-pools
(https://github.com/agentzh/nginx-systemtap-toolkit#ngx-lua-conn-pools
) to ensure that your connection pools are indeed functioning.

Best regards,
-agentzh

huangqi

hi, Agentzh

I use setkeepalived in my code with the 'lua-resty-redis'(https://github.com/openresty/lua-resty-redis)

local ok, err = red:set_keepalive(0, 10000)

if not ok then

ngx.say("failed to set keepalive: ", err)

return

end

but, tail the redis stdout with :

so, what's wrong with that code above?

[12234] 17 Nov 16:18:35.829 - Accepted 10.171.99.210:47152

[12234] 17 Nov 16:18:35.845 - Accepted 10.171.99.210:47154

[12234] 17 Nov 16:18:35.845 - Accepted 10.171.99.210:47156

[12234] 17 Nov 16:18:35.846 - Accepted 10.171.99.210:47158

[12234] 17 Nov 16:18:35.849 - Accepted 10.171.99.210:47160

[12234] 17 Nov 16:18:35.896 - Accepted 10.171.99.210:47162

[12234] 17 Nov 16:18:35.919 - Accepted 10.171.99.210:47165

[12234] 17 Nov 16:18:35.929 - Accepted 10.171.99.210:47167

[12234] 17 Nov 16:18:35.972 - Accepted 10.171.99.210:47169

[12234] 17 Nov 16:18:36.114 - Accepted 10.171.99.210:47171

[12234] 17 Nov 16:18:34.869 - Client closed connection

[12234] 17 Nov 16:18:34.870 - Accepted 10.171.99.210:47078

[12234] 17 Nov 16:18:34.870 - Accepted 10.171.99.210:47079

[12234] 17 Nov 16:18:34.871 - Client closed connection

[12234] 17 Nov 16:18:34.872 - Client closed connection

[12234] 17 Nov 16:18:34.872 - Accepted 10.171.99.210:47082

[12234] 17 Nov 16:18:34.873 - Accepted 10.171.99.210:47084

[12234] 17 Nov 16:18:34.875 - Accepted 10.171.99.210:47086

[12234] 17 Nov 16:18:34.876 - Client closed connection

[12234] 17 Nov 16:18:34.877 - Accepted 10.171.99.210:47088

[12234] 17 Nov 16:18:34.877 - Accepted 10.171.99.210:47090

[12234] 17 Nov 16:18:34.878 - Accepted 10.171.99.210:47091

[12234] 17 Nov 16:18:34.878 - Accepted 10.171.99.210:47094

[12234] 17 Nov 16:18:34.878 - Client closed connection

[12234] 17 Nov 16:18:34.879 - Client closed connection

[12234] 17 Nov 16:18:34.879 - Accepted 10.171.99.210:47096

[12234] 17 Nov 16:18:34.896 - Accepted 10.171.99.210:47098

[12234] 17 Nov 16:18:34.900 - Accepted 10.171.99.210:47100

[12234] 17 Nov 16:18:34.904 - Accepted 10.171.99.210:47102

[12234] 17 Nov 16:18:34.928 - Accepted 10.171.99.210:47104

[12234] 17 Nov 16:18:34.930 - Accepted 10.171.99.210:47106

[12234] 17 Nov 16:18:34.933 - Accepted 10.171.99.210:47108

[12234] 17 Nov 16:18:34.965 - Client closed connection

[12234] 17 Nov 16:18:34.966 - Client closed connection

[12234] 17 Nov 16:18:34.967 - Client closed connection

[12234] 17 Nov 16:18:34.968 - Client closed connection

Jerome Lafon於 2013年11月26日星期二UTC+8下午10時07分59秒寫道：

Thank you, I will use these scripts as soon as I will be able to install systemtap.
I think I found my mistake:
I'm using postgresql and redis in a LUA module. I was doing only one time a new() and on each function call I was doing a connect().
Now I do a new() and a connect on each function call and I use keepalive pool. It seems to boost the number of requests per second.

Best regards
Jerome

Le mardi 26 novembre 2013 00:12:49 UTC+1, agentzh a écrit :
Hello!

On Mon, Nov 25, 2013 at 2:59 PM, jerome lafon wrote:
> Thank you for your reply, I was expecting that behavior but after some
> benchmarks with weighttp on my side it's less efficient to use keepalive
> connections instead of open/close on each request.
> I don't understand why.
>

The simplest way to validate your benchmark results is to do on-CPU
and off-CPU flamegraphs for one of your Nginx workers being loaded if
you are on Linux:

https://github.com/agentzh/nginx-systemtap-toolkit#sample-bt
https://github.com/agentzh/nginx-systemtap-toolkit#sample-bt-off-cpu

It's easy to make mistakes when doing benchmarks :)

Also,

1. always keep an eye on your nginx's error log file, and
2. use tools like netstat and ngx-lua-conn-pools
(https://github.com/agentzh/nginx-systemtap-toolkit#ngx-lua-conn-pools
) to ensure that your connection pools are indeed functioning.

Best regards,
-agentzh

agentzh

Hello!

On Mon, Nov 17, 2014 at 12:44 AM, Justin Huang wrote:
> hi, Agentzh
>

Please do not capitalize my nick. Thank you.

> local ok, err = red:set_keepalive(0, 10000)
> if not ok then
>     ngx.say("failed to set keepalive: ", err)

You'd better make the error handling above use ngx.log(ngx.ERR, ...)
instead of ngx.say() so that you can see if there's any problems in
your nginx error logs. The example in the doc is just for
demonstration purposes.

Also, do not set your connection pool capacity too large. Your redis
server's performance usually degrades a lot of with a lot of idle
connections.

Also, it's a good idea to use a sensible timeout value (instead of 0
for unlimited) for the idle in-pool connections (for the same reason
stated above).

> but, tail the redis stdout with :
>

Do you mean the clients frequently close the connections? If yes, then
you should check

1. if some of your redis operations have errors (proper error logging
comes into play again ;)), and
2. if all your code paths for non-error cases call set_keepalive() properly.

BTW, please always give a (brief) summary of your problem. The redis
logs themselves won't talk, really. And I'm not good at guessing
others' intentions :)

Best regards,
-agentzh

huangqi

Sorry for my indecent behavior...and thank you for your suggestion.

but I wonder where is the correct position of ‘ red:set_keepalive’? bottom of the lua script?

Best regards,

-Justin

On Tuesday, 18 November 2014 04:18:19 UTC+8, agentzh wrote:

Hello!

On Mon, Nov 17, 2014 at 12:44 AM, Justin Huang wrote:
> hi, Agentzh
>

Please do not capitalize my nick. Thank you.

> local ok, err = red:set_keepalive(0, 10000)
> if not ok then
> ngx.say("failed to set keepalive: ", err)

You'd better make the error handling above use ngx.log(ngx.ERR, ...)
instead of ngx.say() so that you can see if there's any problems in
your nginx error logs. The example in the doc is just for
demonstration purposes.

Also, do not set your connection pool capacity too large. Your redis
server's performance usually degrades a lot of with a lot of idle
connections.

Also, it's a good idea to use a sensible timeout value (instead of 0
for unlimited) for the idle in-pool connections (for the same reason
stated above).

> but, tail the redis stdout with :
>

Do you mean the clients frequently close the connections? If yes, then
you should check

1. if some of your redis operations have errors (proper error logging
comes into play again ;)), and
2. if all your code paths for non-error cases call set_keepalive() properly.

BTW, please always give a (brief) summary of your problem. The redis
logs themselves won't talk, really. And I'm not good at guessing
others' intentions :)

Best regards,
-agentzh

agentzh

Hello!

On Tue, Nov 18, 2014 at 7:05 AM, Justin Huang wrote:
> but I wonder where is the correct position of ‘ red:set_keepalive’? bottom
> of the lua script?
>

This is documented:

https://github.com/openresty/lua-resty-redis#set_keepalive

"Only call this method in the place you would have called the close
method instead. Calling this method will immediately turn the current
redis object into the closed state. Any subsequent operations other
than connect() on the current objet will return the closed error."

And it was also my previous suggestion:

"check ... if all your code paths for non-error cases call
set_keepalive() properly."

Regards,
-agentzh

jiezhan1

hi agentzh

a closed socket can be called connect method? I always thought it was not allowed。 whether lua socket close method call underlying socket’s close method? thanks agentzh.

On Wednesday, November 19, 2014 at 3:57:41 AM UTC+8, Yichun Zhang (agentzh) wrote:

Hello!

On Tue, Nov 18, 2014 at 7:05 AM, Justin Huang wrote:
> but I wonder where is the correct position of ‘ red:set_keepalive’? bottom
> of the lua script?
>

This is documented:

https://github.com/openresty/lua-resty-redis#set_keepalive

"Only call this method in the place you would have called the close
method instead. Calling this method will immediately turn the current
redis object into the closed state. Any subsequent operations other
than connect() on the current objet will return the closed error."

And it was also my previous suggestion:

"check ... if all your code paths for non-error cases call
set_keepalive() properly."

Regards,
-agentzh