Re: A couple of ideas ...

agentzh · 2013-11-22T04:07:28+00:00

Because the new mainline version of openresty is not ready to release atm :) It's expected to be out soon. Ok thanks!r.

Re: A couple of ideas ...

agentzh

Hello!

On Wed, Nov 20, 2013 at 8:55 PM, Mark Selby wrote:
> My use case, and I think a common one, for OpenResty is accessing Redis for
> cached content. Would it be a good use case to have "sorted params"
> available, so that
>
> /blah?q=cheese&page=3
> can easily become
> /blah?page=3&q=cheese
>
> This seems tricky to do in LUA for some reason, but would obviously be
> valuable for more cache hits on API's etc .... maybe with the ability to
> fairly easily ignore irrelevant params
> "irrelevant=&blah=somethingNotIrrelevant"
>

This is not tricky to do in Lua at all (BTW, it's Lua, not LUA). See
the documentation below for an example:

http://wiki.nginx.org/HttpSRCacheModule#Cache_Key_Preprocessing

Although in that example the arguments are not really sorted by keys,
the order of keys should be consistent due to the table hash algorithm
in Lua. Even if that's not good enough for you, it's not hard to do
explicit sorting yourself in Lua anyway.

> To chuck something into the pot, the ngx.sleep is brilliant - I'm now
> rewriting for the case where the cache gets cleared, and the first backend
> hit writes to Redis with metadata saying { pending: 100 } straight away
> (before processing anything), so on a busy site the first cache miss adds
> this for the appropriate URL, and subsequent request see this and ngx.sleep
> for that number of milliseconds before checking the cache again - avoiding
> lots of backend hits for the same content.
>

This sounds like exactly what the lua-resty-lock library is doing:

    https://github.com/agentzh/lua-resty-lock

Maybe you don't have to reinvent the wheel here ;)

I'm cc'ing the openresty-en mailing list:
https://groups.google.com/group/openresty-en And you're recommended to
join the list and post further questions there (unless your mail has
sensitive information).

Thanks!
-agentzh

mark

Got a crude implementation working for the alphabetical parameters, posted for reference :

location /test {

rewrite_by_lua '

local sortedUri = ""

local sortedKeys = {}

local args = ngx.req.get_uri_args()

ngx.header.Content_Type = "text/plain"

for k in pairs(args) do

sortedKeys[#sortedKeys + 1] = k

end

table.sort(sortedKeys)

for k, v in pairs(sortedKeys) do

if #sortedUri > 0 then

sortedUri = sortedUri.."&"

end

sortedUri = sortedUri..v.."="..args[v]

end

ngx.req.set_uri_args(sortedUri)

echo $args;

}

I must admit that because I only use LUA rarely it still makes me angry :-)

On 22 November 2013 04:07, Yichun Zhang (agentzh) <age...@gmail.com> wrote:

Hello!

On Wed, Nov 20, 2013 at 8:55 PM, Mark Selby wrote:
> My use case, and I think a common one, for OpenResty is accessing Redis for
> cached content. Would it be a good use case to have "sorted params"
> available, so that
>
> /blah?q=cheese&page=3
> can easily become
> /blah?page=3&q=cheese
>
> This seems tricky to do in LUA for some reason, but would obviously be
> valuable for more cache hits on API's etc .... maybe with the ability to
> fairly easily ignore irrelevant params
> "irrelevant=&blah=somethingNotIrrelevant"
>

This is not tricky to do in Lua at all (BTW, it's Lua, not LUA). See
the documentation below for an example:

http://wiki.nginx.org/HttpSRCacheModule#Cache_Key_Preprocessing

Although in that example the arguments are not really sorted by keys,
the order of keys should be consistent due to the table hash algorithm
in Lua. Even if that's not good enough for you, it's not hard to do
explicit sorting yourself in Lua anyway.

> To chuck something into the pot, the ngx.sleep is brilliant - I'm now
> rewriting for the case where the cache gets cleared, and the first backend
> hit writes to Redis with metadata saying { pending: 100 } straight away
> (before processing anything), so on a busy site the first cache miss adds
> this for the appropriate URL, and subsequent request see this and ngx.sleep
> for that number of milliseconds before checking the cache again - avoiding
> lots of backend hits for the same content.
>

This sounds like exactly what the lua-resty-lock library is doing:

https://github.com/agentzh/lua-resty-lock

Maybe you don't have to reinvent the wheel here ;)

I'm cc'ing the openresty-en mailing list:
https://groups.google.com/group/openresty-en And you're recommended to
join the list and post further questions there (unless your mail has
sensitive information).

Thanks!
-agentzh

agentzh

Hello!

On Thu, Nov 21, 2013 at 6:31 PM, Mark Selby wrote:
>   location /test {
>     rewrite_by_lua '
>       local sortedUri = ""
>       local sortedKeys = {}
>       local args = ngx.req.get_uri_args()
>
>       ngx.header.Content_Type = "text/plain"
>       for k in pairs(args) do
>         sortedKeys[#sortedKeys + 1] = k

You should avoid calling the '#' operator in every loop iteration.

>       end
>       table.sort(sortedKeys)
>       for k, v in pairs(sortedKeys) do

You could have used ipairs() here, which is much more efficient.

>         if #sortedUri > 0 then

Again, you should avoid calling '#' in every loop iteration.

>           sortedUri = sortedUri.."&"

You should have used Lua table and table.concat to do string
concatenation here. Use of s = s .. "blah" in a loop is very
expensive.

>         end
>         sortedUri = sortedUri..v.."="..args[v]

You should take into account multi-value and boolean-value argument.
Read the documentation of ngx.req.get_uri_args() more carefully:

https://github.com/chaoslawful/lua-nginx-module#ngxreqget_uri_args

This line could easily break.

>
> I must admit that because I only use LUA rarely it still makes me angry :-)
>

I must admit your Lua code also makes me "angry".

Also, I've already told you, it is Lua, not LUA! It is not an acronym
at all. Please don't make me say this again.

Finally, if you don't like Lua, just stop using it. Nobody is forcing
you to use it after all.

Regards,
-agentzh

agentzh

Hello!

On Thu, Nov 21, 2013 at 8:37 PM, Yichun Zhang (agentzh) wrote:
>>         end
>>         sortedUri = sortedUri..v.."="..args[v]
>

Sorry, forgot to mention that you should also call ngx.escape_uri()
here to escape (any) special bytes in args[v]. See

    https://github.com/chaoslawful/lua-nginx-module#ngxescape_uri

Regards,
-agentzh

mark

I'm using two server blocks, one for web and one for an API. The API caches to Redis db 0, the web to Redis db 1.

Is there a way to have 2 Redis connection pools (for each server block) and have each one staying connected to the appropriate db to avoid the extra "select" command on each request?

I've had a read around and lots of experimenting but it escapes me ....

Thanks.

On 22 November 2013 12:37, Yichun Zhang (agentzh) <age...@gmail.com> wrote:

Hello!

On Thu, Nov 21, 2013 at 6:31 PM, Mark Selby wrote:
> location /test {
> rewrite_by_lua '
> local sortedUri = ""
> local sortedKeys = {}
> local args = ngx.req.get_uri_args()
>
> ngx.header.Content_Type = "text/plain"
> for k in pairs(args) do
> sortedKeys[#sortedKeys + 1] = k

You should avoid calling the '#' operator in every loop iteration.

> end
> table.sort(sortedKeys)
> for k, v in pairs(sortedKeys) do

You could have used ipairs() here, which is much more efficient.

> if #sortedUri > 0 then

Again, you should avoid calling '#' in every loop iteration.

> sortedUri = sortedUri.."&"

You should have used Lua table and table.concat to do string
concatenation here. Use of s = s .. "blah" in a loop is very
expensive.

> end
> sortedUri = sortedUri..v.."="..args[v]

You should take into account multi-value and boolean-value argument.
Read the documentation of ngx.req.get_uri_args() more carefully:

https://github.com/chaoslawful/lua-nginx-module#ngxreqget_uri_args

This line could easily break.

>
> I must admit that because I only use LUA rarely it still makes me angry :-)
>

I must admit your Lua code also makes me "angry".

Also, I've already told you, it is Lua, not LUA! It is not an acronym
at all. Please don't make me say this again.

Finally, if you don't like Lua, just stop using it. Nobody is forcing
you to use it after all.

Regards,
-agentzh

agentzh

Hello!

On Sat, Nov 23, 2013 at 5:44 PM, Mark Selby wrote:
> I'm using two server blocks, one for web and one for an API. The API caches
> to Redis db 0, the web to Redis db 1.
>
> Is there a way to have 2 Redis connection pools (for each server block) and
> have each one staying connected to the appropriate db to avoid the extra
> "select" command on each request?
>

Yes, sure. Just specify the "pool" option for the connect() call, see

    https://github.com/agentzh/lua-resty-redis#connect

Basically you can use two pool names, "db1" and "db2", so that you'll
have two separate connection pools.

You can also call the get_reused_times() method to determine if the
connection pool is from the pool for is just newly established:

   https://github.com/agentzh/lua-resty-redis#get_reused_times

Only run "select" when the current connection is newly established.

It does not matter if there're two server blocks or just one. It's
quite flexible.

BTW 1: you can use the ngx-lua-conn-pools tool to inspect all the
connection pools in one of your Nginx worker process if you're on
Linux:

    https://github.com/agentzh/nginx-systemtap-toolkit#ngx-lua-conn-pools

BTW 2: it's recommended to use more descriptive email titles for your
new questions on the mailing list.

Regards,
-agentzh