Open Question on Design for implementation

mattwells83 · 2013-06-06T18:17:40+00:00

Hello! On Fri, Jun 7, 2013 at 12:00 PM, agentzh wrote: > On Wed, Jun 5, 2013 at 6:32 AM, aviram wrote: >> I'm implementing a proxy using Nginx and...

Open Question on Design for implementation

mattwells83

Hello

I am in the R&D stages of a service i'm currently constructing and wondered if I might ask feedback from users more experienced than myself:

I am creating a (mildly) intelligent cache that wraps a basic php script, in essence this script performs redirects

In order to perform these redirects upstream I would like to connect to redis, obtain a key, set a cookie and make the redirect from within lua (at which point I submit to a gearman queue to later record the redirect action)

I have a few questions with regards to the approach I should take, specifically:

- I would like an HA redis connection so was considering how I could make use of the upstream directive, but I will need to know the serving redis instance that was used, im unsure of how to get this from within my lua script (accessing upstream_addr for instance in conjunction with redis2_pass and ngx.location.capture) - perhaps it would be better to maintain a table of the available redis instances and connect directly with lua-resty-redis ?)

- are there any users using gearman in production at any scale? what kind of performance should I be looking to get and are there any configuration 'gotchas' I should be aware of?

Many thanks indeed

/Matt

brian

On Jun 6, 2013, at 6:17 AM, NinjaPenguin <mattw...@gmail.com> wrote:

> redis connection so was considering how I could make use of the upstream directive, but I will need to know the serving redis instance that was used, im unsure of how to get this from within my lua script 

Use lua-resty-redis. Faster and you can know exactly which one you connect to.

mattwells83

Thanks very much Brian, I will indeed go for this!

I'll init_by_lua a shared dict containing the servers in the pool and choose one at random on each request, I can then remove from this pool manually if a connection fails - that sound sensible?

On Thursday, 6 June 2013 14:34:30 UTC+1, Brian Akins wrote:

On Jun 6, 2013, at 6:17 AM, NinjaPenguin <mattw...@gmail.com> wrote:

> redis connection so was considering how I could make use of the upstream directive, but I will need to know the serving redis instance that was used, im unsure of how to get this from within my lua script

Use lua-resty-redis. Faster and you can know exactly which one you connect to.

peter_booth

Matt,

Just curious, what functionality do you get from the Redis call? You said that you are building a cache, but that it's also a redirector. Openresty/nginx is awesome, very rich functionality. I've found that the vanilla nginx proxy cache can be very powerful when supplemented by lua,

Sent from my iPhone

On Jun 6, 2013, at 6:17 AM, NinjaPenguin <mattw...@gmail.com> wrote:

Hello

I am in the R&D stages of a service i'm currently constructing and wondered if I might ask feedback from users more experienced than myself:

I am creating a (mildly) intelligent cache that wraps a basic php script, in essence this script performs redirects

In order to perform these redirects upstream I would like to connect to redis, obtain a key, set a cookie and make the redirect from within lua (at which point I submit to a gearman queue to later record the redirect action)

I have a few questions with regards to the approach I should take, specifically:

- I would like an HA redis connection so was considering how I could make use of the upstream directive, but I will need to know the serving redis instance that was used, im unsure of how to get this from within my lua script (accessing upstream_addr for instance in conjunction with redis2_pass and ngx.location.capture) - perhaps it would be better to maintain a table of the available redis instances and connect directly with lua-resty-redis ?)

- are there any users using gearman in production at any scale? what kind of performance should I be looking to get and are there any configuration 'gotchas' I should be aware of?

Many thanks indeed

/Matt

.

agentzh

Hello!

On Thu, Jun 6, 2013 at 2:12 PM, NinjaPenguin wrote:
> I'll init_by_lua a shared dict containing the servers in the pool and choose
> one at random on each request, I can then remove from this pool manually if
> a connection fails - that sound sensible?
>

You can surely do that.

But be careful when you decide to drop a server from the server list
because the connection error can be ephemeral due to occasional packet
drop or other errors in either the kernel or on the wire, in which
case the backend server is not really down. You may want to set up a
threshold for the number of (successive) connection failures for each
backend server or something like that.

Best regards,
-agentzh

mattwells83

@Peter - its a great question, and in actuality I envisage being able to remove the requirement for redis in the future. Currently it is being used as a basic UID generator (we get a, keyspaced, unique id from it (using the atomic inc). In the future I could do this purely from within Nginx using the shared dict, but there are a few complexities around this (like the persistence of the value after restart, the limited number of keyspaces available to me etc..)

@agentzh - fantastic points this would be a great idea - thanks

As my work is progressing I have run into an additional question - i've been adding a number of tests to the code as I've been going, but there are a few areas in which I am using the nginx api specifically (regexps for example) - im wondering if there is a standard method for which to test this? (I can obviously mock out certain aspects but areas like regexp, where im wanting to actually test my implementation logic im not sure how I can actually test this 'offline' so to speak)

Thanks for all the feedback so far guys

/Matt

On Friday, 7 June 2013 04:20:12 UTC+1, agentzh wrote:

Hello!

On Thu, Jun 6, 2013 at 2:12 PM, NinjaPenguin wrote:
> I'll init_by_lua a shared dict containing the servers in the pool and choose
> one at random on each request, I can then remove from this pool manually if
> a connection fails - that sound sensible?
>

You can surely do that.

But be careful when you decide to drop a server from the server list
because the connection error can be ephemeral due to occasional packet
drop or other errors in either the kernel or on the wire, in which
case the backend server is not really down. You may want to set up a
threshold for the number of (successive) connection failures for each
backend server or something like that.

Best regards,
-agentzh

agentzh

Hello!

On Fri, Jun 7, 2013 at 2:33 AM, NinjaPenguin wrote:
> As my work is progressing I have run into an additional question - i've been
> adding a number of tests to the code as I've been going, but there are a few
> areas in which I am using the nginx api specifically (regexps for example) -
> im wondering if there is a standard method for which to test this? (I can
> obviously mock out certain aspects but areas like regexp, where im wanting
> to actually test my implementation logic im not sure how I can actually test
> this 'offline' so to speak)
>

The recommended way to test ngx_lua-based applications is to run a
local nginx server and then beat it from the HTTP client side by means
of Perl or Python or Ruby or something like that. For example, you may
want to take a look at liseen's QAT test framework that has been used
by Qunar.com for testing their nginx-based aplications (and others):

    https://github.com/liseen/QAT

You may also find my Test::Nginx scaffold interesting, which is used
by myself to test ngx_lua, lua-resty-*, and other openresty
components:

    http://search.cpan.org/perldoc?Test%3A%3ANginx

Best regards,
-agentzh

brian

On Fri, Jun 7, 2013 at 3:33 AM, NinjaPenguin <mattw...@gmail.com> wrote:

In the future I could do this purely from within Nginx using the shared dict, but there are a few complexities around this (like the persistence of the value after restart, the limited number of keyspaces available to me etc..)

I would just use redis, FWIW. In my experience, it winds up being less code and complexity. I also have instances I get better overall app performance by using redis than using shared dict, because the spin locks in the shared dict block the entire nginx process, whereas redis network calls do not. Also, redis has some nice features, so once you start using it for one thing, you start using it for many things.

peter_booth

Why do you need to use either to produce uids? Presumably you want this to be very fast? There are robust uuid libraries available that can produce unique uids without need for any interaction between instances or shared state.

Sent from my iPhone

On Jun 7, 2013, at 4:38 PM, Brian Akins <br...@akins.org> wrote:

On Fri, Jun 7, 2013 at 3:33 AM, NinjaPenguin <mattw...@gmail.com> wrote:

In the future I could do this purely from within Nginx using the shared dict, but there are a few complexities around this (like the persistence of the value after restart, the limited number of keyspaces available to me etc..)

I would just use redis, FWIW. In my experience, it winds up being less code and complexity. I also have instances I get better overall app performance by using redis than using shared dict, because the spin locks in the shared dict block the entire nginx process, whereas redis network calls do not. Also, redis has some nice features, so once you start using it for one thing, you start using it for many things.

.

brian

On Jun 8, 2013, at 12:54 AM, Peter Booth <peter...@me.com> wrote:

> Why do you need to use either to produce uids? Presumably you want this to be very fast? There are robust uuid libraries available that can produce unique uids without need for any interaction between instances or shared state.

If on Linux, this works: 

io.open("/proc/sys/kernel/random/uuid"):read())

http://lua-users.org/lists/lua-l/2004-06/msg00587.html

mattwells83

Hey guys

Thanks so much for all the responses

@agentzh - thanks so much for the testing recommendations, I'm going to check out the QAT framework this afternoon

@Peter

> Why do you need to use either to produce uids? Presumably you want this to be very fast? There are robust uuid libraries available that can produce unique uids without need for any interaction between instances or shared state.

Thats a really great question, this work is currently a proof of concept which will hopefully develop, at first, into an intelligent upstream cache to the application. The application currently exists (and is rather large) - the keys generated via redis are widespread and used in various other areas so currently I need to stick with this format of key generation, it is however an incredibly valid point that UUID's generated from within the system itself would open up a raft of performance (we also have limitations on the length of the UID's generated currently..)

@Brian

> I would just use redis, FWIW. In my experience, it winds up being less code and complexity. I also have instances I get better overall app performance by using redis than using shared dict, because the spin locks in the shared dict block the entire nginx process, whereas redis network calls do not. Also, redis has some nice features, so once you start using it for one thing, you start using it for many things

I'm really interested by this - do you know of anywhere that talks about the locks in detail so that I could understand the potential impact of them on performance? - do the locks only apply on write?

Also +1 on the redis usefulness - we're big fans of it currently and make a lot of use of it - the embedded lua scripting has opened up a whole new raft of possibilities as well

One question I'd love to ask you is that of performance - we have an existing application I developed which is again an upstream cache, it relies heavily on redis but in load testing it is the connection to redis itself that seems to be our lowest hanging fruit in terms of bottlenecks - we can get around 17.5K req/s into the nginx instance above which we start seing resource unavailable on the unix socket (we connect via UDS as opposed to TCP/IP) - i'd be really keen to hear of your experiences with it and if you'd done any load testing yourself? I could of course look at distributing the load across a number of redis instances, or perhaps a proxy such as twoemproxy but we're currently plenty happy with the 17,5K/sec :)

Thanks so much for the help so far guys, really appreciated

/Matt

On Saturday, 8 June 2013 12:52:25 UTC+1, Brian Akins wrote:

On Jun 8, 2013, at 12:54 AM, Peter Booth <peter...@me.com> wrote:

> Why do you need to use either to produce uids? Presumably you want this to be very fast? There are robust uuid libraries available that can produce unique uids without need for any interaction between instances or shared state.

If on Linux, this works:

io.open("/proc/sys/kernel/random/uuid"):read())

http://lua-users.org/lists/lua-l/2004-06/msg00587.html

brian

On Jun 9, 2013, at 10:13 AM, NinjaPenguin <mattw...@gmail.com> wrote:

> I'm really interested by this - do you know of anywhere that talks about the locks in detail so that I could understand the potential impact of them on performance? - do the locks only apply on write?

Locks apply on any access to a shared dict. The locks are spin locks.  You can look at the archives of this list for some of my discussions about it. With a spin lock, the entire nginx process blocks waiting on the lock. For certain busy applications this causes overall application slowness.  Technically redis blocks internally, but nginx is able to use all non-blocking network IO talking to it, so while redis is blocking, my other requests (running in other Lua stats in nginx) are able to continue.

It's on my list to write a simpler shared dict implementation that doesn't use spin locks, however, redis fits that need for me now, so it's not a priority right now.

> One question I'd love to ask you is that of performance - we have an existing application I developed which is again an upstream cache, it relies heavily on redis but in load testing it is the connection to redis itself that seems to be our lowest hanging fruit in terms of bottlenecks - we can get around 17.5K req/s into the nginx instance above which we start seing resource unavailable on the unix socket (we connect via UDS as opposed to TCP/IP) - i'd be really keen to hear of your experiences with it and if you'd done any load testing yourself? I could of course look at distributing the load across a number of redis instances, or perhaps a proxy such as twoemproxy but we're currently plenty happy with the 17,5K/sec :)

Use multiple redis instances and shard them - assuming you have more than a few hot keys. In most of our use cases, we've found that one redis per every 2 nginx worker processes seems to give us the best performance.  just write a simple sharding module for redis in Lua, I used the crc of the key because its generally fast enough and is built in. for these local instances, I don't have to worry about any sort of consistent hashing algorithm as the data is all transient and we do not change the number of local redis instances once we've deployed it.  Also, I set redis to never write to disk - like I said all this cache data is transient and we don't restart redis processes in general.