Duplicate traffic to staging environment using cosocket based library

vvarfolomeev · 2015-03-31T23:11:17+00:00

> Do you have more than one light threads working on the same downstream > cosocket object? If yes, then it's very likely to be related to > https:/...

Duplicate traffic to staging environment using cosocket based library

vvarfolomeev

Hello Yichun Zhang,

Can you please provide a way to duplicate every requests to two different upstreams - production and staging environment. Clients requests goes to openresty and mirroring to two different upstreams(prod and stg). I would like to ignore staging responses and forward to the client response from production upstream only. Also it should be coscocket solution since it would be nice to avoid any delay for production environment if staging environment is unavailable.

So what i wrote is pretty common task about trafic duplication. You allready discussed similiar issue here: https://www.ruby-forum.com/topic/6179585 but i am unable to write lua script which will use coscoket library https://github.com/bakins/lua-resty-http-simple for this purporse.

As for know i have tried the following ways:

lua cosocket based library lua-resty-http-simple - probably best way to go but need help to set it up
lua ngx.location.capture

This kind of solution does work:

ngx.req.read_body()

local arguments = ngx.var.args

r1 = ngx.location.capture('/prod/', { args = arguments, share_all_vars = true })

ngx.print(r1.body)

ngx.eof()

r2 = ngx.location.capture('/dev/', { args = arguments, share_all_vars = true })

but i am not happy with this because of ngx.eof() and possible delay to production environment in case of staging is unavailable.

tcpcopy

Another option is go with tcpcopy project: https://github.com/session-replay-tools/tcpcopy

It also does work for some time in production environment but i have copied about half of the production trafic(i monitor incoming/outgoing network interface activity to figure this out)

openresty + zeroMQ http://glipho.com/albert/http-traffic-duplication-with-nginx Looking promising, but i did not find any use case in production systems
gor https://github.com/buger/gor/ looking like tcpcopy project but i did not test it yet.

It would be nice if you can make some example of lua script for this kind of task here : http://openresty.org/download/agentzh-nginx-tutorials-en.html. Sure i will be not the only person who benefit from it. From my site i can provide extra details or perform troubleshooting steps.

Regards Vladimir

agentzh

Hello!

On Tue, Mar 31, 2015 at 8:11 AM, Vladimir Varfolomeev wrote:
> Another option is go with tcpcopy project:
> https://github.com/session-replay-tools/tcpcopy
> It also does work for some time in production environment but i have copied
> about half of the production trafic(i monitor incoming/outgoing network
> interface activity to figure this out)
>

IMHO, it's better done on the lower level than nginx. Even if tcpcopy
does not do the job, it should be something on a similar level.

If you really want to do this in nginx, then the ngx.location.capture
+ ngx.eof approach is the simplest approach to my knowledge. Other
than that, you can use ngx.timer.at + some version of lua-resty-http
to do the duplicated off-line requests asynchronously.

Regards,
-agentzh

lordnynex

agentzh,

Couldn't you do this with capture_multi and never read the response? Also, what happens if you spawn a thread and never wait() for it?

On Tue, Mar 31, 2015 at 12:40 PM, Yichun Zhang (agentzh) <age...@gmail.com> wrote:

Hello!

On Tue, Mar 31, 2015 at 8:11 AM, Vladimir Varfolomeev wrote:
> Another option is go with tcpcopy project:
> https://github.com/session-replay-tools/tcpcopy
> It also does work for some time in production environment but i have copied
> about half of the production trafic(i monitor incoming/outgoing network
> interface activity to figure this out)
>

IMHO, it's better done on the lower level than nginx. Even if tcpcopy
does not do the job, it should be something on a similar level.

If you really want to do this in nginx, then the ngx.location.capture
+ ngx.eof approach is the simplest approach to my knowledge. Other
than that, you can use ngx.timer.at + some version of lua-resty-http
to do the duplicated off-line requests asynchronously.

Regards,
-agentzh

.

david.birdsong

I've found success by using the tcpcopy approach, but I've found it's much easier to configure and reason about a system that uses some sort of offline queueing which drives a set of workers to simulate user traffic. This way I can investigate the server's response from the client's perspective and, if needed, I can mutate the request on the queue client.

I use log_by_lua to write to Heka for logging, then use Heka + NSQ for creating work. The latter part means I can leverage the work put into the streaming log system that's already in place.

You could swap in Heka with a number of stream systems and NSQ with any number of queue systems. Even log_by_lua's not strictly necessary, parsing nginx logs could work too.

On Tue, Mar 31, 2015 at 12:40 PM Yichun Zhang (agentzh) <age...@gmail.com> wrote:

Hello!

On Tue, Mar 31, 2015 at 8:11 AM, Vladimir Varfolomeev wrote:
> Another option is go with tcpcopy project:
> https://github.com/session-replay-tools/tcpcopy
> It also does work for some time in production environment but i have copied
> about half of the production trafic(i monitor incoming/outgoing network
> interface activity to figure this out)
>

IMHO, it's better done on the lower level than nginx. Even if tcpcopy
does not do the job, it should be something on a similar level.

If you really want to do this in nginx, then the ngx.location.capture
+ ngx.eof approach is the simplest approach to my knowledge. Other
than that, you can use ngx.timer.at + some version of lua-resty-http
to do the duplicated off-line requests asynchronously.

Regards,
-agentzh.

agentzh

Hello!

On Tue, Mar 31, 2015 at 12:54 PM, Lord Nynex wrote:
> Also,
> what happens if you spawn a thread and never wait() for it?
>

The request handler creating the light threads will automatically wait
for those light threads to complete unless the light threads are
detached from the request handler by means of ngx.timer.at().

Regards,
-agentzh