Debugging proxied requests

david.richardson · 2014-04-04T01:43:00+00:00

That is awesome. Even with a restricted set of providers it will be a great addition to open-restyOn Friday, April 4, 2014 6:09:44 AM UTC-5, Brian Akins wrot...

Debugging proxied requests

david.richardson

Our application uses an upstream data provider (couchdb) returning responses with 500kb mime enclosures (PDFs) that are sometimes being corrupted/truncated. We need to monitor requests coming into nginx, out-going upstream, and the corresponding response, to debug the problem.

Errors are random, so most traffic will be just noise. Would like to watch traffic in real time. Any suggestions on logging within nginx to do this? Or is best bet an external packet sniffer in promiscuous mode?

david

lazy.dogtown

you could use tcpdump to capture the traffic and a tool like wireshark
to analyze the captures. (easy to implement, difficult to analyze)

or you could use some lua-code to evaluate the reponses and alert on
failures (difficult to implement, easy to analyze)

regards,

mex



2014-04-03 19:42 GMT+02:00  <david.r...@enquora.com>:
> Our application uses an upstream data provider (couchdb) returning responses
> with 500kb mime enclosures (PDFs) that are sometimes being
> corrupted/truncated. We need to monitor requests coming into nginx,
> out-going upstream, and the corresponding response, to debug the problem.
>
> Errors are random, so most traffic will be just noise. Would like to watch
> traffic in real time. Any suggestions on logging within nginx to do this? Or
> is best bet an external packet sniffer in promiscuous mode?
>
> david.

agentzh

Hello!

On Thu, Apr 3, 2014 at 10:42 AM,  david.richardson wrote:
> Our application uses an upstream data provider (couchdb) returning responses
> with 500kb mime enclosures (PDFs) that are sometimes being
> corrupted/truncated. We need to monitor requests coming into nginx,
> out-going upstream, and the corresponding response, to debug the problem.
>
> Errors are random, so most traffic will be just noise. Would like to watch
> traffic in real time. Any suggestions on logging within nginx to do this?

It looks like a perfect use case for systemtap (or dtrace):

    https://sourceware.org/systemtap/

We've been using systemtap to trace rare bad things with great success
in production across our global network in the last year or two. You
can take a look at some of our systemtap tools for reference:

    https://github.com/agentzh/nginx-systemtap-toolkit#readme

    https://github.com/agentzh/stapxx#samples

No need to patch nginx nor reload it. Do everything safely on the fly,
with minimal impact.

Best regards,
-agentzh