Getting request and response size in lua

kirilk · 2013-01-03T21:17:56+00:00

I have used one POST call inside ngnix sub request to out side resource. May be this can help you to make your own. I m using a request to send a mail via ma...

Getting request and response size in lua

kirilk

Hi,

I am trying to implement a basic nginx requests statistics module in lua. I want to get request and response size without getting the whole response in memory? As far as I can see after nginx 1.2.5 I can use bytes_sent common variable from http_log_module for response size but I can't see how I can get request size without getting the whole request. Any ideas what could be done?

Cheers,

Kiril

agentzh

Hello!

On Thu, Jan 3, 2013 at 5:17 AM, Kiril Kalchev <ki...@cloudxcel.com> wrote:
> I am trying to implement a basic nginx requests statistics module in lua. I
> want to get request and response size without getting the whole response in
> memory? As far as I can see after nginx 1.2.5 I can use bytes_sent common
> variable from http_log_module for response size but I can't see how I can
> get request size without getting the whole  request. Any ideas what could be
> done?
>

Case #1

If the request body has not yet been read *and* chunked encoding is
not used for the request body data, then you can just read the
Content-Length request header (that is,
r->headers_in.content_length_n). The request header size can be got by
reading r->request_length directly, you need to add that up, that is,

    r->headers_in.content_length_n + r->request_length

If chunked encoding is used for the request body, then the request
body length cannot be known before reading it.

Case #2

If the request body has already been read, then you can directly read
r->request_length (where r is the current ngx_http_request_t object)
in C.

It seems that the $request_length (fake) variable in
ngx_http_log_module is not exported as a real nginx variable at least
in nginx 1.2.6. The $request_length variable's get handler just reads
r->request_length.

Best regards,
-agentzh

kirilk

Hello,

I made a patch for openresty-1.2.4 to export all log specific variables. I sent the same patch for nginx-1.3.10 to official nginx developers.
I am attaching the one for openresty-1.2.4 here if someone need it.

Attachment: http_variables.patch.openresty
Description: Binary data

Regards,
Kiril

On Jan 4, 2013, at 2:49 AM, agentzh wrote:

> Hello!
> 
> On Thu, Jan 3, 2013 at 5:17 AM, Kiril Kalchev <ki...@cloudxcel.com> wrote:
>> I am trying to implement a basic nginx requests statistics module in lua. I
>> want to get request and response size without getting the whole response in
>> memory? As far as I can see after nginx 1.2.5 I can use bytes_sent common
>> variable from http_log_module for response size but I can't see how I can
>> get request size without getting the whole  request. Any ideas what could be
>> done?
>> 
> 
> Case #1
> 
> If the request body has not yet been read *and* chunked encoding is
> not used for the request body data, then you can just read the
> Content-Length request header (that is,
> r->headers_in.content_length_n). The request header size can be got by
> reading r->request_length directly, you need to add that up, that is,
> 
>    r->headers_in.content_length_n + r->request_length
> 
> If chunked encoding is used for the request body, then the request
> body length cannot be known before reading it.
> 
> Case #2
> 
> If the request body has already been read, then you can directly read
> r->request_length (where r is the current ngx_http_request_t object)
> in C.
> 
> It seems that the $request_length (fake) variable in
> ngx_http_log_module is not exported as a real nginx variable at least
> in nginx 1.2.6. The $request_length variable's get handler just reads
> r->request_length.
> 
> Best regards,
> -agentzh.
> 
>

agentzh

Hello!

On Fri, Jan 4, 2013 at 6:16 AM, Kiril Kalchev wrote:
>
> I made a patch for openresty-1.2.4 to export all log specific variables. I sent the same patch for nginx-1.3.10 to official nginx developers.
> I am attaching the one for openresty-1.2.4 here if someone need it.
>

Thank you for your contribution :)

But a few comments:

1. Regarding $request_length, I'd rather implement an ngx.req.len()
method to retrieve r->request_length, which could be much faster than
retrieving an Nginx variable from within Lua :)

2. The $bytes_sent variable is already in nginx 1.2.6 and
ngx_openresty 1.2.6.1 will be out soon.

3. The $content_length variable is not necessary, because we can
already get it in Lua via

    ngx.var.http_content_length

or

    ngx.req.get_headers()["content-length"]

See https://github.com/chaoslawful/lua-nginx-module#ngxreqget_headers
for more details.

4. The $request_time variable is not necessary because we can already
read the value in a much faster way in Lua via ngx.req.start_time():

    https://github.com/chaoslawful/lua-nginx-module#ngxreqstart_time

5. We can also call ngx.localtime() and ngx.now() and ngx.today() for
at leat $time_local, I think :)

Best regards,
-agentzh

kirilk

You are right, that patch is for nginx developers, I also sent it to them. I attached it here because I saw some other people who have the same problem.
Thank you for the quick and informative response.

Thanks,
Kiril

On Jan 4, 2013, at 9:23 PM, agentzh wrote:

> Hello!
> 
> On Fri, Jan 4, 2013 at 6:16 AM, Kiril Kalchev wrote:
>> 
>> I made a patch for openresty-1.2.4 to export all log specific variables. I sent the same patch for nginx-1.3.10 to official nginx developers.
>> I am attaching the one for openresty-1.2.4 here if someone need it.
>> 
> 
> Thank you for your contribution :)
> 
> But a few comments:
> 
> 1. Regarding $request_length, I'd rather implement an ngx.req.len()
> method to retrieve r->request_length, which could be much faster than
> retrieving an Nginx variable from within Lua :)
> 
> 2. The $bytes_sent variable is already in nginx 1.2.6 and
> ngx_openresty 1.2.6.1 will be out soon.
> 
> 3. The $content_length variable is not necessary, because we can
> already get it in Lua via
> 
>    ngx.var.http_content_length
> 
> or
> 
>    ngx.req.get_headers()["content-length"]
> 
> See https://github.com/chaoslawful/lua-nginx-module#ngxreqget_headers
> for more details.
> 
> 4. The $request_time variable is not necessary because we can already
> read the value in a much faster way in Lua via ngx.req.start_time():
> 
>    https://github.com/chaoslawful/lua-nginx-module#ngxreqstart_time
> 
> 5. We can also call ngx.localtime() and ngx.now() and ngx.today() for
> at leat $time_local, I think :)
> 
> Best regards,
> -agentzh.
> 
>