access request body when proxy_request_buffering is off!!!

hadi.abbasi.programmer · 2019-08-27T18:52:01+00:00

So i thought i would share this here to get some feedback help bring to light some improvements and bugs and to get the ball rolling on how we can make this...

access request body when proxy_request_buffering is off!!!

hadi.abbasi.programmer

Hello...
I have developed a proxy server using openresty and I am looking for a way to send my clients request body (Upload data) directly to their target upstreams!
so it can be solved by setting:

proxy_request_buffering off;

in this way, nginx will send request body immediately to the upstream!

some of proxied hosts can set anti virus scan = on! so I need to buffer and save their request body, it can be solved using:

lua_need_request_body on;
client_max_body_size 300m;
client_body_buffer_size 300m;

I need to set client_max_body_size = 300mb because some of my hosts need to upload files up to 300MB!
when proxy_request_buffering is off , our client_body_buffer_size must be equal to the client_max_body_size to access to the request body data using:

ngx.req.read_body()
local data = ngx.var.request_body

in this case, I can get request body because both client_body_buffer_size and client_max_body_size are equal!
so without any additional code, it will send clients request body to the upstream for none anti virus hosts, otherwise when the host antivirus is enabled, we can do:

ngx.req.read_body()
local data = ngx.var.request_body

then we can write request body on a temp file and scan it using antivirus and then we can set it as request body using:

ngx.req.set_body_data(data)

but server ram usage is high! because when client_body_buffer_size is smaller than client_max_body_size and client body size exceeds client_body_buffer_size , nginx will create a temp file and will buffer it,
so my question is:

CAN I GET THE REQUEST BODY BY SETTING:

proxy_request_buffering off;
lua_need_request_body on;
client_max_body_size 300m;
client_body_buffer_size 64k;
.
.
.
ngx.req.read_body()
local data = ngx.var.request_body
print(data) -- currently it prints nil value when client_max_body_size is not equal to client_body_buffer_size!

even access to tmp buffered files can solve my problem!
thanks a lot...
Best,
Hadi

piotrprz

Please refer to the documentation at https://github.com/openresty/lua-nginx-module#ngxreqget_body_file - it explains how accessing request body works depending on its relative size to your caches.

W dniu wtorek, 27 sierpnia 2019 12:52:01 UTC+2 użytkownik Hadi Abbasi napisał:

Hello...
I have developed a proxy server using openresty and I am looking for a way to send my clients request body (Upload data) directly to their target upstreams!
so it can be solved by setting:

proxy_request_buffering off;

in this way, nginx will send request body immediately to the upstream!
some of proxied hosts can set anti virus scan = on! so I need to buffer and save their request body, it can be solved using:

lua_need_request_body on; client_max_body_size 300m; client_body_buffer_size 300m;

I need to set client_max_body_size = 300mb because some of my hosts need to upload files up to 300MB!
when proxy_request_buffering is off , our client_body_buffer_size must be equal to the client_max_body_size to access to the request body data using:

ngx.req.read_body() local data = ngx.var.request_body

in this case, I can get request body because both client_body_buffer_size and client_max_body_size are equal!
so without any additional code, it will send clients request body to the upstream for none anti virus hosts, otherwise when the host antivirus is enabled, we can do:

ngx.req.read_body() local data = ngx.var.request_body

then we can write request body on a temp file and scan it using antivirus and then we can set it as request body using:

ngx.req.set_body_data(data)

but server ram usage is high! because when client_body_buffer_size is smaller than client_max_body_size and client body size exceeds client_body_buffer_size , nginx will create a temp file and will buffer it,
so my question is:

CAN I GET THE REQUEST BODY BY SETTING:

proxy_request_buffering off; lua_need_request_body on; client_max_body_size 300m; client_body_buffer_size 64k; . . . ngx.req.read_body() local data = ngx.var.request_body print(data) -- currently it prints nil value when client_max_body_size is not equal to client_body_buffer_size!

even access to tmp buffered files can solve my problem!
thanks a lot...
Best,
Hadi

hadi.abbasi.programmer

Thanks a lot Piotr...
yeah it's all I Need!
when the request body buffer size is less than body max size , by setting request body buffer off, it will create tmp file in nginx request body tmp folder and we can access tmp files until the end of request,
so I can pass its url to my local anti virus and it can check the file!
some times there are 3 meta data lines which contains boundary and ... information in this tmp files but I think it's OK and anti virus can check these tmp files...
Thank You so much and Good Luck...
all the Best,
Hadi

On Thursday, August 29, 2019 at 12:42:11 AM UTC+4:30, Piotr Przybylski wrote:

Please refer to the documentation at https://github.com/openresty/lua-nginx-module#ngxreqget_body_file - it explains how accessing request body works depending on its relative size to your caches.