inspecting file upload header

yahel.bahat · 2015-05-13T22:16:20+00:00

Looks nice. Have you thought about adding cluster support as well?Thanks. Yes, for now I left it as a TODO: https://github.com/pintsized/lua-resty-redis-conn...

inspecting file upload header

yahel.bahat

Hi

I'm try to make a proxy using Nginx Lua, that inspect uploaded files.
the goal is to have a list of file types (such as image/png, image/gif) that is permitted to be upload.

Trying to use: https://github.com/openresty/lua-resty-upload, i get the content type, if the file is permitted to pass to the original server, i can't seem to find a way to rewrite the file data to the socket.
is this can be done that way ?

maybe other way ?

thanks

agentzh

Hello!

On Wed, May 13, 2015 at 10:16 PM, yahel wrote:
> Trying to use: https://github.com/openresty/lua-resty-upload, i get the
> content type, if the file is permitted to pass to the original server, i
> can't seem to find a way to rewrite the file data to the socket.
> is this can be done that way ?
>

Two approaches:

1. Read the request body using nginx's builtin request body reader (by
calling ngx.req.read_body, for example), and then parse the read data
via ngx.req.get_body_data and/or ngx.req.get_body_file yourself.

2. Read the request body via lua-resty-upload using its own request
body reader, and then re-assemble the request body for other nginx
modules (including ngx_proxy) via the ngx.req.init_body,
ngx.req.append_body, and ngx.req.finish_body.

We use the first approach at CloudFlare for WAF but the second
approach should work as well and has its own benefit (like streaming
body processing).

Just my 2 cents.

Regards,
-agentzh

P.S. Sorry for my late reply. I've been on vacation lately.

yahel.bahat

Thanks.

that's exactly what i was looking for.

On Friday, May 29, 2015 at 12:04:59 PM UTC+3, agentzh wrote:

Hello!

On Wed, May 13, 2015 at 10:16 PM, yahel wrote:
> Trying to use: https://github.com/openresty/lua-resty-upload, i get the
> content type, if the file is permitted to pass to the original server, i
> can't seem to find a way to rewrite the file data to the socket.
> is this can be done that way ?
>

Two approaches:

1. Read the request body using nginx's builtin request body reader (by
calling ngx.req.read_body, for example), and then parse the read data
via ngx.req.get_body_data and/or ngx.req.get_body_file yourself.

2. Read the request body via lua-resty-upload using its own request
body reader, and then re-assemble the request body for other nginx
modules (including ngx_proxy) via the ngx.req.init_body,
ngx.req.append_body, and ngx.req.finish_body.

We use the first approach at CloudFlare for WAF but the second
approach should work as well and has its own benefit (like streaming
body processing).

Just my 2 cents.

Regards,
-agentzh

P.S. Sorry for my late reply. I've been on vacation lately.

yahel.bahat

Hi agentzh

I was wondering, how do you handle large files uploads, when the requests body is written to temp file ?

do you inspect the file in a non-blocking manner ? if so, how it could be achieved ?

On Monday, June 1, 2015 at 8:53:03 AM UTC+3, yahel wrote:

Thanks.

that's exactly what i was looking for.

On Friday, May 29, 2015 at 12:04:59 PM UTC+3, agentzh wrote:
Hello!

On Wed, May 13, 2015 at 10:16 PM, yahel wrote:
> Trying to use: https://github.com/openresty/lua-resty-upload, i get the
> content type, if the file is permitted to pass to the original server, i
> can't seem to find a way to rewrite the file data to the socket.
> is this can be done that way ?
>

Two approaches:

1. Read the request body using nginx's builtin request body reader (by
calling ngx.req.read_body, for example), and then parse the read data
via ngx.req.get_body_data and/or ngx.req.get_body_file yourself.

2. Read the request body via lua-resty-upload using its own request
body reader, and then re-assemble the request body for other nginx
modules (including ngx_proxy) via the ngx.req.init_body,
ngx.req.append_body, and ngx.req.finish_body.

We use the first approach at CloudFlare for WAF but the second
approach should work as well and has its own benefit (like streaming
body processing).

Just my 2 cents.

Regards,
-agentzh

P.S. Sorry for my late reply. I've been on vacation lately.

agentzh

Hello!

On Mon, Jun 8, 2015 at 9:05 PM, yahel wrote:
> I was wondering, how do you handle large files uploads, when the requests
> body is written to temp file ?

While reading the data from the wire. Like read chunk A, write it
right away to disk; then read chunk B, write it to disk again, and so
on.

> do you inspect the file in a non-blocking manner ?

I don't understand this question. What do you mean by "inspecting the
file". Should you just flush data into the file?

BTW, there is no such thing as "nonblocking file I/O". File I/O is
always blocking. So be careful.

> if so, how it could be
> achieved ?
>

See above.

Regards,
-agentzh