setting content_range in header_filter_by_lua

pongongxlua · 2014-03-12T23:27:01+00:00

I understand now! Thank you. I did not make the connection earlier between the 206 and the range_filter module rewriting the 200 to 206.regardsPongo On Thu,...

setting content_range in header_filter_by_lua

pongongxlua

Hi,

in the 206 response, i need to modify modify the value of the content_range. But it looks like this header is added /after/ header_filter_by_lua_file runs!

If i do ngx.header.content_range = "bytes 11-1000/21635765"; then in the response i see the value i set in lower case, as well as one from nginx:

TP/1.1 206 Partial Content

Server: nginx/1.4.5

Date: Wed, 12 Mar 2014 15:23:56 GMT

Content-Type: text/plain

Content-Length: 991

Last-Modified: Tue, 11 Mar 2014 14:53:27 GMT

Connection: keep-alive

ETag: "531f2367-14a22b6"

content-range: bytes 11-1000/21635765

Content-Range: bytes 10-1000/21635766

(i just put in random numbers in the code for this test, please ignore the correctness)

If i change to code to:

ngx.header["Content-Range"] = "bytes 11-1000/21635765";

then i get:

TP/1.1 206 Partial Content

Server: nginx/1.4.5

Date: Wed, 12 Mar 2014 15:24:36 GMT

Content-Type: text/plain

Content-Length: 991

Last-Modified: Tue, 11 Mar 2014 14:53:27 GMT

Connection: keep-alive

ETag: "531f2367-14a22b6"

Content-Range: bytes 11-1000/21635765

Content-Range: bytes 10-1000/21635766

What is the correct way to change the value of the Content-Range header? And out of curiosity, why do i get the header twice?

thanks

Pongo

pongongxlua

According to the wiki document, ngx.exit is disabled in header_filter_by_lua. BUT, according to the release notes from openresty-mainline-version-1581:

feature: allow use of ngx.exit() in the context of

         header_filter_by_lua* to perform a "filter finalization".
         but in this context ngx.exit() is an asynchronous operation
         and returns immediately.

calling ngx.exit(ngx.HTTP_OK) right after setting the content range "finalised" the output. I hope this is the correct way to solve this :)

thanks

Pongo

On Wednesday, March 12, 2014 4:27:01 PM UTC+1, Pongo wrote:

Hi,

in the 206 response, i need to modify modify the value of the content_range. But it looks like this header is added /after/ header_filter_by_lua_file runs!

If i do ngx.header.content_range = "bytes 11-1000/21635765"; then in the response i see the value i set in lower case, as well as one from nginx:

TP/1.1 206 Partial Content
Server: nginx/1.4.5
Date: Wed, 12 Mar 2014 15:23:56 GMT
Content-Type: text/plain
Content-Length: 991
Last-Modified: Tue, 11 Mar 2014 14:53:27 GMT
Connection: keep-alive
ETag: "531f2367-14a22b6"
content-range: bytes 11-1000/21635765
Content-Range: bytes 10-1000/21635766

(i just put in random numbers in the code for this test, please ignore the correctness)

If i change to code to:
ngx.header["Content-Range"] = "bytes 11-1000/21635765";

then i get:

TP/1.1 206 Partial Content
Server: nginx/1.4.5
Date: Wed, 12 Mar 2014 15:24:36 GMT
Content-Type: text/plain
Content-Length: 991
Last-Modified: Tue, 11 Mar 2014 14:53:27 GMT
Connection: keep-alive
ETag: "531f2367-14a22b6"
Content-Range: bytes 11-1000/21635765
Content-Range: bytes 10-1000/21635766

What is the correct way to change the value of the Content-Range header? And out of curiosity, why do i get the header twice?

thanks
Pongo

agentzh

Hello!

On Wed, Mar 12, 2014 at 8:27 AM, Pongo wrote:
>   in the 206 response, i need to modify modify the value of the
> content_range. But it looks like this header is added /after/
> header_filter_by_lua_file runs!
>

Yes, the standard ngx_http_range_filter_module always runs *after*
ngx_lua module's output filters (see the results here:
https://github.com/agentzh/nginx-systemtap-toolkit#ngx-header-filters
). And the standard range filter module does not check if there is
already a Content-Range response header. That's why you see an extra
one appended.

If you are generating the 206 response all by yourself, then you can
turn off the standard range filter module in your location by
configuring the following line in your location:

    max_ranges 0;

See also http://nginx.org/en/docs/http/ngx_http_core_module.html#max_ranges

BTW, using ngx.exit() in header_filter_by_lua for this very purpose is
not recommended. The "filter finalization" feature is for early
request aborts in the header filter.

Regards,
-agentzh

pongongxlua

Hi again,

Unfortunately i am not generating the content. I am getting a mp4 file over a slow network, and i know the file size before hand (can be computed from mp4 moov header as well). The users want to see the file in the browser while it is being copied. I wanted to intercept the 206 that nginx sends back so i can set the "correct" / final filesize in the content-range header, otherwise the video element assumes duration based on bytes currently available. I did not want to duplicate all the super-efficient effort that nginx does to serve the file myself. Is there a better way?

thanks

Pongo

On Wednesday, March 12, 2014 11:36:38 PM UTC+1, agentzh wrote:

Hello!

On Wed, Mar 12, 2014 at 8:27 AM, Pongo wrote:
> in the 206 response, i need to modify modify the value of the
> content_range. But it looks like this header is added /after/
> header_filter_by_lua_file runs!
>

Yes, the standard ngx_http_range_filter_module always runs *after*
ngx_lua module's output filters (see the results here:
https://github.com/agentzh/nginx-systemtap-toolkit#ngx-header-filters
). And the standard range filter module does not check if there is
already a Content-Range response header. That's why you see an extra
one appended.

If you are generating the 206 response all by yourself, then you can
turn off the standard range filter module in your location by
configuring the following line in your location:

max_ranges 0;

See also http://nginx.org/en/docs/http/ngx_http_core_module.html#max_ranges

BTW, using ngx.exit() in header_filter_by_lua for this very purpose is
not recommended. The "filter finalization" feature is for early
request aborts in the header filter.

Regards,
-agentzh

pongongxlua

Alternately, is it possible to ensure that the ngx_lua module's output filters are run at the tail end of the phase, similar to the rewrite filter?

thanks

Pongo

On Thursday, March 13, 2014 12:59:22 PM UTC+1, Pongo wrote:

Hi again,

Unfortunately i am not generating the content. I am getting a mp4 file over a slow network, and i know the file size before hand (can be computed from mp4 moov header as well). The users want to see the file in the browser while it is being copied. I wanted to intercept the 206 that nginx sends back so i can set the "correct" / final filesize in the content-range header, otherwise the video element assumes duration based on bytes currently available. I did not want to duplicate all the super-efficient effort that nginx does to serve the file myself. Is there a better way?

thanks
Pongo

On Wednesday, March 12, 2014 11:36:38 PM UTC+1, agentzh wrote:
Hello!

On Wed, Mar 12, 2014 at 8:27 AM, Pongo wrote:
> in the 206 response, i need to modify modify the value of the
> content_range. But it looks like this header is added /after/
> header_filter_by_lua_file runs!
>

Yes, the standard ngx_http_range_filter_module always runs *after*
ngx_lua module's output filters (see the results here:
https://github.com/agentzh/nginx-systemtap-toolkit#ngx-header-filters
). And the standard range filter module does not check if there is
already a Content-Range response header. That's why you see an extra
one appended.

If you are generating the 206 response all by yourself, then you can
turn off the standard range filter module in your location by
configuring the following line in your location:

max_ranges 0;

See also http://nginx.org/en/docs/http/ngx_http_core_module.html#max_ranges

BTW, using ngx.exit() in header_filter_by_lua for this very purpose is
not recommended. The "filter finalization" feature is for early
request aborts in the header filter.

Regards,
-agentzh

agentzh

Hello!

On Thu, Mar 13, 2014 at 5:28 AM, Pongo wrote:
> Alternately, is it possible to ensure that the ngx_lua module's output
> filters are run at the tail end of the phase, similar to the rewrite filter?
>

Not possible without hacking the source of the nginx core. AFAIK, the
nginx core ensures that all 3rd-party output filter modules always run
before its standard range filter module.

I don't see why you cannot implement an efficient and also flexible
content generator here in Lua yourself.

Regards,
-agentzh

pongongxlua

Apologies for a late reply, i have been unwell.

On Thursday, March 13, 2014 8:57:09 PM UTC+1, agentzh wrote:

Hello!

On Thu, Mar 13, 2014 at 5:28 AM, Pongo wrote:
> Alternately, is it possible to ensure that the ngx_lua module's output
> filters are run at the tail end of the phase, similar to the rewrite filter?
>

Not possible without hacking the source of the nginx core. AFAIK, the
nginx core ensures that all 3rd-party output filter modules always run
before its standard range filter module.

That is too bad. Looking at the headers generated by nginx when i use exit, it looks like all the important ones are in there; but i'll take your advice to heart and avoid this.

I don't see why you cannot implement an efficient and also flexible
content generator here in Lua yourself.

Pure laziness :) I learnt lua just so i could use your fantastic module (thanks, btw, it is superb work!).

The reason i do not want to write a content generator is because i have to serve several hundred megabytes for each client and nginx has excellent support for doing just this. Do not wish to duplicate all that.

best regards

Pongo

Regards,
-agentzh