Re: Question about how to filter response body in ngx_lua module

agentzh

On Fri, May 4, 2012 at 5:42 AM, Roman Shishkin <r.sh...@gmail.com> wrote:
>
>
> 2012/5/3 agentzh <age...@gmail.com>
>>
>> On Thu, May 3, 2012 at 5:36 AM, Roman Shishkin <r.sh...@gmail.com>
>> wrote:
>> >> Also consider the following:
>> >>
>>
>> Which version of ngx_lua and nginx are you using?
>> --
>
>
> I'm using pure nginx-1.0.15 with only lua module  and nginx1.2 with
> subs_filter.
> They are working the same way.
>
> Here is test results:
>
> 1. location /lua {
>        charset GBR;
>        content_by_lua '
>       ngx.header.content_type = "text/html; charset=UTF-8"
>            ngx.say("hi");
>        ';
>    }
>
> Connection:
> keep-alive
> Content-Type:
> text/html; charset=UTF-8
> Date:
> Thu, 03 May 2012 21:08:36 GMT
> Server:
> nginx/1.0.15
> Transfer-Encoding:
> chunked
>
>
>
> 2. location /lua {
>        charset GBR;
>        content_by_lua '
>        ngx.header.content_type = "text/html"
>            ngx.say("hi");
>        ';
>    }
>
> Content-Type:
> text/html; charset=GBR
> Date:
> Thu, 03 May 2012 21:05:45 GMT
> Server:
> nginx/1.0.15
>
> Transfer-Encoding:
> chunked
>
> Previosly i made a typo - "text/html;" so test was falied, but now it's OK
>
> And test #3 !!!
>
> 3. location /lua {
>        charset GBR;
>        content_by_lua '
>            ngx.say("hi");
>        ';
>    }
>
> Connection:
> keep-alive
> Content-Type:
> text/plain; charset=GBR
> Date:
> Thu, 03 May 2012 21:12:00 GMT
> Server:
> nginx/1.0.15
> Transfer-Encoding:
> chunked
>
>
> Test 3. failed if we capture proxy_pass location, so working Lua
> configuration must be the folowing:

You're going to override the charset value in the upstream response
header Content-Type? No, the "charset" directive will not do that for
you because it does not support overriding existing charset value.
Consider the following example:

    server {
        listen 8879;
        default_type 'text/plain';

        location /cookie {
            charset UTF-8;
            add_header Set-Cookie foo=3;
            add_header Set-Cookie bar=4;
            echo cookie;
        }
    }

    server {
        listen 1984;

        location /cookie {
            charset GBK;
            proxy_set_header  Accept-Encoding  "";
            proxy_pass http://127.0.0.1:8879$request_uri;
        }
    }

Then accessing the /cookie location in the second virtual server gives:

    $ curl -i localhost:1984/cookie
    HTTP/1.1 200 OK
    Server: nginx/1.2.0
    Date: Fri, 04 May 2012 03:54:14 GMT
    Content-Type: text/plain; charset=UTF-8
    Content-Length: 7
    Connection: keep-alive
    Set-Cookie: foo=3
    Set-Cookie: bar=4

    cookie

We can see that, even for standard ngx_proxy module + charset
directive, the upstream charset value cannot be overridden. So this
will also apply to ngx_lua.

> -----------------------------
> -- Catch Nginx proxy pass location
> local res = ngx.location.capture("/proxy-pass");
>
> -- Copy subrequest status
> ngx.status = res.status;
>
> --  Put headers from subrequest
> if res.header then
>     for key, value in pairs(res.header) do
>         ngx.header[key] = value;
>     end
> end
>
> -- Manualy fix charset issue
> if ngx.header["Content-Type"] then
>     local newheader, n = ngx.re.sub(res.header["Content-Type"],
> "charset=ISO-8859-1", " utf-8", "io");

Apparently you need to override the old charset value in the
Content-Type response header here.

Best regards,
-agentzh

>     if newheader then
>         ngx.header["Content-Type"] = newheader;
>     end
> end
>
> ngx.print(res.body);
>

r.shishkin

Hello!

I am using at the server level override charset function like this:

charset utf-8;

override_charset on;

source_charset iso-8859-1;

charset_map iso-8859-1 utf-8 { }

But you absolutely right about native Nginx behavior.

Here is a quote from doc http://wiki.nginx.org/HttpCharsetModule

It is necessary to note that if the response was obtained in the subquery then, independent of directive override_charset, is always carried out reencoding from encoding of the response into encoding of basic demand.

Thank you for clear explanation, Zhang!

Please, don't forget about body_filter function.

My next step is about Lua perfomance on Nginx.
Seeya in nginx list!

2012/5/4 agentzh <age...@gmail.com>

You're going to override the charset value in the upstream response
header Content-Type? No, the "charset" directive will not do that for
you because it does not support overriding existing charset value.
Consider the following example:

server {
listen 8879;
default_type 'text/plain';

location /cookie {
charset UTF-8;
add_header Set-Cookie foo=3;
add_header Set-Cookie bar=4;
echo cookie;
}
}

server {
listen 1984;

location /cookie {
charset GBK;
proxy_set_header Accept-Encoding "";
proxy_pass http://127.0.0.1:8879$request_uri;
}
}

Then accessing the /cookie location in the second virtual server gives:

$ curl -i localhost:1984/cookie
HTTP/1.1 200 OK
Server: nginx/1.2.0
Date: Fri, 04 May 2012 03:54:14 GMT
Content-Type: text/plain; charset=UTF-8
Content-Length: 7
Connection: keep-alive
Set-Cookie: foo=3
Set-Cookie: bar=4

cookie

We can see that, even for standard ngx_proxy module + charset
directive, the upstream charset value cannot be overridden. So this
will also apply to ngx_lua.

--
Best regards

Roman Shishkin