String replacement in body_filter_by_lua

guilherme.e · 2012-11-10T06:05:58+00:00

Hi, if someone is interested in i just made an openresty recipe for system integration framework chef.It is available at : https://github.com/jney/nginx...

String replacement in body_filter_by_lua

guilherme.e

Hello!

I'm trying to make string replacement in body_filter_by_lua, but the replaced content is being truncated. The mainly purpose of this is to append, dynamically (per website), some _javascript_s in response body.

body_filter_by_lua
'
replacestr = "<script>sdsdsdsdsd</script></head>"
ngx.arg[1] = ngx.re.sub(ngx.arg[1],"</head>", replacestr)
';

The original html version:

<html>
<head>
</head>
<body>
test
</body>
</html>

After body_filter_by_lua:

<html>
<head>
<script>sdsdsdsd

It looks like body_filter_by_lua is truncating in 16bytes.

Any idea?

I made this kind of replacement in response body using nginx_substitution_filter (3rd party), and it worked, but it's less dynamic than lua (I cannot use ngx.ctx to use variables from other phases, for example).

What is the recommended approach in this case?

Best regards,

Guilherme

agentzh

Hello!

On Fri, Nov 9, 2012 at 2:05 PM, Guilherme wrote:
> I'm trying to make string replacement in body_filter_by_lua, but the
> replaced content is being truncated. The mainly purpose of this is to
> append, dynamically (per website), some javascripts in response body.
>
>
> body_filter_by_lua
> '
>     replacestr = "<script>sdsdsdsdsd</script></head>"
>     ngx.arg[1] = ngx.re.sub(ngx.arg[1],"</head>", replacestr)
> ';
>
> The original html version:
>
> <html>
> <head>
> </head>
> <body>
> test
> </body>
> </html>
>
> After body_filter_by_lua:
>
> <html>
> <head>
> <script>sdsdsdsd
>
> It looks like body_filter_by_lua is truncating in 16bytes.
>

Does your original response contain a Content-Length response header?
If yes, then you should clear it first in header_filter_by_lua,
because you change the response body length.

You can try something like this:

    location = /test.html {
        header_filter_by_lua '
            ngx.header.content_length = nil
        ';

        body_filter_by_lua
        '
            replacestr = "<script>sdsdsdsdsd</script></head>"
            ngx.arg[1] = ngx.re.sub(ngx.arg[1],"</head>", replacestr)
        ';
    }

Best regards,
-agentzh

guilherme.e

Great! Now it worked!

Tks,

Guilherme

On Fri, Nov 9, 2012 at 9:14 PM, agentzh <age...@gmail.com> wrote:

Hello!

On Fri, Nov 9, 2012 at 2:05 PM, Guilherme wrote:
> I'm trying to make string replacement in body_filter_by_lua, but the
> replaced content is being truncated. The mainly purpose of this is to
> append, dynamically (per website), some _javascript_s in response body.
>
>
> body_filter_by_lua
> '
> replacestr = "<script>sdsdsdsdsd</script></head>"
> ngx.arg[1] = ngx.re.sub(ngx.arg[1],"</head>", replacestr)
> ';
>
> The original html version:
>
> <html>
> <head>
> </head>
> <body>
> test
> </body>
> </html>
>
> After body_filter_by_lua:
>
> <html>
> <head>
> <script>sdsdsdsd
>
> It looks like body_filter_by_lua is truncating in 16bytes.
>

Does your original response contain a Content-Length response header?
If yes, then you should clear it first in header_filter_by_lua,
because you change the response body length.

You can try something like this:

location = /test.html {
header_filter_by_lua '
ngx.header.content_length = nil
';

body_filter_by_lua
'
replacestr = "<script>sdsdsdsdsd</script></head>"
ngx.arg[1] = ngx.re.sub(ngx.arg[1],"</head>", replacestr)
';
}

Best regards,
-agentzh

.

kroptgye

I have very strange bug with this.

When i replace like that

local str = ngx.arg[1]

local script = "<script>alert('Hello');</script></body>"

ngx.arg[1] = ngx.re.gsub(str,'</body>',script)

So when i proxy site like nginx.com(here config)

location = / {

proxy_pass http://nginx.com/;

proxy_set_header Host "nginx.com";

proxy_ignore_headers Expires Cache-Control;

header_filter_by_lua 'ngx.header.content_length = nil';

body_filter_by_lua_file conf/filter.lua;

}

location ~ \.(png|jpg|jpeg|gif|js|css|ico|bmp)$ {

rewrite ^ http://nginx.com/$request_uri? permanent;

}

Than i see result in browser, but when i try to proxy something like google.co.uk or some other sites.

location = / {

proxy_pass http://www.google.co.uk/;

proxy_set_header Host "www.google.co.uk";

proxy_ignore_headers Expires Cache-Control;

header_filter_by_lua 'ngx.header.content_length = nil';

body_filter_by_lua_file conf/filter.lua;

}

location ~ \.(png|jpg|jpeg|gif|js|css|ico|bmp)$ {

rewrite ^ http://www.google.co.uk/$request_uri? permanent;

}

Ain't see changes in html from browsers(Chrome,SrWare,Firefox,IE,Opera) but when i load page from curl or wget i see that html has changed.

So i decided(maybe mistake) that webbrowser ignore(how?) html from openresty and load from origin source. But how to deal with that?

Thanks.

On Saturday, November 10, 2012 3:14:25 AM UTC+4, agentzh wrote:

Hello!

On Fri, Nov 9, 2012 at 2:05 PM, Guilherme wrote:
> I'm trying to make string replacement in body_filter_by_lua, but the
> replaced content is being truncated. The mainly purpose of this is to
> append, dynamically (per website), some _javascript_s in response body.
>
>
> body_filter_by_lua
> '
> replacestr = "<script>sdsdsdsdsd</script></head>"
> ngx.arg[1] = ngx.re.sub(ngx.arg[1],"</head>", replacestr)
> ';
>
> The original html version:
>
> <html>
> <head>
> </head>
> <body>
> test
> </body>
> </html>
>
> After body_filter_by_lua:
>
> <html>
> <head>
> <script>sdsdsdsd
>
> It looks like body_filter_by_lua is truncating in 16bytes.
>

Does your original response contain a Content-Length response header?
If yes, then you should clear it first in header_filter_by_lua,
because you change the response body length.

You can try something like this:

location = /test.html {
header_filter_by_lua '
ngx.header.content_length = nil
';

body_filter_by_lua
'
replacestr = "<script>sdsdsdsdsd</script></head>"
ngx.arg[1] = ngx.re.sub(ngx.arg[1],"</head>", replacestr)
';
}

Best regards,
-agentzh

agentzh

Hello!

On Mon, Jul 8, 2013 at 8:05 PM, Viktor malezink wrote:
> Ain't see changes in html from browsers(Chrome,SrWare,Firefox,IE,Opera) but
> when i load page from curl or wget i see that html has changed.
> So i decided(maybe mistake) that webbrowser ignore(how?) html from openresty
> and load from origin source. But how to deal with that?

One major difference between curl/wget and your web browsers is that
the former does not send the "Accept-Encoding: gzip" request header by
default. If your backend server compresses the HTTP response, nginx
will not automatically uncompress the data before feeding it into its
output filter chain. You can also test this case with curl by
specifying the --compressed command-line option, as in

    curl --compressed http://localhost:8080/

One work-around is to disable gzip compression on your backend servers
by adding the following line to your nginx.conf:

    proxy_set_header Accept-Encoding "";

And you can still enable gzip compression on your nginx side because
the standard ngx_gzip module runs *after* body_filter_by_lua.

Regards,
-agentzh

kroptgye

You are amazingly clever and experienced man, and yeah, you absolutely right.

It's unfortunate that my ignorance is so great.

Thanks for the enlightenment.

On Tuesday, July 9, 2013 7:21:46 AM UTC+4, agentzh wrote:

Hello!

On Mon, Jul 8, 2013 at 8:05 PM, Viktor malezink wrote:
> Ain't see changes in html from browsers(Chrome,SrWare,Firefox,IE,Opera) but
> when i load page from curl or wget i see that html has changed.
> So i decided(maybe mistake) that webbrowser ignore(how?) html from openresty
> and load from origin source. But how to deal with that?

One major difference between curl/wget and your web browsers is that
the former does not send the "Accept-Encoding: gzip" request header by
default. If your backend server compresses the HTTP response, nginx
will not automatically uncompress the data before feeding it into its
output filter chain. You can also test this case with curl by
specifying the --compressed command-line option, as in

curl --compressed http://localhost:8080/

One work-around is to disable gzip compression on your backend servers
by adding the following line to your nginx.conf:

proxy_set_header Accept-Encoding "";

And you can still enable gzip compression on your nginx side because
the standard ngx_gzip module runs *after* body_filter_by_lua.

Regards,
-agentzh