How to remove all js or any other tag.

kroptgye · 2013-07-16T22:33:30+00:00

Hello! On Tue, Jul 16, 2013 at 5:34 AM, Антон Рубцов wrote: > Do you have plans for red...

How to remove all js or any other tag.

kroptgye

Hello

I test with

local text = ngx.arg[1]

text = ngx.re.gsub(text,"<script(.-)>(.-)</script>","")

ngx.arg[1] = text

It stay all scripts.

But when i use

local text = ngx.arg[1]

text = string.gsub(text,"<script(.-)>(.-)</script>","")

ngx.arg[1] = text

It replace almost all scripts(but not all).

What is right and fastest way to do?

Best regards Viktor.M

agentzh

Hello!

On Tue, Jul 16, 2013 at 7:33 AM, Viktor malezink wrote:
> I test with
>
> local text = ngx.arg[1]
> text = ngx.re.gsub(text,"<script(.-)>(.-)</script>","")
> ngx.arg[1] = text
>
> It stay all scripts.
> But when i use
>
> local text = ngx.arg[1]
> text = string.gsub(text,"<script(.-)>(.-)</script>","")
> ngx.arg[1] = text
>
> It replace almost all scripts(but not all).
> What is right and fastest way to do?

Lua's own string.gsub function does not use perl regex syntax; it uses
a completely different pattern notation. You should check out the Lua
documentation for the right regex syntax:

    http://www.lua.org/manual/5.1/manual.html#5.4.1

BTW,

1. I don't really understand the ".-" thing in your regexes. Is that a
typo and you really mean ".*?" here?

2. if you use static regexes with ngx.re.gsub (or other ngx.re.*
functions), then you'd better always specify the "jo" regex options
for best performance. For instance,

    text = ngx.re.gsub(text,"<script(.-)>(.-)</script>","", "jo")

3. Your regex substitutions may miss the script tags spanning multiple
data chunks. Always remember the data is feed into your body filter in
chunks and you cannot assume where the chunk boundary is. See also my
ngx_replace_filter module which automatically handle all the corner
cases with a true NFA-based regex engine:
https://github.com/agentzh/replace-filter-nginx-module

Regards,
-agentzh

quae

On 16 July 2013 18:38, Yichun Zhang (agentzh) <age...@gmail.com> wrote:

1. I don't really understand the ".-" thing in your regexes. Is that a
typo and you really mean ".*?" here?

"-" in lua patterns is the shortest match (i.e. non-greedy)

kroptgye

Hmm, so is there any method to get full request body, connect all the chunks together and read? Its not only replace or filter case, as example i collect info about some words on page and their frequency on page. As example i think to collect with gmatch and store in lua var. Then sync with timer. But if request body is placed as chunks, and I can not get access to all of chunks,it loses any sense to collect word frequency info coz it would be full? How to solve that? Thanks.

Best regards Viktor.M

On Wednesday, July 17, 2013 2:38:04 AM UTC+4, agentzh wrote:

Hello!

On Tue, Jul 16, 2013 at 7:33 AM, Viktor malezink wrote:
> I test with
>
> local text = ngx.arg[1]
> text = ngx.re.gsub(text,"<script(.-)>(.-)</script>","")
> ngx.arg[1] = text
>
> It stay all scripts.
> But when i use
>
> local text = ngx.arg[1]
> text = string.gsub(text,"<script(.-)>(.-)</script>","")
> ngx.arg[1] = text
>
> It replace almost all scripts(but not all).
> What is right and fastest way to do?

Lua's own string.gsub function does not use perl regex syntax; it uses
a completely different pattern notation. You should check out the Lua
documentation for the right regex syntax:

http://www.lua.org/manual/5.1/manual.html#5.4.1

BTW,

1. I don't really understand the ".-" thing in your regexes. Is that a
typo and you really mean ".*?" here?

2. if you use static regexes with ngx.re.gsub (or other ngx.re.*
functions), then you'd better always specify the "jo" regex options
for best performance. For instance,

text = ngx.re.gsub(text,"<script(.-)>(.-)</script>","", "jo")

3. Your regex substitutions may miss the script tags spanning multiple
data chunks. Always remember the data is feed into your body filter in
chunks and you cannot assume where the chunk boundary is. See also my
ngx_replace_filter module which automatically handle all the corner
cases with a true NFA-based regex engine:
https://github.com/agentzh/replace-filter-nginx-module

Regards,
-agentzh

agentzh

Hello!

On Wed, Jul 17, 2013 at 1:07 PM, Viktor malezink wrote:
> Hmm, so is there any method to get full request body, connect all the chunks
> together and read?

You can use subrequests (i.e., via the ngx.location.capture function)
to do full response buffering. See

    http://wiki.nginx.org/HttpLuaModule#ngx.location.capture

But I must warn you that full buffering has great impact on
performance, especially on memory usage, if large response bodies are
involved. And that's why Nginx's output filter works in a streaming
fashion, that is, processing by data chunks.

> Its not only replace or filter case, as example i collect
> info about some words on page and their frequency on page. As example i
> think to collect with gmatch and store in lua var. Then sync with timer. But
> if request body is placed as chunks, and I can not get access to all of
> chunks,it loses any sense to collect word frequency info coz it would be
> full? How to solve that?

You can use the ngx.ctx table to store per-request context information in Lua:

    http://wiki.nginx.org/HttpLuaModule#ngx.ctx

This should not be a reason that you switch to full response buffering ;)

Best regards,
-agentzh

kroptgye

So ngx.arg[1] = string.len(ngx.arg[1] will return size of current chunk? Or size of full html?

And if

Nginx output filters may be called multiple times for a single request because response body may be delivered in chunks. Thus, the Lua code specified by in this directive may also run multiple times in the lifetime of a single HTTP request.

Thats mean i can do something like that.

if ngx.arg[2] == false then
ngx.ctx.current_html = ngx.ctx.current_html..ngx.arg[1]
else
ngx.ctx.current_html = ngx.re.gsub(ngx.ctx.current_html,"<script\\b[^>]*>([\\s\\S]*?)<\\/script>","","jo") --gsub just as «full content» sensative example.
ngx.arg[1] = ngx.ctx.current_html
end

Or i miss something important?

Thanks.

On Thursday, July 18, 2013 12:23:32 AM UTC+4, agentzh wrote:

Hello!

On Wed, Jul 17, 2013 at 1:07 PM, Viktor malezink wrote:
> Hmm, so is there any method to get full request body, connect all the chunks
> together and read?

You can use subrequests (i.e., via the ngx.location.capture function)
to do full response buffering. See

http://wiki.nginx.org/HttpLuaModule#ngx.location.capture

But I must warn you that full buffering has great impact on
performance, especially on memory usage, if large response bodies are
involved. And that's why Nginx's output filter works in a streaming
fashion, that is, processing by data chunks.

> Its not only replace or filter case, as example i collect
> info about some words on page and their frequency on page. As example i
> think to collect with gmatch and store in lua var. Then sync with timer. But
> if request body is placed as chunks, and I can not get access to all of
> chunks,it loses any sense to collect word frequency info coz it would be
> full? How to solve that?

You can use the ngx.ctx table to store per-request context information in Lua:

http://wiki.nginx.org/HttpLuaModule#ngx.ctx

This should not be a reason that you switch to full response buffering ;)

Best regards,
-agentzh

agentzh

Hello!

On Wed, Jul 17, 2013 at 2:53 PM, Viktor malezink <kro...@gmail.com> wrote:
> So ngx.arg[1] = string.len(ngx.arg[1] will return size of current chunk? Or
> size of full html?

The size of the current chunk, of course. You've already quoted the
documentation for ngx.arg[1] below yourself.

> And if
>>
>> Nginx output filters may be called multiple times for a single request
>> because response body may be delivered in chunks. Thus, the Lua code
>> specified by in this directive may also run multiple times in the lifetime
>> of a single HTTP request.
>
> Thats mean i can do something like that.
>
>> if ngx.arg[2] == false then
>>
>> ngx.ctx.current_html = ngx.ctx.current_html..ngx.arg[1]
>>

Don't do this, string concatenation is very expensive, even more
expensive than full response body buffering itself.

BTW, even when ngx.arg[2] is true, ngx.arg[1] could contain data. They
are not exclusive at all.

Also, it's always easy to test everything yourself by means of
ngx_echo which can easily emit multiple small data chunks. For
example,

    echo hello;
    echo world;

will emit 3 data chunks with the last one does not contain data but
the "eof" flag only. You can use this to test your body filters. FWIW,
I'm also using this ngx_echo module to test my ngx_replace_filter
module. Check out its test suite below:

    https://github.com/agentzh/replace-filter-nginx-module/tree/master/t

Regards,
-agentzh