ngx.re.match for multiline string returns null

rohit.c.joshi · 2015-11-30T23:12:32+00:00

Hello! On Mon, Nov 30, 2015 at 6:10 AM, RJoshi wrote: > As part of my proxy integration with Oracle coherence cache server, I am > using ngx.locati...

ngx.re.match for multiline string returns null

rohit.c.joshi

Hello,

I am trying to use ngx.re.match for multi-line string returns null. It works fine with single line string. I tried using options "m" but doesn't help

### Single Line ###

local s="\n\n<key>abcd</key>fdfdsds"

local m=ngx.re.match(s,"<key>(.*)</key>", "jo")

print(m[1])

-- abcd

###Multi Line###

but if I change string multiline, it returns null

local s=[[ \n\n<key>abcd

</key>fdfdsds]]

local m=ngx.re.match(s,"<key>(.*)</key>", "jom")

print(m[1])

-- m is null

lordnynex

Hello,

You should probably not use a greedy regex for this. It is inefficient and can have undesired results. I also recommend using a regex coach on the data first. I use https://chrome.google.com/webstore/detail/regexr/igncokdengclanbghanhfichlnhlpihl?hl=en

Make sure you have a build of PCRE that has jit enabled.

I don't think there is any performance benefit to jit compiling this greedy regex.

If this regex runs in a buffered context you will need to assemble the full buffer before matching.

Here is a more efficient working example

$ resty -e '
local s=[[ \n\n<key>abcd
 </key>fdfdsds]]
local m=ngx.re.match(s,[[<key>\s*(.+?)\s*</key>]], "m")
print(m[1])
'
abcd

On Mon, Nov 30, 2015 at 7:12 AM, RJoshi <rohit....@gmail.com> wrote:

Hello,
I am trying to use ngx.re.match for multi-line string returns null. It works fine with single line string. I tried using options "m" but doesn't help

### Single Line ###
local s="\n\n<key>abcd</key>fdfdsds"
local m=ngx.re.match(s,"<key>(.*)</key>", "jo")

print(m[1])
-- abcd

###Multi Line###
but if I change string multiline, it returns null

local s=[[ \n\n<key>abcd
</key>fdfdsds]]

local m=ngx.re.match(s,"<key>(.*)</key>", "jom")
print(m[1])
-- m is null

.

rohit.c.joshi

Thanks for your help. Yes, I have PCRE jit enabled as I am using ./configure --with-pcre-jit

I have a soap request body from which I am trying to extract value for three different elements.

e.g. Each element could be multiline string

<msg-id>1234</msgid>

<body>xxxxx</body>

Currently, I am using string:find function three times but wanted to make it efficient.

local token_start, token_end, key = request_body:find('<key>(.*)</key>')

local token_start, token_end, msg_id = request_body:find('<msg-id>(.*)</msg-id>', token_end)

local token_start, token_end, body = request_body:find('<body>(.*)</body>', token_end)

Below works fine for single line but doesn't work with multi-line.

local m=ngx.re.match(s,"<key>(.*)</key>.*<msg-id>(.*)</msg-id>.*<body>(.*)</body>", "jo")

print (m[1] .. m[2] .. m[3])

Your suggestion work fine for single match. How can I match all three elements?

On Monday, 30 November 2015 16:23:12 UTC-5, Lord Nynex wrote:

Hello,

You should probably not use a greedy regex for this. It is inefficient and can have undesired results. I also recommend using a regex coach on the data first. I use https://chrome.google.com/webstore/detail/regexr/igncokdengclanbghanhfichlnhlpihl?hl=en

Make sure you have a build of PCRE that has jit enabled.
I don't think there is any performance benefit to jit compiling this greedy regex.
If this regex runs in a buffered context you will need to assemble the full buffer before matching.

Here is a more efficient working example
$ resty -e '
local s=[[ \n\n<key>abcd
 </key>fdfdsds]]
local m=ngx.re.match(s,[[<key>\s*(.+?)\s*</key>]], "m")
print(m[1])
'
abcd
On Mon, Nov 30, 2015 at 7:12 AM, RJoshi <rohit....@gmail.com> wrote:
Hello,
I am trying to use ngx.re.match for multi-line string returns null. It works fine with single line string. I tried using options "m" but doesn't help

### Single Line ###
local s="\n\n<key>abcd</key>fdfdsds"
local m=ngx.re.match(s,"<key>(.*)</key>", "jo")

print(m[1])
-- abcd

###Multi Line###
but if I change string multiline, it returns null

local s=[[ \n\n<key>abcd
</key>fdfdsds]]

local m=ngx.re.match(s,"<key>(.*)</key>", "jom")
print(m[1])
-- m is null

.

moonbingbing

"." means any character except newline. so ".*" can not match new lines.

look this:

MingdeMac-mini:bin ming$ ./resty -e '

> local s = [[\n\n<key>abdc

> </key>asdasdasd]]

> local m = ngx.re.match(s, [[<key>([\S\s]*)</key>]],"jom")

> print (m[1])

> '

abdc

you can use http://regexpal.com/ to test your regular _expression_ before write code.

2015-12-01 6:19 GMT+08:00 RJoshi <rohit....@gmail.com>:

Thanks for your help. Yes, I have PCRE jit enabled as I am using ./configure --with-pcre-jit
I have a soap request body from which I am trying to extract value for three different elements.

e.g. Each element could be multiline string
<key>abc</key>
<msg-id>1234</msgid>
<body>xxxxx</body>

Currently, I am using string:find function three times but wanted to make it efficient.

local token_start, token_end, key = request_body:find('<key>(.*)</key>')
local token_start, token_end, msg_id = request_body:find('<msg-id>(.*)</msg-id>', token_end)
local token_start, token_end, body = request_body:find('<body>(.*)</body>', token_end)

Below works fine for single line but doesn't work with multi-line.

local m=ngx.re.match(s,"<key>(.*)</key>.*<msg-id>(.*)</msg-id>.*<body>(.*)</body>", "jo")
print (m[1] .. m[2] .. m[3])
Your suggestion work fine for single match. How can I match all three elements?

On Monday, 30 November 2015 16:23:12 UTC-5, Lord Nynex wrote:
Hello,

You should probably not use a greedy regex for this. It is inefficient and can have undesired results. I also recommend using a regex coach on the data first. I use https://chrome.google.com/webstore/detail/regexr/igncokdengclanbghanhfichlnhlpihl?hl=en

Make sure you have a build of PCRE that has jit enabled.
I don't think there is any performance benefit to jit compiling this greedy regex.
If this regex runs in a buffered context you will need to assemble the full buffer before matching.

Here is a more efficient working example

$ resty -e ' local s=[[ \n\n<key>abcd </key>fdfdsds]] local m=ngx.re.match(s,[[<key>\s*(.+?)\s*</key>]], "m") print(m[1]) ' abcd

On Mon, Nov 30, 2015 at 7:12 AM, RJoshi <rohit....@gmail.com> wrote:
Hello,
I am trying to use ngx.re.match for multi-line string returns null. It works fine with single line string. I tried using options "m" but doesn't help

### Single Line ###
local s="\n\n<key>abcd</key>fdfdsds"
local m=ngx.re.match(s,"<key>(.*)</key>", "jo")

print(m[1])
-- abcd

###Multi Line###
but if I change string multiline, it returns null

local s=[[ \n\n<key>abcd
</key>fdfdsds]]

local m=ngx.re.match(s,"<key>(.*)</key>", "jom")
print(m[1])
-- m is null

.
rohit.c.joshi

Perfect. Thanks for your help. I am able to exctract all three tokens with this.

a=ngx.re.match(s,[[<key>([\S\s]*)</key>[\S\s]*<msg-id>([\S\s]*)</msg-id>[\S\s]*<body>([\S\s]*)</body>]], "jom")
print(a[1] .. "|" .. a[2] .. "|" .. a[3])

On Monday, 30 November 2015 21:19:00 UTC-5, WenMing wrote:
"." means any character except newline. so ".*" can not match new lines.

look this:

MingdeMac-mini:bin ming$ ./resty -e '
> local s = [[\n\n<key>abdc
> </key>asdasdasd]]
> local m = ngx.re.match(s, [[<key>([\S\s]*)</key>]],"jom")
> print (m[1])
> '
abdc

you can use http://regexpal.com/ to test your regular _expression_ before write code.

2015-12-01 6:19 GMT+08:00 RJoshi <rohit....@gmail.com>:
Thanks for your help. Yes, I have PCRE jit enabled as I am using ./configure --with-pcre-jit

I have a soap request body from which I am trying to extract value for three different elements.

e.g. Each element could be multiline string
<key>abc</key>
<msg-id>1234</msgid>
<body>xxxxx</body>

Currently, I am using string:find function three times but wanted to make it efficient.

local token_start, token_end, key = request_body:find('<key>(.*)</key>')
local token_start, token_end, msg_id = request_body:find('<msg-id>(.*)</msg-id>', token_end)
local token_start, token_end, body = request_body:find('<body>(.*)</body>', token_end)

Below works fine for single line but doesn't work with multi-line.

local m=ngx.re.match(s,"<key>(.*)</key>.*<msg-id>(.*)</msg-id>.*<body>(.*)</body>", "jo")
print (m[1] .. m[2] .. m[3])

Your suggestion work fine for single match. How can I match all three elements?

On Monday, 30 November 2015 16:23:12 UTC-5, Lord Nynex wrote:
Hello,

You should probably not use a greedy regex for this. It is inefficient and can have undesired results. I also recommend using a regex coach on the data first. I use https://chrome.google.com/webstore/detail/regexr/igncokdengclanbghanhfichlnhlpihl?hl=en

Make sure you have a build of PCRE that has jit enabled.
I don't think there is any performance benefit to jit compiling this greedy regex.
If this regex runs in a buffered context you will need to assemble the full buffer before matching.

Here is a more efficient working example

$ resty -e ' local s=[[ \n\n<key>abcd </key>fdfdsds]] local m=ngx.re.match(s,[[<key>\s*(.+?)\s*</key>]], "m") print(m[1]) ' abcd

On Mon, Nov 30, 2015 at 7:12 AM, RJoshi <rohit....@gmail.com> wrote:
Hello,
I am trying to use ngx.re.match for multi-line string returns null. It works fine with single line string. I tried using options "m" but doesn't help

### Single Line ###
local s="\n\n<key>abcd</key>fdfdsds"
local m=ngx.re.match(s,"<key>(.*)</key>", "jo")

print(m[1])
-- abcd

###Multi Line###
but if I change string multiline, it returns null

local s=[[ \n\n<key>abcd
</key>fdfdsds]]

local m=ngx.re.match(s,"<key>(.*)</key>", "jom")
print(m[1])
-- m is null

.

m4manas

We can achieve the same using lua string.find as well. What is more optimal solution using ngx.re.match or string.find of lua? Any suggestions.

rohit.c.joshi

ngx.re.match with "jo" option is significantly faster than string.match but ngx.re.match without "jo" option seem to be slower.

e.g in below test, ngx.re.match with "jo" takes 3 second vs 10 seconds for string.match on my laptop.

local ngx_re_match = ngx.re.match
local string_match = string.match
local s=[[\n5463463463565fhfghfghfgdhfhgfhhdfg\n<key>abcd</key>fdfdsdsfgdfhhfghgfdhgfhfgh<msgid>1234</msgid>4656456645366456464363466dfdfd <v1:Cmd Name=\"1\">dsfdsfsdfsfsf</v1:Cmd>]]
local regex = "<key>(.*)</key>.*<msgid>(.*)</msgid>.*<v1:Cmd.*>(.*)</v1:Cmd>"
local regex_jo = "jo"
local count = 500000
local t1= os.time()
for i = 1, count do
--print(ngx_re_match(s,"<key>(.*)</key>.*<msgid>(.*)</msgid>.*<v1:Cmd.*>(.*)</v1:Cmd>","jo"))
local a =ngx_re_match(s,regex, regex_jo)
end
print(os.time()-t1)

local t1= os.time()
for i = 1, count do
--print(string.match(s,regex))
local a = string_match(s,regex, 1, regex_jo)
end
print(os.time()-t1)

On Tuesday, 1 December 2015 21:25:37 UTC-5, Maanas Royy wrote:

We can achieve the same using lua string.find as well. What is more optimal solution using ngx.re.match or string.find of lua? Any suggestions.

agentzh

Hello! On Thu, Dec 3, 2015 at 12:11 PM, RJoshi wrote: > e.g in below test, ngx.re.match with "jo" takes 3 second vs 10 seconds for > string.match on my laptop. > Well, using lua-resty-core can make things even faster if you are not already. See https://github.com/openresty/lua-resty-core Regards, -agentzh