Fast reading of CSV files

bogdan · 2017-04-24T17:02:50+00:00

Yes! That helped. Thank you very much.I tried 1.1.0e at first (thinking "if a newer version is required, I might as well go with the very latest"),...

Fast reading of CSV files

bogdan

Hello, everybody

I am currently having this task at hand and I would like your opinion to optimize it: we need to read a collection of CSV files to extract one or many parameter values (columns) from them. The files are each about 280 KB in size and have somewhere around 1440 records (one for every minute, for one day). We need to be able to read up to 180 files at a time and extract an array of values for each parameter.

These are the methods I tested:
- Method 1 (file IO):

    while ts <= t_end do
        < compose file path -
 f_path >
        local t, err = chdir(f_path)
        if t then
            local f_name = < 
compose file name - f_name >
            local f, err = io.open(f_path
 .. f_name, "r")

            if not f then
                < log info >
            else
          
      local l = f:read()
                while l do
                    < process line l >


                    l = f:read()
                end

       
         f:close()
            end
        end

        ts = ts + 86400 -- advance one day
    end

- Method 2 (internal request; using gmatch):

    while ts <= t_end do
        < compose file name and request 
url >
        local res = ngx.location.capture(<internal
 url>)        if res.status == 200 then
            for l in res.body:gmatch("[^\r\n]+")
 do
                <process line l>
            end        end

  

        ts = ts + 86400 -- advance one day

    end

- Method 3 (internal request; using find)

    while ts <= t_end do
        < compose file name and request 
url >

     
     local res = ngx.location.capture(<internal
 url>)
        if res.status == 200 then
            local p = 1
            local s, e = find(res.body, '\n', p, true);
            while s do
                local l = sub(res.body, p, 
e-1)
             
   <process line l>

                p = e + 1

                s, e = find(res.body, '\n', p, true)
            end
        end

        ts = ts + 86400 -- 
advance one day
    end

The configuration for the internal request url is this:

location /<some_path> {
    internal;
    log_not_found off;
    alias 
../<some other path>;
}

Method 2 seems slowest and methods 1 and 3 seems comparable in terms of performance (execution time).
I would like to know which of these methods is preferrable and if there is a better way to do this or a way to optimize this even more.

Thank you

Kind regards,
Bogdan

rpaprocki

Using ngx.location.capture can be slow for large data sets, as it buffers the entire response in memory (as part of the "res" variable), so depending on your set size and usage patterns, reading in and processing the data via `ngx.tcp.socket` may be more performant (this would require some redesign about how your serve your data, though).

Note that string.gmatch is not JIT-able (http://wiki.luajit.org/NYI), as is the entire "io" library, so this would be a less performant approach to parsing out data. The string methods used in the third approach seem like they might be the best approach, again, assuming you're okay with working on the whole data set as a single variable.

On Mon, Apr 24, 2017 at 2:02 AM, Bogdan Irimia <bo...@digitair.ro> wrote:

Hello, everybody

I am currently having this task at hand and I would like your opinion to optimize it: we need to read a collection of CSV files to extract one or many parameter values (columns) from them. The files are each about 280 KB in size and have somewhere around 1440 records (one for every minute, for one day). We need to be able to read up to 180 files at a time and extract an array of values for each parameter.

These are the methods I tested:
- Method 1 (file IO):

while ts <= t_end do
< compose file path - f_path >

local t, err = chdir(f_path)
if t then
local f_name = < compose file name - f_name >
local f, err = io.open(f_path .. f_name, "r")
if not f then
< log info >
else
local l = f:read()
while l do
< process line l >

l = f:read()
end

f:close()
end
end

ts = ts + 86400 -- advance one day
end

- Method 2 (internal request; using gmatch):

while ts <= t_end do
< compose file name and request url >
local res = ngx.location.capture(<internal url>)
if res.status == 200 then
for l in res.body:gmatch("[^\r\n]+") do
<process line l>
end
end

ts = ts + 86400 -- advance one day

end

- Method 3 (internal request; using find)

while ts <= t_end do
< compose file name and request url >
local res = ngx.location.capture(<internal url>)
if res.status == 200 then

local p = 1
local s, e = find(res.body, '\n', p, true);
while s do
local l = sub(res.body, p, e-1)
<process line l>

p = e + 1
s, e = find(res.body, '\n', p, true)
end

end

ts = ts + 86400 -- advance one day
end

The configuration for the internal request url is this:

location /<some_path> {
internal;
log_not_found off;
alias ../<some other path>;
}

Method 2 seems slowest and methods 1 and 3 seems comparable in terms of performance (execution time).
I would like to know which of these methods is preferrable and if there is a better way to do this or a way to optimize this even more.

Thank you

Kind regards,
Bogdan

.

thibaultcha

Hi,

You might also be interested in checking out the 'resty.re.split' method:

https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/re.md#split

Don't hesitate to let us know how it performs against other implementations! I think the main benefit would be that is is through-fully tested and takes care of the find/split approach for you, and is just as JIT-able :)

--
Thibault

On 4/24/17 2:02 AM, Bogdan Irimia wrote:

Hello, everybody

I am currently having this task at hand and I would like your opinion to optimize it: we need to read a collection of CSV files to extract one or many parameter values (columns) from them. The files are each about 280 KB in size and have somewhere around 1440 records (one for every minute, for one day). We need to be able to read up to 180 files at a time and extract an array of values for each parameter.

These are the methods I tested:
- Method 1 (file IO):

while ts <= t_end do
< compose file path - f_path >
local t, err = chdir(f_path)
if t then
local f_name = < compose file name - f_name >
local f, err = io.open(f_path .. f_name, "r")
if not f then
< log info >
else
local l = f:read()
while l do
< process line l >

l = f:read()
end

f:close()
end
end

ts = ts + 86400 -- advance one day
end

- Method 2 (internal request; using gmatch):

while ts <= t_end do
< compose file name and request url >
local res = ngx.location.capture(<internal url>)
if res.status == 200 then
for l in res.body:gmatch("[^\r\n]+") do
<process line l>
end
end ts = ts + 86400 -- advance one day
end

- Method 3 (internal request; using find)

while ts <= t_end do
< compose file name and request url > local res = ngx.location.capture(<internal url>)
if res.status == 200 then

local p = 1
local s, e = find(res.body, '\n', p, true);
while s do
local l = sub(res.body, p, e-1)
<process line l>

p = e + 1
s, e = find(res.body, '\n', p, true)
end

end

ts = ts + 86400 -- advance one day
end

The configuration for the internal request url is this:

location /<some_path> {
internal;
log_not_found off;
alias ../<some other path>;
}

Method 2 seems slowest and methods 1 and 3 seems comparable in terms of performance (execution time).
I would like to know which of these methods is preferrable and if there is a better way to do this or a way to optimize this even more.

Thank you

Kind regards,
Bogda.

bogdan

Thank you for your feedback

We kept the method using ngx.location.capture as it seems a little bit smoother than file IO (uses pread() instead of read()). Also we can skip some checks if the folder exists. The file size is around 300 KB, could be larger, but I estimate it doesn't go beyond 500 KB, so I think this is acceptable regarding the buffering. What do you think?

Also we came to the conclusion that not the reading part is the most time-consuming, but the string processing part. We tried another lua library (ftcsv; it is pure Lua) to parse CSV data, but it was slower (because it supports also quotes and header lines). We migrated to LPeg and performance was improved by almost 30%, which is good. The library is written in C and seems to be optimized, although it is a bit too much for our use-case. And I don't know if it benefits from JIT.
Probably the fastest way would be to implement our own C library, but we don't have the time and experience to do that.

Would you suggest that the ngx.re method could be faster than LPeg? We will give it a try.

Thank you again for your support.
Should you have more suggestions on this topic, don't hesitate to add them, as might be of use for somebody else.

Bogdan

Thibault Charbonnier

Tuesday, April 25, 2017 8:39 PM via Postbox

Hi,

You might also be interested in checking out the 'resty.re.split' method:

https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/re.md#split

Don't hesitate to let us know how it performs against other implementations! I think the main benefit would be that is is through-fully tested and takes care of the find/split approach for you, and is just as JIT-able :)

--
Thibault

On 4/24/17 2:02 AM, Bogdan Irimia wrote:
.

rpaprocki

Hi,

On Wed, Apr 26, 2017 at 12:48 AM, Bogdan Irimia <bo...@digitair.ro> wrote:

Thank you for your feedback

We kept the method using ngx.location.capture as it seems a little bit smoother than file IO (uses pread() instead of read()). Also we can skip some checks if the folder exists. The file size is around 300 KB, could be larger, but I estimate it doesn't go beyond 500 KB, so I think this is acceptable regarding the buffering. What do you think?

Depends on what your usage pattern looks like. Are you doing this request many thousands of times a second? That could be a lot of memory used. Is this just a one-off event? It's probably fine :)

Also we came to the conclusion that not the reading part is the most time-consuming, but the string processing part. We tried another lua library (ftcsv; it is pure Lua) to parse CSV data, but it was slower (because it supports also quotes and header lines). We migrated to LPeg and performance was improved by almost 30%, which is good. The library is written in C and seems to be optimized, although it is a bit too much for our use-case. And I don't know if it benefits from JIT.
Probably the fastest way would be to implement our own C library, but we don't have the time and experience to do that.

Would you suggest that the ngx.re method could be faster than LPeg? We will give it a try.

ngx.re.* has some nontrivial overhead associated with the regex engine, so for basic string operations it might be overkill. But is it JIT-able when you use https://github.com/openresty/lua-resty-core, so that will have a substantial performance benefit for you. I'd say it's worth benchmarking both usages and seeing what's faster for your particular workload.

bogdan

Thank you

Our intention would be to keep the number of ngx.location.capture calls to a minimum, using a caching engine. In the worst case (empty cache), we would have to load a file for each day in an interval requested by the user. We will have a limit of a year or so, so I think a batch of 365 files would be a maximum (for now). This is per user. Indeed, in a case of many concurrent users, this might go big, but we take it a step at a time.

About the parsing method, we measured (basic measurements) ngx.re.* and it seems comparable with LPeg. Taking into account that ngx.re.* is already included in OpenResty's core, we went further with it. I understand there is a bit of overhead because of the regular expressions engine, but LPeg suffered from the same thing (of course). And I'm not sure about it being JIT-able. So ngx.re.* seems the sensible choice.
Thank you for suggesting it.

Regards,
Bogdan

Robert Paprocki

Wednesday, April 26, 2017 11:22 PM via Postbox

Hi,

On Wed, Apr 26, 2017 at 12:48 AM, Bogdan Irimia <bo...@digitair.ro> wrote:
Thank you for your feedback

We kept the method using ngx.location.capture as it seems a little bit smoother than file IO (uses pread() instead of read()). Also we can skip some checks if the folder exists. The file size is around 300 KB, could be larger, but I estimate it doesn't go beyond 500 KB, so I think this is acceptable regarding the buffering. What do you think?

Depends on what your usage pattern looks like. Are you doing this request many thousands of times a second? That could be a lot of memory used. Is this just a one-off event? It's probably fine :)

Also we came to the conclusion that not the reading part is the most time-consuming, but the string processing part. We tried another lua library (ftcsv; it is pure Lua) to parse CSV data, but it was slower (because it supports also quotes and header lines). We migrated to LPeg and performance was improved by almost 30%, which is good. The library is written in C and seems to be optimized, although it is a bit too much for our use-case. And I don't know if it benefits from JIT.
Probably the fastest way would be to implement our own C library, but we don't have the time and experience to do that.

Would you suggest that the ngx.re method could be faster than LPeg? We will give it a try.

ngx.re.* has some nontrivial overhead associated with the regex engine, so for basic string operations it might be overkill. But is it JIT-able when you use https://github.com/openresty/lua-resty-core, so that will have a substantial performance benefit for you. I'd say it's worth benchmarking both usages and seeing what's faster for your particular workload.

.

thibaultcha

FYI: don't hesitate to furthermore optimize the call by taking advantage of the Perl compatible regex options:

ngx_re.split("a:b:c:d", ":", "jo")

See the options documented at:

https://github.com/openresty/lua-nginx-module#ngxrematch

--
Thibault

On 4/27/17 12:39 AM, Bogdan Irimia wrote:

Thank you

Our intention would be to keep the number of ngx.location.capture calls to a minimum, using a caching engine. In the worst case (empty cache), we would have to load a file for each day in an interval requested by the user. We will have a limit of a year or so, so I think a batch of 365 files would be a maximum (for now). This is per user. Indeed, in a case of many concurrent users, this might go big, but we take it a step at a time.

About the parsing method, we measured (basic measurements) ngx.re.* and it seems comparable with LPeg. Taking into account that ngx.re.* is already included in OpenResty's core, we went further with it. I understand there is a bit of overhead because of the regular expressions engine, but LPeg suffered from the same thing (of course). And I'm not sure about it being JIT-able. So ngx.re.* seems the sensible choice.
Thank you for suggesting it.

Regards,
Bogdan

Robert Paprocki

Wednesday, April 26, 2017 11:22 PM via Postbox

Hi,

On Wed, Apr 26, 2017 at 12:48 AM, Bogdan Irimia <bo...@digitair.ro> wrote:

Thank you for your feedback

We kept the method using ngx.location.capture as it seems a little bit smoother than file IO (uses pread() instead of read()). Also we can skip some checks if the folder exists. The file size is around 300 KB, could be larger, but I estimate it doesn't go beyond 500 KB, so I think this is acceptable regarding the buffering. What do you think?

Depends on what your usage pattern looks like. Are you doing this request many thousands of times a second? That could be a lot of memory used. Is this just a one-off event? It's probably fine :)

Also we came to the conclusion that not the reading part is the most time-consuming, but the string processing part. We tried another lua library (ftcsv; it is pure Lua) to parse CSV data, but it was slower (because it supports also quotes and header lines). We migrated to LPeg and performance was improved by almost 30%, which is good. The library is written in C and seems to be optimized, although it is a bit too much for our use-case. And I don't know if it benefits from JIT.
Probably the fastest way would be to implement our own C library, but we don't have the time and experience to do that.

Would you suggest that the ngx.re method could be faster than LPeg? We will give it a try.

ngx.re.* has some nontrivial overhead associated with the regex engine, so for basic string operations it might be overkill. But is it JIT-able when you use https://github.com/openresty/lua-resty-core, so that will have a substantial performance benefit for you. I'd say it's worth benchmarking both usages and seeing what's faster for your particular workload.

.

igor.clark

Hi there, interesting, are those 'jo' options necessary for PCRE JIT to work, if the nginx directive "pcre_jit on;" is already set?

Cheers,
Igor

On 27/04/2017 18:25, Thibault Charbonnier wrote:

FYI: don't hesitate to furthermore optimize the call by taking advantage of the Perl compatible regex options:

ngx_re.split("a:b:c:d", ":", "jo")

See the options documented at:

https://github.com/openresty/lua-nginx-module#ngxrematch

--
Thibault

On 4/27/17 12:39 AM, Bogdan Irimia wrote:

Thank you

Our intention would be to keep the number of ngx.location.capture calls to a minimum, using a caching engine. In the worst case (empty cache), we would have to load a file for each day in an interval requested by the user. We will have a limit of a year or so, so I think a batch of 365 files would be a maximum (for now). This is per user. Indeed, in a case of many concurrent users, this might go big, but we take it a step at a time.

About the parsing method, we measured (basic measurements) ngx.re.* and it seems comparable with LPeg. Taking into account that ngx.re.* is already included in OpenResty's core, we went further with it. I understand there is a bit of overhead because of the regular expressions engine, but LPeg suffered from the same thing (of course). And I'm not sure about it being JIT-able. So ngx.re.* seems the sensible choice.
Thank you for suggesting it.

Regards,
Bogdan

Robert Paprocki

Wednesday, April 26, 2017 11:22 PM via Postbox

Hi,

On Wed, Apr 26, 2017 at 12:48 AM, Bogdan Irimia <bo...@digitair.ro> wrote:

Thank you for your feedback

We kept the method using ngx.location.capture as it seems a little bit smoother than file IO (uses pread() instead of read()). Also we can skip some checks if the folder exists. The file size is around 300 KB, could be larger, but I estimate it doesn't go beyond 500 KB, so I think this is acceptable regarding the buffering. What do you think?

Depends on what your usage pattern looks like. Are you doing this request many thousands of times a second? That could be a lot of memory used. Is this just a one-off event? It's probably fine :)

Also we came to the conclusion that not the reading part is the most time-consuming, but the string processing part. We tried another lua library (ftcsv; it is pure Lua) to parse CSV data, but it was slower (because it supports also quotes and header lines). We migrated to LPeg and performance was improved by almost 30%, which is good. The library is written in C and seems to be optimized, although it is a bit too much for our use-case. And I don't know if it benefits from JIT.
Probably the fastest way would be to implement our own C library, but we don't have the time and experience to do that.

Would you suggest that the ngx.re method could be faster than LPeg? We will give it a try.

ngx.re.* has some nontrivial overhead associated with the regex engine, so for basic string operations it might be overkill. But is it JIT-able when you use https://github.com/openresty/lua-resty-core, so that will have a substantial performance benefit for you. I'd say it's worth benchmarking both usages and seeing what's faster for your particular workload.

.

thibaultcha

Hi,

Yes, those options are to be added separately since there is a Lua-land cache of compiled regexes maintained by lua-resty-core. In short:

1. Don't forget to require "resty.core", as Robert noted.
2. Pass the "jo" options if you have JIT PCRE enabled.
3. Eventually, consider re-using the 'res' table to reduce the GC load, as documented in the last section of the ngx_re.split() API documentation:

https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/re.md

4. Be wary of splitting tokens like "|", are those are, obviously, special character for the PCRE engine.

--
Thibault

On 4/28/17 1:00 AM, Igor Clark wrote:

Hi there, interesting, are those 'jo' options necessary for PCRE JIT to work, if the nginx directive "pcre_jit on;" is already set?

Cheers,
Igor

On 27/04/2017 18:25, Thibault Charbonnier wrote:

FYI: don't hesitate to furthermore optimize the call by taking advantage of the Perl compatible regex options:

ngx_re.split("a:b:c:d", ":", "jo")

See the options documented at:

https://github.com/openresty/lua-nginx-module#ngxrematch

--
Thibault

On 4/27/17 12:39 AM, Bogdan Irimia wrote:

Thank you

Our intention would be to keep the number of ngx.location.capture calls to a minimum, using a caching engine. In the worst case (empty cache), we would have to load a file for each day in an interval requested by the user. We will have a limit of a year or so, so I think a batch of 365 files would be a maximum (for now). This is per user. Indeed, in a case of many concurrent users, this might go big, but we take it a step at a time.

About the parsing method, we measured (basic measurements) ngx.re.* and it seems comparable with LPeg. Taking into account that ngx.re.* is already included in OpenResty's core, we went further with it. I understand there is a bit of overhead because of the regular expressions engine, but LPeg suffered from the same thing (of course). And I'm not sure about it being JIT-able. So ngx.re.* seems the sensible choice.
Thank you for suggesting it.

Regards,
Bogdan

Robert Paprocki

Wednesday, April 26, 2017 11:22 PM via Postbox

Hi,

On Wed, Apr 26, 2017 at 12:48 AM, Bogdan Irimia <bo...@digitair.ro> wrote:

Thank you for your feedback

We kept the method using ngx.location.capture as it seems a little bit smoother than file IO (uses pread() instead of read()). Also we can skip some checks if the folder exists. The file size is around 300 KB, could be larger, but I estimate it doesn't go beyond 500 KB, so I think this is acceptable regarding the buffering. What do you think?

Depends on what your usage pattern looks like. Are you doing this request many thousands of times a second? That could be a lot of memory used. Is this just a one-off event? It's probably fine :)

Also we came to the conclusion that not the reading part is the most time-consuming, but the string processing part. We tried another lua library (ftcsv; it is pure Lua) to parse CSV data, but it was slower (because it supports also quotes and header lines). We migrated to LPeg and performance was improved by almost 30%, which is good. The library is written in C and seems to be optimized, although it is a bit too much for our use-case. And I don't know if it benefits from JIT.
Probably the fastest way would be to implement our own C library, but we don't have the time and experience to do that.

Would you suggest that the ngx.re method could be faster than LPeg? We will give it a try.

ngx.re.* has some nontrivial overhead associated with the regex engine, so for basic string operations it might be overkill. But is it JIT-able when you use https://github.com/openresty/lua-resty-core, so that will have a substantial performance benefit for you. I'd say it's worth benchmarking both usages and seeing what's faster for your particular workload.

.

bogdan

Oh, wow

Great suggestion, thank you!
About escaping the special characters: which are all these special characters? Are these all:
.^$*+?()[{\|
?

And... how should I escape them in ngx.re.split? Like this:
ngx_re.split("a:b:c:d", "\|", "jo")
?

Thank you very much.

Bogdan

Thibault Charbonnier

Friday, April 28, 2017 11:22 AM via Postbox

Hi,

Yes, those options are to be added separately since there is a Lua-land cache of compiled regexes maintained by lua-resty-core. In short:

1. Don't forget to require "resty.core", as Robert noted.
2. Pass the "jo" options if you have JIT PCRE enabled.
3. Eventually, consider re-using the 'res' table to reduce the GC load, as documented in the last section of the ngx_re.split() API documentation:

https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/re.md

4. Be wary of splitting tokens like "|", are those are, obviously, special character for the PCRE engine.

--
Thibault

On 4/28/17 1:00 AM, Igor Clark wrote:
.

bogdan

To be more specific, would this fix work for the separator? Would it be enough?

    if string.find(".^$*+?()[{\\|", separator) then
        separator = table.concat{"\\", 
separator}
    end

Thank you

Thibault Charbonnier

Friday, April 28, 2017 11:22 AM via Postbox

Hi,

Yes, those options are to be added separately since there is a Lua-land cache of compiled regexes maintained by lua-resty-core. In short:

1. Don't forget to require "resty.core", as Robert noted.
2. Pass the "jo" options if you have JIT PCRE enabled.
3. Eventually, consider re-using the 'res' table to reduce the GC load, as documented in the last section of the ngx_re.split() API documentation:

https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/re.md

4. Be wary of splitting tokens like "|", are those are, obviously, special character for the PCRE engine.

--
Thibault

On 4/28/17 1:00 AM, Igor Clark wrote:
.

igor.clark

Oh, OK! Thanks, I hadn't realised that.

Assuming we have #1 and #2 in place, do we need to do this as well:

local ngx_re = require "ngx.re"

in order to take advantage of the 'ngx.re' in 'resty.core' ? Or if we just call ngx.re.*, will that call the "plain" ngx.re version, or the resty.core version?

On 28/04/2017 09:22, Thibault Charbonnier wrote:

Hi,

Yes, those options are to be added separately since there is a Lua-land cache of compiled regexes maintained by lua-resty-core. In short:

1. Don't forget to require "resty.core", as Robert noted.
2. Pass the "jo" options if you have JIT PCRE enabled.
3. Eventually, consider re-using the 'res' table to reduce the GC load, as documented in the last section of the ngx_re.split() API documentation:

    https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/re.md

4. Be wary of splitting tokens like "|", are those are, obviously, special character for the PCRE engine.

--
Thibault

On 4/28/17 1:00 AM, Igor Clark wrote:

Hi there, interesting, are those 'jo' options necessary for PCRE JIT to work, if the nginx directive "pcre_jit on;" is already set?

Cheers,
Igor

On 27/04/2017 18:25, Thibault Charbonnier wrote:

FYI: don't hesitate to furthermore optimize the call by taking advantage of the Perl compatible regex options:

    ngx_re.split("a:b:c:d", ":", "jo")

See the options documented at:

    https://github.com/openresty/lua-nginx-module#ngxrematch

--
Thibault

On 4/27/17 12:39 AM, Bogdan Irimia wrote:

Thank you

Our intention would be to keep the number of ngx.location.capture calls to a minimum, using a caching engine. In the worst case (empty cache), we would have to load a file for each day in an interval requested by the user. We will have a limit of a year or so, so I think a batch of 365 files would be a maximum (for now). This is per user. Indeed, in a case of many concurrent users, this might go big, but we take it a step at a time.

About the parsing method, we measured (basic measurements) ngx.re.* and it seems comparable with LPeg. Taking into account that ngx.re.* is already included in OpenResty's core, we went further with it. I understand there is a bit of overhead because of the regular expressions engine, but LPeg suffered from the same thing (of course). And I'm not sure about it being JIT-able. So ngx.re.* seems the sensible choice.
Thank you for suggesting it.

Regards,
Bogdan

Robert Paprocki

Wednesday, April 26, 2017 11:22 PM via Postbox

Hi,

On Wed, Apr 26, 2017 at 12:48 AM, Bogdan Irimia <bo...@digitair.ro> wrote:

Thank you for your feedback

We kept the method using ngx.location.capture as it seems a little bit smoother than file IO (uses pread() instead of read()). Also we can skip some checks if the folder exists. The file size is around 300 KB, could be larger, but I estimate it doesn't go beyond 500 KB, so I think this is acceptable regarding the buffering. What do you think?

Depends on what your usage pattern looks like. Are you doing this request many thousands of times a second? That could be a lot of memory used. Is this just a one-off event? It's probably fine :)

Also we came to the conclusion that not the reading part is the most time-consuming, but the string processing part. We tried another lua library (ftcsv; it is pure Lua) to parse CSV data, but it was slower (because it supports also quotes and header lines). We migrated to LPeg and performance was improved by almost 30%, which is good. The library is written in C and seems to be optimized, although it is a bit too much for our use-case. And I don't know if it benefits from JIT.
Probably the fastest way would be to implement our own C library, but we don't have the time and experience to do that.

Would you suggest that the ngx.re method could be faster than LPeg? We will give it a try.

ngx.re.* has some nontrivial overhead associated with the regex engine, so for basic string operations it might be overkill. But is it JIT-able when you use https://github.com/openresty/lua-resty-core, so that will have a substantial performance benefit for you. I'd say it's worth benchmarking both usages and seeing what's faster for your particular workload.

.

rpaprocki

On Fri, Apr 28, 2017 at 10:47 AM, Igor Clark <igor...@gmail.com> wrote:

Oh, OK! Thanks, I hadn't realised that.

Assuming we have #1 and #2 in place, do we need to do this as well:
local ngx_re = require "ngx.re"
in order to take advantage of the 'ngx.re' in 'resty.core' ? Or if we just call ngx.re.*, will that call the "plain" ngx.re version, or the resty.core version?

The latter. Actually, don't do the former at all, as "ngx.re" isn't a distinct module, but a namespace. Requiring resty.core will simply overwrite the appropriate functions, without any extra work (see https://www.cryptobells.com/resty-core-the-good-the-bad-and-the-jit-y/ for a bit of a detailed write up on some of the mechanics at work).

What you're describing is commonly done to cache a function call as an upvalue. There is some performance benefit here. This would look like:

local re_split = ngx.re.split

There is no need to require additional modules. So, one would 'require "resty.core"' in an init_by_lua context, and simply use ngx.re.split as normal.

igor.clark

Got it. Thank you.

Good bit of reading there, too. Cheers :-)

On 28/04/2017 20:36, Robert Paprocki wrote:

On Fri, Apr 28, 2017 at 10:47 AM, Igor Clark <igor...@gmail.com> wrote:
Oh, OK! Thanks, I hadn't realised that.

Assuming we have #1 and #2 in place, do we need to do this as well:
local ngx_re = require "ngx.re"
in order to take advantage of the 'ngx.re' in 'resty.core' ? Or if we just call ngx.re.*, will that call the "plain" ngx.re version, or the resty.core version?
The latter. Actually, don't do the former at all, as "ngx.re" isn't a distinct module, but a namespace. Requiring resty.core will simply overwrite the appropriate functions, without any extra work (see https://www.cryptobells.com/resty-core-the-good-the-bad-and-the-jit-y/ for a bit of a detailed write up on some of the mechanics at work).

What you're describing is commonly done to cache a function call as an upvalue. There is some performance benefit here. This would look like:

local re_split = ngx.re.split

There is no need to require additional modules. So, one would 'require "resty.core"' in an init_by_lua context, and simply use ngx.re.split as normal.
.

thibaultcha

The 'ngx_re.split()' function is not part of ngx_lua's ngx.re.* API. It is a separate module to be loaded from lua-resty-core (hence why ngx_re vs ngx.re):

local ngx_re = require "ngx.re"

local res, err = ngx_re.split("a,b,c,d", ",")

I believe this is already detailed in the API's documentation :)

On 4/28/17 3:01 PM, Igor Clark wrote:

Got it. Thank you.

Good bit of reading there, too. Cheers :-)

On 28/04/2017 20:36, Robert Paprocki wrote:

On Fri, Apr 28, 2017 at 10:47 AM, Igor Clark <igor...@gmail.com> wrote:

Oh, OK! Thanks, I hadn't realised that.

Assuming we have #1 and #2 in place, do we need to do this as well:

local ngx_re = require "ngx.re"

in order to take advantage of the 'ngx.re' in 'resty.core' ? Or if we just call ngx.re.*, will that call the "plain" ngx.re version, or the resty.core version?

The latter. Actually, don't do the former at all, as "ngx.re" isn't a distinct module, but a namespace. Requiring resty.core will simply overwrite the appropriate functions, without any extra work (see https://www.cryptobells.com/resty-core-the-good-the-bad-and-the-jit-y/ for a bit of a detailed write up on some of the mechanics at work).

What you're describing is commonly done to cache a function call as an upvalue. There is some performance benefit here. This would look like:

local re_split = ngx.re.split

There is no need to require additional modules. So, one would 'require "resty.core"' in an init_by_lua context, and simply use ngx.re.split as normal.

.

rpaprocki

Whups, I mis-remember the implementation here. Thank Thibault for the correction!

On Fri, Apr 28, 2017 at 3:42 PM, Thibault Charbonnier <thiba...@fastmail.com> wrote:

The 'ngx_re.split()' function is not part of ngx_lua's ngx.re.* API. It is a separate module to be loaded from lua-resty-core (hence why ngx_re vs ngx.re):

local ngx_re = require "ngx.re"

local res, err = ngx_re.split("a,b,c,d", ",")

I believe this is already detailed in the API's documentation :)

On 4/28/17 3:01 PM, Igor Clark wrote:
Got it. Thank you.

Good bit of reading there, too. Cheers :-)

On 28/04/2017 20:36, Robert Paprocki wrote:
On Fri, Apr 28, 2017 at 10:47 AM, Igor Clark <igor...@gmail.com> wrote:
Oh, OK! Thanks, I hadn't realised that.

Assuming we have #1 and #2 in place, do we need to do this as well:
local ngx_re = require "ngx.re"
in order to take advantage of the 'ngx.re' in 'resty.core' ? Or if we just call ngx.re.*, will that call the "plain" ngx.re version, or the resty.core version?
The latter. Actually, don't do the former at all, as "ngx.re" isn't a distinct module, but a namespace. Requiring resty.core will simply overwrite the appropriate functions, without any extra work (see https://www.cryptobells.com/resty-core-the-good-the-bad-and-the-jit-y/ for a bit of a detailed write up on some of the mechanics at work).

What you're describing is commonly done to cache a function call as an upvalue. There is some performance benefit here. This would look like:

local re_split = ngx.re.split

There is no need to require additional modules. So, one would 'require "resty.core"' in an init_by_lua context, and simply use ngx.re.split as normal.
.