init_by_lua and io.open

dsgnvictor · 2016-08-23T07:40:48+00:00

Hi,I have a scenario as follow: we prepare to use nginx for backend server load balance. The sessions in the backend server have states. So, we need to enabl...

init_by_lua and io.open

dsgnvictor

In an "init_by_lua" block I'm getting a pointer to a file content:

local fd = assert ( io.open ("/var/local/test,dat" , "rb" ) )
records = assert ( fd:read ( "*a" ) )

and can use the "records" global in different locations and "content_by_lua":

data = records:byte(offset+1)
--ffi casts

Works great.

But when and how should I release the "fd" file handle ?

Is there a known trick maybe ? :-)

Any help would be welcomed!

Victor

rpaprocki

Hi,

On Mon, Aug 22, 2016 at 4:40 PM, Victor <dsgn...@gmail.com> wrote:

In an "init_by_lua" block I'm getting a pointer to a file content:
local fd = assert ( io.open ("/var/local/test,dat" , "rb" ) ) records = assert ( fd:read ( "*a" ) )
and can use the "records" global in different locations and "content_by_lua":
data = records:byte(offset+1) --ffi casts
Works great.
But when and how should I release the "fd" file handle ?
Is there a known trick maybe ? :-)
Any help would be welcomed!
Victor

io.read returns the contents of the file as a string, not a file pointer. so here your variable `records` contains the whole contents of the file as a string, not a pointer. And once this data is read, you don't need to keep the file handle open, so you can close it immediately following your call to fd:read() (though i believe the file is closed once the variable associated with it falls out of scope).

As for how: fd:close(). Easy :)

dsgnvictor

Thanks a lot!
I knew about "fd:close()". But I'm not that happy that io.read returns string and a pointer :-)

For a file over 200MB a pointer would be more sensible in my opinion.

So maybe a memory map file ? Would it make sense ? And how to make it released on nginx exit ?

On Tue, Aug 23, 2016 at 3:00 AM, Robert Paprocki <rpap...@fearnothingproductions.net> wrote:

Hi,

On Mon, Aug 22, 2016 at 4:40 PM, Victor <dsgn...@gmail.com> wrote:
In an "init_by_lua" block I'm getting a pointer to a file content:
local fd = assert ( io.open ("/var/local/test,dat" , "rb" ) ) records = assert ( fd:read ( "*a" ) )
and can use the "records" global in different locations and "content_by_lua":
data = records:byte(offset+1) --ffi casts
Works great.
But when and how should I release the "fd" file handle ?
Is there a known trick maybe ? :-)
Any help would be welcomed!
Victor

io.read returns the contents of the file as a string, not a file pointer. so here your variable `records` contains the whole contents of the file as a string, not a pointer. And once this data is read, you don't need to keep the file handle open, so you can close it immediately following your call to fd:read() (though i believe the file is closed once the variable associated with it falls out of scope).

As for how: fd:close(). Easy :)

.

rpaprocki

Hi,

On Mon, Aug 22, 2016 at 6:50 PM, Wiktor Sadowski <dsgn...@gmail.com> wrote:

Thanks a lot!
I knew about "fd:close()". But I'm not that happy that io.read returns string and a pointer :-)
For a file over 200MB a pointer would be more sensible in my opinion.

How exactly would this work? Would you be reading from various portions of the file from disk for every access? That would absolutely destroy performance.

So maybe a memory map file ? Would it make sense ? And how to make it released on nginx exit ?

I don't think Lua provides an interface to mmap a file. You will probably need to write a native C module for something like that.

You also may want to examine your approach here. Something like redis might make a lot more sense than seeking all over a bare file, even if loaded entirely into memory. Maybe describe a bit more of what you're trying to accomplish so we can better understand the problem? In general, native Lua file IO is not performant resty ecosystem, and should be avoided.

dsgnvictor

Thank you very much again.
I have quite big data file with structure inspired by MaxMind DB - https://maxmind.github.io/MaxMind-DB/. Access to records is very fast with pointer math and there is no memory overhead - I already tried in C.
You might be right suggesting redis but allocating enough memory with ffi and moving there the content of the file on "init_by_lua" would work better in such cases. Faster and less resources eaten probably.
But will try both solutions :-)

v

On Tuesday, August 23, 2016 at 4:27:34 PM UTC+2, rpaprocki wrote:

Hi,

On Mon, Aug 22, 2016 at 6:50 PM, Wiktor Sadowski <dsgnv...@gmail.com> wrote:
Thanks a lot!
I knew about "fd:close()". But I'm not that happy that io.read returns string and a pointer :-)
For a file over 200MB a pointer would be more sensible in my opinion.

How exactly would this work? Would you be reading from various portions of the file from disk for every access? That would absolutely destroy performance.

So maybe a memory map file ? Would it make sense ? And how to make it released on nginx exit ?

I don't think Lua provides an interface to mmap a file. You will probably need to write a native C module for something like that.

You also may want to examine your approach here. Something like redis might make a lot more sense than seeking all over a bare file, even if loaded entirely into memory. Maybe describe a bit more of what you're trying to accomplish so we can better understand the problem? In general, native Lua file IO is not performant resty ecosystem, and should be avoided.