About lua shared dict using

xieqin1 · 2016-03-18T18:31:18+00:00

Hello! On Thu, Mar 17, 2016 at 11:00 AM, N K wrote: > Does the lua-resty-memcached module support memcached binary protocol? Not yet. > When using t...

About lua shared dict using

xieqin1

Hi, I am using a piece of C++ code via luajit ffi, currently I use it like this:

1 module define:

local ffi = require("ffi")

ffi.cdef[[

void* init(const char*, const char*, const char*, const char*, int) asm("so export name");

float* extract(void*, const char*) asm("so export name");

]]

local test = ffi.load("my.so")

local _M = {}

function _M.extract(instance, file)

local result = test.extract(instance, file)

return result

end

function _M.init()

local instance = test.init(....)

return instance

end

return _M

2 init_by_lua:

local mo = require("init_extract")

instance = mo.init()

3 content_by_lua:

local mo = require("init_extract")

local feature = mo.extract(instance, content)

It worked fine and the access is ok.

I use a global variable "instance" to share this instance in nginx work process, as each instance cost plenty of memory, if each work process get a copy of it will course memory insufficient. I observed that, using shell command, each work process as well as the mater process consumed equal memory.

So I plan to use lua shared dict to hold this 'instance' reduce memory use, but I encountered problem;

1 module define:

same as above

2 init_by_lua:

local mo = require("init_extract")

local shared_data = ngx.shared.shared_data

shared_data:set("instance", instance)

3 content_by_lua:

local shared_data = ngx.shared.shared_data

local instance = shared_data:get("instance")

local feature = mo.extract(instance, content)

when I restart nginx, I observed that each work process as well as the mater process consumed equal memory as above，I wondering that if it really provide the advantage of reducing memory use.

And when I access it, nginx collapsed reported:

worker process XXXXX exited on signal 11...

Any one has any suggestions?

xieqin1

Sad no one reply me~~~~

在 2016年3月18日星期五 UTC+8下午6:31:18，xie...@gmail.com写道：

Hi, I am using a piece of C++ code via luajit ffi, currently I use it like this:

1 module define:
local ffi = require("ffi")

ffi.cdef[[
void* init(const char*, const char*, const char*, const char*, int) asm("so export name");
float* extract(void*, const char*) asm("so export name");
]]

local test = ffi.load("my.so")

local _M = {}

function _M.extract(instance, file)
local result = test.extract(instance, file)
return result
end

function _M.init()
local instance = test.init(....)
return instance
end

return _M

2 init_by_lua:
local mo = require("init_extract")
instance = mo.init()

3 content_by_lua:
local mo = require("init_extract")
local feature = mo.extract(instance, content)

It worked fine and the access is ok.
I use a global variable "instance" to share this instance in nginx work process, as each instance cost plenty of memory, if each work process get a copy of it will course memory insufficient. I observed that, using shell command, each work process as well as the mater process consumed equal memory.

So I plan to use lua shared dict to hold this 'instance' reduce memory use, but I encountered problem;
1 module define:
same as above

2 init_by_lua:
local mo = require("init_extract")

local shared_data = ngx.shared.shared_data
shared_data:set("instance", instance)

3 content_by_lua:
local shared_data = ngx.shared.shared_data
local instance = shared_data:get("instance")
local feature = mo.extract(instance, content)

when I restart nginx, I observed that each work process as well as the mater process consumed equal memory as above，I wondering that if it really provide the advantage of reducing memory use.
And when I access it, nginx collapsed reported:
worker process XXXXX exited on signal 11...

Any one has any suggestions?

guanglinlv

1. each nginx worker take a isolate copy of global variable defined in `init_by_lua`, it's expected!

2. shared dict is not support complex type variable, it can be lua booleans,numbers,string,or nil.

I'm not very understand your said, what is your instance? how much memory will be cost?

Thanks.

在 2016年3月18日星期五 UTC+8下午6:31:18，xie...@gmail.com写道：

Hi, I am using a piece of C++ code via luajit ffi, currently I use it like this:

1 module define:
local ffi = require("ffi")

ffi.cdef[[
void* init(const char*, const char*, const char*, const char*, int) asm("so export name");
float* extract(void*, const char*) asm("so export name");
]]

local test = ffi.load("my.so")

local _M = {}

function _M.extract(instance, file)
local result = test.extract(instance, file)
return result
end

function _M.init()
local instance = test.init(....)
return instance
end

return _M

2 init_by_lua:
local mo = require("init_extract")
instance = mo.init()

3 content_by_lua:
local mo = require("init_extract")
local feature = mo.extract(instance, content)

It worked fine and the access is ok.
I use a global variable "instance" to share this instance in nginx work process, as each instance cost plenty of memory, if each work process get a copy of it will course memory insufficient. I observed that, using shell command, each work process as well as the mater process consumed equal memory.

So I plan to use lua shared dict to hold this 'instance' reduce memory use, but I encountered problem;
1 module define:
same as above

2 init_by_lua:
local mo = require("init_extract")

local shared_data = ngx.shared.shared_data
shared_data:set("instance", instance)

3 content_by_lua:
local shared_data = ngx.shared.shared_data
local instance = shared_data:get("instance")
local feature = mo.extract(instance, content)

when I restart nginx, I observed that each work process as well as the mater process consumed equal memory as above，I wondering that if it really provide the advantage of reducing memory use.
And when I access it, nginx collapsed reported:
worker process XXXXX exited on signal 11...

Any one has any suggestions?

xieqin1

I see , so I can't use shared dict to hold this nstance to reduce memory use.

Currently I'm using global variable defined in `init_by_lua`, which cost plenty of memory for each nginx worker take a isolate copy of global variable.

Instance is a class generated by C++ code, and it holds a static member which takes a lot of memoey(500M).

在 2016年3月21日星期一 UTC+8下午4:53:58，Guanglin Lv写道：

1. each nginx worker take a isolate copy of global variable defined in `init_by_lua`, it's expected!
2. shared dict is not support complex type variable, it can be lua booleans,numbers,string,or nil.

I'm not very understand your said, what is your instance? how much memory will be cost?

Thanks.

在 2016年3月18日星期五 UTC+8下午6:31:18，xie...@gmail.com写道：
Hi, I am using a piece of C++ code via luajit ffi, currently I use it like this:

1 module define:
local ffi = require("ffi")

ffi.cdef[[
void* init(const char*, const char*, const char*, const char*, int) asm("so export name");
float* extract(void*, const char*) asm("so export name");
]]

local test = ffi.load("my.so")

local _M = {}

function _M.extract(instance, file)
local result = test.extract(instance, file)
return result
end

function _M.init()
local instance = test.init(....)
return instance
end

return _M

2 init_by_lua:
local mo = require("init_extract")
instance = mo.init()

3 content_by_lua:
local mo = require("init_extract")
local feature = mo.extract(instance, content)

It worked fine and the access is ok.
I use a global variable "instance" to share this instance in nginx work process, as each instance cost plenty of memory, if each work process get a copy of it will course memory insufficient. I observed that, using shell command, each work process as well as the mater process consumed equal memory.

So I plan to use lua shared dict to hold this 'instance' reduce memory use, but I encountered problem;
1 module define:
same as above

2 init_by_lua:
local mo = require("init_extract")

local shared_data = ngx.shared.shared_data
shared_data:set("instance", instance)

3 content_by_lua:
local shared_data = ngx.shared.shared_data
local instance = shared_data:get("instance")
local feature = mo.extract(instance, content)

when I restart nginx, I observed that each work process as well as the mater process consumed equal memory as above，I wondering that if it really provide the advantage of reducing memory use.
And when I access it, nginx collapsed reported:
worker process XXXXX exited on signal 11...

Any one has any suggestions?

guanglinlv

在 2016年3月22日星期二 UTC+8上午10:57:13，xie...@gmail.com写道：

I see , so I can't use shared dict to hold this nstance to reduce memory use.
Currently I'm using global variable defined in `init_by_lua`, which cost plenty of memory for each nginx worker take a isolate copy of global variable.
Instance is a class generated by C++ code, and it holds a static member which takes a lot of memoey(500M).

oh, the static member is so large! so, why do not you share a pointer of that memory reside in instance . use the pointer to read the content in multiple worker.

you have to take care the race condition if you want to change that large static member .

rpaprocki

Depending on your use case,instead of using a global variable you should store you data in a separate lua module. If this is read-only data, if you write the data during the init phase your data should be able to employ copy-on-write. See https://github.com/openresty/lua-nginx-module/blob/master/README.markdown#data-sharing-within-an-nginx-worker for more info.

If the data is read-write then yes you will need to duplicate it for each worker, there's no trying to cheat around this ;)

On Mar 21, 2016, at 19:57, xie...@gmail.com wrote:

I see , so I can't use shared dict to hold this nstance to reduce memory use.
Currently I'm using global variable defined in `init_by_lua`, which cost plenty of memory for each nginx worker take a isolate copy of global variable.
Instance is a class generated by C++ code, and it holds a static member which takes a lot of memoey(500M).

在 2016年3月21日星期一 UTC+8下午4:53:58，Guanglin Lv写道：
1. each nginx worker take a isolate copy of global variable defined in `init_by_lua`, it's expected!
2. shared dict is not support complex type variable, it can be lua booleans,numbers,string,or nil.

I'm not very understand your said, what is your instance? how much memory will be cost?

Thanks.

在 2016年3月18日星期五 UTC+8下午6:31:18，xie...@gmail.com写道：
Hi, I am using a piece of C++ code via luajit ffi, currently I use it like this:

1 module define:
local ffi = require("ffi")

ffi.cdef[[
void* init(const char*, const char*, const char*, const char*, int) asm("so export name");
float* extract(void*, const char*) asm("so export name");
]]

local test = ffi.load("my.so")

local _M = {}

function _M.extract(instance, file)
local result = test.extract(instance, file)
return result
end

function _M.init()
local instance = test.init(....)
return instance
end

return _M

2 init_by_lua:
local mo = require("init_extract")
instance = mo.init()

3 content_by_lua:
local mo = require("init_extract")
local feature = mo.extract(instance, content)

It worked fine and the access is ok.
I use a global variable "instance" to share this instance in nginx work process, as each instance cost plenty of memory, if each work process get a copy of it will course memory insufficient. I observed that, using shell command, each work process as well as the mater process consumed equal memory.

So I plan to use lua shared dict to hold this 'instance' reduce memory use, but I encountered problem;
1 module define:
same as above

2 init_by_lua:
local mo = require("init_extract")

local shared_data = ngx.shared.shared_data
shared_data:set("instance", instance)

3 content_by_lua:
local shared_data = ngx.shared.shared_data
local instance = shared_data:get("instance")
local feature = mo.extract(instance, content)

when I restart nginx, I observed that each work process as well as the mater process consumed equal memory as above，I wondering that if it really provide the advantage of reducing memory use.
And when I access it, nginx collapsed reported:
worker process XXXXX exited on signal 11...

Any one has any suggestions?

.

agentzh

Hello!

On Fri, Mar 18, 2016 at 3:31 AM,  xieqin1 wrote:
> when I restart nginx,  I observed that each work process as well as the
> mater process consumed equal memory as above，I  wondering that if it really
> provide the advantage of reducing memory use.

You know about the copy-on-write feature of modern *NIX systems?
Basically you cannot simply add the memory footprint of each process
to conclude the total memory usage in a Linux or alike.

> And when I  access it, nginx collapsed reported:
>
> worker process XXXXX exited on signal 11...
>

Seems like there's some bugs in your C++ code or FFI binding. Try
tools like gdb and valgrind to debug memory issues.

Regards,
-agentzh

xieqin1

It is read only, but each worker need to access C++ functions in it via ffi.

And I defined a module for it, yet the module‘s data is initiate in phase init_by_lua.

Maybe I could try you suggestion to initiate it in module define as https://github.com/openresty/lua-nginx-module/blob/master/README.markdown#data-sharing-within-an-nginx-worker shows.

But my question is, the shared dict really can not hold C++ pointer? If it can, what is the proper way to realize it, haven't seen any examples yet.

Thanks, good advice anyway.

在 2016年3月28日星期一 UTC+8上午3:17:48，rpaprocki写道：

Depending on your use case,instead of using a global variable you should store you data in a separate lua module. If this is read-only data, if you write the data during the init phase your data should be able to employ copy-on-write. See https://github.com/openresty/lua-nginx-module/blob/master/README.markdown#data-sharing-within-an-nginx-worker for more info.

If the data is read-write then yes you will need to duplicate it for each worker, there's no trying to cheat around this ;)

On Mar 21, 2016, at 19:57, xie...@gmail.com wrote:

I see , so I can't use shared dict to hold this nstance to reduce memory use.
Currently I'm using global variable defined in `init_by_lua`, which cost plenty of memory for each nginx worker take a isolate copy of global variable.
Instance is a class generated by C++ code, and it holds a static member which takes a lot of memoey(500M).

在 2016年3月21日星期一 UTC+8下午4:53:58，Guanglin Lv写道：
1. each nginx worker take a isolate copy of global variable defined in `init_by_lua`, it's expected!
2. shared dict is not support complex type variable, it can be lua booleans,numbers,string,or nil.

I'm not very understand your said, what is your instance? how much memory will be cost?

Thanks.

在 2016年3月18日星期五 UTC+8下午6:31:18，xie...@gmail.com写道：
Hi, I am using a piece of C++ code via luajit ffi, currently I use it like this:

1 module define:
local ffi = require("ffi")

ffi.cdef[[
void* init(const char*, const char*, const char*, const char*, int) asm("so export name");
float* extract(void*, const char*) asm("so export name");
]]

local test = ffi.load("my.so")

local _M = {}

function _M.extract(instance, file)
local result = test.extract(instance, file)
return result
end

function _M.init()
local instance = test.init(....)
return instance
end

return _M

2 init_by_lua:
local mo = require("init_extract")
instance = mo.init()

3 content_by_lua:
local mo = require("init_extract")
local feature = mo.extract(instance, content)

It worked fine and the access is ok.
I use a global variable "instance" to share this instance in nginx work process, as each instance cost plenty of memory, if each work process get a copy of it will course memory insufficient. I observed that, using shell command, each work process as well as the mater process consumed equal memory.

So I plan to use lua shared dict to hold this 'instance' reduce memory use, but I encountered problem;
1 module define:
same as above

2 init_by_lua:
local mo = require("init_extract")

local shared_data = ngx.shared.shared_data
shared_data:set("instance", instance)

3 content_by_lua:
local shared_data = ngx.shared.shared_data
local instance = shared_data:get("instance")
local feature = mo.extract(instance, content)

when I restart nginx, I observed that each work process as well as the mater process consumed equal memory as above，I wondering that if it really provide the advantage of reducing memory use.
And when I access it, nginx collapsed reported:
worker process XXXXX exited on signal 11...

Any one has any suggestions?

.

xieqin1

Wow, you are my idol, my nansheng, mua.

What is your suggestion, to use lua shared dict or to get it done as Mr. rpaprocki said?

My real confusion is, the shared dict really can not hold C++ pointer? If it can, what is the proper way to realize it, haven't seen any examples yet.

在 2016年3月28日星期一 UTC+8上午3:23:29，agentzh写道：

Hello!

On Fri, Mar 18, 2016 at 3:31 AM, xieqin1 wrote:
> when I restart nginx, I observed that each work process as well as the
> mater process consumed equal memory as above，I wondering that if it really
> provide the advantage of reducing memory use.

You know about the copy-on-write feature of modern *NIX systems?
Basically you cannot simply add the memory footprint of each process
to conclude the total memory usage in a Linux or alike.

> And when I access it, nginx collapsed reported:
>
> worker process XXXXX exited on signal 11...
>

Seems like there's some bugs in your C++ code or FFI binding. Try
tools like gdb and valgrind to debug memory issues.

Regards,
-agentzh

xieqin1

BINGO， I read your link, which says:

Note that this data sharing is on a per-worker basis and not on a per-server basis. That is, when there are multiple nginx worker processes under an Nginx master, data sharing cannot cross the process boundary between these workers.

If I want to reduce memory cost(change share per work to share per server), I have to use this share dict which I encountered bug as I posted, awful..

在 2016年3月28日星期一 UTC+8上午3:17:48，rpaprocki写道：

Depending on your use case,instead of using a global variable you should store you data in a separate lua module. If this is read-only data, if you write the data during the init phase your data should be able to employ copy-on-write. See https://github.com/openresty/lua-nginx-module/blob/master/README.markdown#data-sharing-within-an-nginx-worker for more info.

If the data is read-write then yes you will need to duplicate it for each worker, there's no trying to cheat around this ;)

On Mar 21, 2016, at 19:57, xie...@gmail.com wrote:

I see , so I can't use shared dict to hold this nstance to reduce memory use.
Currently I'm using global variable defined in `init_by_lua`, which cost plenty of memory for each nginx worker take a isolate copy of global variable.
Instance is a class generated by C++ code, and it holds a static member which takes a lot of memoey(500M).

在 2016年3月21日星期一 UTC+8下午4:53:58，Guanglin Lv写道：
1. each nginx worker take a isolate copy of global variable defined in `init_by_lua`, it's expected!
2. shared dict is not support complex type variable, it can be lua booleans,numbers,string,or nil.

I'm not very understand your said, what is your instance? how much memory will be cost?

Thanks.

在 2016年3月18日星期五 UTC+8下午6:31:18，xie...@gmail.com写道：
Hi, I am using a piece of C++ code via luajit ffi, currently I use it like this:

1 module define:
local ffi = require("ffi")

ffi.cdef[[
void* init(const char*, const char*, const char*, const char*, int) asm("so export name");
float* extract(void*, const char*) asm("so export name");
]]

local test = ffi.load("my.so")

local _M = {}

function _M.extract(instance, file)
local result = test.extract(instance, file)
return result
end

function _M.init()
local instance = test.init(....)
return instance
end

return _M

2 init_by_lua:
local mo = require("init_extract")
instance = mo.init()

3 content_by_lua:
local mo = require("init_extract")
local feature = mo.extract(instance, content)

It worked fine and the access is ok.
I use a global variable "instance" to share this instance in nginx work process, as each instance cost plenty of memory, if each work process get a copy of it will course memory insufficient. I observed that, using shell command, each work process as well as the mater process consumed equal memory.

So I plan to use lua shared dict to hold this 'instance' reduce memory use, but I encountered problem;
1 module define:
same as above

2 init_by_lua:
local mo = require("init_extract")

local shared_data = ngx.shared.shared_data
shared_data:set("instance", instance)

3 content_by_lua:
local shared_data = ngx.shared.shared_data
local instance = shared_data:get("instance")
local feature = mo.extract(instance, content)

when I restart nginx, I observed that each work process as well as the mater process consumed equal memory as above，I wondering that if it really provide the advantage of reducing memory use.
And when I access it, nginx collapsed reported:
worker process XXXXX exited on signal 11...

Any one has any suggestions?

.

roidinev

agentzh suggest ref the lua-resty-limit-traffic module example for the struct using the sharedict.

but for the void pointer,how?

On Tuesday, March 29, 2016 at 5:41:15 PM UTC+8, xie...@gmail.com wrote:

BINGO， I read your link, which says:
Note that this data sharing is on a per-worker basis and not on a per-server basis. That is, when there are multiple nginx worker processes under an Nginx master, data sharing cannot cross the process boundary between these workers.
If I want to reduce memory cost(change share per work to share per server), I have to use this share dict which I encountered bug as I posted, awful..

在 2016年3月28日星期一 UTC+8上午3:17:48，rpaprocki写道：
Depending on your use case,instead of using a global variable you should store you data in a separate lua module. If this is read-only data, if you write the data during the init phase your data should be able to employ copy-on-write. See https://github.com/openresty/lua-nginx-module/blob/master/README.markdown#data-sharing-within-an-nginx-worker for more info.

If the data is read-write then yes you will need to duplicate it for each worker, there's no trying to cheat around this ;)

On Mar 21, 2016, at 19:57, xie...@gmail.com wrote:

I see , so I can't use shared dict to hold this nstance to reduce memory use.
Currently I'm using global variable defined in `init_by_lua`, which cost plenty of memory for each nginx worker take a isolate copy of global variable.
Instance is a class generated by C++ code, and it holds a static member which takes a lot of memoey(500M).

在 2016年3月21日星期一 UTC+8下午4:53:58，Guanglin Lv写道：
1. each nginx worker take a isolate copy of global variable defined in `init_by_lua`, it's expected!
2. shared dict is not support complex type variable, it can be lua booleans,numbers,string,or nil.

I'm not very understand your said, what is your instance? how much memory will be cost?

Thanks.

在 2016年3月18日星期五 UTC+8下午6:31:18，xie...@gmail.com写道：
Hi, I am using a piece of C++ code via luajit ffi, currently I use it like this:

1 module define:
local ffi = require("ffi")

ffi.cdef[[
void* init(const char*, const char*, const char*, const char*, int) asm("so export name");
float* extract(void*, const char*) asm("so export name");
]]

local test = ffi.load("my.so")

local _M = {}

function _M.extract(instance, file)
local result = test.extract(instance, file)
return result
end

function _M.init()
local instance = test.init(....)
return instance
end

return _M

2 init_by_lua:
local mo = require("init_extract")
instance = mo.init()

3 content_by_lua:
local mo = require("init_extract")
local feature = mo.extract(instance, content)

It worked fine and the access is ok.
I use a global variable "instance" to share this instance in nginx work process, as each instance cost plenty of memory, if each work process get a copy of it will course memory insufficient. I observed that, using shell command, each work process as well as the mater process consumed equal memory.

So I plan to use lua shared dict to hold this 'instance' reduce memory use, but I encountered problem;
1 module define:
same as above

2 init_by_lua:
local mo = require("init_extract")

local shared_data = ngx.shared.shared_data
shared_data:set("instance", instance)

3 content_by_lua:
local shared_data = ngx.shared.shared_data
local instance = shared_data:get("instance")
local feature = mo.extract(instance, content)

when I restart nginx, I observed that each work process as well as the mater process consumed equal memory as above，I wondering that if it really provide the advantage of reducing memory use.
And when I access it, nginx collapsed reported:
worker process XXXXX exited on signal 11...

Any one has any suggestions?

.