LuaJIT FFI restart issues

lordnynex · 2015-04-11T03:32:07+00:00

You could also try http://nginx.org/en/docs/http/ngx_http_empty_gif_module.html from the standard nginx distribution. I think it's enabled by default in...

LuaJIT FFI restart issues

lordnynex

Hello,

I've implemented ffi bindings around libbloom (https://github.com/jvirkki/libbloom).

Theres a lot of problems with the implementation because the init function allocates a small amount of memory and later in the execution I call bloom_free().

Since implementing this, if I restart nginx, it fails to restart the first time and I have to call nginx start to make sure the server comes back up.

I'm assuming this is because many instances of the bloom object are never free'd?

My long term plan is to modify the library to handle memory allocation from FFI directly, but I don't have time to do it right now.

Is this normal behavior? Is there a simple work around?

ffi.cdef[[

typedef struct {

int entries;

double error;

int bits;

int bytes;

int hashes;

double bpe;

unsigned char * bf;

int ready;

} bloom_t;

int bloom_init(bloom_t * bloom, int entries, double error);

int bloom_check(bloom_t * bloom, const void * buffer, int len);

int bloom_add(bloom_t * bloom, const void * buffer, int len);

void bloom_print(bloom_t * bloom);

void bloom_free(bloom_t * bloom);

]]

lib = ffi.load("libbloom.so")

self:filterObj = ffi.new("bloom_t")

lib.bloom_init(self:filterObj, ...)

...

lib.bloom_free(self:filterObj)

jon.keys

On Friday, April 10, 2015 at 3:32:08 PM UTC-4, Lord Nynex wrote:

Hello,

I've implemented ffi bindings around libbloom (https://github.com/jvirkki/libbloom).

Theres a lot of problems with the implementation because the init function allocates a small amount of memory and later in the execution I call bloom_free().

Since implementing this, if I restart nginx, it fails to restart the first time and I have to call nginx start to make sure the server comes back up.

I'm assuming this is because many instances of the bloom object are never free'd?

Do you see this same restart failure behavior if you don't use the bloom stuff?

My long term plan is to modify the library to handle memory allocation from FFI directly, but I don't have time to do it right now.

You could let lua manage it for you using ffi.gc

Is this normal behavior? Is there a simple work around?

ffi.cdef[[
typedef struct {
int entries;
double error;
int bits;
int bytes;
int hashes;

double bpe;
unsigned char * bf;
int ready;
} bloom_t;

int bloom_init(bloom_t * bloom, int entries, double error);
int bloom_check(bloom_t * bloom, const void * buffer, int len);
int bloom_add(bloom_t * bloom, const void * buffer, int len);
void bloom_print(bloom_t * bloom);
void bloom_free(bloom_t * bloom);
]]

lib = ffi.load("libbloom.so")

self:filterObj = ffi.new("bloom_t")
lib.bloom_init(self:filterObj, ...)

...

lib.bloom_free(self:filterObj)

lordnynex

Thanks for the reply! Foreword: I am completely new to FFI. I understand whats going on reasonably well but am still learning some of the pitfalls.

No I do not see a restart failure during restart without the bloom filters.

I've read about ffi.gc probably about fifty times and I guess I'm still confused. All the examples I look at seem to call free on an object. In my case, libbloom provides an init/destroy method that calls calloc/free respectively. Will it work as intended to set the GC function to this destructor? If I can, this would clean up a lot of explicit calls to destroy.

Is GC called on restart (What I'm assuming is SIG_INT)? If so, it seems like this would be helpful.

On Thu, Apr 16, 2015 at 12:03 PM, Jon Keys <jon...@gmail.com> wrote:

On Friday, April 10, 2015 at 3:32:08 PM UTC-4, Lord Nynex wrote:
Hello,

I've implemented ffi bindings around libbloom (https://github.com/jvirkki/libbloom).

Theres a lot of problems with the implementation because the init function allocates a small amount of memory and later in the execution I call bloom_free().

Since implementing this, if I restart nginx, it fails to restart the first time and I have to call nginx start to make sure the server comes back up.

I'm assuming this is because many instances of the bloom object are never free'd?

Do you see this same restart failure behavior if you don't use the bloom stuff?

My long term plan is to modify the library to handle memory allocation from FFI directly, but I don't have time to do it right now.

You could let lua manage it for you using ffi.gc

Is this normal behavior? Is there a simple work around?

ffi.cdef[[
typedef struct {
int entries;
double error;
int bits;
int bytes;
int hashes;

double bpe;
unsigned char * bf;
int ready;
} bloom_t;

int bloom_init(bloom_t * bloom, int entries, double error);
int bloom_check(bloom_t * bloom, const void * buffer, int len);
int bloom_add(bloom_t * bloom, const void * buffer, int len);
void bloom_print(bloom_t * bloom);
void bloom_free(bloom_t * bloom);
]]

lib = ffi.load("libbloom.so")

self:filterObj = ffi.new("bloom_t")
lib.bloom_init(self:filterObj, ...)

...

lib.bloom_free(self:filterObj)

.

jon.keys

On Thursday, April 16, 2015 at 3:17:27 PM UTC-4, Lord Nynex wrote:

Thanks for the reply! Foreword: I am completely new to FFI. I understand whats going on reasonably well but am still learning some of the pitfalls.

No I do not see a restart failure during restart without the bloom filters.

I've read about ffi.gc probably about fifty times and I guess I'm still confused. All the examples I look at seem to call free on an object. In my case, libbloom provides an init/destroy method that calls calloc/free respectively. Will it work as intended to set the GC function to this destructor? If I can, this would clean up a lot of explicit calls to destroy.

Yes, that's what it is meant to do.

Based on your code above it will look something like:

ffi.gc(filterObj, ffi.C.bloom_free)

I used the same method in our hmac lib and it is working quite well for us: https://github.com/jkeys089/lua-resty-hmac/blob/master/lib/resty/hmac.lua#L101

Is GC called on restart (What I'm assuming is SIG_INT)? If so, it seems like this would be helpful.

Your finalizer will be called by the lua garbage collector after the object is no longer referenced.

Off the top of my head I don't actually know how OpenResty handles a restart with regards to the lua vm (i.e. does it gracefully close the lua vm which would, in turn, call the finalizer). I think it should be pretty easy to run a quick test just to see if your finalizer gets called on restart.

On Thu, Apr 16, 2015 at 12:03 PM, Jon Keys <jon....@gmail.com> wrote:

On Friday, April 10, 2015 at 3:32:08 PM UTC-4, Lord Nynex wrote:
Hello,

I've implemented ffi bindings around libbloom (https://github.com/jvirkki/libbloom).

Theres a lot of problems with the implementation because the init function allocates a small amount of memory and later in the execution I call bloom_free().

Since implementing this, if I restart nginx, it fails to restart the first time and I have to call nginx start to make sure the server comes back up.

I'm assuming this is because many instances of the bloom object are never free'd?

Do you see this same restart failure behavior if you don't use the bloom stuff?

My long term plan is to modify the library to handle memory allocation from FFI directly, but I don't have time to do it right now.

You could let lua manage it for you using ffi.gc

Is this normal behavior? Is there a simple work around?

ffi.cdef[[
typedef struct {
int entries;
double error;
int bits;
int bytes;
int hashes;

double bpe;
unsigned char * bf;
int ready;
} bloom_t;

int bloom_init(bloom_t * bloom, int entries, double error);
int bloom_check(bloom_t * bloom, const void * buffer, int len);
int bloom_add(bloom_t * bloom, const void * buffer, int len);
void bloom_print(bloom_t * bloom);
void bloom_free(bloom_t * bloom);
]]

lib = ffi.load("libbloom.so")

self:filterObj = ffi.new("bloom_t")
lib.bloom_init(self:filterObj, ...)

...

lib.bloom_free(self:filterObj)

.