Memory not released with double pointers (FFI)

kaspersky_us · 2016-03-14T08:53:39+00:00

On Wed, Mar 16, 2016 at 12:28 PM Yichun Zhang (agentzh) <age...@gmail.com> wrote:Hello! On Sun, Mar 13, 2016 at 5:52 AM, David Birdsong wrote: >...

Memory not released with double pointers (FFI)

kaspersky_us

Greetings,

I have a memory problem integrating C++ code into Openresty through FFI. I isolated the issue to the use of double pointers. Lua calls the following function (wrapped in C) with amount = 10000000:

void allocate(int amount) const {

int** p = allocate_data(amount);

free_data(p, amount);

}

The corresponding functions are:

int** allocate_data(int size) {

int** p = new int*[size];

for (int i = 0; i < size; i++) {

p[i] = new int(-1);

}

return p;

}

void free_data(int** p, int size) {

for (int i = 0; i < size; i++) {

delete p[i];

}

delete [] p;

}

If I run apache bench for 10 seconds with 10 connections (ab -t 10 -c 10 "http://localhost:8077/allocate"), it will use ~1GB of memory, but will not release it after completion.

Note that with single pointers, this issue doesn't occur.

I'm using openresty/1.9.7.3 and the compiler is g++ (GCC) 5.3.0 on Ubuntu 14.0.4 LTS.

A minimum functional example is attached.

Thanks for any explanation you could provide.

worker_processes auto;
events {
    worker_connections 4000;
    use epoll;
    multi_accept on;
}
daemon off;

error_log logs/error.log;
http {
    server {
        access_log off;
        listen 8077;
        location /allocate {
            content_by_lua '
                local ffi_allocate = require "ffi_allocate";
                lib_allocate(100000000)
                ngx.say("{}")
            ';
        }
    }
}

Attachment: Makefile
Description: Binary data

#include <cstddef>


int** allocate_data(int size) {

    int** p = new int*[size];
    for (int i = 0; i < size; i++) {
        p[i] = new int(-1);
    }
    return p;

}

void free_data(int** p, int size) {
    for (int i = 0; i < size; i++) {
        delete p[i];
    }
    delete [] p;
}

class CAllocate {
public:

    CAllocate() {

    }

    virtual ~CAllocate() {

    }
public:

    void allocate(int amount) const {
        int** p = allocate_data(amount);
        free_data(p, amount);

    }

};

extern "C" void* instantiate() {

    CAllocate* instance = new CAllocate();
    return instance;
}

extern "C" void release(void* instance) {
    if (instance == NULL) {
        return;
    }
    delete (CAllocate*) instance;

}

extern "C" void allocate(void* instance, int amount) {
    if (instance == NULL) {
        return;
    }
    ((CAllocate*) instance)->allocate(amount);
}

Attachment: ffi_allocate.lua
Description: Binary data

kaspersky_us

One important additional information: the amount of memory that is not released varies linearly with the number of workers. In this example, 20 workers were used. If less workers are started by nginx, less memory is used. Workers seem to be holding on to something.

On Sunday, March 13, 2016 at 5:53:39 PM UTC-7, kasper...@yahoo.com wrote:

Greetings,

I have a memory problem integrating C++ code into Openresty through FFI. I isolated the issue to the use of double pointers. Lua calls the following function (wrapped in C) with amount = 10000000:

void allocate(int amount) const {
int** p = allocate_data(amount);
free_data(p, amount);
}

The corresponding functions are:

int** allocate_data(int size) {
int** p = new int*[size];
for (int i = 0; i < size; i++) {
p[i] = new int(-1);
}
return p;
}

void free_data(int** p, int size) {
for (int i = 0; i < size; i++) {
delete p[i];
}
delete [] p;
}

If I run apache bench for 10 seconds with 10 connections (ab -t 10 -c 10 "http://localhost:8077/allocate"), it will use ~1GB of memory, but will not release it after completion.

Note that with single pointers, this issue doesn't occur.

I'm using openresty/1.9.7.3 and the compiler is g++ (GCC) 5.3.0 on Ubuntu 14.0.4 LTS.

A minimum functional example is attached.

Thanks for any explanation you could provide.

agentzh

Hello!

On Sun, Mar 13, 2016 at 5:53 PM, kaspersky_us wrote:
> I have a memory problem integrating C++ code into Openresty through FFI. I
> isolated the issue to the use of double pointers. Lua calls the following
> function (wrapped in C) with amount = 10000000:
>

It seems that you allocate a CAllocate object in your Lua code but
there is nothing who is responsible for freeing it. Thus an obvious
memory leak.

FFI programming is essentially like C programming. No hand-holding
here. And I don't think it's interesting for us to debug issues in
your own code for you :)

Regards,
-agentzh

kaspersky_us

Hello agentzh,

First thanks for being so generous with your time, it's amazing work! Please be sure that there is no expectation of hand-holding, I wrote the code example from scratch to illustrate an issue we are facing. Was trying to be very specific with a complete example.

With that said, as you pointed out, our example was not correct - something was missing - but that wasn't the problem.

An update example - even more minimal - is below. We identified that the issue does *not* occur on MAC OS, but does on Ubuntu - with the exact same code. Perhaps it is actually not due to the OS but to the compiler?

For Mac OS it is: Apple LLVM version 7.0.2 (clang-700.1.81)

And for Ubuntu: gcc version 5.3.0 (GCC)

c code:

#include <stdlib.h>

int** allocate_data(int size) {

int** p = (int**)malloc(sizeof(int*)*size);

for (int i = 0; i < size; i++) {

p[i] = (int*)malloc(sizeof(int));

}

return p;

}

void free_data(int** p, int size) {

for (int i = 0; i < size; i++) {

free(p[i]);

}

free(p);

}

lua module:

ffi = require("ffi")

ffi.cdef[[

int** allocate_data(int size);

void free_data(int** p, int size);

]]

function lib_allocate(a)

local module = ffi.load("Allocate",false)

local p = module.allocate_data(a);

module.free_data(p,a);

end

nginx.conf lua blocks:

init_by_lua_block {

local ffi_allocate = require "ffi_allocate";

}

location /allocate {

content_by_lua_block {

local ffi_allocate = require "ffi_allocate";

lib_allocate(50000000)

ngx.say("{}")

}

On linux, memory is not released, whereas this issue doesn't occur on Mac OS.

If you have any pointer as to what to try next, would be greatly appreciated.

- DK

On Wednesday, March 16, 2016 at 12:26:20 PM UTC-7, agentzh wrote:

Hello!

On Sun, Mar 13, 2016 at 5:53 PM, kaspersky_us wrote:
> I have a memory problem integrating C++ code into Openresty through FFI. I
> isolated the issue to the use of double pointers. Lua calls the following
> function (wrapped in C) with amount = 10000000:
>

It seems that you allocate a CAllocate object in your Lua code but
there is nothing who is responsible for freeing it. Thus an obvious
memory leak.

FFI programming is essentially like C programming. No hand-holding
here. And I don't think it's interesting for us to debug issues in
your own code for you :)

Regards,
-agentzh

kaspersky_us

Hello,

The root cause of the issue has been identified. It has nothing to do with Openresty or LuaJIT/FFI, but with certain design choices of malloc / glibc on Linux. For those who might be interested, you can check out this post (it is in regards to C++ but applicable to C as well): http://stackoverflow.com/questions/26020580/huge-stdvectorstdvector-does-not-release-all-memory-on-destruction

Thanks,

- DK

On Wednesday, March 16, 2016 at 6:05:34 PM UTC-7, kasper...@yahoo.com wrote:

Hello agentzh,

First thanks for being so generous with your time, it's amazing work! Please be sure that there is no expectation of hand-holding, I wrote the code example from scratch to illustrate an issue we are facing. Was trying to be very specific with a complete example.

With that said, as you pointed out, our example was not correct - something was missing - but that wasn't the problem.

An update example - even more minimal - is below. We identified that the issue does *not* occur on MAC OS, but does on Ubuntu - with the exact same code. Perhaps it is actually not due to the OS but to the compiler?

For Mac OS it is: Apple LLVM version 7.0.2 (clang-700.1.81)
And for Ubuntu: gcc version 5.3.0 (GCC)

c code:

#include <stdlib.h>

int** allocate_data(int size) {
int** p = (int**)malloc(sizeof(int*)*size);
for (int i = 0; i < size; i++) {
p[i] = (int*)malloc(sizeof(int));
}
return p;
}

void free_data(int** p, int size) {
for (int i = 0; i < size; i++) {
free(p[i]);
}
free(p);
}

lua module:

ffi = require("ffi")
ffi.cdef[[
int** allocate_data(int size);
void free_data(int** p, int size);
]]

function lib_allocate(a)
local module = ffi.load("Allocate",false)
local p = module.allocate_data(a);
module.free_data(p,a);
end

nginx.conf lua blocks:

init_by_lua_block {
local ffi_allocate = require "ffi_allocate";
}

location /allocate {
content_by_lua_block {
local ffi_allocate = require "ffi_allocate";
lib_allocate(50000000)
ngx.say("{}")
}
}

On linux, memory is not released, whereas this issue doesn't occur on Mac OS.

If you have any pointer as to what to try next, would be greatly appreciated.

- DK

On Wednesday, March 16, 2016 at 12:26:20 PM UTC-7, agentzh wrote:
Hello!

On Sun, Mar 13, 2016 at 5:53 PM, kaspersky_us wrote:
> I have a memory problem integrating C++ code into Openresty through FFI. I
> isolated the issue to the use of double pointers. Lua calls the following
> function (wrapped in C) with amount = 10000000:
>

It seems that you allocate a CAllocate object in your Lua code but
there is nothing who is responsible for freeing it. Thus an obvious
memory leak.

FFI programming is essentially like C programming. No hand-holding
here. And I don't think it's interesting for us to debug issues in
your own code for you :)

Regards,
-agentzh