C/C++ code from openresty

kaspersky_us · 2016-02-08T04:53:54+00:00

Hello! On Fri, Feb 5, 2016 at 11:05 AM, Jim Robinson wrote: > What tools do developers here use for profiling their OpenResty based > services? The S...

C/C++ code from openresty

kaspersky_us

Hello,

I'm trying to use an existing C++ piece of code with openresty, and was able to make it work by transforming it into a .so library with a C wrapper. So far so good. The code uses large data structure (around 5GB) and successfully integrated with openresty, as follows:

init_by_lua '

-- FFI module import

ffi = require("ffi")

-- Declaration of the C functions to be imported

ffi.cdef[[

void* ccode_instantiate(const char* xml);

void ccode_release(void* instance);

char* ccode_execute(void* instance,const char* text,int32_t limit,uint32_t range);

void free(void *ptr);

]]

-- Load the module globally

ffi.load("ccode",true)

-- Creation of the global class instance

instance = ffi.C.ccode_instantiate("${CCODE_HOME}/ccode.xml")

content_by_lua '

local args = ngx.req.get_uri_args()

local ref = args["ref"] -- ref#

if not ref then

return ngx.exit(ngx.HTTP_BAD_REQUEST)

end

local l = tonumber((args["l"] or "10")) -- limit

local r = tonumber((args["r"] or "1")) -- range

local json = ffi.C.ccode_execute(instance,ref,l,r)

ngx.say(ffi.string(json))

ffi.C.free(json) -- Frees the C string

Upon initialization, an XML is read, and external files are read to build the data structure in memory. This all works well, except I made one obvious design mistake apparently: the solution above is single threaded, because the main executable is global, and the workers just call it and have to wait for the current request to complete because requests are blocking. Obviously not what I want.

I think I know the solution, but wanted to confirm before I start refactoring, and I have a a few questions.

1. Can someone confirm that to fix this, I need to separate the data structure from the execution and have my C data structure instantiated globally, then call the executable from the workers that will leverage the global data structure?

2. What is the right way to implement a C++ (with C wrapper) data structure as a global resource (LUA tables are not going to work) to be called by C executables in workers?

3. Is this the best way to do this? Performance is of the essence so I am looking for the fastest performing solution.

4. I considered adding something like boost.asio to my code and leave it as a global resource (as it is now), but seems to me that this would defeat the purpose of leveraging openresty and my C++ code would then be pretty much self sufficient to service requests and could be used standalone, yet would lack the optimizations put in openresty as well as other facilities it provides.

Thanks for your help!

kaspersky_us

One more (important) detail: the global data structure is read-only, so there is no concurrency issues.

On Sunday, February 7, 2016 at 12:53:54 PM UTC-8, kasper...@yahoo.com wrote:

Hello,

I'm trying to use an existing C++ piece of code with openresty, and was able to make it work by transforming it into a .so library with a C wrapper. So far so good. The code uses large data structure (around 5GB) and successfully integrated with openresty, as follows:

init_by_lua '
  -- FFI module import
  ffi = require("ffi")
  -- Declaration of the C functions to be imported
  ffi.cdef[[
      void* ccode_instantiate(const char* xml);
      void ccode_release(void* instance);
      char* ccode_execute(void* instance,const char* text,int32_t limit,uint32_t range);
      void free(void *ptr);
  ]]
  -- Load the module globally
  ffi.load("ccode",true)
  -- Creation of the global class instance
  instance = ffi.C.ccode_instantiate("${CCODE_HOME}/ccode.xml")
';

content_by_lua '
  local args = ngx.req.get_uri_args()
  local ref = args["ref"] -- ref#
  if not ref then
      return ngx.exit(ngx.HTTP_BAD_REQUEST)
  end
  local l = tonumber((args["l"] or "10")) -- limit
  local r = tonumber((args["r"] or "1")) -- range
  local json = ffi.C.ccode_execute(instance,ref,l,r)
  ngx.say(ffi.string(json))
  ffi.C.free(json) -- Frees the C string
';

Upon initialization, an XML is read, and external files are read to build the data structure in memory. This all works well, except I made one obvious design mistake apparently: the solution above is single threaded, because the main executable is global, and the workers just call it and have to wait for the current request to complete because requests are blocking. Obviously not what I want.

I think I know the solution, but wanted to confirm before I start refactoring, and I have a a few questions.

1. Can someone confirm that to fix this, I need to separate the data structure from the execution and have my C data structure instantiated globally, then call the executable from the workers that will leverage the global data structure?

2. What is the right way to implement a C++ (with C wrapper) data structure as a global resource (LUA tables are not going to work) to be called by C executables in workers?

3. Is this the best way to do this? Performance is of the essence so I am looking for the fastest performing solution.

4. I considered adding something like boost.asio to my code and leave it as a global resource (as it is now), but seems to me that this would defeat the purpose of leveraging openresty and my C++ code would then be pretty much self sufficient to service requests and could be used standalone, yet would lack the optimizations put in openresty as well as other facilities it provides.

Thanks for your help!

agentzh

Hello!

On Sun, Feb 7, 2016 at 12:53 PM, kaspersky_us via openresty-en wrote:
> Upon initialization, an XML is read, and external files are read to build
> the data structure in memory. This all works well, except I made one obvious
> design mistake apparently: the solution above is single threaded, because
> the main executable is global, and the workers just call it and have to wait
> for the current request to complete because requests are blocking.

NGINX worker processes work independently (usually). Is your C++ call
blocking on I/O or CPU computations?

> 1. Can someone confirm that to fix this, I need to separate the data
> structure from the execution and have my C data structure instantiated
> globally, then call the executable from the workers that will leverage the
> global data structure?
>

Loading a (read-only) data structure in init_by_lua* can give a chance
to allow the operating system's Copy-On-Write (COW) feature to share a
lot of physical memory pages among all the NGINX worker processes. So
your usage should be memory efficient.

> 2. What is the right way to implement a C++ (with C wrapper) data structure
> as a global resource (LUA tables are not going to work) to be called by C
> executables in workers?
>

The recommended way is to wrap your FFI details in a proper Lua module
(.lua file) and then require() it in init_by_lua* so that COW can take
effect. And later in your request-time handlers (like
content_by_lua*), just require() that module again and call its Lua
functions (which result in C++ calls eventually).

This works because require() caches already loaded Lua modules on the
Lua VM level (i.e., the NGINX worker process level, if lua_code_cache
is not turned off).

> 3. Is this the best way to do this? Performance is of the essence so I am
> looking for the fastest performing solution.
>

I suggest you always sample your busy NGINX worker processes and plot
some flame graphs to analyze:

    https://openresty.org/#Profiling

> 4. I considered adding something like boost.asio to my code and leave it as
> a global resource (as it is now), but seems to me that this would defeat the
> purpose of leveraging openresty and my C++ code would then be pretty much
> self sufficient to service requests and could be used standalone, yet would
> lack the optimizations put in openresty as well as other facilities it
> provides.
>

Boost.Asio seems to have its own I/O scheduler and event loop which
will block NGINX's event loop unless it's running in separate OS
threads (which is a usually waste of system resources and adds
complexity).

It's recommended to delegate your network I/O to ngx_lua's cosocket
API and make your own C/C++ code focus on CPU-intensive operations
only.

Regards,
-agentzh

kaspersky_us

Thanks agentzh!

My C++ code is blocking on I/O, thus I essentially built a single threaded solution. I need to separate my (complex and large) data structure from the execution logic.

Based on your reply, you confirmed that COW should allow implicit sharing of the data, and thus will ensure memory efficiency.

One thing I didn't know, is the need to wrap the FFI details in a proper Lua module (.lua file) and then require() it in both init_by_lua and request time handlers. Thanks for mentioning.

On Monday, February 8, 2016 at 11:20:56 AM UTC-8, agentzh wrote:

Hello!

On Sun, Feb 7, 2016 at 12:53 PM, kaspersky_us via openresty-en wrote:
> Upon initialization, an XML is read, and external files are read to build
> the data structure in memory. This all works well, except I made one obvious
> design mistake apparently: the solution above is single threaded, because
> the main executable is global, and the workers just call it and have to wait
> for the current request to complete because requests are blocking.

NGINX worker processes work independently (usually). Is your C++ call
blocking on I/O or CPU computations?

> 1. Can someone confirm that to fix this, I need to separate the data
> structure from the execution and have my C data structure instantiated
> globally, then call the executable from the workers that will leverage the
> global data structure?
>

Loading a (read-only) data structure in init_by_lua* can give a chance
to allow the operating system's Copy-On-Write (COW) feature to share a
lot of physical memory pages among all the NGINX worker processes. So
your usage should be memory efficient.

> 2. What is the right way to implement a C++ (with C wrapper) data structure
> as a global resource (LUA tables are not going to work) to be called by C
> executables in workers?
>

The recommended way is to wrap your FFI details in a proper Lua module
(.lua file) and then require() it in init_by_lua* so that COW can take
effect. And later in your request-time handlers (like
content_by_lua*), just require() that module again and call its Lua
functions (which result in C++ calls eventually).

This works because require() caches already loaded Lua modules on the
Lua VM level (i.e., the NGINX worker process level, if lua_code_cache
is not turned off).

> 3. Is this the best way to do this? Performance is of the essence so I am
> looking for the fastest performing solution.
>

I suggest you always sample your busy NGINX worker processes and plot
some flame graphs to analyze:

https://openresty.org/#Profiling

> 4. I considered adding something like boost.asio to my code and leave it as
> a global resource (as it is now), but seems to me that this would defeat the
> purpose of leveraging openresty and my C++ code would then be pretty much
> self sufficient to service requests and could be used standalone, yet would
> lack the optimizations put in openresty as well as other facilities it
> provides.
>

Boost.Asio seems to have its own I/O scheduler and event loop which
will block NGINX's event loop unless it's running in separate OS
threads (which is a usually waste of system resources and adds
complexity).

It's recommended to delegate your network I/O to ngx_lua's cosocket
API and make your own C/C++ code focus on CPU-intensive operations
only.

Regards,
-agentzh

kaspersky_us

Dear agentzh,

I followed your advice and my solution still uses a single core on my machine. Below is my config. I reiterate that the underlying C++ code is I/O blocking however my understanding is that what is below will enable COW and workers can execute their individual process (call to the cat function).

A few additional details, the nginx config is setup with "worker_processes auto;" and "worker_connections 4000;" and can take advantage of all CPU cores if a static file is called.

Appreciate any help you can provide.

-- begin mod.lua:

-- FFI module import

ffi = require("ffi")

-- Declaration of the C functions to be imported

ffi.cdef[[

void* ccode_instantiate(const char* xml);

void ccode_release(void* instance);

char* ccode_execute(void* instance,const char* text,int32_t limit,uint32_t range);

void free(void *ptr);

]]

-- Load the module globally

ffi.load("ccode",true)

-- Creation of the global class instance

instance = ffi.C.ccode_instantiate("${CCODE_HOME}/ccode.xml")

function cta(ref,l,r)

local json = ffi.C.ccode_execute(instance,ref,l,r)

return ffi.string(json)

-- ffi.C.free(json)

end

-- end mod.lua

init_by_lua '

local mod = require "mod";

content_by_lua '

local mod = require "mod";

local args = ngx.req.get_uri_args()

local ref = args["ref"] -- Query

if not ref then

return ngx.exit(ngx.HTTP_BAD_REQUEST)

end

local l = tonumber((args["l"] or "10")) -- limit

local r = tonumber((args["r"] or "1")) -- range

ngx.say(cta(ref,l,r))

On Monday, February 8, 2016 at 3:23:54 PM UTC-8, kasper...@yahoo.com wrote:

Thanks agentzh!

My C++ code is blocking on I/O, thus I essentially built a single threaded solution. I need to separate my (complex and large) data structure from the execution logic.

Based on your reply, you confirmed that COW should allow implicit sharing of the data, and thus will ensure memory efficiency.

One thing I didn't know, is the need to wrap the FFI details in a proper Lua module (.lua file) and then require() it in both init_by_lua and request time handlers. Thanks for mentioning.

On Monday, February 8, 2016 at 11:20:56 AM UTC-8, agentzh wrote:
Hello!

On Sun, Feb 7, 2016 at 12:53 PM, kaspersky_us via openresty-en wrote:
> Upon initialization, an XML is read, and external files are read to build
> the data structure in memory. This all works well, except I made one obvious
> design mistake apparently: the solution above is single threaded, because
> the main executable is global, and the workers just call it and have to wait
> for the current request to complete because requests are blocking.

NGINX worker processes work independently (usually). Is your C++ call
blocking on I/O or CPU computations?

> 1. Can someone confirm that to fix this, I need to separate the data
> structure from the execution and have my C data structure instantiated
> globally, then call the executable from the workers that will leverage the
> global data structure?
>

Loading a (read-only) data structure in init_by_lua* can give a chance
to allow the operating system's Copy-On-Write (COW) feature to share a
lot of physical memory pages among all the NGINX worker processes. So
your usage should be memory efficient.

> 2. What is the right way to implement a C++ (with C wrapper) data structure
> as a global resource (LUA tables are not going to work) to be called by C
> executables in workers?
>

The recommended way is to wrap your FFI details in a proper Lua module
(.lua file) and then require() it in init_by_lua* so that COW can take
effect. And later in your request-time handlers (like
content_by_lua*), just require() that module again and call its Lua
functions (which result in C++ calls eventually).

This works because require() caches already loaded Lua modules on the
Lua VM level (i.e., the NGINX worker process level, if lua_code_cache
is not turned off).

> 3. Is this the best way to do this? Performance is of the essence so I am
> looking for the fastest performing solution.
>

I suggest you always sample your busy NGINX worker processes and plot
some flame graphs to analyze:

https://openresty.org/#Profiling

> 4. I considered adding something like boost.asio to my code and leave it as
> a global resource (as it is now), but seems to me that this would defeat the
> purpose of leveraging openresty and my C++ code would then be pretty much
> self sufficient to service requests and could be used standalone, yet would
> lack the optimizations put in openresty as well as other facilities it
> provides.
>

Boost.Asio seems to have its own I/O scheduler and event loop which
will block NGINX's event loop unless it's running in separate OS
threads (which is a usually waste of system resources and adds
complexity).

It's recommended to delegate your network I/O to ngx_lua's cosocket
API and make your own C/C++ code focus on CPU-intensive operations
only.

Regards,
-agentzh

agentzh

Hello!

On Tue, Feb 9, 2016 at 1:41 AM, kaspersky_us wrote:
> I followed your advice and my solution still uses a single core on my
> machine. Below is my config. I reiterate that the underlying C++ code is I/O
> blocking however my understanding is that what is below will enable COW and
> workers can execute their individual process (call to the cat function).
>

My suggestions:

1. Use the ps command to check how many nginx worker processes do you
really have. If you only have only one worker, then only one CPU core
can be utilized.

2. Use the off-CPU flame graph too to analyze what is blocking your
nginx workers (guessing it's a huge lock in your C++ stack):

    https://openresty.org/#Profiling

3. Put the line "accept_mutex off;" to your events {} block in
nginx.conf so that incoming concurrent requests can distribute more
evenly across all your nginx worker processes.

4. Ensure you have enough request concurrency level. Use of "ab -k -c
500" can be good, for example. If your client requests' concurrency
level is small enough, then you can only keep one worker busy anyway.

Best regards,
-agentzh