Fwd: Issues in using Vowpal Wabbit as library with nginx lua

itspuneet59 · 2015-08-28T15:17:59+00:00

Hello! On Thu, Aug 27, 2015 at 4:34 AM, Tobin Baker wrote: > Never mind, I figured out that ngx.say() forces response headers to be sent > immediately...

Fwd: Issues in using Vowpal Wabbit as library with nginx lua

itspuneet59

Hello,

I am stuck at a problem with lua C interfacing, if someone can help me in understanding the cause, that will be a great help.

I am using ngx_openresty-1.7.7.2 to serve http requests, nginx is running with 24 worker processes. I am trying to use vowpal wabbit as library using lua C interface. So the scenario is, http requests are received on nginx, content_by_lua directive calls lua code, and that lua code calls vowpal wabbits, VW_ReadExampleA and VW_Predict API's using lua's C interface. So, normally everything works well. But when we do "nginx -s reload", sometimes the call to VW library gets stuck and never returns, because of which the process which was supposed to shut down after reload doesn't shutdown. Because of this there are lots of idle worker processes in the system and the memory footprint is increasing. Can somebody throw some light, what can be causing this?

Also, when I attach starce to the process which is stuck, I get this

futex(0x8f4b5c, FUTEX_WAIT_PRIVATE, 1, NULL

Thanks,

Puneet

agentzh

Hello!

On Fri, Aug 28, 2015 at 3:17 PM, Puneet Agarwal wrote:
> I am using ngx_openresty-1.7.7.2 to serve http requests, nginx is running
> with 24 worker processes. I am trying to use vowpal wabbit as library using
> lua C interface. So the scenario is, http requests are received on nginx,
> content_by_lua directive calls lua code, and that lua code calls vowpal
> wabbits, VW_ReadExampleA and VW_Predict API's using lua's C interface. So,
> normally everything works well. But when we do "nginx -s reload", sometimes
> the call to VW library gets stuck and never returns, because of which the
> process which was supposed to shut down after reload doesn't shutdown.
> Because of this there are lots of idle worker processes in the system and
> the memory footprint is increasing. Can somebody throw some light, what can
> be causing this?

Seems like you're loading your C library in init_by_lua (i.e., within
the nginx master process during configuration loading) and your C
library does not support overlapped reloading.

You can try loading your C library in the worker processes instead.

Well, there might be deeper issues in your C library or your own C
binding. You'd better use tools like Valgrind or Address Sanitizer to
debug it.

>
> Also, when I attach starce to the process which is stuck, I get this
> futex(0x8f4b5c, FUTEX_WAIT_PRIVATE, 1, NULL
>

This is user-space locking. Maybe some of your C code involved enters
a deadlock? You can try tools like gdb and pstack to fetch the
userspace backtrace to get more context for the hang.

Well, just my 2 cents.

Regards,
-agentzh

itspuneet59

Thanks Yichun for highlighting probable polems.

Few more question,

1. Can we require the library formed with C code for lua C interface in worker process?

I was doing it like this in the global part of the lua file.

local vowpalwabbit = require("vowpalwabbit") -- vowpalwabbit is the library name.

local x = initVW() - this initializes the library

But it does not work says can not find global initVW.

while initVW is a proper function executed in worker process

Thanks,

Puneet

On Fri, Aug 28, 2015 at 8:13 PM, Yichun Zhang (agentzh) <age...@gmail.com> wrote:

Hello!

On Fri, Aug 28, 2015 at 3:17 PM, Puneet Agarwal wrote:
> I am using ngx_openresty-1.7.7.2 to serve http requests, nginx is running
> with 24 worker processes. I am trying to use vowpal wabbit as library using
> lua C interface. So the scenario is, http requests are received on nginx,
> content_by_lua directive calls lua code, and that lua code calls vowpal
> wabbits, VW_ReadExampleA and VW_Predict API's using lua's C interface. So,
> normally everything works well. But when we do "nginx -s reload", sometimes
> the call to VW library gets stuck and never returns, because of which the
> process which was supposed to shut down after reload doesn't shutdown.
> Because of this there are lots of idle worker processes in the system and
> the memory footprint is increasing. Can somebody throw some light, what can
> be causing this?

Seems like you're loading your C library in init_by_lua (i.e., within
the nginx master process during configuration loading) and your C
library does not support overlapped reloading.

You can try loading your C library in the worker processes instead.

Well, there might be deeper issues in your C library or your own C
binding. You'd better use tools like Valgrind or Address Sanitizer to
debug it.

>
> Also, when I attach starce to the process which is stuck, I get this
> futex(0x8f4b5c, FUTEX_WAIT_PRIVATE, 1, NULL
>

This is user-space locking. Maybe some of your C code involved enters
a deadlock? You can try tools like gdb and pstack to fetch the
userspace backtrace to get more context for the hang.

Well, just my 2 cents.

Regards,
-agentzh
.

Puneet Agarwal
Software Engineer
Samsung India Software Operations, Bangalore

aapo.talvensaari

On Monday, 31 August 2015 11:11:23 UTC+3, itspuneet59 wrote:

1. Can we require the library formed with C code for lua C interface in worker process?

I was doing it like this in the global part of the lua file.
local vowpalwabbit = require("vowpalwabbit") -- vowpalwabbit is the library name.
local x = initVW() - this initializes the library

But it does not work says can not find global initVW.

In general it is ok. Just remember that the library does not do blocking.

E.g. file access or network access. For network you can utilize ngx.tcp instead.

If it does blocking, it will work, but will affect scalability of OpenResty, aka blocking

the worker from processing other requests.

I think that your problem is that you have not defined initVW before you used it. In Lua

the function definitions need to be before you use them (even on the same file). Just

make sure you have:

local function initVW(...) ... end

Before you are actually calling it.

agentzh

Hello!

On Mon, Aug 31, 2015 at 4:11 PM, Puneet Agarwal wrote:
> local vowpalwabbit = require("vowpalwabbit") -- vowpalwabbit is the library
> name.
> local x = initVW()  - this initializes the library
>
> But it does not work says can not find global initVW.

The syntax is wrong. Generally, using a Lua module requires the
following syntax:

    local foo = require "foo"
    -- somewhere later on
    foo.do_something()

Never never use Lua globals and never never use module() to define
your own modules. See

    https://github.com/openresty/lua-nginx-module#lua-variable-scope

Otherwise you are doomed to get such "nil value" errors while reading
nonexisitent Lua globals.

Check out the official lua-resty-* libraries (like
lua-resty-memcached) for how to define Lua modules properly in the
real world.

Regards,
-agentzh