Sending huge amounts of data through ngx_http_lua_send_chain_link

rick.richardson · 2015-09-02T02:21:56+00:00

Thank you very much!在 2015年9月2日星期三 UTC+8上午11:09:07，agentzh写道：Hello...

Sending huge amounts of data through ngx_http_lua_send_chain_link

rick.richardson

Hey all. I am writing an app that streams huge chunks of pcaps down http. I am using this function (https://gist.github.com/rrichardson/577b0e0977d05ebe22f4) in a loop.
The loop is iterating through mmap'd chunks of captured packet data, and copying them into 10k buffers which I then pass to write_to_ngx.

What happens is the outbound socket starts to fill up, so writev fails to write the complete amount:

2015/09/01 13:55:25 [debug] 31686#0: *1 writev: 1898 of 9380

2015/09/01 13:55:25 [debug] 31686#0: *1 http write filter 000000000133B6B0

2015/09/01 13:55:25 [debug] 31686#0: *1 http copy filter: -2 "/pcap/?"

2015/09/01 13:55:25 [debug] 31686#0: *1 lua reuse free buf chain, but reallocate memory because 9400 >= 24, cl:000000000133B230, p:000000000133B290

2015/09/01 13:55:25 [debug] 31686#0: *1 malloc: 000000000139C090:9400

2015/09/01 13:55:25 [debug] 31686#0: *1 lua write response

2015/09/01 13:55:25 [debug] 31686#0: *1 http output filter "/pcap/?"

2015/09/01 13:55:25 [debug] 31686#0: *1 http copy filter: "/pcap/?"

2015/09/01 13:55:25 [debug] 31686#0: *1 lua capture body filter, uri "/pcap/"

2015/09/01 13:55:25 [debug] 31686#0: *1 http postpone filter "/pcap/?" 000000000133B230

2015/09/01 13:55:25 [debug] 31686#0: *1 http chunk: 9400

2015/09/01 13:55:25 [debug] 31686#0: *1 write old buf t:1 f:0 0000000001399880, pos 0000000001399FE4, size: 7480 file: 0, size: 0

2015/09/01 13:55:25 [debug] 31686#0: *1 write old buf t:0 f:0 000000000133B6C0, pos 00000000004FAD27, size: 2 file: 0, size: 0

2015/09/01 13:55:25 [debug] 31686#0: *1 write new buf t:1 f:0 000000000133B618, pos 000000000133B618, size: 6 file: 0, size: 0

2015/09/01 13:55:25 [debug] 31686#0: *1 write new buf t:1 f:0 000000000139C090, pos 000000000139C090, size: 9400 file: 0, size: 0

2015/09/01 13:55:25 [debug] 31686#0: *1 write new buf t:0 f:0 0000000000000000, pos 00000000004FAD27, size: 2 file: 0, size: 0

After that, every time I write the buffers keep accumulating and it doesn't seem to attempt a writev again (at least not in my logs). The download just hangs.

I guess some code somewhere should do the polite thing and let the socket clear out before attempting more writes, but I'm not certain about the best way to go about this.

Is there a way to simply deschedule my lua handler and reschedule it later after the socket is cleared?

If there is no way to do that, can I just sleep when I get back a can't-write response from send_chain_link?

Thanks in advance,

Rick

agentzh

Hello!

On Wed, Sep 2, 2015 at 2:21 AM, Rick Richardson wrote:
> After that, every time I write the buffers keep accumulating and it doesn't
> seem to attempt a writev again (at least not in my logs).  The download just
> hangs.
>

That's why the ngx_lua core registers a write event handler on the
downstream connection to flush the send buffers again when there is a
write event.

I guess you're not using ngx_lua core's schedule.

Note that sending data on the nginx C land is asynchronous.

> I guess some code somewhere should do the polite thing and let the socket
> clear out before attempting more writes, but I'm not certain about the best
> way to go about this.
>

Right. Essentially it's like registering r->write_event_handler to our
own callback and then call ngx_http_output_filter() (and
ngx_chain_update_chains etc) with a NULL chain link argument when you
run out of busy output buffers. Well, it's a little bit complicated,
and the ngx_lua's code should be consulted anyway.

Well, I must say, calling ngx_lua's I/O primitives outside ngx_lua's
I/O scheduler is generally a bad idea and is subject to issues.

> Is there a way to simply deschedule my lua handler and reschedule it later
> after the socket is cleared?
>

The simplest way is just to use ngx_lua's I/O scheduler instead of
doing "monkey-patching".

> If there is no way to do that, can I just sleep when I get back a
> can't-write response from send_chain_link?
>

Sleeping is bad practice for nonblocking I/O scheduling in an event
driven world since it will introduce unnecessary delays :) We are
already using I/O multiplexing like epoll/kqueue/etc; why not just use
it? ;)

Regards,
-agentzh

rick.richardson

Thanks for your reply. And thanks for confirming what I kind of already understood. Apologies for the dumb questions.

So if I were to do two things in a loop in my lua handler: 1. Fetch a blob of data (in a ffi.C call) then 2. write it to ngx via something like ngx.say(), that should work?

I asked about this approach in this thread : https://groups.google.com/forum/#!searchin/openresty-en/rick$20richardson/openresty-en/7Bl21a9X4jQ/mpHFoDYBCAAJ

But the problem is I need to be able to pass to ngx_http_lua_ngx_echo an array of binary data, which ngx.print and ngx.say don't currently use.
I wrote something like it here: https://gist.github.com/rrichardson/57540f275cd0f03ee8d6

Would that be a reasonable approach?

Thanks again.

On Tuesday, September 1, 2015 at 10:50:16 PM UTC-4, agentzh wrote:

Hello!

On Wed, Sep 2, 2015 at 2:21 AM, Rick Richardson wrote:
> After that, every time I write the buffers keep accumulating and it doesn't
> seem to attempt a writev again (at least not in my logs). The download just
> hangs.
>

That's why the ngx_lua core registers a write event handler on the
downstream connection to flush the send buffers again when there is a
write event.

I guess you're not using ngx_lua core's schedule.

Note that sending data on the nginx C land is asynchronous.

> I guess some code somewhere should do the polite thing and let the socket
> clear out before attempting more writes, but I'm not certain about the best
> way to go about this.
>

Right. Essentially it's like registering r->write_event_handler to our
own callback and then call ngx_http_output_filter() (and
ngx_chain_update_chains etc) with a NULL chain link argument when you
run out of busy output buffers. Well, it's a little bit complicated,
and the ngx_lua's code should be consulted anyway.

Well, I must say, calling ngx_lua's I/O primitives outside ngx_lua's
I/O scheduler is generally a bad idea and is subject to issues.

> Is there a way to simply deschedule my lua handler and reschedule it later
> after the socket is cleared?
>

The simplest way is just to use ngx_lua's I/O scheduler instead of
doing "monkey-patching".

> If there is no way to do that, can I just sleep when I get back a
> can't-write response from send_chain_link?
>

Sleeping is bad practice for nonblocking I/O scheduling in an event
driven world since it will introduce unnecessary delays :) We are
already using I/O multiplexing like epoll/kqueue/etc; why not just use
it? ;)

Regards,
-agentzh

agentzh

Hello!

On Wed, Sep 2, 2015 at 11:17 PM, Rick Richardson wrote:
> Thanks for your reply. And thanks for confirming what I kind of already
> understood.  Apologies for the dumb questions.
>
> So if I were to do two things in a loop in my lua handler: 1. Fetch a blob
> of data (in a ffi.C call) then 2. write it to ngx via something like
> ngx.say(), that should work?
>

If I understand your intentions correctly, I think we could just
expose a variant of the ngx.print() API that accepts raw C buffers in
the form of FFI cdata. This way we can avoid allocating new
(temporary) Lua strings just to accommodate the API of ngx.print. The
similar idea also applies to other existing Lua APIs of ngx_lua,
including the shdict API and the cosocket API.

You're welcome to submit a patch to ngx_lua (and possibly
lua-resty-core too) or get your employer to sponsor the development
work :)

Thanks!
-agentzh

rick.richardson

OK. The good news : I have created a ngx.write() method, which takes a lightuserdata or number for the buffer, and a number for the size of the buffer. The code can be found here: https://github.com/12sidedtech/lua-nginx-module/blob/master/src/ngx_http_lua_output.c#L43

The bad news is that when I call it in a loop from my lua code, I still get the same errors as before, basically the socket's write buffer fills up and never gets re-registered, so the buffer chains keep piling up.
My lua code which is running in lapis (so in a content_by_lua directive) looks basically like this:

local result = 0

while true do

result = C.ngx_http_lua_pcap_cursor_next(cursor[0], entry, pktlen)

if result ~= 0 then break end

ngx.write(entry[0], pktlen[0]))

end

My assumption was that if I ran this within a lua handler loop, it would do the right things with regards to waiting for the socket to be writable. So my new question is: how can I wait for the socket to become writable to continue writing?

The relevant nginx logs are here, when writev starts failing to send all of its data:

2015/09/15 20:27:05 [debug] 13576#0: *1 writev: 2240 of 11457

2015/09/15 20:27:05 [debug] 13576#0: *1 http write filter 0000000000A82880

2015/09/15 20:27:05 [debug] 13576#0: *1 http copy filter: -2 "/pcap/?"

2015/09/15 20:27:05 [debug] 13576#0: *1 lua allocate new chainlink and new buf of size 10964, cl:0000000000A82870

2015/09/15 20:27:05 [debug] 13576#0: *1 malloc: 0000000000AE7390:10964

2015/09/15 20:27:05 [debug] 13576#0: *1 lua write response

2015/09/15 20:27:05 [debug] 13576#0: *1 http output filter "/pcap/?"

2015/09/15 20:27:05 [debug] 13576#0: *1 http copy filter: "/pcap/?"

2015/09/15 20:27:05 [debug] 13576#0: *1 lua capture body filter, uri "/pcap/"

2015/09/15 20:27:05 [debug] 13576#0: *1 http postpone filter "/pcap/?" 0000000000A82870

2015/09/15 20:27:05 [debug] 13576#0: *1 http chunk: 10964

2015/09/15 20:27:05 [debug] 13576#0: *1 write old buf t:1 f:0 0000000000AE45E0, pos 0000000000AE4E9A, size: 9215 file: 0, size: 0

2015/09/15 20:27:05 [debug] 13576#0: *1 write old buf t:0 f:0 0000000000A828A0, pos 0000000000502167, size: 2 file: 0, size: 0

2015/09/15 20:27:05 [debug] 13576#0: *1 write new buf t:1 f:0 0000000000A827F8, pos 0000000000A827F8, size: 6 file: 0, size: 0

2015/09/15 20:27:05 [debug] 13576#0: *1 write new buf t:1 f:0 0000000000AE7390, pos 0000000000AE7390, size: 10964 file: 0, size: 0

2015/09/15 20:27:05 [debug] 13576#0: *1 write new buf t:0 f:0 0000000000000000, pos 0000000000502167, size: 2 file: 0, size: 0

On Thursday, September 3, 2015 at 3:41:50 AM UTC-4, agentzh wrote:

Hello!

On Wed, Sep 2, 2015 at 11:17 PM, Rick Richardson wrote:
> Thanks for your reply. And thanks for confirming what I kind of already
> understood. Apologies for the dumb questions.
>
> So if I were to do two things in a loop in my lua handler: 1. Fetch a blob
> of data (in a ffi.C call) then 2. write it to ngx via something like
> ngx.say(), that should work?
>

If I understand your intentions correctly, I think we could just
expose a variant of the ngx.print() API that accepts raw C buffers in
the form of FFI cdata. This way we can avoid allocating new
(temporary) Lua strings just to accommodate the API of ngx.print. The
similar idea also applies to other existing Lua APIs of ngx_lua,
including the shdict API and the cosocket API.

You're welcome to submit a patch to ngx_lua (and possibly
lua-resty-core too) or get your employer to sponsor the development
work :)

Thanks!
-agentzh

agentzh

Hello!

On Wed, Sep 16, 2015 at 8:35 AM, Rick Richardson wrote:
> OK. The good news : I have created a ngx.write() method, which takes a
> lightuserdata or number for the buffer, and a number for the size of the
> buffer. The code can be found here:
> https://github.com/12sidedtech/lua-nginx-module/blob/master/src/ngx_http_lua_output.c#L43
>

Cool.

> The bad news is that when I call it in a loop from my lua code, I still get
> the same errors as before, basically the socket's write buffer fills up and
> never gets re-registered, so the buffer chains keep piling up.
> My lua code which is running in lapis (so in a content_by_lua directive)
> looks basically like this:
>
> local result = 0
> while true do
>       result = C.ngx_http_lua_pcap_cursor_next(cursor[0], entry, pktlen)
>
>       if result ~= 0 then break end
>       ngx.write(entry[0], pktlen[0]))
> end

The nginx output API is asynchronous (like the cosocket send API). So
you need to call ngx.flush(true) to make it synchronous (note it's
still nonblocking). Please see

https://github.com/openresty/lua-nginx-module#ngxflush

for more details. (Well, you need the `true` flush argument here to
enforce waiting.)

Regards,
-agentzh