binary data response

rick.richardson · 2015-08-05T02:28:57+00:00

On Tuesday, 18 August 2015 10:48:01 UTC+3, Samarjit Uppal wrote:Just an update. HANA provides http connectivity also. If possible, please make it a lib...

binary data response

rick.richardson

I have a large slab of mmap'd memory that I need to send to the client in a chunked response to a request. What is the best way to do that?

Is there an object in lua which could take a pointer and a length that can turn into something that can be sent by ngx.print, or is there some alternate function I could use?

rick.richardson

I just found https://groups.google.com/forum/#!searchin/openresty-en/write$20memory/openresty-en/XmlJxuZGrw4/nQyBS0f5z4gJ

Which is very similar, but with a different goal in mind. To disambiguate:

In response to a request I need to mmap a file, and then respond with some arbitrary subset of that file for some arbitary length dictated by the request. (and then close mmap)
The files are very large and there are potentially terabytes of them, so I can't preload, or send entire files.

I don't mind writing any or all of the above operations in C, in fact I kind of expected to. I'm just trying to figure out how to tell nginx "here is a void* and a length, please send this down to the client in a chunk"

On Tuesday, August 4, 2015 at 2:28:57 PM UTC-4, Rick Richardson wrote:

I have a large slab of mmap'd memory that I need to send to the client in a chunked response to a request. What is the best way to do that?

Is there an object in lua which could take a pointer and a length that can turn into something that can be sent by ngx.print, or is there some alternate function I could use?

lordnynex

Hello,

Is it not possible to use range requests with nginx thread pooling? https://tools.ietf.org/html/rfc7233 (warning, I have no clue if nginx will read the entire file I'm guessing it won't)

You could easily use an FFI type but the exact details seem like they'd best be done by you. If the chunk is small'ish it shouldn't be a problem but I'm not sure when you start reading data into lua land. You may have a fair bit of memory pressure. I've never done this before. Either way the client is specifying ranges, which nginx supports out of the box.

It may be beneficial to implement this as an nginx module since most of your work will be done in C anyways. I did find http://wiki.nginx.org/HttpSliceModule which appears to be specific to tengine but may give you what you need.

Unrelated to openresty, you could use a private implementation of bit torrent. I also recently stumbled accross https://github.com/facebook/wdt which is really awesome.

On Tue, Aug 4, 2015 at 12:13 PM, Rick Richardson <rick.ri...@gmail.com> wrote:

I just found https://groups.google.com/forum/#!searchin/openresty-en/write$20memory/openresty-en/XmlJxuZGrw4/nQyBS0f5z4gJ

Which is very similar, but with a different goal in mind. To disambiguate:

In response to a request I need to mmap a file, and then respond with some arbitrary subset of that file for some arbitary length dictated by the request. (and then close mmap)
The files are very large and there are potentially terabytes of them, so I can't preload, or send entire files.

I don't mind writing any or all of the above operations in C, in fact I kind of expected to. I'm just trying to figure out how to tell nginx "here is a void* and a length, please send this down to the client in a chunk"

On Tuesday, August 4, 2015 at 2:28:57 PM UTC-4, Rick Richardson wrote:
I have a large slab of mmap'd memory that I need to send to the client in a chunked response to a request. What is the best way to do that?

Is there an object in lua which could take a pointer and a length that can turn into something that can be sent by ngx.print, or is there some alternate function I could use?

.

rick.richardson

Thanks for pointing out the slice module, I was unaware of its existence. It is almost exactly what I need, except I need to send down multiple slices per request, and I need to calculate the files and locations from each request. This is why I was hoping to stream the contents out by using something akin to ngx.print()

I'll look in to seeing how hard it would be to modify HttpSliceModule for my purposes.

On Tuesday, August 4, 2015 at 3:24:34 PM UTC-4, Lord Nynex wrote:

Hello,

Is it not possible to use range requests with nginx thread pooling? https://tools.ietf.org/html/rfc7233 (warning, I have no clue if nginx will read the entire file I'm guessing it won't)

You could easily use an FFI type but the exact details seem like they'd best be done by you. If the chunk is small'ish it shouldn't be a problem but I'm not sure when you start reading data into lua land. You may have a fair bit of memory pressure. I've never done this before. Either way the client is specifying ranges, which nginx supports out of the box.

It may be beneficial to implement this as an nginx module since most of your work will be done in C anyways. I did find http://wiki.nginx.org/HttpSliceModule which appears to be specific to tengine but may give you what you need.

Unrelated to openresty, you could use a private implementation of bit torrent. I also recently stumbled accross https://github.com/facebook/wdt which is really awesome.

On Tue, Aug 4, 2015 at 12:13 PM, Rick Richardson <rick.ri...@gmail.com> wrote:
I just found https://groups.google.com/forum/#!searchin/openresty-en/write$20memory/openresty-en/XmlJxuZGrw4/nQyBS0f5z4gJ

Which is very similar, but with a different goal in mind. To disambiguate:

In response to a request I need to mmap a file, and then respond with some arbitrary subset of that file for some arbitary length dictated by the request. (and then close mmap)
The files are very large and there are potentially terabytes of them, so I can't preload, or send entire files.

I don't mind writing any or all of the above operations in C, in fact I kind of expected to. I'm just trying to figure out how to tell nginx "here is a void* and a length, please send this down to the client in a chunk"

On Tuesday, August 4, 2015 at 2:28:57 PM UTC-4, Rick Richardson wrote:
I have a large slab of mmap'd memory that I need to send to the client in a chunked response to a request. What is the best way to do that?

Is there an object in lua which could take a pointer and a length that can turn into something that can be sent by ngx.print, or is there some alternate function I could use?

.

rick.richardson

Looking at the source for ngx_http_lua_ngx_print I see that it has a case for lua_lightuserdata (as long as it is null)

I'm beginning to think that the shortest path to success for me is to create lua_userdata objects out of my memory chunks, and update ngx_http_lua_ngx_echo to handle userdata.

Thoughts?

On Tuesday, August 4, 2015 at 3:37:15 PM UTC-4, Rick Richardson wrote:

Thanks for pointing out the slice module, I was unaware of its existence. It is almost exactly what I need, except I need to send down multiple slices per request, and I need to calculate the files and locations from each request. This is why I was hoping to stream the contents out by using something akin to ngx.print()

I'll look in to seeing how hard it would be to modify HttpSliceModule for my purposes.

On Tuesday, August 4, 2015 at 3:24:34 PM UTC-4, Lord Nynex wrote:
Hello,

Is it not possible to use range requests with nginx thread pooling? https://tools.ietf.org/html/rfc7233 (warning, I have no clue if nginx will read the entire file I'm guessing it won't)

You could easily use an FFI type but the exact details seem like they'd best be done by you. If the chunk is small'ish it shouldn't be a problem but I'm not sure when you start reading data into lua land. You may have a fair bit of memory pressure. I've never done this before. Either way the client is specifying ranges, which nginx supports out of the box.

It may be beneficial to implement this as an nginx module since most of your work will be done in C anyways. I did find http://wiki.nginx.org/HttpSliceModule which appears to be specific to tengine but may give you what you need.

Unrelated to openresty, you could use a private implementation of bit torrent. I also recently stumbled accross https://github.com/facebook/wdt which is really awesome.

On Tue, Aug 4, 2015 at 12:13 PM, Rick Richardson <rick.ri...@gmail.com> wrote:
I just found https://groups.google.com/forum/#!searchin/openresty-en/write$20memory/openresty-en/XmlJxuZGrw4/nQyBS0f5z4gJ

Which is very similar, but with a different goal in mind. To disambiguate:

In response to a request I need to mmap a file, and then respond with some arbitrary subset of that file for some arbitary length dictated by the request. (and then close mmap)
The files are very large and there are potentially terabytes of them, so I can't preload, or send entire files.

I don't mind writing any or all of the above operations in C, in fact I kind of expected to. I'm just trying to figure out how to tell nginx "here is a void* and a length, please send this down to the client in a chunk"

On Tuesday, August 4, 2015 at 2:28:57 PM UTC-4, Rick Richardson wrote:
I have a large slab of mmap'd memory that I need to send to the client in a chunked response to a request. What is the best way to do that?

Is there an object in lua which could take a pointer and a length that can turn into something that can be sent by ngx.print, or is there some alternate function I could use?

.

agentzh

Hello!

On Wed, Aug 5, 2015 at 2:28 AM, Rick Richardson wrote:
> I have a large slab of mmap'd memory that I need to send to the client in a
> chunked response to a request.  What is the best way to do that?
>
> Is there an object in lua which could take a pointer and a length that can
> turn into something that can be sent by ngx.print, or is there some
> alternate function I could use?
>

I suggest the use of ffi.string() to convert raw C pointer contents to
a Lua string first before feeding it to ngx.print(). And yeah, I
assume you're aleady using LuaJIT FFI in this context.

Eventually we could expose a lower level ngx.print() variant that
accepts raw cdata pointers and length arguments to eliminate the
intermediate Lua strings altogether (well, patches welcome :))

Regards,
-agentzh

rick.richardson

I wanted to avoid the creation of the Lua String because of the additional alloc it would do.

I have in fact, created an ngx.write() which takes a LUA_TLIGHTUSERDATA and a LUA_TNUMBER and works like ngx.print. I am testing it now.

I reckon it should probably take a table of userdata,number tuples so it can inject multiple buffers per chain

Obviously this solution needs to be used with flush(true) so I can ensure that the data has been written so that I can unmap the memory. Maybe this could be tied to some sort of GC or destructor mechanism so that a free function can be specified. Unfortunately that is beyond my current lua_ffi skillet :)

On Tuesday, August 4, 2015 at 9:19:37 PM UTC-4, agentzh wrote:

Hello!

On Wed, Aug 5, 2015 at 2:28 AM, Rick Richardson wrote:
> I have a large slab of mmap'd memory that I need to send to the client in a
> chunked response to a request. What is the best way to do that?
>
> Is there an object in lua which could take a pointer and a length that can
> turn into something that can be sent by ngx.print, or is there some
> alternate function I could use?
>

I suggest the use of ffi.string() to convert raw C pointer contents to
a Lua string first before feeding it to ngx.print(). And yeah, I
assume you're aleady using LuaJIT FFI in this context.

Eventually we could expose a lower level ngx.print() variant that
accepts raw cdata pointers and length arguments to eliminate the
intermediate Lua strings altogether (well, patches welcome :))

Regards,
-agentzh

agentzh

Hello!

On Wed, Aug 5, 2015 at 10:51 PM, Rick Richardson wrote:
> I wanted to avoid the creation of the Lua String because of the additional
> alloc it would do.
>

Understood.

> I have in fact, created an ngx.write()   which takes a LUA_TLIGHTUSERDATA
> and a LUA_TNUMBER  and works like ngx.print.  I am testing it now.
> I reckon it should probably take a table of  userdata,number tuples so it
> can inject multiple buffers per chain
>

Sounds good to me :)

> Obviously this solution needs to be used with flush(true)  so I can ensure
> that the data has been written so that I can unmap the memory.  Maybe this
> could be tied to some sort of GC or destructor mechanism so that a free
> function can be specified.  Unfortunately that is beyond my current lua_ffi
> skillet :)

Oh, no. I suggest you always copy the data from your memory zones to
the buffers used for chain link sending. nginx sends data to the
downstream asynchronously by default (unless you explicitly call
ngx.flush, which is expensive per se).

As compared to dynamic allocations and socket operations, data copying
is (often) not really worth saving in a nontrivial setup. On the other
hand, trying too hard to save data copying can easily lead to (overly)
complicated and fragile memory management.

Well, just my 2 cents.

Regards,
-agentzh

rick.richardson

Cool! I took your advice.. I have it almost working but I'm running into a snafu, likely due to something stupid I'm doing.

Here is the gist: https://gist.github.com/rrichardson/57540f275cd0f03ee8d6

The first chunk of data I send out looks exactly right. The chunk size is correct. However, when I attempt to send the second chunk, by calling ngx_write again, I get this:

2015/08/14 11:32:00 [debug] 31394#0: *7 lua reuse free buf chain, but reallocate memory because 18360 >= 9180, cl:00000000014FC610, p:000000000155A840

2015/08/14 11:32:00 [debug] 31394#0: *7 free: 000000000155A840

*** glibc detected *** nginx: worker process: double free or corruption (out): 0x000000000155a840 ***

On Wednesday, August 5, 2015 at 11:10:44 AM UTC-4, agentzh wrote:

Hello!

On Wed, Aug 5, 2015 at 10:51 PM, Rick Richardson wrote:
> I wanted to avoid the creation of the Lua String because of the additional
> alloc it would do.
>

Understood.

> I have in fact, created an ngx.write() which takes a LUA_TLIGHTUSERDATA
> and a LUA_TNUMBER and works like ngx.print. I am testing it now.
> I reckon it should probably take a table of userdata,number tuples so it
> can inject multiple buffers per chain
>

Sounds good to me :)

> Obviously this solution needs to be used with flush(true) so I can ensure
> that the data has been written so that I can unmap the memory. Maybe this
> could be tied to some sort of GC or destructor mechanism so that a free
> function can be specified. Unfortunately that is beyond my current lua_ffi
> skillet :)

Oh, no. I suggest you always copy the data from your memory zones to
the buffers used for chain link sending. nginx sends data to the
downstream asynchronously by default (unless you explicitly call
ngx.flush, which is expensive per se).

As compared to dynamic allocations and socket operations, data copying
is (often) not really worth saving in a nontrivial setup. On the other
hand, trying too hard to save data copying can easily lead to (overly)
complicated and fragile memory management.

Well, just my 2 cents.

Regards,
-agentzh

rick.richardson

So what I'm doing is calling ngx_write in a loop with a buffer that is around 9180 bytes.

(I'm generating pcaps via another application and streaming them down via http, the buffer size might change slightly depending on the size of the packets).
Is there a better approach than repeatedly calling ngx_lua_chain_get_free_buf?

On Friday, August 14, 2015 at 11:45:50 AM UTC-4, Rick Richardson wrote:

Cool! I took your advice.. I have it almost working but I'm running into a snafu, likely due to something stupid I'm doing.

Here is the gist: https://gist.github.com/rrichardson/57540f275cd0f03ee8d6

The first chunk of data I send out looks exactly right. The chunk size is correct. However, when I attempt to send the second chunk, by calling ngx_write again, I get this:

2015/08/14 11:32:00 [debug] 31394#0: *7 lua reuse free buf chain, but reallocate memory because 18360 >= 9180, cl:00000000014FC610, p:000000000155A840
2015/08/14 11:32:00 [debug] 31394#0: *7 free: 000000000155A840
*** glibc detected *** nginx: worker process: double free or corruption (out): 0x000000000155a840 ***

On Wednesday, August 5, 2015 at 11:10:44 AM UTC-4, agentzh wrote:
Hello!

On Wed, Aug 5, 2015 at 10:51 PM, Rick Richardson wrote:
> I wanted to avoid the creation of the Lua String because of the additional
> alloc it would do.
>

Understood.

> I have in fact, created an ngx.write() which takes a LUA_TLIGHTUSERDATA
> and a LUA_TNUMBER and works like ngx.print. I am testing it now.
> I reckon it should probably take a table of userdata,number tuples so it
> can inject multiple buffers per chain
>

Sounds good to me :)

> Obviously this solution needs to be used with flush(true) so I can ensure
> that the data has been written so that I can unmap the memory. Maybe this
> could be tied to some sort of GC or destructor mechanism so that a free
> function can be specified. Unfortunately that is beyond my current lua_ffi
> skillet :)

Oh, no. I suggest you always copy the data from your memory zones to
the buffers used for chain link sending. nginx sends data to the
downstream asynchronously by default (unless you explicitly call
ngx.flush, which is expensive per se).

As compared to dynamic allocations and socket operations, data copying
is (often) not really worth saving in a nontrivial setup. On the other
hand, trying too hard to save data copying can easily lead to (overly)
complicated and fragile memory management.

Well, just my 2 cents.

Regards,
-agentzh

rick.richardson

actually.. there was a bug in my code, where I wasn't clearing the total size, so it was doubling every iteration. I fixed that and I'm closer.
However, I'm still not clear as to how that would cause an error in ngx_lua_chain_get_free_buf.

My only other issue now is that it seems like it is inserting CR/LFs between chunks that I write, which is adversely effecting the correctness of the pcap file I am producing.

Is there something I can do about that?

On Friday, August 14, 2015 at 11:57:32 AM UTC-4, Rick Richardson wrote:

So what I'm doing is calling ngx_write in a loop with a buffer that is around 9180 bytes.
(I'm generating pcaps via another application and streaming them down via http, the buffer size might change slightly depending on the size of the packets).
Is there a better approach than repeatedly calling ngx_lua_chain_get_free_buf?

On Friday, August 14, 2015 at 11:45:50 AM UTC-4, Rick Richardson wrote:
Cool! I took your advice.. I have it almost working but I'm running into a snafu, likely due to something stupid I'm doing.

Here is the gist: https://gist.github.com/rrichardson/57540f275cd0f03ee8d6

The first chunk of data I send out looks exactly right. The chunk size is correct. However, when I attempt to send the second chunk, by calling ngx_write again, I get this:

2015/08/14 11:32:00 [debug] 31394#0: *7 lua reuse free buf chain, but reallocate memory because 18360 >= 9180, cl:00000000014FC610, p:000000000155A840
2015/08/14 11:32:00 [debug] 31394#0: *7 free: 000000000155A840
*** glibc detected *** nginx: worker process: double free or corruption (out): 0x000000000155a840 ***

On Wednesday, August 5, 2015 at 11:10:44 AM UTC-4, agentzh wrote:
Hello!

On Wed, Aug 5, 2015 at 10:51 PM, Rick Richardson wrote:
> I wanted to avoid the creation of the Lua String because of the additional
> alloc it would do.
>

Understood.

> I have in fact, created an ngx.write() which takes a LUA_TLIGHTUSERDATA
> and a LUA_TNUMBER and works like ngx.print. I am testing it now.
> I reckon it should probably take a table of userdata,number tuples so it
> can inject multiple buffers per chain
>

Sounds good to me :)

> Obviously this solution needs to be used with flush(true) so I can ensure
> that the data has been written so that I can unmap the memory. Maybe this
> could be tied to some sort of GC or destructor mechanism so that a free
> function can be specified. Unfortunately that is beyond my current lua_ffi
> skillet :)

Oh, no. I suggest you always copy the data from your memory zones to
the buffers used for chain link sending. nginx sends data to the
downstream asynchronously by default (unless you explicitly call
ngx.flush, which is expensive per se).

As compared to dynamic allocations and socket operations, data copying
is (often) not really worth saving in a nontrivial setup. On the other
hand, trying too hard to save data copying can easily lead to (overly)
complicated and fragile memory management.

Well, just my 2 cents.

Regards,
-agentzh

agentzh

Hello!

On Fri, Aug 14, 2015 at 11:57 PM, Rick Richardson wrote:
>  Is there a better approach than repeatedly calling
> ngx_lua_chain_get_free_buf?
>

What's wrong with that? Please note that there is buffer cycling
automatically going on with ngx_lua_chain_get_free_buf and
ngx_http_lua_send_chain_link.

> My only other issue now is that it seems like it is inserting CR/LFs between chunks that I write, which is adversely effecting the
> correctness of the pcap file I am producing.

No CR LF should be inserted in response bodies. There may be something
wrong in your C code or packet inspection it seems.

If you can reproduce it with the stock ngx_lua, please let me know and
provide a minimal and standalone test case :)

egards,
-agentzh