streaming zlib interface using luajit ffi

andreaslubbe · 2016-05-22T00:30:31+00:00

Hello! On Fri, May 20, 2016 at 5:01 PM, Romaboy wrote: > I was confused that workers can't interact so I want to write a module. I'm not sure about this...

streaming zlib interface using luajit ffi

andreaslubbe

Some of my upstreams send gzipped responses even though Accept-Encoding is empty, so I have to handle/ungzip them before I can process them further.

Currently, I am using lua-zlib to do that by instantiating ctx.inflate = zlib.inflate() in the header phase and calling it in the body filter phase.

Out of interest, I would like to substitute it with a luajit binding. Is anyone aware of such a project? The closest I have found is luapower's zlib library but it does not have a streaming API https://github.com/luapower/zlib

Because I couldn't find anything, I've thrown together a very rough first draft of what that could look like at https://github.com/luapower/zlib/pull/2

I lack the understanding and ffi experience to really implement, but maybe someone can join me? ;)

hamish.forbes

Hey,

I wrote https://github.com/hamishforbes/lua-ffi-zlib for almost exactly that purpose, we've been using in production for a while now.

I haven't tried it using the body_filter_by_lua stuff, but I think it should be workable I think.

We use resty-http and this lib to stream upstream responses through zlib.

It just takes 2 functions which provide chunks of input or receive chunks of output.

If you don't actually need to manipulate the response with Lua and just fix your encoding problem have you tried the nginx gunzip module?

http://nginx.org/en/docs/http/ngx_http_gunzip_module.html

Hamish

On Saturday, 21 May 2016 17:30:31 UTC+1, AL wrote:

Some of my upstreams send gzipped responses even though Accept-Encoding is empty, so I have to handle/ungzip them before I can process them further.
Currently, I am using lua-zlib to do that by instantiating ctx.inflate = zlib.inflate() in the header phase and calling it in the body filter phase.
Out of interest, I would like to substitute it with a luajit binding. Is anyone aware of such a project? The closest I have found is luapower's zlib library but it does not have a streaming API https://github.com/luapower/zlib
Because I couldn't find anything, I've thrown together a very rough first draft of what that could look like at https://github.com/luapower/zlib/pull/2
I lack the understanding and ffi experience to really implement, but maybe someone can join me? ;)

andreaslubbe

That's a great library, I was not aware of it!

The top-level API seems similar to that of luapower, in the sense that you pass in an input function that handles the reading. In my case, I would like to reverse the control because I have to wait for the next body_filter to be called - similar to what I have tried to describe here https://github.com/luapower/zlib/pull/2/files#diff-e7b5fbba0a09c5894a1c4f1167bbb800R67. What would be the idiomatic way of using your binding to achieve that?

On Monday, May 23, 2016 at 4:05:36 PM UTC+2, Hamish Forbes wrote:

Hey,

I wrote https://github.com/hamishforbes/lua-ffi-zlib for almost exactly that purpose, we've been using in production for a while now.

I haven't tried it using the body_filter_by_lua stuff, but I think it should be workable I think.
We use resty-http and this lib to stream upstream responses through zlib.
It just takes 2 functions which provide chunks of input or receive chunks of output.

If you don't actually need to manipulate the response with Lua and just fix your encoding problem have you tried the nginx gunzip module?
http://nginx.org/en/docs/http/ngx_http_gunzip_module.html

Hamish

On Saturday, 21 May 2016 17:30:31 UTC+1, AL wrote:
Some of my upstreams send gzipped responses even though Accept-Encoding is empty, so I have to handle/ungzip them before I can process them further.
Currently, I am using lua-zlib to do that by instantiating ctx.inflate = zlib.inflate() in the header phase and calling it in the body filter phase.
Out of interest, I would like to substitute it with a luajit binding. Is anyone aware of such a project? The closest I have found is luapower's zlib library but it does not have a streaming API https://github.com/luapower/zlib
Because I couldn't find anything, I've thrown together a very rough first draft of what that could look like at https://github.com/luapower/zlib/pull/2
I lack the understanding and ffi experience to really implement, but maybe someone can join me? ;)

hamish.forbes

Yeah I think I wrote it based off the luapower one originally actually... It was a while ago!

I think you'd have to use the 'advanced' functions rather than the simple wrapper.

Create your own stream and stash it in ngx.ctx and then each time body_filter is run you would have to create buffers and input/output functions then call flate manually.

When you run out of input bytes in that iteration of body_filter you exit and wait for the next one?

On Monday, 23 May 2016 15:42:58 UTC+1, AL wrote:

That's a great library, I was not aware of it!
The top-level API seems similar to that of luapower, in the sense that you pass in an input function that handles the reading. In my case, I would like to reverse the control because I have to wait for the next body_filter to be called - similar to what I have tried to describe here https://github.com/luapower/zlib/pull/2/files#diff-e7b5fbba0a09c5894a1c4f1167bbb800R67. What would be the idiomatic way of using your binding to achieve that?

On Monday, May 23, 2016 at 4:05:36 PM UTC+2, Hamish Forbes wrote:
Hey,

I wrote https://github.com/hamishforbes/lua-ffi-zlib for almost exactly that purpose, we've been using in production for a while now.

I haven't tried it using the body_filter_by_lua stuff, but I think it should be workable I think.
We use resty-http and this lib to stream upstream responses through zlib.
It just takes 2 functions which provide chunks of input or receive chunks of output.

If you don't actually need to manipulate the response with Lua and just fix your encoding problem have you tried the nginx gunzip module?
http://nginx.org/en/docs/http/ngx_http_gunzip_module.html

Hamish

On Saturday, 21 May 2016 17:30:31 UTC+1, AL wrote:
Some of my upstreams send gzipped responses even though Accept-Encoding is empty, so I have to handle/ungzip them before I can process them further.
Currently, I am using lua-zlib to do that by instantiating ctx.inflate = zlib.inflate() in the header phase and calling it in the body filter phase.
Out of interest, I would like to substitute it with a luajit binding. Is anyone aware of such a project? The closest I have found is luapower's zlib library but it does not have a streaming API https://github.com/luapower/zlib
Because I couldn't find anything, I've thrown together a very rough first draft of what that could look like at https://github.com/luapower/zlib/pull/2
I lack the understanding and ffi experience to really implement, but maybe someone can join me? ;)

andreaslubbe

I've whipped up a naive attempt given your suggestion. It works great for responses that are transmitted in one full buffer, but when it consists of two or more, it fails with this error while processing the first buffer:

FLATE: buffer error

It happens in the second iteration of the loop (the first iteration read the input buffer) where the mode has been set to Z_FINISH. Could you help me figure out how to get it working?

Here is the code:

-- header_filter

ctx.inflateStream, ctx.inbuf, ctx.outbuf = luajit_zlib.createStream(9999)

-- body_filter

local chunk = ngx.arg[1]

local inflated = ""

local init = luajit_zlib.initInflate(ctx.inflateStream)

local input = function() local a = chunk chunk = nil return a end

local output = function(data) inflated = inflated .. data end

if init == zlib.Z_OK then

local ok, err = luajit_zlib.flate(zlib.inflate, zlib.inflateEnd, input, output, 9999, ctx.inflateStream, ctx.inbuf, ctx.outbuf)

if not ok then error(err) end

else

-- Init error

zlib.inflateEnd(inflateStream)

error("INIT: ".. luajit_zlib.zlib_err(init))

end

On Monday, May 23, 2016 at 4:50:13 PM UTC+2, Hamish Forbes wrote:

Yeah I think I wrote it based off the luapower one originally actually... It was a while ago!

I think you'd have to use the 'advanced' functions rather than the simple wrapper.
Create your own stream and stash it in ngx.ctx and then each time body_filter is run you would have to create buffers and input/output functions then call flate manually.
When you run out of input bytes in that iteration of body_filter you exit and wait for the next one?

On Monday, 23 May 2016 15:42:58 UTC+1, AL wrote:
That's a great library, I was not aware of it!
The top-level API seems similar to that of luapower, in the sense that you pass in an input function that handles the reading. In my case, I would like to reverse the control because I have to wait for the next body_filter to be called - similar to what I have tried to describe here https://github.com/luapower/zlib/pull/2/files#diff-e7b5fbba0a09c5894a1c4f1167bbb800R67. What would be the idiomatic way of using your binding to achieve that?

On Monday, May 23, 2016 at 4:05:36 PM UTC+2, Hamish Forbes wrote:
Hey,

I wrote https://github.com/hamishforbes/lua-ffi-zlib for almost exactly that purpose, we've been using in production for a while now.

I haven't tried it using the body_filter_by_lua stuff, but I think it should be workable I think.
We use resty-http and this lib to stream upstream responses through zlib.
It just takes 2 functions which provide chunks of input or receive chunks of output.

If you don't actually need to manipulate the response with Lua and just fix your encoding problem have you tried the nginx gunzip module?
http://nginx.org/en/docs/http/ngx_http_gunzip_module.html

Hamish

On Saturday, 21 May 2016 17:30:31 UTC+1, AL wrote:
Some of my upstreams send gzipped responses even though Accept-Encoding is empty, so I have to handle/ungzip them before I can process them further.
Currently, I am using lua-zlib to do that by instantiating ctx.inflate = zlib.inflate() in the header phase and calling it in the body filter phase.
Out of interest, I would like to substitute it with a luajit binding. Is anyone aware of such a project? The closest I have found is luapower's zlib library but it does not have a streaming API https://github.com/luapower/zlib
Because I couldn't find anything, I've thrown together a very rough first draft of what that could look like at https://github.com/luapower/zlib/pull/2
I lack the understanding and ffi experience to really implement, but maybe someone can join me? ;)

hamish.forbes

Ah yeah.. Looking at it more closely I think you'd have to re-implement most of the logic inside zlib.flate() because it expects to be able to run to the end of the input stream.

Which is not possible within body_filter.

(also you need to call initInflate only once, after you create the stream)

In each body_filter iteration you need to run the innermost loop within zlib.flate until it runs out of input data, but then wait for the next body_filter iteration in the outer loop.

Tricky!

I'm not sure my lib is any better structured than the luapower one for your particular use case.

Have you tried the gunzip module approach instead? Unless you do need to manipulate the uncompressed response.

On Monday, 23 May 2016 20:04:58 UTC+1, AL wrote:

I've whipped up a naive attempt given your suggestion. It works great for responses that are transmitted in one full buffer, but when it consists of two or more, it fails with this error while processing the first buffer:
FLATE: buffer error
It happens in the second iteration of the loop (the first iteration read the input buffer) where the mode has been set to Z_FINISH. Could you help me figure out how to get it working?

Here is the code:
-- header_filter
ctx.inflateStream, ctx.inbuf, ctx.outbuf = luajit_zlib.createStream(9999)

-- body_filter
local chunk = ngx.arg[1]
local inflated = ""
local init = luajit_zlib.initInflate(ctx.inflateStream)
local input = function() local a = chunk chunk = nil return a end
local output = function(data) inflated = inflated .. data end
if init == zlib.Z_OK then
local ok, err = luajit_zlib.flate(zlib.inflate, zlib.inflateEnd, input, output, 9999, ctx.inflateStream, ctx.inbuf, ctx.outbuf)
if not ok then error(err) end
else
-- Init error
zlib.inflateEnd(inflateStream)
error("INIT: ".. luajit_zlib.zlib_err(init))
end

On Monday, May 23, 2016 at 4:50:13 PM UTC+2, Hamish Forbes wrote:
Yeah I think I wrote it based off the luapower one originally actually... It was a while ago!

I think you'd have to use the 'advanced' functions rather than the simple wrapper.
Create your own stream and stash it in ngx.ctx and then each time body_filter is run you would have to create buffers and input/output functions then call flate manually.
When you run out of input bytes in that iteration of body_filter you exit and wait for the next one?

On Monday, 23 May 2016 15:42:58 UTC+1, AL wrote:
That's a great library, I was not aware of it!
The top-level API seems similar to that of luapower, in the sense that you pass in an input function that handles the reading. In my case, I would like to reverse the control because I have to wait for the next body_filter to be called - similar to what I have tried to describe here https://github.com/luapower/zlib/pull/2/files#diff-e7b5fbba0a09c5894a1c4f1167bbb800R67. What would be the idiomatic way of using your binding to achieve that?

On Monday, May 23, 2016 at 4:05:36 PM UTC+2, Hamish Forbes wrote:
Hey,

I wrote https://github.com/hamishforbes/lua-ffi-zlib for almost exactly that purpose, we've been using in production for a while now.

I haven't tried it using the body_filter_by_lua stuff, but I think it should be workable I think.
We use resty-http and this lib to stream upstream responses through zlib.
It just takes 2 functions which provide chunks of input or receive chunks of output.

If you don't actually need to manipulate the response with Lua and just fix your encoding problem have you tried the nginx gunzip module?
http://nginx.org/en/docs/http/ngx_http_gunzip_module.html

Hamish

On Saturday, 21 May 2016 17:30:31 UTC+1, AL wrote:
Some of my upstreams send gzipped responses even though Accept-Encoding is empty, so I have to handle/ungzip them before I can process them further.
Currently, I am using lua-zlib to do that by instantiating ctx.inflate = zlib.inflate() in the header phase and calling it in the body filter phase.
Out of interest, I would like to substitute it with a luajit binding. Is anyone aware of such a project? The closest I have found is luapower's zlib library but it does not have a streaming API https://github.com/luapower/zlib
Because I couldn't find anything, I've thrown together a very rough first draft of what that could look like at https://github.com/luapower/zlib/pull/2
I lack the understanding and ffi experience to really implement, but maybe someone can join me? ;)

hamish.forbes

I think got it working :)

https://gist.github.com/hamishforbes/ea0e9446c3b6add0df559ba4bce2cc12

Theres some cleanup needed in there and probably needs wrapping up in a helper function with some more sanity checks.

Happy to accept accept a pull request to accept another wrapper method if you want to clean that code up and submit something?

Hamish

On Tuesday, 24 May 2016 09:57:19 UTC+1, Hamish Forbes wrote:

Ah yeah.. Looking at it more closely I think you'd have to re-implement most of the logic inside zlib.flate() because it expects to be able to run to the end of the input stream.
Which is not possible within body_filter.

(also you need to call initInflate only once, after you create the stream)

In each body_filter iteration you need to run the innermost loop within zlib.flate until it runs out of input data, but then wait for the next body_filter iteration in the outer loop.
Tricky!
I'm not sure my lib is any better structured than the luapower one for your particular use case.

Have you tried the gunzip module approach instead? Unless you do need to manipulate the uncompressed response.

andreaslubbe

I've left you a quick comment on the gist, I still get errors if there is more than one buffer.

hamish.forbes

Doh!

I've updated the gist, I think that was due to running the inner loop at least once with no input data available.

It's pretty messy but I think thats working now?

On Tuesday, 24 May 2016 14:53:47 UTC+1, AL wrote:

I've left you a quick comment on the gist, I still get errors if there is more than one buffer.

andreaslubbe

Weirdly enough, this won't work on my machine (Ubuntu 16.04).
I've left you a more detailed comment on the gist and put together a self contained repo here: https://github.com/alubbe/ffizlibtest

On Tuesday, May 24, 2016 at 4:16:47 PM UTC+2, Hamish Forbes wrote:

Doh!

I've updated the gist, I think that was due to running the inner loop at least once with no input data available.
It's pretty messy but I think thats working now?

On Tuesday, 24 May 2016 14:53:47 UTC+1, AL wrote:

I've left you a quick comment on the gist, I still get errors if there is more than one buffer.