lua-resty-redist isn't putting binary data correctly in redis

gauravphoenix · 2014-11-21T07:39:53+00:00

Hello! On Thu, Nov 20, 2014 at 4:20 AM, Darshan Deshmane wrote: > I am pretty new to NGINX. > I wan to host a REST service on Ubuntu that uses NGINX H...

lua-resty-redist isn't putting binary data correctly in redis

gauravphoenix

I've following code which uses redis lua to push a binary value to "myList"

ngx.log(ngx.INFO,str2hex(dataToPush))

local ok,err =red:rpush("myList",dataToPush)

I am logging the data in hex format. When I do "LINDEX myList 0" on redis client, the results do not match.

The first few bytes of dataToPush printed in logs are-

\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03\x94\x50\x4d\x4f\xe3\x30\x10\x3d\xb3\xbf

Whereas output LINDEX myList 0 is - (snipped for brevity)

\x1f\x8b\b\x00\x00\x00\x00\x00\x00\x03\x94PMO\xe30\x10=\xb3\xbfb\x1a$\xbc{\xb0\

The code which assigns value to dataToPush is here. The value of dataToPush is basically gzipped response from a server which uses chunk encoding in this particular case .

So I believe that nginx lua redis client isn't putting binary data correctly in redis. When I've all ASCII data in dataToPush, everything looks fine.

manchev

You could use double long brackets [[]] around your strings in this case [1] or you'll need to escape the backslashes before sending to Redis.

[1]: https://github.com/openresty/lua-nginx-module#special-pcre-sequences

Best,
Vladislav

On Fri, Nov 21, 2014 at 1:39 AM, Gaurav Kumar <gaurav...@gmail.com> wrote:

I've following code which uses redis lua to push a binary value to "myList"

ngx.log(ngx.INFO,str2hex(dataToPush)) local ok,err =red:rpush("myList",dataToPush)

I am logging the data in hex format. When I do "LINDEX myList 0" on redis client, the results do not match.

The first few bytes of dataToPush printed in logs are-

\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03\x94\x50\x4d\x4f\xe3\x30\x10\x3d\xb3\xbf

Whereas output LINDEX myList 0 is - (snipped for brevity)

\x1f\x8b\b\x00\x00\x00\x00\x00\x00\x03\x94PMO\xe30\x10=\xb3\xbfb\x1a$\xbc{\xb0\

The code which assigns value to dataToPush is here. The value of dataToPush is basically gzipped response from a server which uses chunk encoding in this particular case .

So I believe that nginx lua redis client isn't putting binary data correctly in redis. When I've all ASCII data in dataToPush, everything looks fine.

.

gauravphoenix

Thanks Vladislav!

Can you please provide me an example? Where exactly do I need to place double [[ ]] ? Do I need to have these the very time strings are assigned i.e. in this file http://pastebin.com/qy7EPFeb ?

Note that my LUA code is in separate file being called by using body_filter_by_lua_file directive.

On Thu, Nov 20, 2014 at 4:09 PM, Vladislav Manchev <man...@bin.bz> wrote:

You could use double long brackets [[]] around your strings in this case [1] or you'll need to escape the backslashes before sending to Redis.

[1]: https://github.com/openresty/lua-nginx-module#special-pcre-sequences

Best,
Vladislav

On Fri, Nov 21, 2014 at 1:39 AM, Gaurav Kumar <gaurav...@gmail.com> wrote:
I've following code which uses redis lua to push a binary value to "myList"

ngx.log(ngx.INFO,str2hex(dataToPush)) local ok,err =red:rpush("myList",dataToPush)

I am logging the data in hex format. When I do "LINDEX myList 0" on redis client, the results do not match.

The first few bytes of dataToPush printed in logs are-

\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03\x94\x50\x4d\x4f\xe3\x30\x10\x3d\xb3\xbf

Whereas output LINDEX myList 0 is - (snipped for brevity)

\x1f\x8b\b\x00\x00\x00\x00\x00\x00\x03\x94PMO\xe30\x10=\xb3\xbfb\x1a$\xbc{\xb0\

The code which assigns value to dataToPush is here. The value of dataToPush is basically gzipped response from a server which uses chunk encoding in this particular case .

So I believe that nginx lua redis client isn't putting binary data correctly in redis. When I've all ASCII data in dataToPush, everything looks fine.

.

manchev

The backslash is used as a character escape in double and single-quoted strings, but not in long strings written with the [[...]] notation. That's how Lua works [1].

So, you can enclose the literal string (if you have the literal string, of course) like:
local a = [[\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03\x94\x50\x4d\x4f\xe3\x30\x10\x3d\xb3\xbf]]

Otherwise just use gsub to replace backslashes with double backslashes and you'll be fine.

[1]: http://www.lua.org/pil/2.4.html

Best,
Vladislav

On Fri, Nov 21, 2014 at 2:28 AM, Gaurav Kumar <gaurav...@gmail.com> wrote:

Thanks Vladislav!
Can you please provide me an example? Where exactly do I need to place double [[ ]] ? Do I need to have these the very time strings are assigned i.e. in this file http://pastebin.com/qy7EPFeb ?

Note that my LUA code is in separate file being called by using body_filter_by_lua_file directive.

On Thu, Nov 20, 2014 at 4:09 PM, Vladislav Manchev <man...@bin.bz> wrote:
You could use double long brackets [[]] around your strings in this case [1] or you'll need to escape the backslashes before sending to Redis.

[1]: https://github.com/openresty/lua-nginx-module#special-pcre-sequences

Best,
Vladislav

On Fri, Nov 21, 2014 at 1:39 AM, Gaurav Kumar <gaurav...@gmail.com> wrote:
I've following code which uses redis lua to push a binary value to "myList"

ngx.log(ngx.INFO,str2hex(dataToPush)) local ok,err =red:rpush("myList",dataToPush)

I am logging the data in hex format. When I do "LINDEX myList 0" on redis client, the results do not match.

The first few bytes of dataToPush printed in logs are-

\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03\x94\x50\x4d\x4f\xe3\x30\x10\x3d\xb3\xbf

Whereas output LINDEX myList 0 is - (snipped for brevity)

\x1f\x8b\b\x00\x00\x00\x00\x00\x00\x03\x94PMO\xe30\x10=\xb3\xbfb\x1a$\xbc{\xb0\

The code which assigns value to dataToPush is here. The value of dataToPush is basically gzipped response from a server which uses chunk encoding in this particular case .

So I believe that nginx lua redis client isn't putting binary data correctly in redis. When I've all ASCII data in dataToPush, everything looks fine.

.

gauravphoenix

The thing is I don't have any string literals. The value of dataToPush is directly being assigned in this code- http://pastebin.com/qy7EPFeb

This is perhaps not an encoding issue but something else?

On Thu, Nov 20, 2014 at 4:51 PM, Vladislav Manchev <man...@bin.bz> wrote:

The backslash is used as a character escape in double and single-quoted strings, but not in long strings written with the [[...]] notation. That's how Lua works [1].

So, you can enclose the literal string (if you have the literal string, of course) like:
local a = [[\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03\x94\x50\x4d\x4f\xe3\x30\x10\x3d\xb3\xbf]]

Otherwise just use gsub to replace backslashes with double backslashes and you'll be fine.

[1]: http://www.lua.org/pil/2.4.html

Best,
Vladislav

On Fri, Nov 21, 2014 at 2:28 AM, Gaurav Kumar <gaurav...@gmail.com> wrote:
Thanks Vladislav!
Can you please provide me an example? Where exactly do I need to place double [[ ]] ? Do I need to have these the very time strings are assigned i.e. in this file http://pastebin.com/qy7EPFeb ?

Note that my LUA code is in separate file being called by using body_filter_by_lua_file directive.

On Thu, Nov 20, 2014 at 4:09 PM, Vladislav Manchev <man...@bin.bz> wrote:
You could use double long brackets [[]] around your strings in this case [1] or you'll need to escape the backslashes before sending to Redis.

[1]: https://github.com/openresty/lua-nginx-module#special-pcre-sequences

Best,
Vladislav

On Fri, Nov 21, 2014 at 1:39 AM, Gaurav Kumar <gaurav...@gmail.com> wrote:
I've following code which uses redis lua to push a binary value to "myList"

ngx.log(ngx.INFO,str2hex(dataToPush)) local ok,err =red:rpush("myList",dataToPush)

I am logging the data in hex format. When I do "LINDEX myList 0" on redis client, the results do not match.

The first few bytes of dataToPush printed in logs are-

\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03\x94\x50\x4d\x4f\xe3\x30\x10\x3d\xb3\xbf

Whereas output LINDEX myList 0 is - (snipped for brevity)

\x1f\x8b\b\x00\x00\x00\x00\x00\x00\x03\x94PMO\xe30\x10=\xb3\xbfb\x1a$\xbc{\xb0\

The code which assigns value to dataToPush is here. The value of dataToPush is basically gzipped response from a server which uses chunk encoding in this particular case .

So I believe that nginx lua redis client isn't putting binary data correctly in redis. When I've all ASCII data in dataToPush, everything looks fine.

.

manchev

You don't seem to understand the problem -- please do read the links from the previous mails, experiment with some code locally and then come back with questions.

Best,
Vladislav

On Fri, Nov 21, 2014 at 2:55 AM, Gaurav Kumar <gaurav...@gmail.com> wrote:

The thing is I don't have any string literals. The value of dataToPush is directly being assigned in this code- http://pastebin.com/qy7EPFeb
This is perhaps not an encoding issue but something else?

On Thu, Nov 20, 2014 at 4:51 PM, Vladislav Manchev <man...@bin.bz> wrote:
The backslash is used as a character escape in double and single-quoted strings, but not in long strings written with the [[...]] notation. That's how Lua works [1].

So, you can enclose the literal string (if you have the literal string, of course) like:
local a = [[\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03\x94\x50\x4d\x4f\xe3\x30\x10\x3d\xb3\xbf]]

Otherwise just use gsub to replace backslashes with double backslashes and you'll be fine.

[1]: http://www.lua.org/pil/2.4.html

Best,
Vladislav

On Fri, Nov 21, 2014 at 2:28 AM, Gaurav Kumar <gaurav...@gmail.com> wrote:
Thanks Vladislav!
Can you please provide me an example? Where exactly do I need to place double [[ ]] ? Do I need to have these the very time strings are assigned i.e. in this file http://pastebin.com/qy7EPFeb ?

Note that my LUA code is in separate file being called by using body_filter_by_lua_file directive.

On Thu, Nov 20, 2014 at 4:09 PM, Vladislav Manchev <man...@bin.bz> wrote:
You could use double long brackets [[]] around your strings in this case [1] or you'll need to escape the backslashes before sending to Redis.

[1]: https://github.com/openresty/lua-nginx-module#special-pcre-sequences

Best,
Vladislav

On Fri, Nov 21, 2014 at 1:39 AM, Gaurav Kumar <gaurav...@gmail.com> wrote:
I've following code which uses redis lua to push a binary value to "myList"

ngx.log(ngx.INFO,str2hex(dataToPush)) local ok,err =red:rpush("myList",dataToPush)

I am logging the data in hex format. When I do "LINDEX myList 0" on redis client, the results do not match.

The first few bytes of dataToPush printed in logs are-

\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03\x94\x50\x4d\x4f\xe3\x30\x10\x3d\xb3\xbf

Whereas output LINDEX myList 0 is - (snipped for brevity)

\x1f\x8b\b\x00\x00\x00\x00\x00\x00\x03\x94PMO\xe30\x10=\xb3\xbfb\x1a$\xbc{\xb0\

The code which assigns value to dataToPush is here. The value of dataToPush is basically gzipped response from a server which uses chunk encoding in this particular case .

So I believe that nginx lua redis client isn't putting binary data correctly in redis. When I've all ASCII data in dataToPush, everything looks fine.

.

gauravphoenix

Vladislav,

I did read the links and Googled before posting. All of the documentation I found including Stack Overflow questions mentions how to escape string literals i.e. when people are typing strings in lua code itself. If this specific case, the value of string isn't typed in the code but rather coming from a third party variable.

I ended up solving this problem by avoiding binaryvalues all together by decompressing gzip data by luazlib. Since luazlib uses streaming, I am finding it quite memory efficient too.

Cheers,

Gaurav

On Friday, November 21, 2014 3:53:47 AM UTC-8, Vladislav Manchev wrote:

You don't seem to understand the problem -- please do read the links from the previous mails, experiment with some code locally and then come back with questions.

Best,
Vladislav

On Fri, Nov 21, 2014 at 2:55 AM, Gaurav Kumar <gaurav...@gmail.com> wrote:
The thing is I don't have any string literals. The value of dataToPush is directly being assigned in this code- http://pastebin.com/qy7EPFeb
This is perhaps not an encoding issue but something else?

On Thu, Nov 20, 2014 at 4:51 PM, Vladislav Manchev <man...@bin.bz> wrote:
The backslash is used as a character escape in double and single-quoted strings, but not in long strings written with the [[...]] notation. That's how Lua works [1].

So, you can enclose the literal string (if you have the literal string, of course) like:
local a = [[\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03\x94\x50\x4d\x4f\xe3\x30\x10\x3d\xb3\xbf]]

Otherwise just use gsub to replace backslashes with double backslashes and you'll be fine.

[1]: http://www.lua.org/pil/2.4.html

Best,
Vladislav

On Fri, Nov 21, 2014 at 2:28 AM, Gaurav Kumar <gaurav...@gmail.com> wrote:
Thanks Vladislav!
Can you please provide me an example? Where exactly do I need to place double [[ ]] ? Do I need to have these the very time strings are assigned i.e. in this file http://pastebin.com/qy7EPFeb ?

Note that my LUA code is in separate file being called by using body_filter_by_lua_file directive.

On Thu, Nov 20, 2014 at 4:09 PM, Vladislav Manchev <man...@bin.bz> wrote:
You could use double long brackets [[]] around your strings in this case [1] or you'll need to escape the backslashes before sending to Redis.

[1]: https://github.com/openresty/lua-nginx-module#special-pcre-sequences

Best,
Vladislav

On Fri, Nov 21, 2014 at 1:39 AM, Gaurav Kumar <gaurav...@gmail.com> wrote:
I've following code which uses redis lua to push a binary value to "myList"

ngx.log(ngx.INFO,str2hex(dataToPush)) local ok,err =red:rpush("myList",dataToPush)

I am logging the data in hex format. When I do "LINDEX myList 0" on redis client, the results do not match.

The first few bytes of dataToPush printed in logs are-

\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03\x94\x50\x4d\x4f\xe3\x30\x10\x3d\xb3\xbf

Whereas output LINDEX myList 0 is - (snipped for brevity)

\x1f\x8b\b\x00\x00\x00\x00\x00\x00\x03\x94PMO\xe30\x10=\xb3\xbfb\x1a$\xbc{\xb0\

The code which assigns value to dataToPush is here. The value of dataToPush is basically gzipped response from a server which uses chunk encoding in this particular case .

So I believe that nginx lua redis client isn't putting binary data correctly in redis. When I've all ASCII data in dataToPush, everything looks fine.

.

agentzh

Hello!

On Thu, Nov 20, 2014 at 3:39 PM, Gaurav Kumar wrote:
> The first few bytes of dataToPush printed in logs are-
> \x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03\x94\x50\x4d\x4f\xe3\x30\x10\x3d\xb3\xbf
> Whereas output LINDEX myList 0 is - (snipped for brevity)
\x1f\x8b\b\x00\x00\x00\x00\x00\x00\x03\x94PMO\xe30\x10=\xb3\xbfb\x1a$\xbc{\xb0\
>

These two look equivalent to me. Are you aware that

1. \x08 is exactly \b?

    $ perl -e 'print "\x08"' | od -c
    0000000  \b
    0000001

2. \x50 is exactly P?

    $ perl -e 'print "\x50"'|od -c
    0000000   P
    0000001

3. \x4d is exactly M?

    $ perl -e 'print "\x4d"'|od -c
    0000000   M
    0000001

I could go on :)

Try compare their binary forms instead of their escaped forms which
may have equivalent variants to lead to confusions.

Regards,
-agentzh

gauravphoenix

Yeah.. it turned out finally that there was issue with data truncation rather than encoding.

On Fri, Nov 21, 2014 at 4:31 PM, Yichun Zhang (agentzh) <age...@gmail.com> wrote:

Hello!

On Thu, Nov 20, 2014 at 3:39 PM, Gaurav Kumar wrote:
> The first few bytes of dataToPush printed in logs are-
> \x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03\x94\x50\x4d\x4f\xe3\x30\x10\x3d\xb3\xbf
> Whereas output LINDEX myList 0 is - (snipped for brevity)
\x1f\x8b\b\x00\x00\x00\x00\x00\x00\x03\x94PMO\xe30\x10=\xb3\xbfb\x1a$\xbc{\xb0\
>

These two look equivalent to me. Are you aware that

1. \x08 is exactly \b?

$ perl -e 'print "\x08"' | od -c
0000000 \b
0000001

2. \x50 is exactly P?

$ perl -e 'print "\x50"'|od -c
0000000 P
0000001

3. \x4d is exactly M?

$ perl -e 'print "\x4d"'|od -c
0000000 M
0000001

I could go on :)

Try compare their binary forms instead of their escaped forms which
may have equivalent variants to lead to confusions.

Regards,
-agentzh
.

agentzh

Hello!

On Fri, Nov 21, 2014 at 4:44 PM, Gaurav Kumar wrote:
> Yeah.. it turned out finally that there was issue with data truncation
> rather than encoding.
>

Truncation due to the nginx error log message length limit?

Regards,
-agentzh

gauravphoenix

It was due to incorrectly decompressing gzip data. I will later debug why streaming unzipping didn't work when remote server uses chunk encoding.

On Fri, Nov 21, 2014 at 4:50 PM, Yichun Zhang (agentzh) <age...@gmail.com> wrote:

Hello!

On Fri, Nov 21, 2014 at 4:44 PM, Gaurav Kumar wrote:
> Yeah.. it turned out finally that there was issue with data truncation
> rather than encoding.
>

Truncation due to the nginx error log message length limit?

Regards,
-agentzh.