Trouble decoding user agent and url strings in nginx/openresty

rohit.yadav · 2013-11-12T20:51:30+00:00

Hello! On Mon, Nov 11, 2013 at 9:57 PM, Porfus wrote: > I use Openresty with Upload module. Use the following configuration: > It's recommended to us...

Trouble decoding user agent and url strings in nginx/openresty

rohit.yadav

Hello,

I'm using nginx/openresty (1.4.3.1) and trying to write user agent
(ngxVar.http_user_agent) and referer url (ngxVar.http_referer) as json
encoded string to Redis from Lua which is then read and decoded by a
Python daemon using the 'json' package.

The issue is with decoding the string at Python end; both the user
agent and referrer url strings are not UTF8 encoded as expected, so I
tried to use ISO-8859-1 which worked for some URLs but broke for other
encoding. What is the default encoding of these strings and how can we
force them to be UTF8 in the json encoded string at Lua end? Looks
like someone else also got such an issue [1].

[1] http://forum.nginx.org/read.php?2,243106,243106#msg-243106

Regards,
Rohit Yadav
Software Engineer, Wingify

http://visualwebsiteoptimizer.com
World's easiest A/B testing software

Wingify Software Pvt. Ltd.
14th Floor, KLJ Tower North,
Netaji Subhash Place,
Pitam Pura, Delhi 110034,
India

agentzh

Hello!

On Tue, Nov 12, 2013 at 4:51 AM, Rohit Yadav wrote:
>
> The issue is with decoding the string at Python end; both the user
> agent and referrer url strings are not UTF8 encoded as expected, so I
> tried to use ISO-8859-1 which worked for some URLs but broke for other
> encoding. What is the default encoding of these strings and how can we
> force them to be UTF8 in the json encoded string at Lua end?

You didn't mention what JSON library you're using on the Lua side.

Assuming you're using lua-cjson, you should always feed valid UTF-8
strings into cjson.encode() yourself. The lua-cjson library does not
do any encoding conversion here. Also, lua-cjson does not bother
escaping multi-byte UTF-8 character sequences. See lua-cjson's manual
for more details:

    http://www.kyne.com.au/~mark/software/lua-cjson-manual.html

> Looks
> like someone else also got such an issue [1].
>
> [1] http://forum.nginx.org/read.php?2,243106,243106#msg-243106
>

Nginx (and ngx_lua) never tries to make sense of the character
encoding in any input data nor does any encoding conversions unless
you explicitly configure them to do it. They just offer the raw
(binary) data from the wire by default. It is up to you to make sense
of the character encoding actually used in the binary strings.

Regards,
-agentzh

rohit.yadav

On Wed, Nov 13, 2013 at 12:47 AM, Yichun Zhang (agentzh)
<age...@gmail.com> wrote:
> Hello!
>
> On Tue, Nov 12, 2013 at 4:51 AM, Rohit Yadav wrote:
>>
>> The issue is with decoding the string at Python end; both the user
>> agent and referrer url strings are not UTF8 encoded as expected, so I
>> tried to use ISO-8859-1 which worked for some URLs but broke for other
>> encoding. What is the default encoding of these strings and how can we
>> force them to be UTF8 in the json encoded string at Lua end?
>
> You didn't mention what JSON library you're using on the Lua side.

Oh, yes that was indeed lua-cjson.

>
> Assuming you're using lua-cjson, you should always feed valid UTF-8
> strings into cjson.encode() yourself. The lua-cjson library does not
> do any encoding conversion here. Also, lua-cjson does not bother
> escaping multi-byte UTF-8 character sequences. See lua-cjson's manual
> for more details:
>
>     http://www.kyne.com.au/~mark/software/lua-cjson-manual.html
>
>> Looks
>> like someone else also got such an issue [1].
>>
>> [1] http://forum.nginx.org/read.php?2,243106,243106#msg-243106
>>
>
> Nginx (and ngx_lua) never tries to make sense of the character
> encoding in any input data nor does any encoding conversions unless
> you explicitly configure them to do it. They just offer the raw
> (binary) data from the wire by default. It is up to you to make sense
> of the character encoding actually used in the binary strings.

Thanks for the information, I was confused if nginx forces any default
encoding as some of url strings had the utf8 encoding while others
were ISO-8859-1. Good this clears up.

Regards.

>
> Regards,
> -agentzh.