Two failed assertions when upgrading from 1.4.2.7 -> 1.4.3.6

leafot · 2013-12-02T12:48:43+00:00

Hello! On Thu, Feb 25, 2016 at 3:39 PM, Erik Wickstrom wrote: > # ./ngx-lua-bt -p 24492 --luajit20 > semantic error: while resolving probe point: iden...

Two failed assertions when upgrading from 1.4.2.7 -> 1.4.3.6

leafot

When upgrading to 1.4.3.6 I came across two instances of failed assertions when issuing http requests through a proxy_pass internal location. (Yes, I know you don't approve but I don't think https is supported in any of the cosocket implementations yet)

I've provided an nginx configuration that reliably fails on my system:

https://gist.github.com/leafo/7745145

I was able to find two separate issues, when accessing /first I get the following:

../ngx_lua-0.9.2/src/ngx_http_lua_headers_in.c:706: ngx_http_lua_rm_header_helper: Assertion `cur->next != ((void *)0)' failed.

2013/12/01 20:34:21 [notice] 1903#0: signal 17 (SIGCHLD) received

2013/12/01 20:34:21 [alert] 1903#0: worker process 1926 exited on signal 6 (core dumped)

2013/12/01 20:34:21 [notice] 1903#0: start worker process 1992

The failure happens on the second request. I inserted some dummy headers, seems to be related to the size of the headers being sent but I didn't do extensive testing.

On /second I get the following error:

../ngx_lua-0.9.2/src/ngx_http_lua_headers_in.c:176: ngx_http_set_header_helper: Assertion `!(r->headers_in.headers.part.next == ((void *)0) && r->headers_in.headers.last != &r->headers_in.headers.part)' failed.

2013/12/01 20:33:34 [notice] 1903#0: signal 17 (SIGCHLD) received

2013/12/01 20:33:34 [alert] 1903#0: worker process 1911 exited on signal 6 (core dumped)

2013/12/01 20:33:34 [notice] 1903#0: start worker process 1926

This one is similar to above but happens when a body is set on the initial request. The failure happens on the last request.

Thanks, tell me if you need any more information.

agentzh

Hello!

On Sun, Dec 1, 2013 at 8:48 PM, leaf corcoran wrote:
> When upgrading to 1.4.3.6 I came across two instances of failed assertions
> when issuing http requests through a proxy_pass internal location.

Yes, modifying subrequest's request headers is known to have memory
issues (for long, even long before the time when the related
assertions were added).

I'll work on a patch to fix the issues here shortly. Thanks for your test case!

> (Yes, I
> know you don't approve but I don't think https is supported in any of the
> cosocket implementations yet)
>

Yeah, I have to speed up the integration of the SSL cosocket patch
from Aviram Cohen :)

Best regards,
-agentzh

agentzh

Hello!

On Tue, Dec 3, 2013 at 10:47 AM, Yichun Zhang (agentzh) wrote:
> Yes, modifying subrequest's request headers is known to have memory
> issues (for long, even long before the time when the related
> assertions were added).
>
> I'll work on a patch to fix the issues here shortly. Thanks for your test case!
>

I've just committed a patch to ngx_lua's git master to fix the issue here:

    https://github.com/chaoslawful/lua-nginx-module/commit/5ed4aa8

With this patch, neither of you test cases gives me assertion
failures. Also, I've added your test cases to the test suite, with
minor modifications to make them more self-contained (you surely don't
want my EC2 test cluster to keep hitting your site ;)).

Please try the latest git master on your side :)

Thanks!
-agentzh

leafot

Thanks for the quick fix, works great.

Happened to notice another change with your recent changes to lua_code_cache off:

https://gist.github.com/leafo/7784297

The two servers print out different values when using the LUA_PATH/LUA_CPATH environment variables. When the code cache is off my environment variable search paths aren't used.

I noticed if I added

env LUA_PATH;

env LUA_CPATH;

It would fix the search paths, is this intended?

On Tue, Dec 3, 2013 at 1:39 PM, Yichun Zhang (agentzh) <age...@gmail.com> wrote:

Hello!

On Tue, Dec 3, 2013 at 10:47 AM, Yichun Zhang (agentzh) wrote:
> Yes, modifying subrequest's request headers is known to have memory
> issues (for long, even long before the time when the related
> assertions were added).
>
> I'll work on a patch to fix the issues here shortly. Thanks for your test case!
>

I've just committed a patch to ngx_lua's git master to fix the issue here:

https://github.com/chaoslawful/lua-nginx-module/commit/5ed4aa8

With this patch, neither of you test cases gives me assertion
failures. Also, I've added your test cases to the test suite, with
minor modifications to make them more self-contained (you surely don't
want my EC2 test cluster to keep hitting your site ;)).

Please try the latest git master on your side :)

Thanks!

-agentzh.

agentzh

Hello!

On Wed, Dec 4, 2013 at 12:48 AM, leaf corcoran wrote:
> Thanks for the quick fix, works great.
>

Yay! :)

> Happened to notice another change with your recent changes to lua_code_cache
> off:
> https://gist.github.com/leafo/7784297
> The two servers print out different values when using the LUA_PATH/LUA_CPATH
> environment variables. When the code cache is off my environment variable
> search paths aren't used.
>

Thank you for the report! Just committed a fix for this:

  https://github.com/chaoslawful/lua-nginx-module/commit/2a12131

Please try it out :)

Best regards,
-agentzh

leafot

Thanks, works great

On Wed, Dec 4, 2013 at 1:57 PM, Yichun Zhang (agentzh) <age...@gmail.com> wrote:

Hello!

On Wed, Dec 4, 2013 at 12:48 AM, leaf corcoran wrote:
> Thanks for the quick fix, works great.
>

Yay! :)

> Happened to notice another change with your recent changes to lua_code_cache
> off:
> https://gist.github.com/leafo/7784297
> The two servers print out different values when using the LUA_PATH/LUA_CPATH
> environment variables. When the code cache is off my environment variable
> search paths aren't used.
>

Thank you for the report! Just committed a fix for this:

https://github.com/chaoslawful/lua-nginx-module/commit/2a12131

Please try it out :)

Best regards,

-agentzh.