Are coroutines getting garbage collected when started in timer context?

gpuhome · 2018-05-15T04:23:39+00:00

Hello. I wanted to make some changes on existing nginx configuration. My previous configuration was like this:Then I added content_by_lua_block as below:My /...

Are coroutines getting garbage collected when started in timer context?

gpuhome

Hi,

I think I have discovered a potential issue which in some scenarios could bring Nginx to a halt...

I have a small application (running on the latest Openresty bundle) which sends hundreds of thousands of HTTP requests out (doing nothing mischievous - just to build maps of the whole world and cache them on my server).

The whole application is running in ngx.timer context (since it is getting started by init_worker_by_lua), and it scans maps cache for missing parts and sends out http requests to my backend, receives back map's data and fills up gaps.

This application uses https://github.com/pintsized/lua-resty-http client - and everything works fine. However, the application slowly slows down - it gets noticeable after first 100000 requests or so - and then it just gets slower and slower (so I need to restart Nginx every 30 minutes - which corresponds to about 2 millions of sent requests).

I did some testing - this issue only exists when HTTP response has some body (even if it is a very small one - about 30 bytes of JSON in my case). Changing outgoing request's method to HEAD completely eliminates the issue.

Looking inside http client source code I have noticed that for each HTTP request it starts HTTP response body reading coroutine (to support data streaming) - looks like it is a major difference between GET and HEAD (for example) requests. Looking in to lua-nginx-module source code - I can not identify a spot where these coroutines are supposed to get collected once they are done running.

So, here is a question - are coroutines getting garbage collected when started in timer context?

Thanks,

Gene

zchao1995

Hello!

I think the garbage isn't relevant with the ngx_lua's contest, any gc objects might be recycled when they were never used. Did you forget to do some cleanup jobs? Like set_keepalive or close?

gpuhome

Hi,

There are no any HTTP connections cleanup related problems - haven't I mentioned that everything works properly for HEAD (or any other HTTP request that doesn't have body)?

Most of the testing was done using simple httpc:request_uri() call - which automatically does set_keepalive() or close() for each request.

> I think the garbage isn't relevant with the ngx_lua's contest, any gc objects might be recycled when they were never used.

I think that you are mistaken over here - while it could be true for normal Lua objects - I do think coroutines are a special case since they are re-implemented in Openresty:

https://github.com/openresty/lua-nginx-module/blob/master/src/ngx_http_lua_coroutine.c

I'm looking at this source code and it is not clear to me how and at what point finished coroutines are supposed to be "collected" (which should free allocated to them resources) - since timer thread is going to run forever...

Thanks,

Gene

On Monday, May 14, 2018 at 9:50:33 PM UTC-4, tokers wrote:

Hello!

I think the garbage isn't relevant with the ngx_lua's contest, any gc objects might be recycled when they were never used. Did you forget to do some cleanup jobs? Like set_keepalive or close?

zchao1995

Hello!

>> I think the garbage isn't relevant with the ngx_lua's contest, any gc objects might be recycled when they were never used.

> I think that you are mistaken over here - while it could be true for normal Lua objects - I do think coroutines are a special case since they are re-implemented in

> Openresty:

> https://github.com/openresty/lua-nginx-module/blob/master/src/ngx_http_lua_coroutine.c

> I'm looking at this source code and it is not clear to me how and at what point finished coroutines are supposed to be "collected" (which should free allocated to them resources) - since timer thread is going to run forever...

Yes, OpenResty wraps the coroutine and schedule by itself, however, the underlying coroutine will be unreferenced after LuaL_unref is called (these coroutine objects are referenced in a Lua table).

gpuhome

Hi,

>> I think that you are mistaken over here - while it could be true for normal Lua objects - I do think coroutines are a special case since they are re-implemented in
>> Openresty:
>>
>> https://github.com/openresty/lua-nginx-module/blob/master/src/ngx_http_lua_coroutine.c
>>
>> I'm looking at this source code and it is not clear to me how and at what point finished coroutines are supposed to be "collected" (which should free allocated to them resources) - since timer thread is going to run forever...

Yes, OpenResty wraps the coroutine and schedule by itself, however, the underlying coroutine will be unreferenced after LuaL_unref is called (these coroutine objects are referenced in a Lua table).

Agree. However, it brings up another question - when and under what conditions LuaL_unref is going to get called for coroutine? For example, I can see it gets called at thread.wait() if thread is zombie. However - I don't see LuaL_unref getting called anywhere inside coroutines API...

zchao1995

Hello!

> Agree. However, it brings up another question - when and under what conditions LuaL_unref is going to get called for coroutine? For example, I can see it gets called at thread.wait() if thread is zombie. However - I don't see LuaL_unref getting called anywhere inside coroutines API...

APIs like ngx.exit will result in finalizing the current Lua coroutine. In addition, when the corresponding Lua code has been executed totally, the Lua coroutine will be destoryed as well. You can see the function ngx_http_lua_run_thread for the details. :)