Hi again,
On Tue, Oct 8, 2013 at 6:21 AM, Yichun Zhang (agentzh)
<age...@gmail.com> wrote:
> Hello!
>
> On Mon, Oct 7, 2013 at 10:03 AM, Rohit Yadav wrote:
>> Thanks a lot, the new gc-objs tool is much better. Though, I need some
>> help to understand the output:
>>
>
> Yeah, we definitely need more documentation for the output :) But I'll
> be more motivated if people ask concrete questions :) So thank you :)
>
>> - The GC total size is the amount of memory garbage collected (like
>> cleaned, freed) or amount of memory in use, held by luajit within the
>> nginx worker?
>> - Help in understanding the GC states: propagate, sweep, sweep-string,
>> pause etc.
>> - What are the objects actually? string, function, table, and proto.
>
> The "objects" are Lua values that actually participate in garbage
> collection (GC).
>
> Primitive Lua values like numbers, booleans, nils, and light user data
> do not participate in GC, so we shall never see them listed in the
> output of the ngx-lj-gc-objs tool.
>
> Another interesting exception is empty Lua strings, they are specially
> handled by LuaJIT and they never appear in the output either.
>
> The following types of objects participate in GC and thus considered
> "GC objects" in LuaJIT 2.0:
>
> * string: Lua strings
> * upvalue: Lua Upvalues
> * thread: Lua threads (i.e., Lua coroutines)
> * proto: Lua function prototypes
> * function: Lua functions (Lua closures) and C functions
> * cdata: cdata created by the FFI API in Lua.
> * table: Lua tables
> * userdata: Lua user data
> * trace: JIT compiled Lua code paths
>
> Note that the space calculated for aggregate objects like "table" and
> "function", only the size of their backbones is calculated. For
> example, for a Lua table, we do not follow the references to its value
> objects and key objects (but we do include the size of the "key" and
> "value" reference pointers themselves). Similarly, for a Lua function
> object, we do not follow its upvalues either. Therefore, we can safely
> add up the sizes of all the GC objects to obtain the total size
> allocated by the LuaJIT GC without repeating anyone.
>
> It's worth mentioning that the LuaJIT GC may also allocate some space
> for some components in the VM itself (on demand):
>
> * the JIT compiler state (i.e., the "JIT state size" item in the tool's output).
> * the state for FFI C types (i.e., the "C type state size" item in the output).
> * global state temporary buffer (i.e., the "global state tmpbuf size" item).
>
> These allocations may not happen for trivial examples. For example, if
> the JIT compiler is disabled, we won't see nonzero "JIT state size".
> Similarly, if we don't use FFI in our Lua code at all, we won't see
> nonzero "C type state size" either.
>
>> - Is there a way to dispose off lua objects from lua code (like force
>> garbage collection, like C++'s delete?).
>>
>
> Like any language with a proper GC, any unused Lua objects will be
> automatically freed by the LuaJIT GC. You make a Lua object become
> unused by avoid referencing the object from any "GC root objects"
> either directly or indirectly.
>
> You force the LuaJIT GC to free all the unsed objects at once by calling
>
> collectgarbage()
>
> This forces the LuaJIT (or standard Lua interpreter) GC to run a
> complete collection cycle. See
> http://www.lua.org/manual/5.1/manual.html#pdf-collectgarbage for more
> details.
>
Thanks for a detailed reply, it really helps to understand the details.
> But it's usually very expensive to call and it's usually not required
> unless your Lua code is really really heavy on allocations (sometimes
> it's due to programming mistakes in the Lua code) because LuaJIT (and
> the standard Lua 5.1 interpreter) uses an incremental GC.
>
>> For a simple test, where for a url using lua we're printing output
>> (simply ngx.print('string')), I saw an increase in string, table and
>> function objects everytime I request the url (and it runs the lua
>> code). I don't know how to interpret this data.
>>
>
> Apparently you did not keep requesting :) It's fine for Lua objects to
> stay a little longer then needed. You'll see the number of objects
> going down after dozens of requests :) Basically it's fine and also
> normal to see the GC count (or the memory allocated by the GC) going
> up and down during the lifetime of the process.
While I used loader.io to send about two tests of 50k requests. You're
right, it does clear up with time.
> That's how a typical incremental GC works. The LuaJIT GC cycle
> consists of various different phases. And all these phases are divided
> into many small pieces that interleave with normal Lua code execution.
> For example, when the GC cycle is at the "sweep-string" phase,
> non-string GC objects will not freed at all until the GC cycle later
> enters the "sweep" phase.
>
> The ngx-lj-gc-objs tool is good at finding out the largest GC objects
> in your already bloated Nginx worker processes.
>
> For your simple Lua handlers calling ngx.print(), the newly allocated
> Lua thread object is the Lua main "light thread" for your request
> handler (be it content_by_lua, access_by_lua, or rewrite_by_lua), the
> newly allocated Lua table is the global environment table for the
> request handler, and the newly allocated function is from the closure
> factory for the current Lua thread for the current request handler (we
> need a separate Lua function for a separate global variable
> environment for each request handler).
>
>> In my case I'm using shared lua dictionary to store things to be
>> shared among nginx workers using Lua. I see that nginx workers keep
>> eating RAM and only a reload saves the system from thrashing (this is
>> bad as it increases system load and http requests take more time to
>> process, so more latencies).
>>
>
> We also use ngx_lua's shared dictionary heavily online and the LuaJIT
> GC count (from the ngx-lj-gc tool) remains around 700KB ~ 1MB per
> worker for one ngx_lua-based service and 20MB ~ 26MB per worker for
> another service under really heavy traffic.
>
>> For populating the shared lua dictionary, I'm using (lua-resty-mysql)
>> mysql to populate initially where I flush all the shared keys in the
>> dictionary, when I do that I see for a test setup (with four nginx
>> workers) tremendous increase in the GC objects and overall GC total
>> size each time I run the build cache url.
>>
>
> The ngx-lj-gc and ngx-lj-gc-objs tools are not designed for debugging
> individual URL queries due to the non-determinism of the GC on micro
> levels. You should load your nginx workers by tools like ab and
> weighttp and make your nginx worker eat up a *lot* of memory. The
> more, the merrier. After that, run ngx-lj-gc-objs on your largest
> nginx worker process :)
I've another question to ask you; we do a lot of regex matches within
lua and have a large lua_regex_cache_max_entries value, could this
contribute to a lot of RAM consumption as the
lua_regex_cache_max_entries is held per worker and we see that when
openresty is reloaded a lot of RAM gets freed?
>
> Could you provide the original output from the ngx-lj-gc-objs tool for
> your largest nginx worker? Without seeing the details, I cannot give
> any suggestions for further steps :)
Yes, so this was done on a VM and not the actual production system;
Output on clean run;
Start tracing 123056 (/var/www/dev/bin/nginx)
GC total size: 47089 bytes
GC state: sweep-string
421 string objects: max=342, avg=27, min=18, sum=11509 (in bytes)
305 function objects: max=144, avg=32, min=20, sum=9920 (in bytes)
57 table objects: max=3104, avg=292, min=32, sum=16696 (in bytes)
5 proto objects: max=200, avg=159, min=126, sum=799 (in bytes)
4 userdata objects: max=64, avg=46, min=40, sum=184 (in bytes)
2 upvalue objects: max=24, avg=24, min=24, sum=48 (in bytes)
1 thread objects: max=424, avg=424, min=424, sum=424 (in bytes)
global state tmpbuf size: 301 bytes
My GC walker detected for total 47089 bytes.
354 microseconds elapsed in the probe handler.
Start tracing 123056 (/var/www/dev/bin/nginx)
Output after 100k http requests;
GC total size: 156682 bytes
GC state: pause
867 string objects: max=540, avg=31, min=18, sum=27415 (in bytes)
559 function objects: max=144, avg=29, min=20, sum=16696 (in bytes)
223 upvalue objects: max=24, avg=24, min=24, sum=5352 (in bytes)
211 table objects: max=6176, avg=167, min=32, sum=35264 (in bytes)
126 userdata objects: max=1384, avg=58, min=29, sum=7390 (in bytes)
114 proto objects: max=2357, avg=344, min=90, sum=39293 (in bytes)
17 thread objects: max=832, avg=788, min=424, sum=13400 (in bytes)
JIT state size: 2400 bytes
global state tmpbuf size: 216 bytes
My GC walker detected for total 156682 bytes.
871 microseconds elapsed in the probe handler.
Thanks for all your help agentzh, you're awesome as always.
Regards.
>
> Best regards,
> -agentzh.
