Hello!
On Thu, Apr 9, 2015 at 3:19 PM, Lord Nynex wrote:
> I don't want to send the lua land flame graph because it reveals some
> sensitive information. (we are under a cloudflare mutual NDA if you want to
> look at it in private) This project basically just finds intersections
> between flat tables. { "aaa", "bbb", "ccc" } intersected against { "ccc",
> "ddd", "eee" }. But there are hundreds of tables.
>
Creating many such short-lived little Lua tables put pressure on
LuaJIT's GC, which should be eliminated.
> Right now the biggest culprit is I'm using Moses
> (https://github.com/Yonaba/Moses) intersect()
> (https://github.com/Yonaba/Moses/blob/master/moses.lua#L919) in a pairs()
> loop. This is a huge problem in the flame graph. The intersect function
> returns a bunch of anonymous iterator functions which appear to be quite
> expensive.
>
Yes, anonymous functions cannot be JIT compiled (yet) and also
requires dynamic allocations (again, GC burden) :)
> In this case, the pairs() loops are unavoidable by design. Is there any
> performance win I can get for fast intersection? Would it be worth
> implementing in C and passing two tables to an FFI func?
>
If we can avoid short-lived tables (and other GC objects) altogether,
then it will make a huge difference :)
I'm not sure about your detailed requirements but the rule of thumb is
to avoid using Lua for very complicated and GC-intensive computations.
Such computations better go onto the C land if feasible. And yeah, FFI
is your friend :) Well, just my 2 cents :)
Regards,
-agentzh