Yichun Zhang (agentzh) wrote on Saturday, October 17, 2015 8:01 AM
>On Sat, Oct 17, 2015 at 4:01 PM, Aapo Talvensaari wrote:
>> Basically this, but with small changes:
>>
>> I usually append to table with this:
>> buf[#buf+1] = line
>>
>> And in case you know the index you don't need to even use #buf (using
>> this #-length operator is just convenient in some more complex cases).
>Well, I'm against both for the sake of performance. Both table.insert(tb, elem) and tb[#tb + 1] are an O(n) operation since both of them need to >calculate the size of the Lua (array) table, which is not readily available according to the current table implementation in the standard Lua 5.1 >interpreter or LuaJIT 2. So if you use such things in a simple loop, it can easily become O(n^2) over all. I used to spot such things in real->world production flame graphs in the form of hot lj_tab_len() frames. LuaJIT 2 uses lj_tab_len to calculate the size of Lua array table, which is >called from within both table.insert (without an explicit index argument) and #tb. (It might be worth mentioning that #str does not have this >issue since Lua strings store their lengths explicitly in memory, unlike null-terminated C strings).
>
>The recommended way is to use a local Lua variable to track the current size of the Lua (array) table in the loop yourself, like
>
> local sz = 0
> local sz = {}
> for _, s in ipairs(...) do
> if mytest(s) then
> sz = sz + 1
> tb[sz] = i
> end
> end
>
Very helpful, thanks
>Or better, pre-allocate the lua table if you can (roughly) predict the size of the resulting table to avoid table auto growth (which is also an >O(n) operation with potential more expensive memory dynamic allocations). That is, to replace the line
>
> local tb = {}
>
>with
>
> local tb = tb_new(m, 0)
>
>where tb_new is initialized in the top level scope like this:
>
> local tb_new = require "table.new" -- this requires LuaJIT v2.1
>
That's a great tip, thanks
>> And you don't need to append '\n' to the table, because you can use:
>> table.concat(table, "\n")
>
>Well, I must say that it's (usually) more efficient to feed a Lua
> (array) table directly into calls like ngx.say() since it can collect the string pieces in the Lua table directly on the C land without creating >an intermediate (long) Lua string object that is almost immediately discarded. So manually inserting the "\n" string to the table in the loop and >finally inserting that table directly to
>ngx.say() can be more efficient (unless the result of this table.concat call never changes, which is case when loading a single invariant request >with tools like ab or weighttp). So be careful while benchmarking this, since Lua's string intern'ing might be helping you too much to be >realistic. Well, just a warning.
I suppose if you're using a fixed-size array, you could assign the '\n' to the odd positions just one time.
>> If you know the number of entries, you can preallocate the table with
>> table.new(narray, nhash), and you can reuse the table with table.clear(tab).
>Oh, yeah, I've mentioned the table.new() trick above as well. But be careful with table reuse via table.clear() across multiple requests.
>Since you may have race conditions with concurrent requests if you are not careful enough. Just a caveat. I meant to opensource a lua-resty->table-pool library I wrote a while back which can manage the table recycling for you.
With this caveat are you essentially warning against sharing the table between light threads? Or is there some non-obvious problem with using a local table?
----------------------------------------------------------------------
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
