Hello!
On Thu, Nov 1, 2012 at 1:40 PM, milspec wrote:
> I've seen the new function get_keys.
>
> Could you enhance this function to accept the key from where to
> begin/continue?
>
> keys=get_keys(MAX_ELEMENTS, "start_key");
>
> Why?
>
> Imagine you have 100,000 elements in shm and you want to iterate over
> all of them. Then you might run into memory problems. Instead you just grab
> lets say 100 items a time and tell get_keys the next time at which key to
> continue.
>
> Keys in SHM ordered by its crc32-representation: "hello", "mouse", "dog",
> "cat", "water"
> max keys per get_keys: 2
>
> you do:
>
> keys1_till_2=get_keys(2, ""); --"hello", "mouse"
>
> In keys1_till_2, the last key is: "mouse", now you do:
>
> keys3_till_4=get_keys(2, "mouse"); --"dog", "cat"
>
> keys5_till_6=get_keys(2,"cat"); ---"water"
>
> keys7_till_8=get_keys(2,"water"); ---no more left.
>
>
> In case a key (its crc32-representation) is no longer there and you want to
> continue to traverse,
> just seek for the nearest bigger / smaller crc32-key (depending from which
> direction the traversal happens),
> so you could traverse the full SHM without a permanent lock even when in the
> meantime keys got
> deleted.
>
> I understand that the keys that get returned are not ordered alphabetically,
> they are ordered by their
> crc32-representation which is ok.
>
> In case you don't want to program it, maybe the patch contributor of the
> get_keys patch could do?
> I have no contact details, you could forward this email.
>
I've been thinking along the same line in fact ;)
Well, the current implementation of get_keys() does not traverse the
rbtree directly. Instead, it traverses the LRU queue from the most
recently used items to the least recently used ones. See here:
https://github.com/chaoslawful/lua-nginx-module/blob/master/src/ngx_http_lua_shdict.c#L624
So maybe we could just add an optional "offset" argument to get_keys
to specify where to begin in the current iteration for the queue. But
compared to your approach, duplicate items are more likely to get
returned by a later invocation if other workers actively modify the
store. I'm not sure about the practical value.
Also cc'ing Brian Akins and the openresty-en mailing list:
https://groups.google.com/group/openresty-en
Best regards,
-agentzh
