NGINX Shared dictionary - Data size

gbshiva · 2016-07-27T11:02:37+00:00

Cheers thanks for that.. was curious as some folks have been actually blocking the requests with Proxy header returning 403 or 400 errors for them.On Saturda...

NGINX Shared dictionary - Data size

gbshiva

We are using shared dictionary to store large amounts of data. The data set size is about 20 - 25 M records with the file size containing all the records = 2.5GB.

When i load the data from this file ( Key / Value) NGINX seem to occupy a lot of memory ( duff / cache) approx 10GB. I have checked the code and ensure there is no duplicate entries added etc.

Am i missing something here.

What is the overhead of storing data into shared dictionary. For every element stored how many bytes of additional data does nginx add to track expiry etc.

Appreciate any help on this .

moonbingbing

you can use ngx-shm( https://github.com/openresty/nginx-systemtap-toolkit#ngx-shm) to analyzes all the shared memory zones.

2016-07-27 11:02 GMT+08:00 Shivakumar Gokaram <gbs...@gmail.com>:

We are using shared dictionary to store large amounts of data. The data set size is about 20 - 25 M records with the file size containing all the records = 2.5GB.
When i load the data from this file ( Key / Value) NGINX seem to occupy a lot of memory ( duff / cache) approx 10GB. I have checked the code and ensure there is no duplicate entries added etc.

Am i missing something here.
What is the overhead of storing data into shared dictionary. For every element stored how many bytes of additional data does nginx add to track expiry etc.

Appreciate any help on this .

.

agentzh

Hello!

On Tue, Jul 26, 2016 at 8:02 PM, Shivakumar Gokaram wrote:
> We are using shared dictionary to store large amounts of data. The data set
> size is about 20 - 25 M records with the file size containing all the
> records = 2.5GB.
> When i load the data from this file ( Key / Value) NGINX seem to occupy a
> lot of memory ( duff / cache) approx 10GB.  I have checked the code and
> ensure there is no duplicate entries added etc.
>

Because there are many different memory metrics in a modern *NIX
system, what kind of memory metric are you referring to? It is virtual
memory size (VIRT) or residence size (RSS)?

Your latest private email seems to provide more useful details, please
allow me to quote it here:

> I want to store a file with key / value pairs and the file size is about 3.1GB ( output of ls-lah).
>  The file contains 69M lines of the format  key,val.
> I am reading this file in lua and storing into nginx shared dictionary.  I am facing No Memory error when
> i set the LUA Dictionary size to anything less <12GB.  It works fine if the size is set of dic 12GB+.
>  This is almost 4x the size of my data file.

Okay, this paragraph contains much more detailed information. So we
can do the math here:

3.1GB file with 69M lines of key-value pairs per line, which means
each key-value pair is about 48.2 bytes on average. Excluding the
comma and trailing new line, then it should be about 46 bytes each for
the real data.

You said that lua_shared_dict requires 12GB of room to accommodate all
these 69M key-value pairs, which means for each key-value pair should
need 186.74 bytes on average. Let's subtract out the true data size,
46 bytes, then we have an average per-pair memory overhead of about
140 bytes.

The true memory overhead (including red-black tree, expiration queues
and timestamps, key length field and value length field) for each
key-value pair in lua_shared_dict is currently 76 bytes on a typical
64-bit Linux system, plus a little memory overhead of the nginx slab
allocator (which should be very small for each pair). So what you're
seeing is 1.8x as much as the real overhead. But it's still not wildly
divergent. Not sure if it's calculation error or anything else.

Oh, BTW, I've just rearranged the lua_shared_dict node struct layout
and saves 8 byte overhead for each key-value pair on x86_64 systems at
least:

https://github.com/openresty/lua-nginx-module/commit/da08f59a

So now it's just 68 bytes per key-value pair. You can try this patch
out though do not expect to see a dramatic difference, obviously ;)

Regards,
-agentzh

gbshiva

Thank you of the detailed explanation and the current overhead for each entry. I will dig a bit more into the error we faced and compare the RSS and VIRTM for the worker process.

Will also give a shot at the patch below.

Shiva

On Monday, October 3, 2016 at 2:10:23 AM UTC-4, agentzh wrote:

Hello!

On Tue, Jul 26, 2016 at 8:02 PM, Shivakumar Gokaram wrote:
> We are using shared dictionary to store large amounts of data. The data set
> size is about 20 - 25 M records with the file size containing all the
> records = 2.5GB.
> When i load the data from this file ( Key / Value) NGINX seem to occupy a
> lot of memory ( duff / cache) approx 10GB. I have checked the code and
> ensure there is no duplicate entries added etc.
>

Because there are many different memory metrics in a modern *NIX
system, what kind of memory metric are you referring to? It is virtual
memory size (VIRT) or residence size (RSS)?

Your latest private email seems to provide more useful details, please
allow me to quote it here:

> I want to store a file with key / value pairs and the file size is about 3.1GB ( output of ls-lah).
> The file contains 69M lines of the format key,val.
> I am reading this file in lua and storing into nginx shared dictionary. I am facing No Memory error when
> i set the LUA Dictionary size to anything less <12GB. It works fine if the size is set of dic 12GB+.
> This is almost 4x the size of my data file.

Okay, this paragraph contains much more detailed information. So we
can do the math here:

3.1GB file with 69M lines of key-value pairs per line, which means
each key-value pair is about 48.2 bytes on average. Excluding the
comma and trailing new line, then it should be about 46 bytes each for
the real data.

You said that lua_shared_dict requires 12GB of room to accommodate all
these 69M key-value pairs, which means for each key-value pair should
need 186.74 bytes on average. Let's subtract out the true data size,
46 bytes, then we have an average per-pair memory overhead of about
140 bytes.

The true memory overhead (including red-black tree, expiration queues
and timestamps, key length field and value length field) for each
key-value pair in lua_shared_dict is currently 76 bytes on a typical
64-bit Linux system, plus a little memory overhead of the nginx slab
allocator (which should be very small for each pair). So what you're
seeing is 1.8x as much as the real overhead. But it's still not wildly
divergent. Not sure if it's calculation error or anything else.

Oh, BTW, I've just rearranged the lua_shared_dict node struct layout
and saves 8 byte overhead for each key-value pair on x86_64 systems at
least:

https://github.com/openresty/lua-nginx-module/commit/da08f59a

So now it's just 68 bytes per key-value pair. You can try this patch
out though do not expect to see a dramatic difference, obviously ;)

Regards,
-agentzh

gbshiva

I did some additional tests on this shared dictionary and looks like data fragmentation is creating a huge impact.

Tests conducted

1. Created a file of 6GB with 90M rows all of equal length. Loaded this file into a shared dictionary on size 7GB and it works.

2. Created a file of 6GB with 90M but with different lengths in each row. This time when loading into shared dictionary i got a out of memory error. Increased the Shared dictionary size to 15GB still no luck. I was finally able to load the data when setting the dictionary size to 25GB.

Sample python script to generate data

import os

import sys

import logging

import traceback

from optparse import OptionParser

if __name__ == "__main__":

outputfile="fixlen.data"

f = open(outputfile,'w')

altflag=True

for x in range(0,90000000):

data="">

if altflag:

f.write(data+"|"+data+data+data+data+data+data+"\n")

altflag=False

else:

altflag=True

f.write(data+"|"+data+data+data+data+data+data+data+data+data+data+data+"\n")

f.close()

Load this file into Shared dictionary by splitting every line at '|'

LUA CODE:

local count=0
ngx.say("Clearing data from cache ")
fcctcccache:flush_all()
fcctcccache:flush_expired()


ngx.say("Loading Data from Filename = "..pfilename)

for line in io.lines(pfilename) do
    local i = string.find(line,"|",0)
    if i~= nil then
        local key = string.sub(line,1,i-1)
        local val = string.sub(line,i+1)
        local succ, err, forcible = fcctcccache:set(key,val)
        if succ ~= nil and succ == false then
            ngx.say("Error Loading Data to cache. ")
            ngx.say(err)
            ngx.exit(200)
        end
    end
    count=count+1
end

Is there any way to to rebalance the data or set page sizes or anything to efficiently utilize memory and avoid this.

Shiva

On Monday, October 3, 2016 at 11:40:23 PM UTC-4, Shivakumar Gokaram wrote:

Hi

Thank you of the detailed explanation and the current overhead for each entry. I will dig a bit more into the error we faced and compare the RSS and VIRTM for the worker process.
Will also give a shot at the patch below.

Shiva

On Monday, October 3, 2016 at 2:10:23 AM UTC-4, agentzh wrote:
Hello!

On Tue, Jul 26, 2016 at 8:02 PM, Shivakumar Gokaram wrote:
> We are using shared dictionary to store large amounts of data. The data set
> size is about 20 - 25 M records with the file size containing all the
> records = 2.5GB.
> When i load the data from this file ( Key / Value) NGINX seem to occupy a
> lot of memory ( duff / cache) approx 10GB. I have checked the code and
> ensure there is no duplicate entries added etc.
>

Because there are many different memory metrics in a modern *NIX
system, what kind of memory metric are you referring to? It is virtual
memory size (VIRT) or residence size (RSS)?

Your latest private email seems to provide more useful details, please
allow me to quote it here:

> I want to store a file with key / value pairs and the file size is about 3.1GB ( output of ls-lah).
> The file contains 69M lines of the format key,val.
> I am reading this file in lua and storing into nginx shared dictionary. I am facing No Memory error when
> i set the LUA Dictionary size to anything less <12GB. It works fine if the size is set of dic 12GB+.
> This is almost 4x the size of my data file.

Okay, this paragraph contains much more detailed information. So we
can do the math here:

3.1GB file with 69M lines of key-value pairs per line, which means
each key-value pair is about 48.2 bytes on average. Excluding the
comma and trailing new line, then it should be about 46 bytes each for
the real data.

You said that lua_shared_dict requires 12GB of room to accommodate all
these 69M key-value pairs, which means for each key-value pair should
need 186.74 bytes on average. Let's subtract out the true data size,
46 bytes, then we have an average per-pair memory overhead of about
140 bytes.

The true memory overhead (including red-black tree, expiration queues
and timestamps, key length field and value length field) for each
key-value pair in lua_shared_dict is currently 76 bytes on a typical
64-bit Linux system, plus a little memory overhead of the nginx slab
allocator (which should be very small for each pair). So what you're
seeing is 1.8x as much as the real overhead. But it's still not wildly
divergent. Not sure if it's calculation error or anything else.

Oh, BTW, I've just rearranged the lua_shared_dict node struct layout
and saves 8 byte overhead for each key-value pair on x86_64 systems at
least:

https://github.com/openresty/lua-nginx-module/commit/da08f59a

So now it's just 68 bytes per key-value pair. You can try this patch
out though do not expect to see a dramatic difference, obviously ;)

Regards,
-agentzh

agentzh

Hello!

On Thu, Nov 10, 2016 at 9:24 PM, Shivakumar Gokaram wrote:
> 2. Created a file of 6GB with 90M but with different lengths in each row.
> This time when loading into shared dictionary i got a out of memory error.
> Increased the Shared dictionary size to 15GB still no luck. I was finally
> able to load the data when setting the dictionary size to 25GB.
>

How different in the lengths? What are the maximum and minimum?

>
> Is there any way to to rebalance the data or set page sizes or anything to
> efficiently utilize memory and avoid this.
>

Some suggestions:

1. Since the insertion order of your Lua loop determines the
allocation order and thus their physical positions in the shm zone,
you can arrange your data insertion order offline in some way to fill
each 4KB memory page (which is the default on Linux) as much as
possible. We cannot easily do this automatically in the slab allocator
ourselves since the optimal algorithm's complexity is too high to be
feasible on this level.

2. You can experiment with different page sizes by patching the nginx
core's src/core/ngx_slab.c source files Basically you should replace
ngx_pagesize, ngx_pagesize_shift, and possibly other constants there
with your own values. Currently they are hard-coded and cannot be
configured.

Regards,
-agentzh

gbshiva

I will try this out and let me you the outcome.

Shiva

On Tuesday, July 26, 2016 at 11:02:37 PM UTC-4, Shivakumar Gokaram wrote:

We are using shared dictionary to store large amounts of data. The data set size is about 20 - 25 M records with the file size containing all the records = 2.5GB.
When i load the data from this file ( Key / Value) NGINX seem to occupy a lot of memory ( duff / cache) approx 10GB. I have checked the code and ensure there is no duplicate entries added etc.

Am i missing something here.
What is the overhead of storing data into shared dictionary. For every element stored how many bytes of additional data does nginx add to track expiry etc.

Appreciate any help on this .