Periodic drop in throughput (about 120s intervals), Lua GC?

ruslan.talpa · 2016-10-27T21:39:14+00:00

this is a re-worked report i prepared that some might find useful. CAUTION: there are several seriously troubling events su...

Periodic drop in throughput (about 120s intervals), Lua GC?

ruslan.talpa

I am testing a system around openresty and i am hitting the box (t2.micro) with about 900rps and i see periodic drops in RPS at about 120s intervals.

no other errors is logs and the requests are all 200 OK.

I am not doing anything special in lua, very few things, and the request is proxied to an upstream.

Memory is relatively constant.

Is this coming from the GC? Any way to control even a bit it's behaviour?

Anything other happening in openresty at 2 min intervals?

Thanks

ruslan.talpa

Any way to play with these values at runtime (server startup), or can they be controlled only at compile time (and only by going to that specific file and changing them)?

/* Various tunables. */

#define LUAI_MAXSTACK 65500 /* Max. # of stack slots for a thread (<64K). */

#define LUAI_MAXCSTACK 8000 /* Max. # of stack slots for a C func (<10K). */

#define LUAI_GCPAUSE 200 /* Pause GC until memory is at 200%. */

#define LUAI_GCMUL 200 /* Run GC at 200% of allocation speed. */

#define LUA_MAXCAPTURES 32 /* Max. pattern captures. */

ruslan.talpa

Further investigation shows the 120 is not really fixed, the "drops" happens when the requests get at about 400/s, the time does not matter.

also things like

init_by_lua_block {

collectgarbage( ... )

}

don't seem to have an impact on the drop

ruslan.talpa

Tried nginx directly (in docker) with this config, same results (it's not the upstream, i get 900/rps going there directly).

Everything that is commented has been tried, not influence.

worker_processes 2;

#worker_rlimit_nofile 2048;

events {

# multi_accept on;

worker_connections 1024;

use epoll;

}

http {

resolver 127.0.0.11 ipv6=off;

include mime.types;

# tcp_nodelay off;

# tcp_nopush on;

upstream postgrest {

server postgrest:3000;

keepalive 64;

}

server {

listen 80;

server_name localhost;

charset utf-8;

location /rest/ {

default_type application/json;

# proxy_buffering off;

proxy_pass http://postgrest/; # Reverse proxy to your PostgREST

}

ruslan.talpa

moving discussion here http://stackoverflow.com/questions/26361678/efficiently-using-multiple-docker-containers-in-a-single-host since it's not a openresty thing