cannot create more than 200 capture_multi requests

dlinenbe · 2015-10-06T23:28:45+00:00

Hey, how about that - a debug log that isn't just a bunch of unintelligible nonsense! Thanks for the tip!So after turning on the log, I discovered that m...

cannot create more than 200 capture_multi requests

dlinenbe

Hello,

I am very new to openresty. I am trying to work with ngx.location.capture_multi, and have noticed it is failing when I put more than 200 requests in.

My desire is to be able to use ngx.location.capture_multi with thousands of simultaneous requests.

A conf file which illustrates the error is below. I am running on a single core AWS microinstance with Amazon Linux.

http {

server {

listen 80;

location /hello {

content_by_lua '

ngx.say("hello")

}

location /grid {

content_by_lua '

local args = {};

for i=1,201 do -- works when last index is 200 or lower, fails at 201

args[i] = {"/hello"};

end

local rslt = {ngx.location.capture_multi(args)}

ngx.say("DONE")

}

The error I get, when I bump up the args count to 201 or greater is as follows in my error.log file:

2015/10/06 15:24:11 [notice] 3241#0: signal process started

2015/10/06 15:24:11 [alert] 3230#0: *10129 open socket #7 left in connection 2

2015/10/06 15:24:11 [alert] 3230#0: aborting

2015/10/06 15:24:15 [error] 3246#0: *10131 subrequests cycle while processing "/hello", client: 100.1.46.100, server: , request: "GET /grid HTTP/1.1", host: "54.175.150.16"

2015/10/06 15:24:15 [error] 3246#0: *10131 lua entry thread aborted: runtime error: content_by_lua(nginx.conf:27):6: failed to issue subrequest: -1

stack traceback:

coroutine 0:

[C]: in function 'capture_multi'

content_by_lua(nginx.conf:27):6: in function <content_by_lua(nginx.conf:27):1>, client: 100.1.46.100, server: , request: "GET /grid HTTP/1.1", host: "54.175.150.16"

2015/10/06 15:24:16 [error] 3246#0: *10132 open() "/home/ec2-user/investingengines/build_linux/resty/html/favicon.ico" failed (13: Permission denied), client: 100.1.46.100, server: , request: "GET /favicon.ico HTTP/1.1", host: "54.175.150.16", referrer: "http://54.175.150.16/grid"

[ec2-user@ip-172-31-55-176 resty]$

Thanks for any help

Dave Linenberg

info

I think your lucky getting 200 atm. see http://forum.nginx.org/read.php?29,261343
which I think is the reason here.

On Tuesday, October 6, 2015 at 5:28:45 PM UTC+2, David Linenberg wrote:

Hello,
I am very new to openresty. I am trying to work with ngx.location.capture_multi, and have noticed it is failing when I put more than 200 requests in.
My desire is to be able to use ngx.location.capture_multi with thousands of simultaneous requests.

A conf file which illustrates the error is below. I am running on a single core AWS microinstance with Amazon Linux.

http {
server {
listen 80;

location /hello {
content_by_lua '
ngx.say("hello")
';
}

location /grid {
content_by_lua '
local args = {};
for i=1,201 do -- works when last index is 200 or lower, fails at 201
args[i] = {"/hello"};
end
local rslt = {ngx.location.capture_multi(args)}
ngx.say("DONE")

';
}

}
}

The error I get, when I bump up the args count to 201 or greater is as follows in my error.log file:

2015/10/06 15:24:11 [notice] 3241#0: signal process started
2015/10/06 15:24:11 [alert] 3230#0: *10129 open socket #7 left in connection 2
2015/10/06 15:24:11 [alert] 3230#0: aborting
2015/10/06 15:24:15 [error] 3246#0: *10131 subrequests cycle while processing "/hello", client: 100.1.46.100, server: , request: "GET /grid HTTP/1.1", host: "54.175.150.16"
2015/10/06 15:24:15 [error] 3246#0: *10131 lua entry thread aborted: runtime error: content_by_lua(nginx.conf:27):6: failed to issue subrequest: -1
stack traceback:
coroutine 0:
[C]: in function 'capture_multi'
content_by_lua(nginx.conf:27):6: in function <content_by_lua(nginx.conf:27):1>, client: 100.1.46.100, server: , request: "GET /grid HTTP/1.1", host: "54.175.150.16"
2015/10/06 15:24:16 [error] 3246#0: *10132 open() "/home/ec2-user/investingengines/build_linux/resty/html/favicon.ico" failed (13: Permission denied), client: 100.1.46.100, server: , request: "GET /favicon.ico HTTP/1.1", host: "54.175.150.16", referrer: "http://54.175.150.16/grid"
[ec2-user@ip-172-31-55-176 resty]$

Thanks for any help
Dave Linenberg

lordnynex

Can you tell us a little more about your actual use case? Even if you
could get this many asynchronous requests, this would be extremely
inefficient. There is very likely better ways to accomplish what you want.

On 10/06/2015 08:28 AM, David Linenberg wrote:
> Hello,
> I am very new to openresty.  I am trying to work
> with ngx.location.capture_multi, and have noticed it is failing when I
> put more than 200 requests in.
> My desire is to be able to use ngx.location.capture_multi with thousands
> of simultaneous requests.
> 
> A conf file which illustrates the error is below.  I am running on a
> single core AWS microinstance with Amazon Linux.
> 
> http {
>     server {
>         listen 80;
>        
>         location /hello {
>         content_by_lua '
>             ngx.say("hello")
>         ';
>         }
>         
>         location /grid {
>         content_by_lua '
>             local args = {};
>             for i=1,201 do                                  -- works
> when last index is 200 or lower, fails at 201
>                 args[i] = {"/hello"};
>             end
>             local rslt = {ngx.location.capture_multi(args)}
>             ngx.say("DONE")
>             
>         ';    
>         }
>         
>     }
> }
> 
> The error I get, when I bump up the args count to 201 or greater is as
> follows in my error.log file:
> 
> 2015/10/06 15:24:11 [notice] 3241#0: signal process started
> 2015/10/06 15:24:11 [alert] 3230#0: *10129 open socket #7 left in
> connection 2
> 2015/10/06 15:24:11 [alert] 3230#0: aborting
> 2015/10/06 15:24:15 [error] 3246#0: *10131 subrequests cycle while
> processing "/hello", client: 100.1.46.100, server: , request: "GET /grid
> HTTP/1.1", host: "54.175.150.16"
> *2015/10/06 15:24:15 [error] 3246#0: *10131 lua entry thread aborted:
> runtime error: content_by_lua(nginx.conf:27):6: failed to issue
> subrequest: -1*
> *stack traceback:*
> coroutine 0:
>         [C]: in function 'capture_multi'
>         content_by_lua(nginx.conf:27):6: in function
> <content_by_lua(nginx.conf:27):1>, client: 100.1.46.100, server: ,
> request: "GET /grid HTTP/1.1", host: "54.175.150.16"
> 2015/10/06 15:24:16 [error] 3246#0: *10132 open()
> "/home/ec2-user/investingengines/build_linux/resty/html/favicon.ico"
> failed (13: Permission denied), client: 100.1.46.100, server: , request:
> "GET /favicon.ico HTTP/1.1", host: "54.175.150.16", referrer:
> "http://54.175.150.16/grid";
> [ec2-user@ip-172-31-55-176 resty]$
> 
> Thanks for any help
> Dave Linenbe.

dlinenbe

Background:

I work for a bank where our traders use proprietary interest rate derivative pricing models which employ monte carlo techniques (that are embarassingly parallel)...

I've done this successfully using MPI on several thousand AWS cores - typically 1000 to 3600 separate processors which are distributed across 30 to 100 machines. Typically we spawn up 30 to 100 x 36 node AWS machines, and our calculation times have gone down from a dozen hours for a single desktop 4 core pc risk report to into the minutes or seconds range using the grid. This is important for traders who want to reprice the risk on their trading books a number of times throughout the day - instead of once a night. We use AWS ec2, cloudformation, MPI, and a substantial (for me!) (1 million lines) of C/C++ code. The engine (worker) code is old-school single threaded C (some c++) w/ JSON , JNI & Lua interfaces. The front end is an embedded webserver fused to an MPI root process. The engine code as well as the scheduler code is adaptable to nginx + Lua. We've gotten the code to run in java based grids, on OpenVMS (its origins), on Linux, Windows, in Excel using C# interfaces etc -- so we have the expertise to expose it as we see fit. That's the nice thing about old school C. It really is write once, run everywhere..but I'm digressing.

Anway -

I am investigating replacing a bit of this infrastructure (paricularly the communications MPI layer part) and the C++ master -> slave scheduling code by spawning up a number of webservers on AWS nodes, and farming the calculations out to the servers. Some custom Lua code + load balancing on a "master" nginx server might sufficiently offload our calculations to perhaps 100 nginx servers which each have 36 cores. Or a single "master" nginx server might offload API calls (REST JSON) to 3600 independent simpler webservers (embedded - mongoose, libuv, etc) fused to our C back end.

The main point is that this all needs to be asynchronous - so - we can send a block of asynch requests out to the grid (which was why I was looking at the capture_multi call out) - or we can send non-blocking individual requests out - and poll to see when finished (this is how we do our work now). I'd actually prefer single non-blocking calls.

Any advice on how to send thousands of web requests out to a farm of web servers asynchronously, either all together- in a block, or independently - one by one - using NGINX - and controlling the scheduling of those requests - would be greatly appreciated. (Right now, we control every aspect with a libMongoose/MPI front end scheduler process & an MPI slave node back end processes). Perhaps I can replicate this in openResty/NGINX in the front end/ and/or back-end and eliminate a bit of proprietary code with something simpler.... If not NGINX, I've had luck on AWS using mongoose + libCurl front end, plus mongoose slaves - to replace MPI - but maybe NGINX /openResty has a role?! As far as efficient, our custom MPI solution has no issues dealing with 100s of thousands of distributed calls - but the point of the exercise is to see how much functionality we could possibly get out of a less proprietary, more transparent, web server based, solution...

Thanks,

Dave

On Tuesday, October 6, 2015 at 1:58:43 PM UTC-4, Lord Nynex wrote:

Can you tell us a little more about your actual use case? Even if you
could get this many asynchronous requests, this would be extremely
inefficient. There is very likely better ways to accomplish what you want.

dlinenbe

doesn't seem to work when doing individually with ngx.thread.spawn combined with ngx.location.capture either at values > 200

works:

for i = 1, 200 do

ths[i] = ngx.thread.spawn(function ()

rslt[i] = ngx.location.capture("/l")

end)

end

for i = 1, #ths do

ngx.say(i .. " " .. rslt[i].body)

end

doesn't work:

for i = 1, 201 do

ths[i] = ngx.thread.spawn(function ()

rslt[i] = ngx.location.capture("/l")

end)

end

for i = 1, #ths do

ngx.say(i .. " " .. rslt[i].body)

end

james_9172

Hi David,

You probably need to avoid subrequests for this. You can call out to your servers directly using cosockets instead: https://github.com/pintsized/lua-resty-http

You should be able to handle thousands of simultaneous outbound connections this way.

James.

On 7 October 2015 at 00:23, David Linenberg <dli...@gmail.com> wrote:

doesn't seem to work when doing individually with ngx.thread.spawn combined with ngx.location.capture either at values > 200

works:
for i = 1, 200 do
ths[i] = ngx.thread.spawn(function ()
rslt[i] = ngx.location.capture("/l")
end)
end
for i = 1, #ths do
ngx.say(i .. " " .. rslt[i].body)
end

doesn't work:
for i = 1, 201 do
ths[i] = ngx.thread.spawn(function ()
rslt[i] = ngx.location.capture("/l")
end)
end
for i = 1, #ths do
ngx.say(i .. " " .. rslt[i].body)
end

.

James Hurst

ja...@pintsized.co.uk

agentzh

Hello!

Yeah, with NGINX cores older than 1.9.5 (exclusive), there is a
hard-coded limit, 200, in the NGINX core, on the total number of
subrequests allowed under a single main request.

Starting from NGINX 1.9.5, however, there is no longer any limits on
the total number of subrequests per main request, but only a limit
(50) on the depth of the recursive subrequest chain (which is very
rare in practice).

Best regards,
-agentzh


On Wed, Oct 7, 2015 at 12:55 AM,  <i...@ecsystems.nl> wrote:
> I think your lucky getting 200 atm. see
> http://forum.nginx.org/read.php?29,261343
> which I think is the reason here.
>
>
> On Tuesday, October 6, 2015 at 5:28:45 PM UTC+2, David Linenberg wrote:
>>
>> Hello,
>> I am very new to openresty.  I am trying to work with
>> ngx.location.capture_multi, and have noticed it is failing when I put more
>> than 200 requests in.
>> My desire is to be able to use ngx.location.capture_multi with thousands
>> of simultaneous requests.
>>
>> A conf file which illustrates the error is below.  I am running on a
>> single core AWS microinstance with Amazon Linux.
>>
>> http {
>>     server {
>>         listen 80;
>>
>>         location /hello {
>>         content_by_lua '
>>             ngx.say("hello")
>>         ';
>>         }
>>
>>         location /grid {
>>         content_by_lua '
>>             local args = {};
>>             for i=1,201 do                                  -- works when
>> last index is 200 or lower, fails at 201
>>                 args[i] = {"/hello"};
>>             end
>>             local rslt = {ngx.location.capture_multi(args)}
>>             ngx.say("DONE")
>>
>>         ';
>>         }
>>
>>     }
>> }
>>
>> The error I get, when I bump up the args count to 201 or greater is as
>> follows in my error.log file:
>>
>> 2015/10/06 15:24:11 [notice] 3241#0: signal process started
>> 2015/10/06 15:24:11 [alert] 3230#0: *10129 open socket #7 left in
>> connection 2
>> 2015/10/06 15:24:11 [alert] 3230#0: aborting
>> 2015/10/06 15:24:15 [error] 3246#0: *10131 subrequests cycle while
>> processing "/hello", client: 100.1.46.100, server: , request: "GET /grid
>> HTTP/1.1", host: "54.175.150.16"
>> 2015/10/06 15:24:15 [error] 3246#0: *10131 lua entry thread aborted:
>> runtime error: content_by_lua(nginx.conf:27):6: failed to issue subrequest:
>> -1
>> stack traceback:
>> coroutine 0:
>>         [C]: in function 'capture_multi'
>>         content_by_lua(nginx.conf:27):6: in function
>> <content_by_lua(nginx.conf:27):1>, client: 100.1.46.100, server: , request:
>> "GET /grid HTTP/1.1", host: "54.175.150.16"
>> 2015/10/06 15:24:16 [error] 3246#0: *10132 open()
>> "/home/ec2-user/investingengines/build_linux/resty/html/favicon.ico" failed
>> (13: Permission denied), client: 100.1.46.100, server: , request: "GET
>> /favicon.ico HTTP/1.1", host: "54.175.150.16", referrer:
>> "http://54.175.150.16/grid";
>> [ec2-user@ip-172-31-55-176 resty]$
>>
>> Thanks for any help
>> Dave Linenbe.