Subrequests vs initiating requests with socket.http

gmadarm · 2014-05-15T04:23:45+00:00

Thank you!JeromeLe mercredi 14 mai 2014 18:06:13 UTC+2, Brian Akins a écrit :Use the second one with the explicit require, IMO.

Subrequests vs initiating requests with socket.http

gmadarm

I'm creating an API that should take a parameter and query several other external URIs using the submitted parameter value. For example, the api receives a request for /api?mode=123 then nginx should submit a request to http://ExternalServer.com/?status=123 and parse the response body (e.g.: <status>On</status>). Can this be done using subrequests or is it more efficient to use a lua script with socket.http to initiate the request (or is there a better way)?

thanks,

- G

agentzh

Hello!

On Wed, May 14, 2014 at 1:23 PM, Gordon Madarm wrote:
> I'm creating an API that should take a parameter and query several other
> external URIs using the submitted parameter value. For example, the api
> receives a request for /api?mode=123 then nginx should submit a request to
> http://ExternalServer.com/?status=123 and parse the response body (e.g.:
> <status>On</status>). Can this be done using subrequests or is it more
> efficient to use a lua script with socket.http to initiate the request (or
> is there a better way)?
>

Are you using the socket.http module from the LuaSocket library? If
yes, then it will horribly block your nginx event loop. And you should
avoid that.

Better use those 3rd-party lua-resty-http libraries based on ngx_lua's
nonblocking cosocket API out there, for instance,

    https://github.com/pintsized/lua-resty-http

or

    https://github.com/bakins/lua-resty-http-simple

These libraries are usually more efficient than subrequests, both
memory-wise and CPU-wise. But you should always do your own benchmark
for your own use case anyway :)

Regards,
-agentzh

gmadarm

On Wednesday, May 14, 2014 11:36:37 PM UTC+3, agentzh wrote:

Are you using the socket.http module from the LuaSocket library? If
yes, then it will horribly block your nginx event loop. And you should
avoid that.

Better use those 3rd-party lua-resty-http libraries based on ngx_lua's
nonblocking cosocket API out there, for instance,

https://github.com/pintsized/lua-resty-http

or

https://github.com/bakins/lua-resty-http-simple

Thanks mate! I need to configure nginx/lua to use a HTTP proxy when sending the requests. The LuaSocket library allowed me to do this. I was a bit unclear if this is possible using lua-resty-http or lua-resty-http-simple based on the docs. Can this be done? The lua-resty-upstream module looks like it can be used to proxy requests, but its marked as experimental. Any thoughts on how to implement this?

- G

james_9172

Hi,

Thanks mate! I need to configure nginx/lua to use a HTTP proxy when sending the requests. The LuaSocket library allowed me to do this. I was a bit unclear if this is possible using lua-resty-http or lua-resty-http-simple based on the docs. Can this be done? The lua-resty-upstream module looks like it can be used to proxy requests, but its marked as experimental. Any thoughts on how to implement this?

You can use the "generic" interface in https://github.com/pintsized/lua-resty-http to connect to a proxy rather than the actual origin host, and then send a request with the appropriate Host header. The "proxy" field in the LuaSocket implementation is just a convenience allowing you to override the connection parameters with a proxy instead.

Regards,

--
James Hurst

ja...@pintsized.co.uk

gmadarm

On Thursday, May 15, 2014 11:45:44 AM UTC+3, James Hurst wrote:

Hi,

Thanks mate! I need to configure nginx/lua to use a HTTP proxy when sending the requests. The LuaSocket library allowed me to do this. I was a bit unclear if this is possible using lua-resty-http or lua-resty-http-simple based on the docs. Can this be done? The lua-resty-upstream module looks like it can be used to proxy requests, but its marked as experimental. Any thoughts on how to implement this?

You can use the "generic" interface in https://github.com/pintsized/lua-resty-http to connect to a proxy rather than the actual origin host, and then send a request with the appropriate Host header. The "proxy" field in the LuaSocket implementation is just a convenience allowing you to override the connection parameters with a proxy instead.

Hi James,

Is there a code example you could point me to? Another (newbie) question, I downloaded the lua-resty-http source from github and tried recompiling openresty using:
./configure --add-module=/usr/local/src/lua-resty-http

But that results in the following error:

./configure: error: no /usr/local/src/lua-resty-http/config was found

and indeed there is no config file:
# ls /usr/local/src/lua-resty-http/
lib LICENSE lua-resty-http-0.1-0.rockspec Makefile README.md t util

I tried running make all, but that resulted in:
# make all
make: Nothing to be done for `all'.

I'm sure I'm missing a simple step, any hints for compiling lua-resty-http on ubuntu 14.01?

thank you!

- G

james_9172

On 15 May 2014 09:54, Gordon Madarm <gma...@gmail.com> wrote:

On Thursday, May 15, 2014 11:45:44 AM UTC+3, James Hurst wrote:

Hi,

Thanks mate! I need to configure nginx/lua to use a HTTP proxy when sending the requests. The LuaSocket library allowed me to do this. I was a bit unclear if this is possible using lua-resty-http or lua-resty-http-simple based on the docs. Can this be done? The lua-resty-upstream module looks like it can be used to proxy requests, but its marked as experimental. Any thoughts on how to implement this?

You can use the "generic" interface in https://github.com/pintsized/lua-resty-http to connect to a proxy rather than the actual origin host, and then send a request with the appropriate Host header. The "proxy" field in the LuaSocket implementation is just a convenience allowing you to override the connection parameters with a proxy instead.

Hi James,

Is there a code example you could point me to?

If you scroll down the example code here: https://github.com/pintsized/lua-resty-http#synopsis to the location block marked "genericinterface", you'll see how the connection and request steps are separate, allowing you to connect to your proxy server, and issue a request to it for the host and path that you want.

Another (newbie) question, I downloaded the lua-resty-http source from github and tried recompiling openresty using:
./configure --add-module=/usr/local/src/lua-resty-http

But that results in the following error:

./configure: error: no /usr/local/src/lua-resty-http/config was found

I'm sure I'm missing a simple step, any hints for compiling lua-resty-http on ubuntu 14.01?

No need to compile anything! The Makefile is a bit of a red herring (I only use it for running tests), but Yichun's build system uses these to move modules into the correct location as part of the official releases. Since lua-resty-http is not part of the official release, you need to either:

1) Install it alongside the other Lua modules, e.g. in /usr/local/openresty/lualib

or probably more sensibly...

2) Keep third party Lua modules wherever you want, and include those paths in your lua_package_path directive in your Nginx config file (https://github.com/openresty/lua-nginx-module#lua_package_path), such as:

lua_package_path '/usr/local/src/openresty/lua-resty-http/?.lua;;';

Regards,

--
James Hurst

ja...@pintsized.co.uk

gmadarm

On Thursday, May 15, 2014 12:15:32 PM UTC+3, James Hurst wrote:

No need to compile anything! The Makefile is a bit of a red herring (I only use it for running tests), but Yichun's build system uses these to move modules into the correct location as part of the official releases. Since lua-resty-http is not part of the official release, you need to either:

1) Install it alongside the other Lua modules, e.g. in /usr/local/openresty/lualib

or probably more sensibly...

2) Keep third party Lua modules wherever you want, and include those paths in your lua_package_path directive in your Nginx config file (https://github.com/openresty/lua-nginx-module#lua_package_path), such as:

Hi James,

I don't follow. Assuming I use the more sensible option 2, how do I compile nginx with lua-resty-http support? If I understood you correctly, once the module is compiled I could use it from the /usr/local/src/lua-resty-http directory via the lua_package_path directive. But how do I get past the initial compilation error about the missing config file?

thanks,
- G

james_9172

Hi James,

I don't follow. Assuming I use the more sensible option 2, how do I compile nginx with lua-resty-http support? If I understood you correctly, once the module is compiled I could use it from the /usr/local/src/lua-resty-http directory via the lua_package_path directive. But how do I get past the initial compilation error about the missing config file?

There is no compilation required - Lua modules are interpreted (or dynamically compiled if you're using LuaJIT). Basically ignore the Makefile, it's there to bootstrap the test suite only. Just think of the Lua modules as scripts which are interpreted, and use the lua_package_path directive to tell Nginx where to find modules that you want to use in your code.

So you can freely add / remove Lua modules to a compiled and installed OpenResty system without recompilation (Lua modules are not the same as Nginx modules written in C). You only need to ensure that they can be found in the lua_package_path directive (this can be any paths you like, just manually copy the module somewhere sensible and update that directive to match), and reload Nginx.

Hope that helps,

--
James Hurst

ja...@pintsized.co.uk

gmadarm

On Thursday, May 15, 2014 12:45:52 PM UTC+3, James Hurst wrote:

So you can freely add / remove Lua modules to a compiled and installed OpenResty system without recompilation (Lua modules are not the same as Nginx modules written in C). You only need to ensure that they can be found in the lua_package_path directive (this can be any paths you like, just manually copy the module somewhere sensible and update that directive to match), and reload Nginx.

Hope that helps,

Hi James,

I added the following right outside my server scope in nginx.conf

lua_package_path '/usr/local/src/lua-resty-http/lib/?.lua;;';

FWIW, I also tried without /lib, yet still get this error:
2014/05/15 17:04:20 [error] 20110#0: *429 lua entry thread aborted: runtime error: /usr/local/openresty/nginx/conf/test.lua:69: attempt to index global 'http' (a nil value)
stack traceback:
coroutine 0:

Line 69 of test.lua is part of the HTTP function as shown below:

function HTTP()
      local httpc = http.new()
      local res, err = httpc:request_uri("http://checkip.amazonaws.com", {
        method = "GET",
      })
ngx.status = res.status

      for k,v in pairs(res.headers) do
          --
      end

      ngx.say(res.body)

end

Any hints as to what I'm doing wrong?

thanks,
-G

--
James Hurst
ja...@pintsized.co.uk

james_9172

FWIW, I also tried without /lib, yet still get this error:
2014/05/15 17:04:20 [error] 20110#0: *429 lua entry thread aborted: runtime error: /usr/local/openresty/nginx/conf/test.lua:69: attempt to index global 'http' (a nil value)
stack traceback:
coroutine 0:

Line 69 of test.lua is part of the HTTP function as shown below:

function HTTP()
      local httpc = http.new()
      local res, err = httpc:request_uri("http://checkip.amazonaws.com", {
        method = "GET",
      })
ngx.status = res.status

      for k,v in pairs(res.headers) do
          --
      end

      ngx.say(res.body)

end

Any hints as to what I'm doing wrong?

The clue is in the error message! If Lua says it failed to index a global called "http", then you have to ask yourself what "http" is? You are attempting to access a global variable which has not been defined (is nil).

If you review the examples in my documentation, you'll see that "http" is defined by requiring the "resty.http" module:

local http = require "resty.http"

If you're new to Lua, I can recommend Roberto's "Programming in Lua" book. The first edition, which is largely still relevant, is available for free online: http://www.lua.org/pil/contents.html

--
James Hurst

ja...@pintsized.co.uk

liupengwyy

Hi, agentzh,

Is it possible to use this http module in your later balancer_by_lua block?

Thanks
Liu Peng

On Thursday, May 15, 2014 at 4:36:37 AM UTC+8, agentzh wrote:

Hello!

On Wed, May 14, 2014 at 1:23 PM, Gordon Madarm wrote:
> I'm creating an API that should take a parameter and query several other
> external URIs using the submitted parameter value. For example, the api
> receives a request for /api?mode=123 then nginx should submit a request to
> http://ExternalServer.com/?status=123 and parse the response body (e.g.:
> <status>On</status>). Can this be done using subrequests or is it more
> efficient to use a lua script with socket.http to initiate the request (or
> is there a better way)?
>

Are you using the socket.http module from the LuaSocket library? If
yes, then it will horribly block your nginx event loop. And you should
avoid that.

Better use those 3rd-party lua-resty-http libraries based on ngx_lua's
nonblocking cosocket API out there, for instance,

https://github.com/pintsized/lua-resty-http

or

https://github.com/bakins/lua-resty-http-simple

These libraries are usually more efficient than subrequests, both
memory-wise and CPU-wise. But you should always do your own benchmark
for your own use case anyway :)

Regards,
-agentzh