lua-resty-http POST request timeouts

davidmuscut · 2014-09-30T22:49:47+00:00

Hello! On Mon, Sep 29, 2014 at 1:10 AM, songj songj wrote: > 怎么知道lua_share_dict实际用了多&#x5...

lua-resty-http POST request timeouts

davidmuscut

When using lua-resty-http (https://github.com/pintsized/lua-resty-http) and sending POST requests I get a timeout no matter how long I set the timeout in the lua client (e.g httpc:set_timeout(10000)). The POST request is pretty small (Content-Length: 150), any idea what could be causing the timeout? When I send the same request directly to the backend server everything works normally. When I go through the reverse proxy I get the following in the logs:

[error] 27253#0: *139 lua tcp socket read timed out, client

thanks,
- D

james_9172

Hi,

On 30 September 2014 15:49, <david...@gmail.com> wrote:

When using lua-resty-http (https://github.com/pintsized/lua-resty-http) and sending POST requests I get a timeout no matter how long I set the timeout in the lua client (e.g httpc:set_timeout(10000)). The POST request is pretty small (Content-Length: 150), any idea what could be causing the timeout? When I send the same request directly to the backend server everything works normally. When I go through the reverse proxy I get the following in the logs:

[error] 27253#0: *139 lua tcp socket read timed out, client

How are you sending the request body, as a string or using the client body reader? Can you post a gist of your code? Do you have any error on the backend server?

--
James Hurst

ja...@pintsized.co.uk

davidmuscut

On Tuesday, September 30, 2014 5:56:35 PM UTC+3, James Hurst wrote:

Hi,

On 30 September 2014 15:49, <david...@gmail.com> wrote:
When using lua-resty-http (https://github.com/pintsized/lua-resty-http) and sending POST requests I get a timeout no matter how long I set the timeout in the lua client (e.g httpc:set_timeout(10000)). The POST request is pretty small (Content-Length: 150), any idea what could be causing the timeout? When I send the same request directly to the backend server everything works normally. When I go through the reverse proxy I get the following in the logs:

[error] 27253#0: *139 lua tcp socket read timed out, client

How are you sending the request body, as a string or using the client body reader? Can you post a gist of your code? Do you have any error on the backend server?

Hi James,

The code is below. I don't have access to the backend server logs, but the same request works when I send it directly to the backend server (and update the target IP and Host header). Also note that some POST requests work while others do not and result in the timeout error despite extremely large values set in httpc:set_timeout().

-- This is a generic function that grabs the request args (for both GET and POST requests) and returns them in a string
function GetParams()
    local args, err = ngx.req.get_uri_args()
    local pargs, err = ngx.req.get_post_args()
    for k,v in pairs(pargs) do args[k] = v end
    local parms
    if not args then
        ngx.log(ngx.ERR, "failed to get post args: ", err)
        return
    end
    for key, val in pairs(args) do
        if type(val) == "table" then
            ngx.log(ngx.ERR, key, ": ", table.concat(val, ", "))
            if parms == nil then
                parms = key .. "=" .. val
            else
                parms = parms .. "&" .. key .. "=" .. val
            end
        else
            ngx.log(ngx.ERR, key, ": ", val)
            if parms == nil then
                parms = key .. "=" .. val
            else
                parms = parms .. "&" .. key .. "=" .. val
            end
        end
    end
    return parms
end

-- This function is called when proxying POST requests
function ProxyPOSTRequest(Host, URI, Params, Headers)
    local http = require "resty.http"
    local httpc = http.new()
    httpc:set_timeout(5000)
    local ok, err = httpc:connect(Host, 80)

    local res, err = httpc:request{
        method = "POST",
        headers = Headers,
        path = URI,
        body = Params,
    }

    for k,v in pairs(res.headers) do
        ngx.header[k] = v
        --ngx.say(k, ": ", v)
        ngx.log(ngx.ERR, "Headers: ", k, " ", v)
    end

    local body = res:read_body()
    ngx.say(body)
    local ok, err = httpc:set_keepalive()
    if not ok then
        ngx.say("failed to set keepalive: ", err)
        return
    end
end

thanks!
- D

james_9172

Hi,

The code is below. I don't have access to the backend server logs, but the same request works when I send it directly to the backend server (and update the target IP and Host header). Also note that some POST requests work while others do not and result in the timeout error despite extremely large values set in httpc:set_timeout().

It's hard to tell without seeing the request parameters that you're sending too, but my guess would be it's something to do with your reformatting of the request body (is merging GET and POST something you need to do explicitly, or are you just doing that as part of proxying?). My guess is you're also sending the original request headers unchecked, which include a Content-Length?

So the backend server is perhaps expecting a different length to what you're sending in your new request body. A hunch, but I think it's going to be something like that. Could be something similar with the Transfer-Encoding request header.

I've added some proxy functions to the master branch, with tests and updated the documentation. Can you try them out? And does that cover your case, or do you actually need to modify the request body too?

https://github.com/pintsized/lua-resty-http#proxy

Regards,

--
James Hurst

ja...@pintsized.co.uk

davidmuscut

On Wednesday, October 1, 2014 5:54:23 PM UTC+3, James Hurst wrote:

I've added some proxy functions to the master branch, with tests and updated the documentation. Can you try them out? And does that cover your case, or do you actually need to modify the request body too?

https://github.com/pintsized/lua-resty-http#proxy

Hi James,

Thanks for that. My test case involves rewriting the URI, will that work with your new proxy code? For example, the client requests /foo, my code looks up /foo in a shared dictionary and gets back /bar. I then proxy their request to the backend server using /bar as the URI. I do not modify the request body, only the URI. How can I do that with your proxy code?

thanks!
- D

Regards,

--
James Hurst
ja...@pintsized.co.uk

hamish.forbes

If you're only modifying the URI and not the body then you could either use ngx.req.set_uri before calling the proxy_request() function

or call request with your new uri, e.g. something like this:

local new_uri = whatever

local res, err = httpc:request{
        method = ngx.req.get_method(),
        path =  new_uri .. ngx.var.is_args .. (ngx.var.query_string or ""),
        body = httpc:get_client_body_reader(),
        headers = ngx.req.get_headers(),
    }

if not res then
    -- handle error
else
    httpc:proxy_response(res)
end

The key is to use get_client_body_reader() so that your request body and the content-length header match up :)

On Thursday, 2 October 2014 07:35:16 UTC+1, david...@gmail.com wrote:

On Wednesday, October 1, 2014 5:54:23 PM UTC+3, James Hurst wrote:

I've added some proxy functions to the master branch, with tests and updated the documentation. Can you try them out? And does that cover your case, or do you actually need to modify the request body too?

https://github.com/pintsized/lua-resty-http#proxy

Hi James,

Thanks for that. My test case involves rewriting the URI, will that work with your new proxy code? For example, the client requests /foo, my code looks up /foo in a shared dictionary and gets back /bar. I then proxy their request to the backend server using /bar as the URI. I do not modify the request body, only the URI. How can I do that with your proxy code?

thanks!
- D

Regards,

--
James Hurst
ja...@pintsized.co.uk

james_9172

Hi,

Yes, as Hamish says you either have to rewrite the URI, or make your own version of proxy_request. Personally I'd go for rewriting the URI during the rewrite_by_lua phase, so:

rewrite_by_lua '

-- look up new value for ngx.var.uri

ngx.req.set_uri(new_uri)

...and then content_by_lua simply proxies as if that was the URI requested.

I've just fixed a bug I introduced in the error handling around the request body reader, so you might want pull from master again.

James.

On 2 October 2014 10:08, Hamish Forbes <hamish...@gmail.com> wrote:

If you're only modifying the URI and not the body then you could either use ngx.req.set_uri before calling the proxy_request() function
or call request with your new uri, e.g. something like this:

local new_uri = whatever local res, err = httpc:request{ method = ngx.req.get_method(), path = new_uri .. ngx.var.is_args .. (ngx.var.query_string or ""), body = httpc:get_client_body_reader(), headers = ngx.req.get_headers(), } if not res then -- handle error else httpc:proxy_response(res) end

The key is to use get_client_body_reader() so that your request body and the content-length header match up :)

On Thursday, 2 October 2014 07:35:16 UTC+1, david...@gmail.com wrote:
On Wednesday, October 1, 2014 5:54:23 PM UTC+3, James Hurst wrote:

I've added some proxy functions to the master branch, with tests and updated the documentation. Can you try them out? And does that cover your case, or do you actually need to modify the request body too?

https://github.com/pintsized/lua-resty-http#proxy

Hi James,

Thanks for that. My test case involves rewriting the URI, will that work with your new proxy code? For example, the client requests /foo, my code looks up /foo in a shared dictionary and gets back /bar. I then proxy their request to the backend server using /bar as the URI. I do not modify the request body, only the URI. How can I do that with your proxy code?

thanks!
- D

Regards,

--
James Hurst
ja...@pintsized.co.uk

.

--
James Hurst

ja...@pintsized.co.uk

davidmuscut

On Thursday, October 2, 2014 12:24:32 PM UTC+3, James Hurst wrote:

Hi,

Yes, as Hamish says you either have to rewrite the URI, or make your own version of proxy_request. Personally I'd go for rewriting the URI during the rewrite_by_lua phase, so:

Thanks Hamish and James. During this discussion I realized that there will be at least two cases where I'll need to rewrite parameter names in both GET and POST requests. Assuming I go with your rewriting the URI during the rewrite_by_lua phase suggestion, is that the correct place to rewrite the parameter names as well? How do you recommend I recalculate/set the Content-Length header or will get_client_body_reader() take care of that for me?

- D

hamish.forbes

In that case you'll need to set the Content-Length header to be the same as your modified body length.

get_client_body_reader() returns an iterator function that uses ngx.req.socket(), so it will always return the original request body.

You'll have to get the request body and modify it similar to how you were before, but once you've got the modified body as a string you just set Content-Length to be the length of the string:

local new_uri = whatever
local body = "blah blah, some test body data"

local headers = ngx.req.get_headers()
headers["Content-Length"] = #body

local res, err = httpc:request{
        method = ngx.req.get_method(),
        path =  new_uri .. ngx.var.is_args .. (ngx.var.query_string or ""),
        body = body,
        headers = headers,
    }

On Thursday, 2 October 2014 12:40:40 UTC+1, david...@gmail.com wrote:

On Thursday, October 2, 2014 12:24:32 PM UTC+3, James Hurst wrote:
Hi,

Yes, as Hamish says you either have to rewrite the URI, or make your own version of proxy_request. Personally I'd go for rewriting the URI during the rewrite_by_lua phase, so:

Thanks Hamish and James. During this discussion I realized that there will be at least two cases where I'll need to rewrite parameter names in both GET and POST requests. Assuming I go with your rewriting the URI during the rewrite_by_lua phase suggestion, is that the correct place to rewrite the parameter names as well? How do you recommend I recalculate/set the Content-Length header or will get_client_body_reader() take care of that for me?

- D