streaming performance optimization

mahmutemin · 2016-03-20T06:51:19+00:00

ah, i was using a non-coroutine one (i think)i'm using this one now, and it works! https://github.com/pintsized/lua-resty-httpi'm nearly done wi...

streaming performance optimization

mahmutemin

Hello, having performance problems while proxying without buffering or streaming large video objects using James Hurst's resty.http library, simply doing the following,

local chunksize = chunksize or 8192

local httpc = self.http.new()

httpc:set_timeout(500)

local scheme, host, port, path = unpack(httpc:parse_uri(base_url))

local ok, err = httpc:connect(host, port)

if not ok then

ngx.log(ngx.ERR, err)

return

end

httpc:set_timeout(2000)

local res, err = httpc:request{

method = "GET",

path = string.format("/%s/%s", bucket, key),

headers = {

["Date"] = date_str,

["Authorization"] = get_auth_header,

}

if not res then

ngx.say("failed to request: ", err)

return

end

ngx.status = res.status

local reader = res.body_reader

repeat

local chunk, err = reader(8192)

if err then

ngx_log(ngx_ERR, err)

break

end

if chunk then

local res, err = ngx.print(chunk)

if not res then

ngx_log(ngx_ERR, err)

break

end

ngx.flush(true)

end

until not chunk

httpc:set_keepalive()

Using this implementation I don't see large memory utilizations but not the same for the for cpu usage, done a simple two test by

1. proxying 1GB file by utilizing 1gbps network and observed 50% average cpu usage on nginx worker process,

2. serving the file from local file system by utilizing 1gbps network, and observed nearly zero memory and cpu usage,

I am expecting to see the same cpu usage with lua implementation, I could get the flame graphs with sample-bt systemtap script as you can find attached.

Could you suggest howto optimize such cpu cycles?

Regards,

Emin.

Attachment: a.svg
Description: image/svg

james_9172

Hi Emin,

Do you see similar results if you remove the `ngx.flush(true)` line?

James.

On 19 March 2016 at 22:51, Emin <mahm...@gmail.com> wrote:

Hello, having performance problems while proxying without buffering or streaming large video objects using James Hurst's resty.http library, simply doing the following,

local chunksize = chunksize or 8192

local httpc = self.http.new()
httpc:set_timeout(500)
local scheme, host, port, path = unpack(httpc:parse_uri(base_url))
local ok, err = httpc:connect(host, port)

if not ok then
ngx.log(ngx.ERR, err)
return
end

httpc:set_timeout(2000)
local res, err = httpc:request{
method = "GET",
path = string.format("/%s/%s", bucket, key),
headers = {
["Date"] = date_str,
["Authorization"] = get_auth_header,
},
}

if not res then
ngx.say("failed to request: ", err)
return
end

ngx.status = res.status

local reader = res.body_reader
repeat
local chunk, err = reader(8192)
if err then
ngx_log(ngx_ERR, err)
break
end

if chunk then
local res, err = ngx.print(chunk)
if not res then
ngx_log(ngx_ERR, err)
break
end
ngx.flush(true)
end
until not chunk

httpc:set_keepalive()

Using this implementation I don't see large memory utilizations but not the same for the for cpu usage, done a simple two test by

1. proxying 1GB file by utilizing 1gbps network and observed 50% average cpu usage on nginx worker process,
2. serving the file from local file system by utilizing 1gbps network, and observed nearly zero memory and cpu usage,

I am expecting to see the same cpu usage with lua implementation, I could get the flame graphs with sample-bt systemtap script as you can find attached.

Could you suggest howto optimize such cpu cycles?

Regards,
Emin.

.

James Hurst

ja...@pintsized.co.uk

guanglinlv

hi Emin, i had the same cpu issuein the same stream scene, see https://groups.google.com/forum/#!topic/openresty-en/n-8y0PJSaRY

unfortunately, i can not find the cause reason. but a workaround is using proxy_pass forward the response not the resty-http based on cosocket.

hope to help you.

thanks.

在 2016年3月20日星期日 UTC+8上午6:51:19，Emin写道：

Hello, having performance problems while proxying without buffering or streaming large video objects using James Hurst's resty.http library, simply doing the following,

local chunksize = chunksize or 8192

local httpc = self.http.new()
httpc:set_timeout(500)
local scheme, host, port, path = unpack(httpc:parse_uri(base_url))
local ok, err = httpc:connect(host, port)

if not ok then
ngx.log(ngx.ERR, err)
return
end

httpc:set_timeout(2000)
local res, err = httpc:request{
method = "GET",
path = string.format("/%s/%s", bucket, key),
headers = {
["Date"] = date_str,
["Authorization"] = get_auth_header,
},
}

if not res then
ngx.say("failed to request: ", err)
return
end

ngx.status = res.status

local reader = res.body_reader
repeat
local chunk, err = reader(8192)
if err then
ngx_log(ngx_ERR, err)
break
end

if chunk then
local res, err = ngx.print(chunk)
if not res then
ngx_log(ngx_ERR, err)
break
end
ngx.flush(true)
end
until not chunk

httpc:set_keepalive()

Using this implementation I don't see large memory utilizations but not the same for the for cpu usage, done a simple two test by

1. proxying 1GB file by utilizing 1gbps network and observed 50% average cpu usage on nginx worker process,
2. serving the file from local file system by utilizing 1gbps network, and observed nearly zero memory and cpu usage,

I am expecting to see the same cpu usage with lua implementation, I could get the flame graphs with sample-bt systemtap script as you can find attached.

Could you suggest howto optimize such cpu cycles?

Regards,
Emin.

james_9172

Hi,

I don't see any reason in principle as to why resty.http should perform measurably slower than the proxy module, if used in a comparable way. This being said, I'm obviously keen to be sure of this! I'm no expert with flame graphs, and I'm hoping Yichun might be able to add something here, but it looks to me like the majority of time is spent either in socket receive, or the output filter?

The code example shows 8k chunks being read and then synchronously flushed to the output filter - this just seems like a bad idea to me and I wouldn't be surprised if it performed poorly. I'd be interested to know if CPU utilisation was more normal when using, say, 64k chunks at least and removing any calls to ngx.flush()?

Cheers,

James.

On 21 March 2016 at 01:41, Guanglin Lv <guan...@gmail.com> wrote:

hi Emin, i had the same cpu issuein the same stream scene, see https://groups.google.com/forum/#!topic/openresty-en/n-8y0PJSaRY

unfortunately, i can not find the cause reason. but a workaround is using proxy_pass forward the response not the resty-http based on cosocket.

hope to help you.

thanks.

在 2016年3月20日星期日 UTC+8上午6:51:19，Emin写道：
Hello, having performance problems while proxying without buffering or streaming large video objects using James Hurst's resty.http library, simply doing the following,

local chunksize = chunksize or 8192

local httpc = self.http.new()
httpc:set_timeout(500)
local scheme, host, port, path = unpack(httpc:parse_uri(base_url))
local ok, err = httpc:connect(host, port)

if not ok then
ngx.log(ngx.ERR, err)
return
end

httpc:set_timeout(2000)
local res, err = httpc:request{
method = "GET",
path = string.format("/%s/%s", bucket, key),
headers = {
["Date"] = date_str,
["Authorization"] = get_auth_header,
},
}

if not res then
ngx.say("failed to request: ", err)
return
end

ngx.status = res.status

local reader = res.body_reader
repeat
local chunk, err = reader(8192)
if err then
ngx_log(ngx_ERR, err)
break
end

if chunk then
local res, err = ngx.print(chunk)
if not res then
ngx_log(ngx_ERR, err)
break
end
ngx.flush(true)
end
until not chunk

httpc:set_keepalive()

Using this implementation I don't see large memory utilizations but not the same for the for cpu usage, done a simple two test by

1. proxying 1GB file by utilizing 1gbps network and observed 50% average cpu usage on nginx worker process,
2. serving the file from local file system by utilizing 1gbps network, and observed nearly zero memory and cpu usage,

I am expecting to see the same cpu usage with lua implementation, I could get the flame graphs with sample-bt systemtap script as you can find attached.

Could you suggest howto optimize such cpu cycles?

Regards,
Emin.

.

James Hurst

ja...@pintsized.co.uk

guanglinlv

I think we need pay more attention on the stream scene, it has a very different feature with normal http reuqst.

long roll connection, uncertain size of chunk, small heartbeat chunk etc...

maybe some bad/unneeded resuming coroutine repeated again and again.

Thanks.

在 2016年3月21日星期一 UTC+8下午5:59:12，James Hurst写道：

Hi,

I don't see any reason in principle as to why resty.http should perform measurably slower than the proxy module, if used in a comparable way. This being said, I'm obviously keen to be sure of this! I'm no expert with flame graphs, and I'm hoping Yichun might be able to add something here, but it looks to me like the majority of time is spent either in socket receive, or the output filter?

The code example shows 8k chunks being read and then synchronously flushed to the output filter - this just seems like a bad idea to me and I wouldn't be surprised if it performed poorly. I'd be interested to know if CPU utilisation was more normal when using, say, 64k chunks at least and removing any calls to ngx.flush()?

Cheers,

James.

On 21 March 2016 at 01:41, Guanglin Lv <guang...@gmail.com> wrote:

hi Emin, i had the same cpu issuein the same stream scene, see https://groups.google.com/forum/#!topic/openresty-en/n-8y0PJSaRY

unfortunately, i can not find the cause reason. but a workaround is using proxy_pass forward the response not the resty-http based on cosocket.

hope to help you.

thanks.

在 2016年3月20日星期日 UTC+8上午6:51:19，Emin写道：
Hello, having performance problems while proxying without buffering or streaming large video objects using James Hurst's resty.http library, simply doing the following,

local chunksize = chunksize or 8192

local httpc = self.http.new()
httpc:set_timeout(500)
local scheme, host, port, path = unpack(httpc:parse_uri(base_url))
local ok, err = httpc:connect(host, port)

if not ok then
ngx.log(ngx.ERR, err)
return
end

httpc:set_timeout(2000)
local res, err = httpc:request{
method = "GET",
path = string.format("/%s/%s", bucket, key),
headers = {
["Date"] = date_str,
["Authorization"] = get_auth_header,
},
}

if not res then
ngx.say("failed to request: ", err)
return
end

ngx.status = res.status

local reader = res.body_reader
repeat
local chunk, err = reader(8192)
if err then
ngx_log(ngx_ERR, err)
break
end

if chunk then
local res, err = ngx.print(chunk)
if not res then
ngx_log(ngx_ERR, err)
break
end
ngx.flush(true)
end
until not chunk

httpc:set_keepalive()

Using this implementation I don't see large memory utilizations but not the same for the for cpu usage, done a simple two test by

1. proxying 1GB file by utilizing 1gbps network and observed 50% average cpu usage on nginx worker process,
2. serving the file from local file system by utilizing 1gbps network, and observed nearly zero memory and cpu usage,

I am expecting to see the same cpu usage with lua implementation, I could get the flame graphs with sample-bt systemtap script as you can find attached.

Could you suggest howto optimize such cpu cycles?

Regards,
Emin.

.

--
James Hurst
ja...@pintsized.co.uk

mahmutemin

Actually first tried with library's functions which is not having the ngx.flush() call with 64k chunksize and I had observed that there are plenty of memory usage which may cause problems on unpredicted traffic, and the cpu usage was much than expected, and hit a memory leak which I could say some but not sure the exact reason it is, in some situations upstream was dropping the connection which is started with a Content-Lenght header.

Thanks for the ngx.exec tip, seems it is going to save my day. I am testing proxy module with caching and cache-lock enabled which is good fit for my requirements.

Thanks for the resty.http library which is doing great in my project for the cases I need to read all response body.

Regards,

Emin.

On 21 March 2016 at 12:25, Guanglin Lv <guan...@gmail.com> wrote:

HI

I think we need pay more attention on the stream scene, it has a very different feature with normal http reuqst.

long roll connection, uncertain size of chunk, small heartbeat chunk etc...

maybe some bad/unneeded resuming coroutine repeated again and again.

Thanks.

在 2016年3月21日星期一 UTC+8下午5:59:12，James Hurst写道：
Hi,

I don't see any reason in principle as to why resty.http should perform measurably slower than the proxy module, if used in a comparable way. This being said, I'm obviously keen to be sure of this! I'm no expert with flame graphs, and I'm hoping Yichun might be able to add something here, but it looks to me like the majority of time is spent either in socket receive, or the output filter?

The code example shows 8k chunks being read and then synchronously flushed to the output filter - this just seems like a bad idea to me and I wouldn't be surprised if it performed poorly. I'd be interested to know if CPU utilisation was more normal when using, say, 64k chunks at least and removing any calls to ngx.flush()?

Cheers,

James.

On 21 March 2016 at 01:41, Guanglin Lv <guang...@gmail.com> wrote:

hi Emin, i had the same cpu issuein the same stream scene, see https://groups.google.com/forum/#!topic/openresty-en/n-8y0PJSaRY

unfortunately, i can not find the cause reason. but a workaround is using proxy_pass forward the response not the resty-http based on cosocket.

hope to help you.

thanks.

在 2016年3月20日星期日 UTC+8上午6:51:19，Emin写道：
Hello, having performance problems while proxying without buffering or streaming large video objects using James Hurst's resty.http library, simply doing the following,

local chunksize = chunksize or 8192

local httpc = self.http.new()
httpc:set_timeout(500)
local scheme, host, port, path = unpack(httpc:parse_uri(base_url))
local ok, err = httpc:connect(host, port)

if not ok then
ngx.log(ngx.ERR, err)
return
end

httpc:set_timeout(2000)
local res, err = httpc:request{
method = "GET",
path = string.format("/%s/%s", bucket, key),
headers = {
["Date"] = date_str,
["Authorization"] = get_auth_header,
},
}

if not res then
ngx.say("failed to request: ", err)
return
end

ngx.status = res.status

local reader = res.body_reader
repeat
local chunk, err = reader(8192)
if err then
ngx_log(ngx_ERR, err)
break
end

if chunk then
local res, err = ngx.print(chunk)
if not res then
ngx_log(ngx_ERR, err)
break
end
ngx.flush(true)
end
until not chunk

httpc:set_keepalive()

Using this implementation I don't see large memory utilizations but not the same for the for cpu usage, done a simple two test by

1. proxying 1GB file by utilizing 1gbps network and observed 50% average cpu usage on nginx worker process,
2. serving the file from local file system by utilizing 1gbps network, and observed nearly zero memory and cpu usage,

I am expecting to see the same cpu usage with lua implementation, I could get the flame graphs with sample-bt systemtap script as you can find attached.

Could you suggest howto optimize such cpu cycles?

Regards,
Emin.

.

james_9172

Hi,

Ok, well if the nginx proxy features fit your requirements, then that's all good, but I'd still really like to understand these problems better. We don't see any issues here in production with memory usage, leaks, or CPU usage, and it's been running a fair bit of traffic for the last year or so.

Are you able to post a gist with config, code and also a content example?

If you see higher memory usage, using a fixed chunk size but no manual flush, then I guess this is just Nginx doing whatever it thinks is best with the output buffering. I'd be really surprised to see unpredictable memory usage, or leaks. And if they exist, then obviously it needs to be fixed!

I'm not sure if your case is the same as the one posted by Guanglin, where there are small heartbeat chunks which must be flushed to the client? I presuming this is specific to the content / media itself? If either of you are able to post some examples I'm more than happy to investigate.

Cheers,

James.

On 21 March 2016 at 11:33, Mahmut Emin Taşgetiren <mahm...@gmail.com> wrote:
Actually first tried with library's functions which is not having the ngx.flush() call with 64k chunksize and I had observed that there are plenty of memory usage which may cause problems on unpredicted traffic, and the cpu usage was much than expected, and hit a memory leak which I could say some but not sure the exact reason it is, in some situations upstream was dropping the connection which is started with a Content-Lenght header.

Thanks for the ngx.exec tip, seems it is going to save my day. I am testing proxy module with caching and cache-lock enabled which is good fit for my requirements.

Thanks for the resty.http library which is doing great in my project for the cases I need to read all response body.

Regards,
Emin.

On 21 March 2016 at 12:25, Guanglin Lv <guan...@gmail.com> wrote:

HI

I think we need pay more attention on the stream scene, it has a very different feature with normal http reuqst.

long roll connection, uncertain size of chunk, small heartbeat chunk etc...

maybe some bad/unneeded resuming coroutine repeated again and again.

Thanks.

在 2016年3月21日星期一 UTC+8下午5:59:12，James Hurst写道：
Hi,

I don't see any reason in principle as to why resty.http should perform measurably slower than the proxy module, if used in a comparable way. This being said, I'm obviously keen to be sure of this! I'm no expert with flame graphs, and I'm hoping Yichun might be able to add something here, but it looks to me like the majority of time is spent either in socket receive, or the output filter?

The code example shows 8k chunks being read and then synchronously flushed to the output filter - this just seems like a bad idea to me and I wouldn't be surprised if it performed poorly. I'd be interested to know if CPU utilisation was more normal when using, say, 64k chunks at least and removing any calls to ngx.flush()?

Cheers,

James.

On 21 March 2016 at 01:41, Guanglin Lv <guang...@gmail.com> wrote:

hi Emin, i had the same cpu issuein the same stream scene, see https://groups.google.com/forum/#!topic/openresty-en/n-8y0PJSaRY

unfortunately, i can not find the cause reason. but a workaround is using proxy_pass forward the response not the resty-http based on cosocket.

hope to help you.

thanks.

在 2016年3月20日星期日 UTC+8上午6:51:19，Emin写道：
Hello, having performance problems while proxying without buffering or streaming large video objects using James Hurst's resty.http library, simply doing the following,

local chunksize = chunksize or 8192

local httpc = self.http.new()
httpc:set_timeout(500)
local scheme, host, port, path = unpack(httpc:parse_uri(base_url))
local ok, err = httpc:connect(host, port)

if not ok then
ngx.log(ngx.ERR, err)
return
end

httpc:set_timeout(2000)
local res, err = httpc:request{
method = "GET",
path = string.format("/%s/%s", bucket, key),
headers = {
["Date"] = date_str,
["Authorization"] = get_auth_header,
},
}

if not res then
ngx.say("failed to request: ", err)
return
end

ngx.status = res.status

local reader = res.body_reader
repeat
local chunk, err = reader(8192)
if err then
ngx_log(ngx_ERR, err)
break
end

if chunk then
local res, err = ngx.print(chunk)
if not res then
ngx_log(ngx_ERR, err)
break
end
ngx.flush(true)
end
until not chunk

httpc:set_keepalive()

Using this implementation I don't see large memory utilizations but not the same for the for cpu usage, done a simple two test by

1. proxying 1GB file by utilizing 1gbps network and observed 50% average cpu usage on nginx worker process,
2. serving the file from local file system by utilizing 1gbps network, and observed nearly zero memory and cpu usage,

I am expecting to see the same cpu usage with lua implementation, I could get the flame graphs with sample-bt systemtap script as you can find attached.

Could you suggest howto optimize such cpu cycles?

Regards,
Emin.

.

--
James Hurst
ja...@pintsized.co.uk