Hi,
I want my users to upload files which are getting stored on several backend servers. Files are getting POST'ed.
To make the backend servers dynamic, I first tried to set a variable and use proxy_pass $upload_backend_server.
The variable $upload_backend_server got set with lua (rewrite_by_lua_file).
This worked, but there is one big downside: All user body requests are getting buffered first, before uploaded to the backend servers. This is very bad for files of hundreds of MB or even GBytes.
Therefore I tried to write an own lua function which basically works like the proxy_pass function, meaning it reads the request of the client, tunnels it to the backend server, reads the backend response back and sends it back to the client.
The response of the backend server then informs the client if the upload succeeded, or if there was an error.
Here's part of the code:
- displayError is an own defined function, basically it outputs the error message, sets response code and logs the request.
- internal_ip is an iP of the backend server
if ngx.req.get_method() ~= "POST" then
displayError("no POST request: " .. ngx.req.get_method(), 400)
end
ngx.flush(true)
local sock, err = ngx.req.socket(true) -- get raw request socket
if not sock then
ngx.log(ngx.ERR,"could not get request socket: ", err)
end
local req_headers = ngx.req.get_headers()
if req_headers['expect'] then
sock:send("HTTP/1.1 100 Continue\r\n\r\n") -- I do not know if this is even needed, but it works with curl
ngx.flush(true)
end
-- require content-length client header to see when the request is over
if not req_headers['content-length'] then
displayError("No Content-Length set",400)
end
sock:settimeout(60000)
local tcpsock = ngx.socket.tcp()
local ok, err = tcpsock:connect(internal_ip, 80)
if not ok then
displayError("could not connect to upstream " .. internal_ip .. " : " .. err,500)
end
ngx.log(ngx.NOTICE,"connected to upstream " .. internal_ip)
tcpsock:settimeout(60000) -- 60 seconds timeout for upstream
local bytes, err = tcpsock:send("POST /ul/" .. token .. " HTTP/1.1\r\n")
if not bytes then
displayError("could not send headers to upstream: " .. err)
end
for k, v in pairs(req_headers) do
if k ~= "expect" and k ~= "connection" then
ngx.log(ngx.NOTICE, k .. ": " .. v)
local bytes, err = tcpsock:send(k .. ": " .. v .. "\r\n")
if not bytes then
displayError("could not send headers to upstream: " .. err)
end
end
end
tcpsock:send("Connection: close\r\n\r\n")
local buffer_length = 262144 -- 256kB buffer
local read_bytes = buffer_length
local content_length = tonumber(req_headers['content-length'])
ngx.log(ngx.NOTICE, "content-length: " .. content_length)
while true do
if content_length > buffer_length then
read_bytes = buffer_length
else
read_bytes = content_length
end
content_length = content_length - read_bytes
local data, err, partial = sock:receive(read_bytes)
if not data then
displayError("failed to read the data: " .. err, 500)
end
-- do something with the data here
local bytes, err = tcpsock:send(data)
if not bytes or bytes ~= read_bytes then
displayError("could not send all data to upstream: " .. err,500)
end
-- end do something
if content_length <= 0 then
ngx.log(ngx.NOTICE,"finished")
break
end
end
local data, err, partial = tcpsock:receive('*a')
if not data then
displayError("could not receive response from upstream" .. err, 500)
end
sock:send(data)
ngx.log(ngx.NOTICE,data)
ngx.flush(true)
Please note that this is only a proof of concept and many things need to get improved. I don't even know if this is the way to go or if there's a better way, but we will see in time.
relevant nginx config:
location ~ /upload/(?<uploadtoken>[^/]+) {
if ($request_method = OPTIONS ) {
add_header Access-Control-Allow-Origin "*";
add_header Access-Control-Allow-Methods "GET, POST, OPTIONS";
add_header Access-Control-Allow-Headers "Content-Type, Content-Range, Content-Disposition";
add_header Content-Length 0;
add_header Content-Type text/plain;
return 200;
}
add_header Access-Control-Allow-Origin "*";
add_header Access-Control-Allow-Methods "GET, POST, OPTIONS";
add_header Access-Control-Allow-Headers "Content-Type, Content-Range, Content-Disposition";
add_header Content-Type "text/plain";
#max filesize
client_max_body_size 95500m;
client_body_timeout 12h;
lua_code_cache off;
content_by_lua_file /home/upload/lua/upload.lua;
}
The problem now is, that this is working so far, but there's one strange issue:
Sometimes the transfer speed drops down from a full GBit to only a few kBit/sec or stucks completely for a few seconds. After 5 or 6 seconds, speed goes up to a full GBit again.
I checked everything so far, HDD load on the source, HDD load on the target, used full SSD servers, uploaded directly to RAM, modified the buffer size, but the behaviour didn't change.
So my assumption is that it's something with nginx/lua. Maybe nginx isn't designed for this type of usage, but this is still a strange issue.
Would be great if someone could tell me what to do to get a stable speed, or at least assume why there are these speed drops.
Also, if there is anything which can get improved, please let me know :)
