I thought about this one but I don't think it can help.
Almost all of the request will go through proxy (which backend to use is defined in redis database). But some clients can have more than one backend defined and if for some backend isn't avaliable at this very moment (timeout maybe) would be great to try next one. Currently we just return 504 and when web user refresh the page it will go to the next avaliable backend because the previous was marked dead because of response code. We also scan for possible dead backends but this is done only every few minutes.
We already have some inhouse implementation for proxying with this retry functionality but It's to slow and nginx with lua is just great with this and stable.
I don't know how lua is integrated and what nginx actually allow us to do but the best option would be to somehow repeat the process from header_filter_by_lua back to access_by_lua where we have this redis logic implemented.
Here is just a basic idea what is going inside access_by_lua
1. connect to redis
3. we receive one or more http://172.16.10.x:IP
4. pick random from this list (check also with deadlist shared table)
5. set variable with this backend
6. set some headers
7. use proxy_pass $backend;
in header_filter_by_lua
1. check for ngx.status
2. if there is timeout status add current backend to deadlist
I hope this clears a bit what we would like to do.
Regards,
Uros
