Hello,
DNS lookup failures halt nginx at startup because it has an internal cache to avoid adding dns lookup latency to requests at run time. It is not possible, to explicitly ignore these failures.
On a more positive note, you have some options. DNS lookups use system level (glibc IIRC) functions (getaddrinfo etc). The bright side is these functions are aware of your system resolver and further, are aware of /etc/hosts. If the host you are putting into the upstream block is NOT a CNAME, you could create /etc/hosts entries for them and call it a day. This is the simplest approach, but is pretty inelegant IMHO.
Another possibility is to do what you've described, run a local caching recursive dns server (maybe dnsmasq or a full version of bind or powerdns). In such case, you can use nginx's 'resolver' directive to tell it to direct DNS queries to localhost (or a central DNS server within your cluster).
There are some more modern options as well, that are unrelated to nginx. One example, consul has the ability to generate templates based on configuration changes. In this case, DNS would represent a change indicating to consul it should render a new config file for the upstream block. This is not without it's issues in your situation though, if your setup is such that an A record or CNAME can disappear entirely, there will be a convergence time where nginx will not be able to resolve the missing hostname before consul can render and reload. In this case, nginx will simply produce an error and (depending on your logging) write an error to the error log.
One last option is using openresty's balancer_by_lua directive to implement your own load balancing block. One possibility is to use a cluster central service like consul to do service registration, and having your lua code select an upstream based on consul service names. This is the most dynamic and scalable solution, but requires code and a non-trivial amount of testing.
As an aside, I recommend you attempt to solve the problem externally as well. A server or cluster of servers that have dependencies on names, present a serious issue when those names suddenly disappear. Using DNS to indicate availability or register a service, is simply not a sensible or scalable design.
I hope this helps
-Brandon
