Can't get reverse address lookup working

ssimpson · 2014-03-11T11:08:05+00:00

On Wed, Mar 12, 2014 at 10:40 AM, Yichun Zhang (agentzh) <age...@gmail.com> wrote: Hello! On Wed, Mar 12, 2014 at 12:16 AM, David Birdsong wrote:...

Can't get reverse address lookup working

ssimpson

I have some code in nginx Lua resty that looks like this:

ngx.ctx.resolver = require 'resty.dns.resolver'

-- ips are the local IPs for my DNS servers
local resolver_obj, err = ngx.ctx.resolver:new{ nameservers = ips }
if not resolver_obj then ngx.say("Could not create resolver object: ", err) end

local result, err = resolver_obj:query("127.0.0.1", {qtype = ngx.ctx.resolver.TYPE_PTR})
if not result then ngx.say("Can't lookup reverse address record for ", addr, ": ")

This doesn't seem to work. I keep getting {errcode=3,errstr="name error"} as the value of result. If I change the line to

local result, err = resolver_obj:query("www.uci.edu")

then I get {{address="128.195.188.231",type=1,class=1,name="www.uci.edu",ttl=25},{address="128.195.188.233",type=1,class=1,name="www.uci.edu",ttl=25}}

as the result. Do reverse records work? Thanks.
Scott

agentzh

Hello!

On Mon, Mar 10, 2014 at 8:08 PM,  ssimpson wrote:
> Do reverse records work?

A dedicated API for reverse lookup is still on the TODO list of the
lua-resty-dns library. Patches welcome :)

Thanks!
-agentzh

ssimpson

Bummer. I think for the sake of expediency I'll just make a C call for now. I looked at your Lua code and it looks like you are shifting bits around and generating a DNS header and since my Lua isn't as good as yours I think I'll go the C route. Thanks for your help.
Scott

agentzh

Hello!

On Tue, Mar 11, 2014 at 10:29 AM,  ssimpson wrote:
> Bummer. I think for the sake of expediency I'll just make a C call for now.

No, don't do that. Such C calls will almost always block the NGINX
worker processes (or some OS threads) and ruin the performance.

Regards,
-agentzh

ssimpson

No, don't do that. Such C calls will almost always block the NGINX
worker processes (or some OS threads) and ruin the performance.

Ugh. Well, it is rare that I need to do this. I only need to do this for Googlebot requests (to verify Googlebots, you need to do reverse lookup and then forward lookup) so it shouldn't occur very often. And I only need to block on the reverse lookup. On the forward lookup, your routines work. I may have to do this now and address the Lua code later. For the Lua code, I need to know the octets to pass to DNS and that will be a tedious undertaking. If I get around to it, I'll forward the changes.

ssimpson

Also, I whitelist/blacklist these Googlebot requests. So I ONLY run this reverse code when I first see a new Googlebot request. Therefore, I think I'll be good.

peter_booth

I had been curious about why you did this, thanks for sharing. If it were me, I wouldn't want to be doing any DNS lookups from within Nginx. Instead, I'd prefer to use a batch script to build my whitelist separate from my Nginx process and simply include the text file with list of IPs.

I can certainly relate to the pain that comes with this. The last retail site I supported was being scraped by over 300 distinct robots and spiders from tens of thousands of IPs.

Sent from my iPhone

On Mar 11, 2014, at 6:28 PM, ssi...@elance.com wrote:

Also, I whitelist/blacklist these Googlebot requests. So I ONLY run this reverse code when I first see a new Googlebot request. Therefore, I think I'll be good.

.

ssimpson

If we slow down a Googlebot for a short period that isn't a problem. They don't hit very often. Also, after the first hit they are white or blacklisted in so I'll only do it once. The only bots I care about are Google and Bing. Others are ignored.

On Tuesday, March 11, 2014 5:53:08 PM UTC-7, Peter Booth wrote:

I had been curious about why you did this, thanks for sharing. If it were me, I wouldn't want to be doing any DNS lookups from within Nginx. Instead, I'd prefer to use a batch script to build my whitelist separate from my Nginx process and simply include the text file with list of IPs.

I can certainly relate to the pain that comes with this. The last retail site I supported was being scraped by over 300 distinct robots and spiders from tens of thousands of IPs.

david.birdsong

On Mar 13, 2014 9:36 PM, <ssi...@elance.com> wrote:
>
> If we slow down a Googlebot for a short period that isn't a problem. They don't hit very often. Also, after the first hit they are white or blacklisted in so I'll only do it once. The only bots I care about are Google and Bing. Others are ignored.
>

That's just it, if you tie up the nginx event loop, you're not just slowing down a googlebot, but all current sessions on that nginx worker.
>
> On Tuesday, March 11, 2014 5:53:08 PM UTC-7, Peter Booth wrote:
>>
>> I had been curious about why you did this, thanks for sharing. If it were me, I wouldn't want to be doing any DNS lookups from within Nginx. Instead, I'd prefer to use a batch script to build my whitelist separate from my Nginx process and simply include the text file with list of IPs.
>>
>> I can certainly relate to the pain that comes with this. The last retail site I supported was being scraped by over 300 distinct robots and spiders from tens of thousands of IPs..

ssimpson

On Friday, March 14, 2014 12:44:10 AM UTC-7, David Birdsong wrote:

On Mar 13, 2014 9:36 PM, <ssim...@elance.com> wrote:
>
> If we slow down a Googlebot for a short period that isn't a problem. They don't hit very often. Also, after the first hit they are white or blacklisted in so I'll only do it once. The only bots I care about are Google and Bing. Others are ignored.
>

But this only happens once for each googlebot IP and there aren't that many of them. After verifying the IP, they are whitelisted/blacklisted. I'll take the hit. If you have 100 workers, then 1/100 of the capacity is slightly slowed. I'm not going to prematurely optimize unless it proves to be a problem.

ssimpson

Another problem is that Google doesn't tell you the Googlebot IPs in advance. They are free to change them anytime without notice. You can't know them a priori. See https://support.google.com/webmasters/answer/80553?hl=en.

peter_booth

You're right, they do change, though it's at a glacial speed compared to web time. Openresty is a toolkit that people use in diverse ways, and most problems can be solved in a bunch of ways.

I would want to isolate the process of building my whitelist from the process of serving web pages. Not as an optimization, be it premature or otherwise, but as a separation of concerns.

Philosophically, I'd see the challenges that you're encountering when trying to do it in-process as "maybe the pushback is the problem telling me I'm approaching this the wrong way?"

That's my way of thinking- YMMV

Sent from my iPhone

On Mar 14, 2014, at 11:46 AM, ssi...@elance.com wrote:

Another problem is that Google doesn't tell you the Googlebot IPs in advance. They are free to change them anytime without notice. You can't know them a priori. See https://support.google.com/webmasters/answer/80553?hl=en.

.