Thursday, January 16, 2014

Fernmelder to the rescue

I've been experimenting with mass DNS resolving lately.

Imagine you have some large list of DNS names (FQDN's)
which you want to map to its IPv4 or IPv6 address.
That could be a GB sized zone-file or an enumerated list of
names for some double-flux network when you research
how you can take down a botnet. In either way, sometimes
you need to do that in finite time and clearly
gethostbyname() in a loop is not the way to go!

For asynchronous resolving the glibc already has
getaddrinfo_a() but it turns out that this function is
entirely useless because its using threads. So, for
every request you send, a thread is cloned() which does
not scale well. [The glibc aio_ functions also use threads,
its a pitty that glibc async support is so toast!]

So I hacked up something from scratch that works for me.
Its on my github. The output resembles that from dig
and from the zone files you know.

The problem is to find right parameters for the amount
of requests to send in a row and the amount of usecs you
want to usleep before doing that again. Otherwise you will
just hammer the DNS server and gain no response. The default
values are sane enough that it yields some good result.
The better your uplink to your recursive DNS, the
smaller amount of time you need to usleep().
You can also distribute the requests across multiple DNS
servers by using more than one -N switch. The more reliable
DNS servers you have, the better because you do not run
in any rate limiting.


Anonymous said...

why not c-ares?

Sebastian said...

I tried with various async DNS libs until I found I could write one myself in less time than getting the API of the others right.