I was just debugging my crawler for the Parked Domains Project. Every now and then it dies when doing a DNS query.
After some Web research, I found out there are some known issues with the perl Net::DNS module--it's parsing code can croak on various malformed DNS entries. So I wrapped the call in an eval. It went from
I had already defined a timeout of 10 at the top of the code, i.e.
Anyway, here is the weird part. After adding this eval, my code used about 200% less CPU and my crawler speed increased significantly. That's great and all, but I can't figure out why!
I haven't spent too much time on this yet, but I've verified the code is actually working, and that the only local die messages (from the eval) seem to be the alarm. Any ideas?
Follow me on Twitter.
Or in Google reader.
Or by email.
Oh, and tell me what I should blog about: yegg@alum.mit.edu
After some Web research, I found out there are some known issues with the perl Net::DNS module--it's parsing code can croak on various malformed DNS entries. So I wrapped the call in an eval. It went from
my $query = $res->query($host . '.');to
my $query = '';
eval {
local $SIG{ALRM} = sub { die "alarm\n" };
alarm(10);
$query = $res->query($host . '.');
alarm(0);
};
I had already defined a timeout of 10 at the top of the code, i.e.
my $res = Net::DNS::Resolver->new;
$res->tcp_timeout(10);
$res->udp_timeout(10);
Anyway, here is the weird part. After adding this eval, my code used about 200% less CPU and my crawler speed increased significantly. That's great and all, but I can't figure out why!
I haven't spent too much time on this yet, but I've verified the code is actually working, and that the only local die messages (from the eval) seem to be the alarm. Any ideas?
