January 2009 Archives

I'm starting a new category of posts today for TV and movie clips that I find absolutely funny.  I've wanted to edit together a long video of all these clips, and I see this as a starting point.  So without further ado...

I was just debugging my crawler again for the Parked Domains Project.  Every now and then it still dies despite appropriate eval wraps around areas that could die.For a super weird example, try this (it's in Perl):

use LWP::UserAgent;
my $ua = new LWP::UserAgent();
my $response = '';
eval {
    local $SIG{ALRM} = sub { die "alarm\n" };
    alarm(10);
    my $request = HTTP::Request->new('GET','https://www.whatsyourriskscore.info/');
    $response = $ua->request($request);
    alarm(0);
};
print "TEST!\n";

The print statement never executes!  I'm not really sure as to why, though I traced it pretty far as is explained in the bug report I just filed.  It seems to have something to do with line 208 of LWP/Protocol/http.pm:

# Since this just writes out the header block it should almost
# always succeed to send the whole buffer in a single write all.
my $n = $socket->syswrite($req_buf, length($req_buf));

$written = Net::SSLeay::write_partial( $ssl,$offset,$length,$$buffer );

V1 of the Duck Duck Go iPhone application was released late last night.  You can read more about it, see screen shots, and download it here.

I was just debugging my crawler for the Parked Domains Project.  Every now and then it dies when doing a DNS query. 

After some Web research, I found out there are some known issues with the perl Net::DNS module--it's parsing code can croak on various malformed DNS entries.  So I wrapped the call in an eval.  It went from

my $query = $res->query($host . '.');

to

my $query = '';
eval {
            local $SIG{ALRM} = sub { die "alarm\n" };
            alarm(10);
            $query = $res->query($host . '.');
            alarm(0);
        };

I had already defined a timeout of 10 at the top of the code, i.e.

my $res = Net::DNS::Resolver->new;
$res->tcp_timeout(10);
$res->udp_timeout(10);

Anyway, here is the weird part.  After adding this eval, my code used about 200% less CPU and my crawler speed increased significantly.  That's great and all, but I can't figure out why!

I haven't spent too much time on this yet, but I've verified the code is actually working, and that the only local die messages (from the eval) seem to be the alarm.  Any ideas?