Gabriel Weinberg's Blog

I thought I knew everything you needed to know about Web images.  But, of course, I didn't. Here's what I just learned when launching the new icon bar on the homepage of Duck Duck Go. We wanted the it to function sort of like the Apple dashboard (and on the Web like Schmedley's bottom bar).

Today I started the official blog for Duck Duck Go, aptly named Duck Duck Blog.  The inaugural post is What People Search First at Duck Duck Go.  The general blog feed is here.

Today I am really really disappointed in MIT, my alma mater.  Yesterday they officially announced that they are cutting eight varsity sports.  No longer will we be able to say we offer "the most varsity sports of any university."

The recent surge in twitter press is amazing. A lot of my less technical friends and family have been asking me about it, usually with skeptical tones.  Why would you do that?  You use it--really?

Ever since Eli was born, we've been skyping with my parents regularly.  My parents supposedly bought a high quality webcam, but we weren't seeing them great.  And I know my built-in laptop cam wasn't up to par, so a few days ago I set out to make it all work.

I often get asked about Search Engine Optimization (SEO) because I run a search engine.  The following is currently my best advice with regards to SEO.  Just to be clear, I'm talking about Google here (and by extension Yahoo, Live, etc.) and not Duck Duck Go (my search engine), which has different optimization parameters.

I assume you already know the basics, e.g. don't have duplicate content, have good looking URLs and page titles, etc.  If you don't know the basics, read Google's SEO Starter Guide first.

Last night I sent this tweet: "Finally watching The Fast and the Furious: Tokyo Drift."  Literally within a minute or two TokyoDriftMovie starts following me on twitter.  His bio says:

I'm looking for other people who enjoy The Fast and The Furious: Tokyo Drift Movie as much as I do, maybe that's asking too much. You at least have to enjoy it.

So does he (Rob Jones) have a bot set up that follows anyone who mentions the movie?  His profile links to an amazon store page where you can buy the movie.  Is he doing this for many movies?  If so, he takes really great pride in the details of his work--just check out his twitter stream, e.g. "I stopped keeping track after seeing Tokyo Drift 100 times..."

I often get asked what Duck Duck Go "runs on."  This post basically answers that question by outlining the major moving parts that serve queries, i.e. its architecture.  I'll detail in another post what, in particular, makes it fast, i.e. tunables and other specifics.

Short version: I violated twitter TOS, so my account was suspended (as it should have been).  However, I think what I was doing was worthwhile.  So what I suggest is a new, paid tweets section (akin to direct messages in functionality, but you pay to send them).  So if the receivers never wants to see them, they don't have to.  But if they do, the paid tweets are one-click away...  

***

Last night the Duck Duck Go twitter account was suspended. I was sleep deprived from the birth of my son, but yet still awake being a human baby monitor. So I quietly used the computer to pass the time (like I'm doing now!).

Lacking SSH access (hospital wifi blocks port 22 for some reason) I was essentially blocked from doing any real work. So I decided to mess around with twitter's search feature, which everyone seems to be talking about lately.

It really is cool. There were a ton of people talking about very specific things I'm really interested in, new search engines being one of them. So of course, I wanted to communicate with them.

So what did I do? Naturally, I found the relevant tweets, and replied to them. I'm still not sure if you can actually reply to a particular tweet, but I did click the reply link from each tweet's individual page. They were all tweets about checking out new search engines.

My reply said something like "@username here's another new search engine, which is live now: http://duckduckgo.com. What do you think? Yes, it's ours :)". Some people replied very quickly with feedback, so I know my tweets at least were somewhat relevant. I'm guessing others would have replied too, but I can't tell for sure because my account was suspended shortly afterwards.

So I naturally went to the twitter support page on account suspensions. It does look like I violated this term: "updates consist of duplicate or repeating links and/or text." That's exactly what I did. OK, fine. I sent an apology to them so hopefully my account will be reinstated soon.

To the bigger question though: what should I have done? Not sent the tweets of course...but is there some way to have the same effect without getting my account suspended? For example, should I have just changed up the message each time? Probably wouldn't have mattered. Sent to less people? But all those people seemed to express equal interest.

Thinking about it some more, I don't really see any way in the current twitter scheme to send tweets like this without risking suspension. And yet, there is value here (on both sides). I clearly want the right people to check out the search engine. And people who post tweets about checking out new search engines, presumably, on average, want to check out a new one!

So here's what I suggest, which I'm sure has been suggested before but I'm too lazy to find out where. (And one more nudge won't hurt anyone...):

Twitter should charge me to send tweets like these, i.e. unsolicited business related tweets. The charging part both would cut down on spam and make them money. But wait, isn't it still some form of spam? Agreed, and, as such, it both messes up the twitter user experience for those receiving them (their reply screen) and those people following our twitter stream, which is annoying.

To fix this issue, twitter should also make a new type of message (like DMs) for this stuff. Instead of 'd ' make it 'p '. Each time I make a paid tweet, charge me $0.05. And on the receiver end, put them in their own window so they don't clutter up anything else. That way only the receiver sees them, and only when they want to.

Update: additional comments can be found here.

Yesterday Yahoo! announced upcoming usage fees for its BOSS API.  The announcement has caused quite a stir on the official discussion group and elsewhere.  As the founder of a new search engine that uses BOSS, this news clearly effects me.  Here are my initial thoughts on the matter...

And if you like that...check out these.

I'm starting a new category of posts today for TV and movie clips that I find absolutely funny.  I've wanted to edit together a long video of all these clips, and I see this as a starting point.  So without further ado...

I was just debugging my crawler again for the Parked Domains Project.  Every now and then it still dies despite appropriate eval wraps around areas that could die.For a super weird example, try this (it's in Perl):

use LWP::UserAgent;
my $ua = new LWP::UserAgent();
my $response = '';
eval {
    local $SIG{ALRM} = sub { die "alarm\n" };
    alarm(10);
    my $request = HTTP::Request->new('GET','https://www.whatsyourriskscore.info/');
    $response = $ua->request($request);
    alarm(0);
};
print "TEST!\n";

The print statement never executes!  I'm not really sure as to why, though I traced it pretty far as is explained in the bug report I just filed.  It seems to have something to do with line 208 of LWP/Protocol/http.pm:

# Since this just writes out the header block it should almost
# always succeed to send the whole buffer in a single write all.
my $n = $socket->syswrite($req_buf, length($req_buf));

$written = Net::SSLeay::write_partial( $ssl,$offset,$length,$$buffer );

V1 of the Duck Duck Go iPhone application was released late last night.  You can read more about it, see screen shots, and download it here.

I was just debugging my crawler for the Parked Domains Project.  Every now and then it dies when doing a DNS query. 

After some Web research, I found out there are some known issues with the perl Net::DNS module--it's parsing code can croak on various malformed DNS entries.  So I wrapped the call in an eval.  It went from

my $query = $res->query($host . '.');

to

my $query = '';
eval {
            local $SIG{ALRM} = sub { die "alarm\n" };
            alarm(10);
            $query = $res->query($host . '.');
            alarm(0);
        };

I had already defined a timeout of 10 at the top of the code, i.e.

my $res = Net::DNS::Resolver->new;
$res->tcp_timeout(10);
$res->udp_timeout(10);

Anyway, here is the weird part.  After adding this eval, my code used about 200% less CPU and my crawler speed increased significantly.  That's great and all, but I can't figure out why!

I haven't spent too much time on this yet, but I've verified the code is actually working, and that the only local die messages (from the eval) seem to be the alarm.  Any ideas?

A couple days ago I released two widgets, and since then I've wanted to keep an eye on installations for bug detection and vanity purposes.  Tailing the logs for this purpose was becoming cumbersome, so I whipped up this one-liner to tell me what is going on.

grep [kp].js /var/log/nginx/nginx-access.log | awk '{print $10}' | perl -pe 's/^\"http:\/\/([^\/]+).*\"$/$1/' | sort | uniq -c | sort -n

I'm posting it here to remember it and because it might be useful to you.  Here's what it does.

You are not your job.  You are not your online profiles.  You are not the sum of your reddit and news.yc karma.  But hey, that shouldn't stop you from linking & displaying them on your sites.

After optimizing my Web crawler for the Parked Domains Project, I started crawling so fast that the log process for my DNS server was eating up 20% of one of the CPUs (and wasting a lot of I/O as well).

exec setuidgid daemon multilog -*

exec 2>&1

to

exec 1>/dev/null 2>&1

Now nothing will go to multilog at all.  Of course, you need to restart or HUP the log and run processes 

I spent the last 24 hours optimizing the Web crawler for the Parked Domains Project.  The previous bottleneck was obviously CPU.  After a bunch of profiling and benchmarking, I determined that a particular block of Perl regexp was causing most of the problem.

my @regexp = (
  'cat',
  'dog',
  'bird',
 );

use Regexp::List;
my $regexp  = Regexp::List->new;
my $qr = $regexp->set(modifiers=>'i')->list2re(@regexp);

Like many entrepreneurs, I check the compete.com #s for my Web sites.  I've found that they are way off for relatively low traffic sites.  Take my recently launched startup, Duck Duck Go.

Google recently started pointing out pages with malware in their search results by placing "This site may harm your computer" next to such links and putting intermediary warning pages in between you and the sites.  

But, look at the situation from the user's point of view. As a user, I'd be pretty annoyed if Google sent me to a site it knew was dangerous. Even a short delay would expose some users to that risk, and it doesn't seem justified. 

But, look at the situation from the user's point of view. As a user, I'd be pretty annoyed if Google sent me to a site it knew was dangerous. Even a short delay Keeping the link would expose some users to that risk, and it doesn't seem justified. 

Google must see a significant line between keeping the link with warnings and disabling it that I don't see.  Perhaps you see it their way.  If so, I'd be interested to know more...

A few days ago, I finally shared what I've been working on this past year with the general public: a new general purpose search engine called Duck Duck Go.  (For more info, see this about page.)  It hasn't yet received any press, and I didn't expect any.  Who's going to be first? :)

Google Browser Sync was my favorite Google product.  It syncs browser data between your computers.  In my case, I use it to move between my main Desktop and laptop seamlessly.  My bookmarks, passwords, history, and even my most recently opened tabs were just right there.

I've used online bookmark services in the past all the way back to backflip (remember them?), but Google Browser Sync really took it to a new level for me.  It syncs much more than bookmarks and even more importantly I didn't have to go anywhere--it just worked.  And it worked fast and well.

I recently delayed upgrading to FireFox 3 because it kept saying that Google Browser Sync wasn't updated for it yet.  So I finally searched for why, and found this post from Google saying they are discontinuing my favorite Google product!

Yes, I get there are alternatives.  Yes, I get Google is a private company and can do whatever they want.  Yes, they are releasing code.  But I still just don't get it.  Why?  Why, Google, why?

How much resources does it really take to continue supporting Google Browser Sync?  Why does Mozilla think this is a huge opportunity and you don't?  And even if you don't think it is *as huge*, you can't make a bet on it with one FTE or even a partial FTE out of your 19K+ employees?  What about just to enable its use for FireFox 3?

And how much of a chance did you really give it anyway?  I didn't notice any real promotion, and everyone I've explained it to (yes, non-tech people too) love it, just like me.  Is there anything I can do to change your mind?

Lame! (cut from South Park's AWESOM-O)

After encouraging feedback about WatrCoolr on Hacker News and Proggit, I created Hacker WatrCoolr, which is a hacker oriented version of the original site.

What does that mean?  A lot of little tweaks to make the site more amenable to hackers.  And one big change: the general interest feeds are replaced with hacker interest feeds on Hacker WatrCoolr.  Here are the current ones:

• Hacker News of course (stories that reach the top)
• Techmeme (stories that reach the top)
• RSSmeme, English 12 hours (stories that reach the top)
• reddit, programming (stories that reach the top)
• Digg popular: software, programming, design, tech news & gadgets, aggregated
• Slashdot: developers, books, ask, bsd & it, aggregated
• Yahoo! Technology News Most Emailed (stories that reach the top)
• del.icio.us popular: programming, webdesign, startups, design, tools, software, web2.0, css, reference & development, aggregated
• ReadBurner (stories that reach the top)
  

(And no xkcd.)

These feeds are subject to change based on their continued usefulness and your feedback.  I've been tweaking them for a week or so (as you may guess from reading them).

Everybody says dumb things, and I highly respect Warren Buffett's opinion (how could you not?), but I was astounded when I read this Forbes article yesterday.  The article summarizes highlights from Buffett's latest annual shareholder letter.  Not off-hand comments mind you...this was apparently from his annual letter--his ivory tower!

Here's the real downer. You're unlikely to make anything like 5.3% for the next 92 years. Swallow hard and read on. To achieve 5.3% this century would mean a Dow climbing from roughly 13,000 today to (gulp!) 2 million. As Warren opines: "We are now eight years into this century, and we have racked up less than 2,000 of the 1,988,000 Dow points the market needed to travel in this hundred years to equal the 5.3% of the last." Don't expect 5.3% in the 21st century. That's the bottom line.

I was so incredulous that I went straight to the source.  The following is from page 19 in the 2007 letter.

Let's revisit some data I mentioned two years ago: During the 20th Century, the Dow advanced from 66 to 11,497. This gain, though it appears huge, shrinks to 5.3% when compounded annually...Think now about this century. For investors to merely match that 5.3% market-value gain, the Dow - recently below 13,000 - would need to close at about 2,000,000 on December 31, 2099. We are now eight years into this century, and we have racked up less than 2,000 of the 1,988,000 Dow points the market needed to travel in this hundred years to equal the 5.3% of the last.

It's amusing that commentators regularly hyperventilate at the prospect of the Dow crossing an even number of thousands, such as 14,000 or 15,000. If they keep reacting that way, a 5.3% annual gain for the century will mean they experience at least 1,986 seizures during the next 92 years. While anything is possible, does anyone really believe this is the most likely outcome?

I should mention that people who expect to earn 10% annually from equities during this century - envisioning that 2% of that will come from dividends and 8% from price appreciation - are implicitly forecasting a level of about 24,000,000 on the Dow by 2100. If your adviser talks to you about double digit returns from equities, explain this math to him - not that it will faze him. Many helpers are apparently direct descendants of the queen in Alice in Wonderland, who said: "Why, sometimes I've believed as many as six impossible things before breakfast." Beware the glib helper who fills your head with fantasies while he fills his pockets with fees.

Now there may be reasons not to expect similar returns in this century as compared to the last, but this is certainly not one of them.  I think this is a completely nonsensical (or at best a circular) argument.  Let's parse it.

The argument seems to go like this:

1. Suppose we had similar 5.3% price appreciation in this century like the last.
2. Then the Dow would have to close at 2M by end century.
3. This number is too high to believe.
4. Therefore, there can not be 5.3% price appreciation in this century.
5. And if you believe in even higher returns, the 2M is even higher, so your belief is even crazier.

Yet, he opens the whole argument with this: "During the 20th Century, the Dow advanced from 66 to 11,497. This gain, though it appears huge, shrinks to 5.3% when compounded annually."

Now fast forward to 2107.  Buffett's descendant writes "During the 21th Century, the Dow advanced from 11,497 to 2,000,000. This gain, though it appears huge, shrinks to 5.3% when compounded annually."

Don't you think that 11,497 seemed as crazy to the people in 1900 who were looking at a 66 point Dow as a 2M point Dow now seems to us?  You should because it is the same % growth!  The absolute #s are meaningless, and if people have a problem with the index # getting too high, they can reset it.  Problem solved.

This is not a discussion about whether we are actually in for similar, higher, or lower returns.  Let's leave that for another day.  This is just to say that this argument for lower returns is ridiculous!

Lame! (cut from South Park's AWESOM-O)

Update: additional comments can be found here.

The current debate on Hacker News seems a little too black and white to me.  In my opinion, there is no right way to start a startup (or a business).  How you should start your startup depends on your definition of startup success.  In particular, I believe you should:

1. Define what startup success means to you right now.
2. Make decisions that have the highest probability of achieving that success.
3. Every so often, GOTO 1.

Different people will have different definitions of startup success depending on a number of highly personal attributes, including current monetary wealth, desired monetary wealth, desired lifestyle, what one wants to work on, how much one wants to manage people, how much one wants to impact the world, etc, etc.

Consider the following (knowingly grossly simplified) personal situations:

• An independently wealthy person doing what they love with an indefinite (5Y+) time horizon.
• Someone at a high salaried consulting job who wants to quit to start a startup, but not change his/her lifestyle.
  
• Someone living in a developing country who just wants enough money to not work.
• Someone who wants to quit their job to do a startup, but has a family and so needs $x/month relatively soon to do so.

I posit that these people will (probably) have vastly different definitions of startup success, and thus should be starting their startups in different manners. 

A poll I created the other day I believe exemplifies this point.  I was surprised how many people would sell their startup for under $100K (the highest answer).  Clearly this poll has issues--for one, it would be better statistically if it were worded slightly differently and then given to startup founders right when they are starting out.  But I think the core exemplification still shines through.  If your monetary end goal is $5M vs. $100K, you should probably be doing things a bit differently.

Btw, this comment inspired me to write this post...so thanks for that hugh.