August 2010 Archives

Productive Programming

BIG disclaimer: I'm not formally trained in computer science (aside from two classes at MIT in 2000) and I haven't worked closely with that many programmers or teams (maybe 10 or so).

I've gotten to be a lot more productive as a programmer since I started casually 15+ years ago and seriously about 10 years ago. By productive, I mean shipping something that actually works in as short a time as possible. 

Lately I've been thinking about the origin of all this productivity. I think it all comes down to doing a few things well. Say you want some code that does X.

  1. Tools. Often the quickest way to do X (especially if X is complicated) is to grab something off-the-shelf. My goto repository is CPAN, or even higher level some essentially (to me) black box Apache project. Once I reach the limitations of that tool, then I might jettison it, but when the goal is to ship quickly, nothing beats using existing code.

  2. Simple Algorithms. If I have to do it myself, I initially try to accomplish X in the simplest way possible by breaking it down into trivial steps. I know there is a famous programming quote that pertains to this process, but I can't find it right now--someone in the comments help me out please :)

    his breakdown process has certainly become easier over time for me as more tools/data structures/design patterns/etc. pop into my head immediately just from having used them a lot. But the real key is to push off all complexity until later. Get something working, then add to it incrementally.

    Pushing off complexity is not as easy as it sounds. A lot of things that seem essential are really not. What I do is make a critical path walk through of what is absolutely needed to accomplish X. Anything that improves X but is not on that critical path, I write down on a list to revisit later.

    When you do the initial breakdown, it is quite possible that some of those (still non-trivial) steps can be accomplished by Tools, i.e. you can cobble together some stuff of your own with other people's stuff to get X working quickly. Just because one tool doesn't do X outright, doesn't mean you should do everything yourself.

  3. Debugging. I've seen people waste the most time here, literally aimlessly changing lines of code hoping it will solve their problems. When approaching debugging, I first find out where the problem is, i.e. the exact line of code or config file or whatever that is causing the problem. That should not take long, and before you ask anyone else about it you should know exactly where the problem lies.

    If it is my code and I've been working on it recently, I usually have a pretty good mental model for how it all works and can get to the relevant line almost immediately. If it is not my code, e.g. an open source tool or sysadmin thing, I usually make educated guesses and then essentially do a binary search printing variables out as I go until I see where it messed up.

  4. References. If you are stuck in either Algorithms or Debugging, it is likely people have talked about your problem on the Internet already, e.g. on HN, StackOverflow, a mailing list, tutorial, FAQ or in a book. Knowing how to use references is key to doing things quickly. I can't tell you how many people I've tried to help who needed to RTFM

    Before you ask anyone, you should look online. If you're really stuck, ask someone on IRC or post your own question to an online forum. Then work on something else while you give people time to respond. When you're really stuck, there is no reason to spin your wheels endlessly. I've also found sleeping on it often magically produces a solution in the morning.

Those are the main things that I think contribute to my programming productivity. Other than those, there are some largely idiosyncratic habits I've formed that certainly help me but I'm not sure how useful they would be to you:

  • I run code constantly, literally every change I make I re-run stuff to see if it improved/is working. This process makes it so I usually only have one bug at a time to deal with. It's part of how I work through that whole simple algorithm thing above.

  • I put # For debugging lines in everywhere when debugging and then I leave them there so when I come back I can easily uncomment them and get some decent debugging output for that particular component.

  • I try not to be clever and comment anything that looks complicated with a short explanation right by that piece. I used to try to be a lot more clever, but it never worked out well.

  • I'm not an early adopter of new tools/languages unless they seem to fit X exactly. I'm still using Perl with no framework :). But I have started using nginx, memcached, etc.

Update: more good comments on HN here, including this one from xentronium: "There were awful lot of tasks I tried to achieve via pure coding and rewriting / rewiring blocks here and there. All they actually needed was some analysis on paper." I have had the same experience.

Update2: this post is re-printed in the Software Developer's Journal.

Personal Inflation

Recently there was some interesting discussion on Hacker News (here and here) regarding whether or not you can retire at 30 with a $1-5million (after selling your company).

That question aside, one major input is inflation, and all these models make the assumption that it is the same for everyone. Depending on your personal circumstances, I think your personal inflation rate could vary highly from the average.

It comes down to that the basket of goods you purchase over the time frame in question may look very different than the CPI (official) basket. Here are some extreme examples to highlight the differences:

  • Say you sell your company and buy a house. This expenditure probably dwarfs all your others. If you consider the time period from when you started your company, you might have had a decently negative personal inflation rate due to falling house prices.

  • Or say your biggest expenditure is commodity servers, which drop in price rapidly. If your needs are relatively constant, your personal inflation rate could similarly be negative as these assets depreciate fast. 

  • Taking a longer view, suppose you never have kids (read education expenses), and you buy your house in this economy and live in it indefinitely. Assuming you just buy normal stuff the rest of life (food, etc.) your personal inflation rate should be rather under control. You somewhat cut out a lot of the biggest drivers long term (education, housing) and you gained some negative inflation by buying your biggest expenditure (house) at the right time. 

Rapid prototyping as burnout antidote

Sometimes I get burnt out. Who doesn't? I'm very sensitive to it though because I know it can be devastating to a startup, especially a single founder startup.

Over the past several years I've learned that a burnout antidote, at least for myself, is rapid prototyping of something brand new. It could be a skunk works project for the startup, but more often than not, it is something just off the wall.

Actually, it doesn't even have to be software related at all. Recently I cleaned out and organized our basement.

I didn't think too much about this process (except for that it works for me) until the other day. Upon reflection, I think why it works is the satisfaction of pushing a minimum viable product out the door. 

It's fun and you're left with a great sense of accomplishment. And since it isn't your main thing (startup), you don't feel you have push past that mirage 80% completion and spend the rest of your life on the final 20%.

That's why working on a project related to your startup often doesn't work for this solution. First off, it can't be incremental because then you don't get the satisfaction. But even if it is something brand new, you could feel compelled to keep working on it, especially if it hits users. Now that may not be a bad thing at all; it just may not help with burnout. 

The other thing I've learned to stave off burnout as much as possible is to get a lot of user feedback. Ideas and suggestions from real users really motivates me.

Update: also some good comments on HN.

Thoughts on Yahoo! BOSS Monetization II

Last year (Feb 11, 2009), Yahoo announced upcoming usage fees for its BOSS API. At that time, I wrote up my thoughts on BOSS monetization

That was before the Microsoft/Yahoo search deal. Yesterday, Yahoo! announced more upcoming BOSS changes in light of that deal. Or, they announced that nothing will change... 

BOSS will apparently remain in existence despite Bing's powering of Yahoo! organic results. And BOSS will get a fee structure just like was proposed last year. Actually, it might be a bit better than that because there is talk of a revenue share program that would waive fees, but we'll have to wait for the details on that come out.

While I use BOSS myself for DuckDuckGo, and will probably continue to do so in some capacity, I find the idea of paying for it a bit curious for a few reasons.

  • Downtime. BOSS has had a lot of down time. Recently, it was down for about 24 hours straight. You can't build a reliable search engine on that kind of down time. Moreover, there is no API status dashboard, which I've pushed for, and there is also no reliable way to communicate with Yahoo! about BOSS downtime. They say to use the usergroup, but is anyone really checking it at 4AM? It doesn't seem like it.

    If I'm paying I would hope to have an SLA with some way to reach the ops team. Sometimes it feels like I'm the only one monitoring! I get alerted (by email and text message) when it goes down and report it after I determine it isn't my fault. And I'm pretty much the only one reporting it like this.

  • Bing. If Bing is powering Yahoo's organic results, will it also be powering BOSS? The announcement seems to indicate this will be the case. If so, why not just use Bing's API, which is currently free and would give the exact same results? If not, then why wouldn't we expect it to just degrade in quality as Yahoo spends less time and money on generating their own organic results (e.g. on crawling, indexing, etc.)?

    If I'm paying I would hope to have solid answers to these questions. I hope that they do continue crawling and indexing, because for me I'd like to use reliable independent APIs. Otherwise, the combined entity is essentially a single point of failure, or maybe single+epsilon.

  • Bugs/Roadmap. There have been a lot of outstanding bugs in the BOSS API for a while now, which I have worked around. For example, the API will change your query in some cases and won't tell you. There hasn't been great communication as to a roadmap, and I get the sense that everything was sort of on hold for a long time because of the Microsoft deal. If the commitment to BOSS is serious, please start treating it accordingly.
In any case, DuckDuckGo should be fine!

Update: in hope that this post does not come across as complaining, I want to add that I'm very grateful for the BOSS platform. I don't know where I'd be today without it. At the same time, I'm in a somewhat unique position to comment on it, and I think the above are valid concerns with regards to charging for it that could benefit from more public discussion. 

Update2: Ashim (head of BOSS) just released this update on the forum, clarifying that indeed BOSS Web results will be powered by Bing in the future.

DuckDuckGo now operates a Tor exit enclave

Eff tor.png
Tor enables Internet anonymity by routing your traffic through a series of encrypted relays.

DuckDuckGo now operates one of these relays, and more importantly an exit enclave for DDG search engine traffic. 

That means if you're on Tor, and you access DDG, you'll likely exit through our relay and get service much faster. Tor can be slow, but this should speed it up a bit (when using DuckDuckGo).

I believe this fits right in line with our privacy policy. Using Tor and DDG, you can now be end to end anonymous with your searching. And if you use our encrypted homepage, you can be end to end encrypted as well.

Thanks to Caine Tighe for the idea and for helping to make it happen.

I should point out that people are often scared of operating Tor relays due to potential abuse issues. Those issues scared me off too, at first. But with this exit enclave setup, there are no abuse issues. Only DDG traffic exits from our relay.

Even if you have only 20KB of bandwidth to spare, you can set up a bridge relay that just routes traffic between nodes and never lets anyone out through your relay. That helps the cause, and it is really easy to set up. You can limit bandwidth to just the amount you want to give. Check out the FAQ.

Update: there is now a DuckDuckGo hidden service at 3g2upl4pq6kufc4m.onion

First-timer entrepreneur symptoms and cures

Previously I wrote about wannabe entrepreneur symptoms and cures. I was once a wannabe entrepreneur, and then I was a first-timer.

First-timers are not wannabes--they're actually out there doing stuff. And they can be successful their first time around. I've funded some myself. But it's not the norm.

OK, so it's not the norm to be successful at startups at all. It's even less the norm for first time entrepreneurs though. The reason is pretty obvious--they make first-timer mistakes.

I made all of these mistakes in my first companies.

Symptom: you're asking everyone you meet with to sign an NDA.

Cure: don't do it, and know it is a major turn off to investors, who won't sign it anyway.

Symptom: you went with the first idea that sounded like it could work.

Cure: come up with a bunch of ideas before picking one.

Symptom: your co-founders are not startup types.

Cure: be careful with team formation and work with people who really get the startup lifestyle. Listen to Paul English.

Symptom: you think that if you build it, they will come.

Cure: spend a lot of your time thinking about customer acquisition (for me it's ~50%). Only very rarely will they come on their own early on.

Symptom: you think your idea is going to revolutionize something.

Cure: pick another idea or a more incremental initial focus. You're probably ahead of your time. Maybe you're not, but I was by ten years or so.

Symptom: you're about to give up too quickly.

Cure: you're likely starting a movement and not sparking a powder-keg. If you believe in your idea, give it some time. Do you see signs of life?

Symptom: you are wasting lots of time on tangential crap like premature optimization.

Cure: set goals and focus on critical path.

Here are some others that @dowdle left in a comment:

Other symptoms include: one of the first hires planned is a CFO, hiring a bunch of people with Big Co experience and no start-up experience, raising money without understanding terms, looking for office space prior to funding, putting IPO as the exit strategy.

Message here is that the best path to being an entrepreneur is to apprentice under a real one or go work at a start-up and learn the game. 

I did the CFO one too.

I second his general cure, but would add that you can also get there through good mentor-advisors. I'm not talking about an "Advisory Board" that does nothing, but real mentors you go to every month or so and talk serious strategy.

A new approach to mobile search

iphone.jpgThe new DuckDuckGo App is up on iTunes (and is iPad compatible). Android and BlackBerry versions are also in the works.

The premise behind this search app is simple: going to Web sites on your phone to find information is a pain.

In response, the goal of the DDG app is to get you the information you want in zero clicks, without having to go to any sites.

To do that, we go out to the top links in real time and pull back the most relevant paragraphs for your particular search. These paragraphs go in a section labeled 'Topic Summaries.' They function like normal links, but with the readable paragraphs on top.

In addition, you get all the "Zero-click Info" that you get on the Web site, e.g. snippets from Wikipedia & over 25 other sources, instant answers from us & WolframAlpha, etc. You also get the the content pages we make for the Web site (formatted for mobile), e.g. disambiguation & category pages, related topics, etc. And you get normal links as well, with site icons next to them.

Of course, you don't always get what you're looking for with zero clicks. It's a work in progress, and with your feedback I'm sure it will improve over time. But right now I believe it offers a unique and compelling mobile search experience. It really shines when you just want some quick information on a topic.

The DDG app also protects your privacy. All searches are preformed over an encrypted connection (https/SSL), your search strings are not shared with the sites you click on, and no IP addresses are stored on our servers. In short, it follows our privacy policy.

The !bang syntax, which takes you to 100s of sites directly, and most goodies from the Web site also work. There are links explaining both within the app (on the home screen). 

You will notice there are not currently any ads. We tested some but found them to be too distracting and irrelevant. In the future, we may put up one unobtrusive, relevant context ad per search.

Thank you Chris Heimark for making this app possible and also thank you to all the people who beta tested it and gave us great feedback. I'd really appreciate it if you'd check it out now and give us your feedback as well.

My putty settings. What are yours?


putty.pngI use putty all the time to connect to my servers and local VMware images. I actually develop over putty via SSH (in emacs).

There are a lot of putty settings and I haven't explored them all, but there are some I make sure are always on. These are:

  • Window -> Lines of scrollback -> 10000. The 200 default is sort of ridiculous. I often have long script outputs and want to scroll back.

  • Window -> Behavior -> Window title -> <server name>. I currently have 19 putty windows running. If they weren't individually named by server, it would be a disaster.

  • Terminal -> Bell -> None (bell disabled). If I don't turn that bell off, I get a headache after like 5 min.

  • Window -> Translation -> UTF-8. The default (Latin encoding) causes all sorts of problems if you tail a log with UTF-8 output.

  • Connection -> Data -> Auto-login username -> yegg. Saves time.

  • In the past, I've also messed with default colors (Window -> Colours), but have always gone back to the default. I've also changed the keepalive (Connection -> Seconds between keepalives), but my current servers simply don't drop ever.

What else do you set?

Negotiate terms at the term sheet stage


I've been involved in a few deals now (selling my last company, angel rounds, things involving my portfolio & advisor companies, partnerships, etc.). I've seen a pattern that I hope I can convince you to avoid. Here it is:

  • You get a term sheet, and like some of what you see, e.g. price.
  • For lots of possible reasons (don't want it to go away, pressure, excitement, lack of understanding, etc.) you sign the term sheet to get to the next stage.
  • You then get the draft deal docs based on the term sheet.
  • You start reading through them and consult with your lawyer and see a lot of scary stuff.
  • Some of that stuff was actually in the original term sheet at a high level.
  • You start pushing back on these terms that you don't like.

There are a two main problems with this scenario, from your perspective...

  1. You'll usually end up worse off than if you negotiated the term sheet properly. The problem is you'll get serious push back of the form "you already agreed to that in term sheet." Yes, things are probably still negotiable, but you've lost a lot of your positioning, and frankly, they make a good point. So you'll probably be more likely to get stuck with things you don't like.

  2. You're significantly lowering the probability of closing. Each term in a term sheet gives and takes off of others. So when you try to clean it up after the fact, you're really asking the other side to go back to the term sheet stage and put everything else back on the table. Again, this is doable, but it puts the deal in reverse, and that is never good. You want to maintain momentum.

Luckily, this is an easy issue to resolve. First, don't rush into anything. If you have a term sheet, you certainly have enough time to reasonably understand it and go over it with your lawyers. 

Second, make sure you understand every point in there. For VC/angel stuff, start here. And finally, negotiate. A lot of stuff in there is probably changeable, so figure out what is important to you and push for the deal you want.

After the term sheet stage, I can guarantee you there will be more negotiation on the details, so it really makes sense to really get agreement on the high level stuff.