July 2011 Archives

You can take an external API and run it through your server instead of letting the client call it directly. 

Similarly, you can reset your headers. This can protect privacy by zeroing out search terms (in the case of the Referrer), but you can also set custom headers.

JSONP is a slight modification of JSON where the object is wrapped in a callback function usually specified by you. For example, say you grab a JSON object from somewhere that looks like this:

{"Name": "Cheeso", "Id" : 1823, "Rank": 7}

With JSONP you specify a callback function like 'parseResponse' and then it looks like this:

parseResponse({"Name": "Cheeso", "Id" : 1823, "Rank": 7})

This is useful for two reasons. First, the function will be called automatically when it is done loading. Second, it allows you to get around cross-domain errors.

If the above API was yours it's easy to call within a client-side script. But if it isn't yours, i.e. on another domain, and you try to call it you'll often get lots of cross-domain errors. The way around this is to use JSONP. Then you can do something like this:

<script type="text/javascript"
         src="http://other.server.com/api/?q=param&callback=parseResponse">
 </script>

You could also do that via JS, e.g.

Now here's the problem. Most external APIs don't have JSONP capability. 

No bother, with nginx you can turn JSON into JSONP.

This uses the HTTPEchoModule to wrap the JSON response in the callback for the external API.

The HTTPLogModule allows you to specify log formats within location blocks, which means you can write your API logs to a separate file. It also means if you proxy via a location block as in the examples above, you could give each proxy their own access and error logs with different parameters, e.g. error log level.

Update: good comments on HN.

This is the story of how I made a micro-site in an hour to try to take advantage of a Twitter trend that was related to DuckDuckGo, my startup. Well, related by name...

I first came across it eight hours ago when a tweet by @jordankanarek came across my stream. I didn't pay close enough attention to it then and didn't even realize it contained a hashtag -- probably because I didn't recognize the quote and the hashtag was so long.

Then a few hours later, Jordan actually sent me an email with a link to the hashtag search page. New tweets were loading like wildfire (and still are).

That got my attention! If you know me, you know I love startup micro opportunities. So I immediately went to namecheap and bought the domain replaceawordinafamousquotewithduck.com. Then I fired back this email.

Right after I sent the email I knew that aggregation was the way to go. I previously made some micro-sites to do similar things, e.g. poeet.com (aggregates #haiku).

I quickly grabbed that code and adapted it for the aggregation of these quotes. It needed a bit of changing because the purpose is slightly different. For example, I needed to do more normalization since we were going to rank them. If you're interested, I just put all the code up on github.

After I printed out some basic HTML I remembered that previously my sister (and others) had made some duck movie posters (for something else that never fully launched).  I figured they would be perfect for the micro-site, so I got those and put them up in the left column. 

Here is the final product.

Domain purchase to live site: 1.25hr. Now what?

We have this trending topic and we have this micro-site, but how to blend the two?

Well, of course the first I thing I did was yell from the rooftops as loud as I could. We tweeted it. Posted it to duck.co and IRC. On Facebook. And sent it to the DuckDuckGo spread team. Oh, and I even tried Google+ for good measure, but it didn't take :)

Those did generate some traffic, but nothing like donttrack.us or dontbubble.us, which were picked up by social news sites pretty quickly and then by blogs and more mainstream media.

Then I remembered we were still in the beta program for Twitter ads, and this could be the perfect use for them. So I quickly fired up the ad interface.

Twitter has a number of ad products, but the relevant one here is Promoted Tweets. You select one or more of your tweets and then you can promote them for chosen keywords (including hashtags).

You can either do cost per engagement (CPE) or cost per thousand impressions (CPM). The engagement in CPE is clicks, favorites, retweets and replies, though in reality it is clicks since that is what happens the vast majority of the time. 

The minimum for CPE is $0.10 and the minimum for CPM is $5.00. I figured I wouldn't have any competition, but I didn't know how they judge whether to show anything at all or if they fill in for whatever inventory they have, so I made two campaigns.

First I made a CPE campaign at $0.50 (the suggested bid) and a CPM campaign at $5 (the minimum). Previously my random engagement rates had been about 1%, so that would have put them at parity.

One thing that is cool about the Twitter ad platform is it pretty much immediately kicks into gear. A few minutes later I refreshed the analytics page and I had already spent $120 :). I clicked on the detail page and my engagement rate was ~5%!

Panicking (since the CPE was then 5x more costly than the CPM), I couldn't quickly figure out how to change the bid price (even though it is super-easy), and so I deleted the CPE campaign. I then let the CPM campaign take over, but after refreshing the hashtag search page in various browsers, I wasn't convinced they were showing it all the time. So I re-made the CPE campaign, but this time at $0.15, as the engagement rate had dropped to an equivalent $0.13 (saturation?), and I figured it would fall more and that was about parity.

To my suprise engagement actually climbed, and so after a while longer I lowered it to $0.10 (the minimum) so now both were at the minimum (and parity).

 

For a while thereafter, engagement stayed for both campaigns around 5.5%. Given that they are both promoting the exact same tweet, you would expect them to remain the same, which is why this end end result makes no sense to me.

Why would the CPE campaign be a full % point higher, i.e. 6.13% vs 4.99%? It remains a mystery to me. You might think well, they occurred at different times, but here is the graph of when they were shown.

They generally follow the same pattern, but I suppose they are somewhat different and maybe that end piece really made the difference. Either way, this is amazing engagement: before I was normally getting 1% with 2% max and now I'm at 5-6%! But did it lead to any benefit to DuckDuckGo?

All told (so far), as far as I can tell, it yielded about 302 searches on the site and 60 clicks to the search engine homepage. (We don't track users, but there are referrers in our logs and so I can grep for them; hence, these numbers.)

Hmm...that's not a lot. For reference, we're currently seeing about 250,000 a day.

On the micro-site side, I count 3,605 loads of the bottom (Spiderman 3) image and 3,777 loads of the top (Wall-E) image. That is less clicks than Twitter is reporting in their stats, and doesn't include the ones shared around by me. I don't understand the mismatch there.

So ultimately I spent around $630 on this. That would be (so far) approximately 17c per click to the micro-site and $1.74 to the search engine. 

From that standpoint, probably not worth it. If you include the value of the learning experience, this blog post and the thrill of it all, probably worth it.

Yet the trend is still going and I haven't re-upped the campaigns. Go figure. 

First, it seems hard to grab on to a twitter trend, even if you get pretty close to its message. People are using the hashtag, sure, but they're not inserting your micro-site URL (even it if it just missing a .com). That's just not the nature of the trend, and so you seem to be relying on either a) the site itself going viral or b) people clicking to the hashtag search and then clicking on a url in the stream.

Second. engagement on Twitter can get pretty high (for ads). They told me so originally, but I hadn't seen it "in the wild" until now. So that's cool. Theoretically, I'm sure someone could turn those types of engagement numbers into something very useful, e.g. if you were selling a product that yields more than a few dollars in profit.

Third. there are some open questions as to the Twitter Analytic #s, namely why did the CPM and CPE engagement rates end up different and why the large discrepancy between the Twitter click count and my click count)? 

Update: some good comments on HN.

Update3: Removed the images of the Twitter ad manager at their request, since the beta program was confidential. Looks worse without the pictures :(

My guess so far as to what will have the the biggest impact is that they will grow up expecting everything should be on-demand. People have usually framed that concept in a critical manner, but I think it is a bit more subtle and could turn out very positive.

Yes, kids want things on-demand and it can be annoying, e.g. that particular episode or song wherever they happen to be in the world at that moment. Yet kids have always wanted things on-demand. 

I'm sure I wanted things on-demand when I was two as well. But I usually couldn't get it. We had to wait until the song came on the radio, look in an encyclopedia, etc.

Now my son can often actually get it, and the question is more whether we should give it to him or not. And he knows it. So his default state switches to the expectation that things should be on-demand. It's like his default expectation that everything is a touch screen, which is amusing, but ultimately I doubt it will have much effect on his frame of reference. 

Expecting things on-demand though I think has more far-reaching consequences. I'm just not sure exactly what they are.

You could argue that when you grow up thinking like that, slow processes may come to really bother you. Why can't we do that now? Perhaps that will be good for the world long-term as his generation starts to break down processes and make them faster. Or not...