For the first two years or so, I ran DuckDuckGo out of my basement. Then when people really started using it I set up EC2 for fail-over, and now EC2 is our primary host. We still use these basement servers for a variety of things, however.
Before DuckDuckGo, I operated in a pre-cloud world and had been running my own servers for a number of years. My last company co-located (stored our servers in a data center), which had worked well except for when things went wrong and a reboot didn't solve it. Then I'd have to (usually at inopportune times) drive over to the data center, be escorted in, mess with things, and be charged a lot for doing so.
The main problem with self-hosting (vs co-lo) in my mind had always been crappy Internet. Then in 2006 I moved to our house in a town that approved FIOS a few months later. I had been watching it progress, and we were literally the first person in the area to install it.
I was hopeful and it turned out correct that the crappy Internet problem had been solved. In five years, it has never been fully out except for the time our service got suspended for lack of payment (forgot to update my expired card). Other than that, there is just the usual blip always coincidental with some internal Internet routing thing. In fact, since I have complete network control, the "co-lo" doesn't reboot routers for scheduled maintenance either. The latency is also pretty good.
Note that I'm not advising any of this -- simply telling my story. So why did I do this? Really it was the path of least resistance given my history and experience. I was used to setting up and maintaining servers, and had some laying around. This setup seemed great at the time -- I'd get the benefits of co-location (good Internet) but could mess with things at will. (The other co-lo benefit is power -- I have that covered as well.)
I still like the idea of having a server for non-production use, e.g. this Web site. There is something nice about having it close by. Perhaps it is a false sense, but I feel that the setup is more reliable than any cloud provider. And I get to feel a bit more full stack.
But all that pretty much stops at one server for me. Anything beyond that and it quickly becomes a PITA. Here's a pic from when we bought the rack off of Craigslist: