Hacker Remix

So you want to build your own data center

196 points by dban 5 hours ago | 83 comments

jonatron 4 hours ago

Why would you call colocation "building your own data center"? You could call it "colocation" or "renting space in a data center". What are you building? You're racking. Can you say what you mean?

xiconfjs 3 hours ago

I have to second this. While it takes mich effort and in-depth knowledge do build up from an “empty” cage it’s still far from dealing with everything from building permits, to plan and realize a data center to code including redundant power lines, AC and fibre.

Still kudos going this path in the cloud-centric time we live in.

matt-p 2 hours ago

Yes, the second is much more work, orders of magnitude at least.

j45 2 hours ago

Having been around and through both, setting up a cage or two is very different than the entire facility.

HaZeust 2 hours ago

I think you and GP are in agreement.

ThatGuyRaion 17 minutes ago

Not saying I don't agree with you but most tech businesses that have their own "Data center" usually have a private cage in a Colo.

macintux 3 hours ago

Dealing with power at that scale, arranging your own ISPs, seems a bit beyond your normal colocation project, but I haven’t bee in the data center space in a very long time.

redeux 3 hours ago

I worked for a colo provider for a long time. Many tenants arranged for their own ISPs, especially the ones large enough to use a cage.

chatmasta 2 hours ago

It seems a bit disingenuous but it’s common practice. Even the hyperscalers, who do have their own datacenters, include their colocation servers in the term “datacenter.” Good luck finding the actual, physical location of a server in GCP europe-west2-a (“London”). Maybe it’s in a real Google datacenter in London! Or it could be in an Equinix datacenter in Slough, one room away from AWS eu-west-1.

Cloudflare has also historically used “datacenter” to refer to their rack deployments.

All that said, for the purpose of the blog post, “building your own datacenter” is misleading.

matt-p 2 hours ago

The hyperscalers are absolutely not colo-ing their general purpose compute at Equinix! A cage for routers and direct connect, maybe some limited Edge CDN/compute at most.

Even where they do lease wholesale space, you'd be hard pushed to find examples of more than one in a single building. If you count them as Microsoft, Google, AWS then I'm not sure I can think of a single example off the top of my head. Only really possible if you start including players like IBM or Oracle in that list.

chatmasta 1 hour ago

Maybe leasing wholesale space shouldn’t be considered colocation, but GCP absolutely does this and the Slough datacenter was a real example.

I can’t dig up the source atm but IIRC some Equinix website was bragging about it (and it wasn’t just about direct connect to GCP).

matt-p 1 hour ago

Google doesn't put GCP compute inside Equinx Slough. I could perhaps believe if they have a cage of routers and perhaps even CDN boxes/Edge, but no general cloud compute.

Google and AWS will put routers inside Equinx Slough sure, but that's literally written on the tin, and the only way a carrier hotel could work.

chatmasta 1 hour ago

Then why do they obfuscate the location of their servers? If they were all in Google datacenters, why not let me see where my VM is?

achierius 57 minutes ago

Security reasons, I presume? Otherwise it would be trivial for an adversary to map out their resources by sampling VM rentals over a moderate time-period.

lostlogin 51 minutes ago

I’m very naive on the subject here - what advantage would this give someone?

deelowe 1 hour ago

Hyperscalers use colos all the time for edge presence.

fragmede 25 minutes ago

The best part about adamantly making such a claim is that anybody who knows better also knows better than to break NDA and pull a Warthunder to prove that the CSPs do use colo facilities, so you're not going to get anyone who knows better to disagree with you and say AWS S3 or GCP compute is colo-ed at a specific colo provider.

toddmorey 2 hours ago

Reminds me of the old Rackspace days! Boy we had some war stories:

   - Some EMC guys came to install a storage device for us to test... and tripped over each other and knocked out an entire Rack of servers like a comedy skit. (They uh... didn't win the contract.)
   - Some poor guy driving a truck had a heart attack and the crash took our DFW datecenter offline. (There were ballards to prevent this sort of scenario, but the cement hadn't been poured in them yet.)
   - At one point we temporarily laser-beamed bandwidth across the street to another building
   - There was one day we knocked out windows and purchased box fans because servers were literally catching on fire.

Data center science has... well improved since the earlier days. We worked with Facebook on the OpenCompute Project that had some very forward looking infra concepts at the time.

lostlogin 49 minutes ago

> There was one day we knocked out windows and purchased box fans because servers were literally catching on fire.

Pointing the fans in or out?

jdoss 3 hours ago

This is a pretty decent write up. One thing that comes to mind is why would you write your own internal tooling for managing a rack when Netbox exists? Netbox is fantastic and I wish I had this back in the mid 2000s when I was managing 50+ racks.

https://github.com/netbox-community/netbox

ca508 3 hours ago

we evaluated a lot of commercial and oss offerings before we decided do go build it ourselves - we still have a deploy of netbox somewhere. But our custom tool (Railyard) works so well because it integrates deeply into the our full software, hardware and orchestration stack. The problem with the OSS stuff is that it's almost too generic - you shape the problem to fit its data model vs. solve the problem. We're likely going to fold our tool into Railway itself eventually - want to go on-prem; button click hardware design, commission, deploy and devex. Sorta like what Oxide is doing, but approaching the problem from the opposite side.

matt-p 1 hour ago

It is not that difficult to build it into your app, if you're already storing information about hosts, networking etc. All you're really doing is expanding the scope, netbox is a fine starting point if you're willing to start there and build your systems around it, but if you've already got a system (or you need to do anything that doesn't fit netbox logic) you're probably better off just extending it.

In this case railway will need to care about a lot of extra information beyond just racks, IP addresses and physical servers.

ca508 32 minutes ago

correct; I think the first version of our tool sprung up in the space of a couple of weekends. It wasn't planned, my colleague Pierre who wrote it just had a lot of fun building it.

enahs-sf 8 minutes ago

Curious why California when the kwh is so high here vs Oregon or Washington