There is one word that comes to my mind when I think about how to run a data center, consistency! I have worked with many people and organizations over the years. Recently I have seen a fair number of issues and to summarize them with one word I picked consistency.
In my mind this means right or wrong, if you are going to do something be consistent with it. If you’re using jumpstart or kickstart then put the environment in a revision control system, like CVS or Subversion. This way changes can be tracked and logged. Sometimes it is the simplest things that tip me off that say one system out of ten is different.
For example, when I’m deploying applications on many servers at the same time I use cluster ssh. Once connected I’ll ‘sudo su -’ so I can do what I need to do. If some servers have different root prompts then that is an immediate tip to me that the servers are not all the same.
How do you achieve consistency? Automated scripts/tools. When I deploy the applications I don’t do a lot by hand, except for running some scripts that install the various applications.
Now I’m off to continue the fun I’m having today with ESXi and OpenBSD. I’ve figured out how to create hosted servers from the command line, using ssh. Right now I can easily create an OpenBSD virtual server, power it on, and have the install started all using ssh and the ESXi command line. Next up is to create a fully automated OpenBSD install routine. While the installer is simple and easy, it does require someone answer questions. I want a fully automated and customized environment. I did this a few years ago but am now going to re-visit and improve it.

