Automation tip — adjust a file on a lot of servers

March 31, 2011 By Leave a Comment

I have a customer that has 40 servers that perform a given function. They are comprised of physical machines and Solaris zones. I needed to adjust a file on each of those machines. I was not about to ssh into each machine and then start up vi and adjust the file by hand.

Here is what I did instead

for host in 1 2 3 4 5; do
  for zone in 1 2 3 4 5 6 7 8; do
    ssh -q $host\-$zone 'perl -p -i -e "s/ReplaceMe/WithMe/g" /path/to/file'
  done
done

I’m confident that I’m not the first person do this but I thought it was creative all the same. Combines a PERL one liner with two nested for loops for nice system automation.

See my post on consistency, this is a great example of why it is necessary.

Filed Under: Blogs

ESXi – creating new virtual machines (servers) from the command line

December 24, 2010 By Leave a Comment

I was able to get a server up and running at home again, and given what I want to do, using ESXi is a good solution. When it comes to servers I prefer to do: (a) from the command line and (b) using ssh. First thing I did after getting ESXi installed was to enable their “Tech Support Mode“, and then things got interesting. The command line of ESXi 4.1 is limited, but yet powerful enough to do the job nicely.  After some searching I learned how to create a new server on the command line, power it on/off, register it with ESX and destroy it too.

To create and power on a new server I created the following script

#!/bin/sh

## Most of this taken from http://www.vm-help.com/esx40i/manage_without_VI_client_1.php

mkdir $1

# First make the disk
vmkfstools -c 15G -a lsilogic $1/$1.vmdk

# Now output the vmx file
cat < $1/$1.vmx

config.version = "8"
virtualHW.version = "7"
vmci0.present = "TRUE"
displayName = "$1"
floppy0.present = "FALSE"
numvcpus = "2"
scsi0.present = "TRUE"
scsi0.sharedBus = "none"
scsi0.virtualDev = "lsilogic"
memsize = "256"
scsi0:0.present = "TRUE"
scsi0:0.fileName = "$1.vmdk"
scsi0:0.deviceType = "scsi-hardDisk"
ide1:0.present = "TRUE"
ide1:0.fileName = "/vmfs/volumes/datastore1/ISOs/install48-amd64.iso"
ide1:0.deviceType = "cdrom-image"
ethernet0.present = "TRUE"
ethernet0.virtualDev = "vmxnet"
ethernet0.features = "15"
ethernet0.networkName = "VM Network"
ethernet0.addressType = "generated"
ethernet1.present = "TRUE"
ethernet1.virtualDev = "vmxnet"
ethernet1.features = "15"
ethernet1.networkName = "VM Network 2"
ethernet1.addressType = "generated"
guestOS = "freebsd-64"
EOF

# Now register our new VM
vnum=`vim-cmd solo/registervm /vmfs/volumes/datastore1/$1/$1.vmx`
vim-cmd vmsvc/power.on $vnum

#!/bin/sh
vim-cmd vmsvc/power.off `vim-cmd vmsvc/getallvms |grep $1|awk '{print $1}'`
vim-cmd vmsvc/destroy `vim-cmd vmsvc/getallvms |grep $1|awk '{print $1}'`

Then I went ahead and wrote a one liner to create 10 new machines and then destroy them.  Once I get my completely automated OpenBSD installer finished, then I can adjust the creation of the machines to boot from the network.  Thus all I will have to do is run the script to create the machine and then sit back and wait.  In I’m guessing about 15 minutes I’ll have an up and working OpenBSD system.  Since the install will be automated I can also fully customize the final result.  If I was responsible for a group of say web servers and my company just announced some awesome widget that everyone wants, then I had better be prepared to deploy more servers quickly.  Using the above I could easily accomplish that task.  After all if our customers could not use our website then they will not be happy.

Filed Under: Blogs Tagged With: , ,

To sum up in a single word…. CONSISTENCY

December 24, 2010 By Leave a Comment

There is one word that comes to my mind when I think about how to run a data center, consistency! I have worked with many people and organizations over the years. Recently I have seen a fair number of issues and to summarize them with one word I picked consistency.

In my mind this means right or wrong, if you are going to do something be consistent with it. If you’re using jumpstart or kickstart then put the environment in a revision control system, like CVS or Subversion. This way changes can be tracked and logged. Sometimes it is the simplest things that tip me off that say one system out of ten is different.

For example, when I’m deploying applications on many servers at the same time I use cluster ssh. Once connected I’ll ‘sudo su -’ so I can do what I need to do. If some servers have different root prompts then that is an immediate tip to me that the servers are not all the same.

How do you achieve consistency? Automated scripts/tools. When I deploy the applications I don’t do a lot by hand, except for running some scripts that install the various applications.

Now I’m off to continue the fun I’m having today with ESXi and OpenBSD. I’ve figured out how to create hosted servers from the command line, using ssh. Right now I can easily create an OpenBSD virtual server, power it on, and have the install started all using ssh and the ESXi command line. Next up is to create a fully automated OpenBSD install routine. While the installer is simple and easy, it does require someone answer questions. I want a fully automated and customized environment. I did this a few years ago but am now going to re-visit and improve it.

Filed Under: Blogs Tagged With: , , , ,

Scary tool – dnscat

March 18, 2010 By 9 Comments

The idea of this tool is that you can run just about any program and/or copy files to/from the machine, say an ssh session, using DNS packets to/from the client.  In other words, a workstation sitting on a network somewhere, behind the companies firewalls, IPS/IDS, AV, etc., etc. could communicate with a system on the Internet, using DNS packets.  Thus completely bypassing the security that the company has setup.  Certainly not something that a company would want to happen.  Also hard to detect, since DNS queries are a common thing on a network.

You should know that from a attack/defend perspective, I am currently, much more of a defender than an attacker.  That is to say, when I learn about tools like this, my brain starts to think “okay how do I defend against or stop this?”.

The answer in this case is that the clients (i.e. workstations/other internal devices) are not allowed to get information about any domain on the Internet.  In other words any request from them for say www.google.com or www.cnn.com would get back a response from their DNS server(s) of does not exist, period the end.  The internal DNS servers to the company would perform recursive DNS calls but only for the internal zone.

Then how do they reach the Internet for say www.cnn.com?  The clients must use proxies.  The proxy server(s) are allowed to talk with specific DNS servers inside the DMZ that would allow recursive queries from only the proxy servers and thus enable the proxy to fetch the data from www.cnn.com, for example.

Compared to say an Internet Cafe offering free WiFi service, such a corporate network would seem very restrictive and probably not considered “friendly” to the employees.  On the other hand if I was the owner of the company or the CSO I’d sleep better at night.   Defense in depth is something I learned early on in my days of information security and continue to refine.

Of course these same rules apply within the company as well.  If a particular area needs additional protections, then it should treat the rest of the company just like the Internet and barricade itself off.

What do you think?

Filed Under: security Tagged With: ,

FBI Supply chain compromised :)

March 11, 2010 By Leave a Comment

http://blogs.csoonline.com/the_fbi_supply_chain_illustrated

Funny!

Filed Under: Blogs
« Older Posts