The script shown below is specific to this particular server, it will need to be adjusted to suit the hardware. I put the script on a web server, then I boot the to be installed system from CD, select shell or Live CD.  Then I download the script and execute.  Tip: you can do that in one step

ftp -o - http://server/script.sh | sh

That will download the script and immediately execute it.

In this example the server has 6 disks.  I’ve configured the hardware raid controller such that it is basically a JBOD.  I want ZFS handling this, not hardware. While I would prefer ZFS have the entire disk, I don’t think that is possible today.  I opted to make each disk identical from a layout perspective. My thinking being should a disk fail, any other disk should be bootable, etc..  I also opted to only create a raidz pool with 5 disks, the 6th disk is a spare.  These are not the newest servers or drives, I know they will fail at some point and thus I’m willing to trade space for data protection.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104

#!/bin/sh
 
MP=/mnt
 
zpool export zroot
for DSK in 0 1 2 3 4 5; do
gpart destroy -F da$DSK
gpart create -s gpt da$DSK
gpart add -s 122 -t freebsd-boot da$DSK
gpart add -s 512M -t freebsd-swap -l swap$DSK da$DSK
gpart add -t freebsd-zfs -l disk$DSK da$DSK
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da$DSK
done
 
# Now create the pool
zpool create -f -o altroot=$MP -O canmount=off zroot raidz /dev/gpt/disk0 /dev/gpt/disk1 /dev/gpt/disk2 /dev/gpt/disk3 /dev/gpt/disk4 spare /dev/gpt/disk5
 
 
 
 
# I personally dislike the root of pools being mounted and available.
zfs set mountpoint=none                                         zroot
 
 
 
# Due to the above, we have to specify the mount points for each FS we create,
# this same rule applies post install when we create a FS, must set the mount point
 
zfs create -o mountpoint=/                                                                      zroot/ROOT
zfs create -o mountpoint=/usr                                                                   zroot/usr
zfs create                              -o mountpoint=/var                                      zroot/var
zfs create -o setuid=off                -o mountpoint=/home                                     zroot/home
zfs create              -o setuid=off                       -o mountpoint=/usr/local            zroot/usr/local
zfs create -o exec=off  -o setuid=off                       -o mountpoint=/usr/locall/etc/      zroot/usr/local/etc
zfs create -o exec=off  -o setuid=off                       -o mountpoint=/usr/src              zroot/usr/src
#zfs create -o setuid=off                                    -o mountpoint=/usr/ports            zroot/usr/ports
#zfs create -o exec=off  -o setuid=off                       -o mountpoint=/usr/ports/distfiles  zroot/usr/ports/distfiles
#zfs create -o exec=off  -o setuid=off                       -o mountpoint=/usr/ports/packages   zroot/usr/ports/packages
zfs create -o exec=off  -o setuid=off   -o compression=on   -o mountpoint=/var/crash            zroot/var/crash
zfs create -o exec=off  -o setuid=off                       -o mountpoint=/var/db               zroot/var/db
zfs create -o exec=on   -o setuid=off  -o compression=on    -o mountpoint=/var/db/pkg           zroot/var/db/pkg
zfs create -o exec=off  -o setuid=off                       -o mountpoint=/var/empty            zroot/var/empty
zfs create -o exec=off  -o setuid=off  -o compression=on    -o mountpoint=/var/log              zroot/var/log
zfs create -o exec=off  -o setuid=off  -o compression=on    -o mountpoint=/var/mail             zroot/var/mail
zfs create -o exec=off  -o setuid=off                       -o mountpoint=/var/run              zroot/var/run
zfs create -o exec=on   -o setuid=off                       -o mountpoint=/var/tmp              zroot/var/tmp
 
 
 
cd /usr/freebsd-dist
export DESTDIR=$MP/
echo "base..."; cat base.txz | tar --unlink -xpJf - -C ${DESTDIR:-/}
echo "src..."; cat src.txz | tar --unlink -xpJf - -C ${DESTDIR:-/}
echo "kernel..."; cat kernel.txz | tar --unlink -xpJf - -C ${DESTDIR:-/}
echo "doc..."; cat doc.txz | tar --unlink -xpJf - -C ${DESTDIR:-/}
 
#cat ports.txz | tar --unlink -xpJf - -C ${DESTDIR:-/}
 
chmod 1777 $MP/tmp
chmod 1777 $MP/var/tmp
 
zfs set readonly=on zroot/var/empty
zpool set bootfs=zroot/ROOT zroot
 
cat > $MP/boot/loader.conf << __EOF__
zfs_load="YES"
__EOF__
 
cat > $MP/etc/fstab << __EOF__
 # Device                       Mountpoint              FStype  Options         Dump    Pass#
 /dev/gpt/swap0                 none                    swap    sw              0       0
 /dev/gpt/swap1                 none                    swap    sw              0       0
 /dev/gpt/swap2                 none                    swap    sw              0       0
 /dev/gpt/swap3                 none                    swap    sw              0       0
 /dev/gpt/swap4                 none                    swap    sw              0       0
 /dev/gpt/swap5                 none                    swap    sw              0       0
 tmpfs                          /tmp                    tmpfs   rw,mode=777     0       0
__EOF__
 
 
cat > $MP/etc/rc.conf << __EOF__
# General System Config
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"
clear_tmp_enable="YES"  # Clear /tmp at startup
ifconfig_em0="up"
cloned_interfaces="vlan89"
ifconfig_vlan4="inet 192.168.89.135 netmask 255.255.255.0 vlan 89 vlandev em0"
ifconfig_vlan4_alias0="inet 192.168.89.245 netmask 255.255.255.255"
defaultrouter="192.168.89.1"
hostname="hostname.balius.com"
sshd_enable="YES"
__EOF__
 
 
zfs unmount -af
 
## The next two are "hacks" in my book, without the last line, on reboot
## it gets stuck trying to find zfs:zroot/ROOT, but somehow the -f "fixes" this quirk
#
zpool export zroot
zpool import -f zroot
echo "Now type reboot, remove CD-ROM, etc."

I decided that trading a CDROM only based OS, i.e. SmartOS a for system that I’m more comfortable using and is more general purpose is better for my media server.  I give up a bit of disk space, but gain something I am a lot more familiar with and capable of supporting and keeping up-to-date.

I had 129GB of data sitting on /movies/ that I didn’t want to lose.  I had a FreeBSD 9.1 i386 system test system that had about 127GB of space left in the pool.  I started out using rsync+ssh and transfer the files, slow.  Both servers are connected to a 1Gbit network but still the encryption overhead slowed things down I think.  Then I used tar and netcat, better but still not easy enough.  I pruned my media a bit, down to about 106GB.

Then I figured I’d try using ZFS send / recv over netcat.  I’ve done it once before as a bare metal restore and it worked great then.  The unknown was could I do this using SmartOS (from around 10/2012) as the source and FreeBSD 9.1-RELEASE i386 as the destination.  Turns out the answer is yes!

On the destination system I first created a ZFS file system

zfs create -o mountpoint=/movies zroot/movies

Next I got netcat and ZFS ready to receive

nc -w 300 -l 5600 | zfs recv -Fv zroot/movies

On the source system I created a snapshot first

zfs snapshot zones/movies@before-move

Next was sending the snapshot over

zfs send  -v zones/movies@before-move | nc -w 30 destination-server-IP 5600

It started moving the bits and no complaints from either side.  I started this later at night so I went to bed. When I woke up in the morning I found the process had completed and everything looked good.  I then went about installing FreeBSD 9.1-RELEASE amd64 replacing SmartOS.  That took less than 30 minutes. Then I used the same procedure again to copy the files from the temporary box back to the same hardware with a new OS.

I do enjoy ZFS, yes it is memory hungry, but being able to basically dump a live file system and then send that to another system over the network or other means is awesome! In this case I did it between two different operating systems and architectures. I admit I have not yet tried to stream the content from the rebuilt server.  It could be I’ve completely messed up my data and will have to recreate it from DVDs, in which case I’ll be pissed.  On the other hand had this been critical data I would have tested that it was usable before I rebuilt the existing system. I’m happy it took about an hour to move 106GB of data between the 2 servers, both ways.

The high level steps are as follows:

1. Download and jailbreak using the files at this link http://www.mobileread.com/forums/showthread.php?p=2169819
2. Download and install the USB network from the same place
3. From http://wiki.mobileread.com/wiki/Kindle_Touch_Hacking#Screen_Savers you need to ssh into your KPW and  then do the following

mntroot rw
mv /usr/share/blanket/screensaver /mnt/us/screensaver.original
ln -sfn /mnt/us/screensaver /usr/share/blanket/screensaver

Despite what the directions say in the screensaver hack, the images for KPW are 1024 x 758 and the naming conventions are a little different. The names need to follow the pattern of

bg_medium_ss00.png

Then you can simply attach your KPW to your computer and copy the images you want to use to the screensaver folder.  Hopefully you found this useful.

I took the images that I had been using on my Kindle 3 Keyboard and I noticed the time for the screensaver to be displayed was a lot longer.  I played with it this morning and it looks like if the size is incorrect, then it takes longer for it to display, nearly 7 seconds.  Pictures sized appropriately load in nearly 3 seconds.  Now if only I could get Aperture to export every picture to this exact size without my having to crop it first.

Here is what I did instead

for host in 1 2 3 4 5; do
  for zone in 1 2 3 4 5 6 7 8; do
    ssh -q $host\-$zone 'perl -p -i -e "s/ReplaceMe/WithMe/g" /path/to/file'
  done
done

I’m confident that I’m not the first person do this but I thought it was creative all the same. Combines a PERL one liner with two nested for loops for nice system automation.

See my post on consistency, this is a great example of why it is necessary.

To create and power on a new server I created the following script

#!/bin/sh

## Most of this taken from http://www.vm-help.com/esx40i/manage_without_VI_client_1.php

mkdir $1

# First make the disk
vmkfstools -c 15G -a lsilogic $1/$1.vmdk

# Now output the vmx file
cat < $1/$1.vmx

config.version = "8"
virtualHW.version = "7"
vmci0.present = "TRUE"
displayName = "$1"
floppy0.present = "FALSE"
numvcpus = "2"
scsi0.present = "TRUE"
scsi0.sharedBus = "none"
scsi0.virtualDev = "lsilogic"
memsize = "256"
scsi0:0.present = "TRUE"
scsi0:0.fileName = "$1.vmdk"
scsi0:0.deviceType = "scsi-hardDisk"
ide1:0.present = "TRUE"
ide1:0.fileName = "/vmfs/volumes/datastore1/ISOs/install48-amd64.iso"
ide1:0.deviceType = "cdrom-image"
ethernet0.present = "TRUE"
ethernet0.virtualDev = "vmxnet"
ethernet0.features = "15"
ethernet0.networkName = "VM Network"
ethernet0.addressType = "generated"
ethernet1.present = "TRUE"
ethernet1.virtualDev = "vmxnet"
ethernet1.features = "15"
ethernet1.networkName = "VM Network 2"
ethernet1.addressType = "generated"
guestOS = "freebsd-64"
EOF

# Now register our new VM
vnum=`vim-cmd solo/registervm /vmfs/volumes/datastore1/$1/$1.vmx`
vim-cmd vmsvc/power.on $vnum

#!/bin/sh
vim-cmd vmsvc/power.off `vim-cmd vmsvc/getallvms |grep $1|awk '{print $1}'`
vim-cmd vmsvc/destroy `vim-cmd vmsvc/getallvms |grep $1|awk '{print $1}'`

Then I went ahead and wrote a one liner to create 10 new machines and then destroy them.  Once I get my completely automated OpenBSD installer finished, then I can adjust the creation of the machines to boot from the network.  Thus all I will have to do is run the script to create the machine and then sit back and wait.  In I’m guessing about 15 minutes I’ll have an up and working OpenBSD system.  Since the install will be automated I can also fully customize the final result.  If I was responsible for a group of say web servers and my company just announced some awesome widget that everyone wants, then I had better be prepared to deploy more servers quickly.  Using the above I could easily accomplish that task.  After all if our customers could not use our website then they will not be happy.

In my mind this means right or wrong, if you are going to do something be consistent with it. If you’re using jumpstart or kickstart then put the environment in a revision control system, like CVS or Subversion. This way changes can be tracked and logged. Sometimes it is the simplest things that tip me off that say one system out of ten is different.

For example, when I’m deploying applications on many servers at the same time I use cluster ssh. Once connected I’ll ‘sudo su -’ so I can do what I need to do. If some servers have different root prompts then that is an immediate tip to me that the servers are not all the same.

How do you achieve consistency? Automated scripts/tools. When I deploy the applications I don’t do a lot by hand, except for running some scripts that install the various applications.

Now I’m off to continue the fun I’m having today with ESXi and OpenBSD. I’ve figured out how to create hosted servers from the command line, using ssh. Right now I can easily create an OpenBSD virtual server, power it on, and have the install started all using ssh and the ESXi command line. Next up is to create a fully automated OpenBSD install routine. While the installer is simple and easy, it does require someone answer questions. I want a fully automated and customized environment. I did this a few years ago but am now going to re-visit and improve it.

You should know that from a attack/defend perspective, I am currently, much more of a defender than an attacker.  That is to say, when I learn about tools like this, my brain starts to think “okay how do I defend against or stop this?”.

The answer in this case is that the clients (i.e. workstations/other internal devices) are not allowed to get information about any domain on the Internet.  In other words any request from them for say www.google.com or www.cnn.com would get back a response from their DNS server(s) of does not exist, period the end.  The internal DNS servers to the company would perform recursive DNS calls but only for the internal zone.

Then how do they reach the Internet for say www.cnn.com?  The clients must use proxies.  The proxy server(s) are allowed to talk with specific DNS servers inside the DMZ that would allow recursive queries from only the proxy servers and thus enable the proxy to fetch the data from www.cnn.com, for example.

Compared to say an Internet Cafe offering free WiFi service, such a corporate network would seem very restrictive and probably not considered “friendly” to the employees.  On the other hand if I was the owner of the company or the CSO I’d sleep better at night.   Defense in depth is something I learned early on in my days of information security and continue to refine.

Of course these same rules apply within the company as well.  If a particular area needs additional protections, then it should treat the rest of the company just like the Internet and barricade itself off.

What do you think?

Funny!

Many companies have problems with viruses, and for years I referred to Outlook as “the best virus distribution tool” around.  My point being that if the workstations are not allowed to connect to outside servers/services then that stops a lot of stuff in it tracks.  For example if a workstation became infected and the malicious software was trying to send out spam and the firewall let it out, then out goes the spam.  On the other hand if the firewall rules prevented the workstation from directly talking with SMTP servers outside of the company, then malicious software would have to find a different way.  Maybe instead the malicious software would talk with the companies mail server and let it then send the SPAM out.  As long as the mail server is setup correctly, i.e. anti-spam/virus for outbound messages as well, then this problem would be stopped or at least slowed. Another thing to do, force the use of secure authentication for all users, regardless of location.

Another trick, rate limiting. TCP/IP connection rate limiting and message rate limiting should be put in place on the messaging server.  If an internal machine violates these limits then the machine could be put into a deny all list and an administrator alerted.

I find is surprising that a company, in this day and age, would not have at least have egress filters on their firewall.  I can understand companies not having NSM in place, but I hope to change that as I build out our mananged services offering to include NSM.

Sources familiar with the new initiative said the focus is not figuring out who was behind the recent cyberattacks — doing so is a nearly impossible task after the fact — but building a better defense of Google’s networks, or what its technicians call “information assurance.”

I have two words for Google, “air gap”, at the most basic level.  If the packets can not enter or leave the computer/network than at least the system is secure from over the wire attacks.  Ignoring physical attacks. After all the DoD operates SIPRNet and except for a few cases of people bringing a virus to SIPRNet it is secure.  Of course that is just one small part of a complete “information assurance” program, but a good foundation is required.

Unlike SIPRNet which could have multiple organizations connected which thus opens up potential security challenges, Google could have a less difficult time.  Google has one entity, itself.  If the reports about employees assisting the attackers is true, in a way the air gap is even more important.  On the other hand if an employee(s) wanted to collaborate and get information from a secure network to the open Internet it would not impossible.  If this network is secured properly, then it would be extremely difficult to accomplish, if not impossible. With no CDs, USB device, physical inspections upon entering/leaving the rooms and armed guards the ability to get information off the network either electronically or in hard copy would be difficult.