IT Angst

VCP 5 exam thoughts

2012-01-31T11:52:00.000-08:00

Just passed my VCP 5 this morning. Was pretty tough for me as there were quite a few questions pertaining to vDSwitches, and my experience set is almost entirely Nexus 1000v based, so I"m sure I got those wrong.

Almost no configuration maximum questions (the only one I remember is the maximum LUNs presentable to single host) lots of vswitch scenario, policy questions. Quite a few on memory limits, reservations, and lots of CPU performance troubleshooting. A number of questions on the DCUI and what you can change/modify there, and what items are downloadable off the ESXi host homepage.

All around a tough but fair exam I think. Pretty obvious you couldn't just bootcamp it, you need pretty intimate familiarity with the UIs.

2012 New Year, New Job

2012-01-20T07:46:00.000-08:00

So I changed to LongView Systems from CNRL at the beginning of the year, had 6 years at CNRL, thats a long time for me, it was fun, lots of great learning, but in the end it was just too static to keep me interested and engaged.

Joining the cloud practice at Longview should be really interesting. Alread on-site at a client and about to migrate from an f-block (reference architecture for a vblock as a POC) to a couple vblocks.

More OTV, Jumbo Frames and Vmware fun!

2011-04-04T08:15:00.000-07:00

So we're moving datacenters. Primary and backup both moving in the same short timeframe. Fun.

Our new VMware design relies heavily on OTV from Cisco. 4 blades in each datacenter running the same cluster of virtual machines. We had it tested and working with the secondary DC moved to its new location, but over the weekend we fired up the new primary datacenter and moved the Nexus 7ks to it, while keeping the 6509s in our old primary DC. No VC-ESX communications worked after that.

Now, last week we discovered that OTV adds some packet overhead to communications (we knew it did but didn't realize the repercussions). Vmware secure communications to the VC server are pretty close to the maximum size already (1500 by default). When we tried to add a host to the cluster when the VC was connected to a vlan using OTV, the host would send SHA thumbprint info, but the communication would timeout after that. Thats because OTV adds 70 bytes or so. Pings even work normally, but using the size option (-s, -l depending on client) we found that pings of 1430 size worked, and 1431 size didn't.

So after discovering this we played around with resizing the MTU on vmware and the VC, but decided rather that the switches all should have their MTU fixed. The network team fixed the MTU size on the 7ks, but the 6509s will unfortunately cause OSPF errors if the MTU isn't the same on all the switches. means a big outage, so we're scheduling that.

So, why did it work during testing? Because the 7ks could talk directly to each other without a router (6509) prior to the move, and afterwards they couldn't. Doh.

So, why then couldn't the VC server, which was hosted in the backup DC, not even communicate with the ESX host it resided on? Because of OTV domain ownership. The 7ks in the primary datacenter own all the OTV vlans, and because the 7ks couldn't talk to each other anymore directly, the OTV vlans in the backup DC are broken until the 6509 reboots. Big d'oh.

VMware HA and Cisco OTV

2011-02-23T07:42:00.000-08:00

So we're moving to 2 new data-centers in the next little while, from our existing 2. However the new ones will eventually be active/active, once we get Ontap 8.x whichever one supports metronet clusters.

So in designing our new HA architecture, I had to take into account the fact that we're using HP blades and OTV from cisco, which allows us to have a flat network and have VMs that are portable across the 2 data-centers.

Which seems like a great idea. We'll have cisco 7000 10G switches connecting the ESX hosts in each location, and the 6509s will handle the OTV domain ownership since they control the links between locations as well. Links are redundant, ESX clusters have capacity in both locations, and eventually we'll have metronet clusters for the filers, so storage will be available in both places.

HOWEVER, I was talking about the OTV vlans to our network team today. Seems that in the event of a datacenter outage, an OTV domain owned by the switch in the failed datacenter would be unavailable everywhere, even on the 7000s in the opposite DC. So the HA response for ESX would be to shut everything down, because its heartbeats would fail everywhere.

The solution is having another service console network for the whole cluster, and have its OTV domain be owned by the 6509 in its DC. That way in the event of a DC failure, those hosts could still communicate on their secondary SC network, and they wouldn't fail, and as well they would properly power on the VMs from the other DC, which is what we want.

And we have to be sure we add hosts in a specific order to the HA cluster. The first 5 are HA masters, so we need at least 1 in each location, preferably 2 - so adding 3 from one DC and then 2 from the other to the HA cluster is important.

2010-05-03T08:25:00.000-07:00

So I'm writing this down here to help me remind me to think this out. I think the internet will be the downfall of democracy.

I watched a video from the TED conference with a speaker (he was familiar, but I forget who he was, a journalist) who was taking about science denial, and how everyone is entitled to an opinion, but not their own facts. I think the internet is great, but it does a terrible job (good job, but terrible effect) of empowering people who have an opinion counter to science. Vaccines are a great example of this (although I think we seem to have a handle on that one), but now theres a furor over high-fructose corn syrup. Our soda in Canada has sugar, not HFCS in it, and we still get fat from drinking it. I think its a victim attitude.

Maybe I shouldn't say the internet, maybe I should say facebook will be the downfall, although its existence is necessary for facebook to exist. People with some political agenda think product X is evil, with no scientific background whatsoever, make facebook page to convince others its evil, and those people think its right! no science involved whatsoever.

Anyway, when we stop listening to experts, our system is going to fall into chaos, with only the loudest or most popular voices being listened to. Sean Penn will be president! Or something equally as bad.

2010-03-19T11:49:00.000-07:00

While enabling mailboxes in Symantec Enterprise Vault (software that archives most of you mail except the last 30 days, put in place because we're too nice to tell users to FUCKING DELETE THEIR SHITTY 15 year old emails) this error popped up. Funny.

If it looks like I'm not doing anything, it's because I did it right the first time

2010-01-29T11:17:00.000-08:00

I can't stand make-work projects, and I also can't stand being bored. That means I end up writing a lot of scripts to make my life easier, rather than do something meaningless that someone else will look at twice and then forget about.

I also hate people who get panicked. Calm down and think about the situation before you make a snap decision.

NFS is great.... sorta

2009-11-04T07:08:00.000-08:00

NFS is great technology when it comes to VMware - thin provisioning by default, no HBAs to mess with, dynamic storage allocation, etc.... however there is one pitfall that I've just noticed on our test lab.

With HBAs, when you ran out of space in a presented LUN, only VMs that had snapshots attached would crash, because they were writing a redo log, and no space was available for that.

Now, with NFS and thin-provisioned VMs (and even thin-provisioned VMs on VMFS) all the VMs have redo logs essentially, so if you run out of space on a volume, all the VMs in that volume are going to crash (which could be a large number fyi).

Thankfully, dynamic allocation of NFS volumes should prevent this, given sufficient warning on your storage.

VMworld 2009

2009-09-08T13:49:00.000-07:00

Just got back from VMworld 2009 in San Francisco. Conference was great - good sessions, great labs (although I hear they had some trouble on Monday) and good people. Overall a great value, I'll be going again next year.

Max.

VMware storage migration script

2009-05-06T10:17:00.000-07:00

We're moving from SAN attached to NFS storage on our vmware environment, (and vSphere4 isn't out yet) so I wrote a script that will migrate VMs from current location to the new one (I had to hard-code the new location in, since when using sed the path with vmfs/ messed up sed due to the s/).

Warning: if you have VMs with 2 disks with the same name (if they were created in different directories) then this won't work. It will work for up to 3 disks, and should be easy to make support more.

#!/bin/bash
HOMEDIR=/vmfs/volumes/d42ddbc9-93beeb73
for VMNAME in `vmware-cmd -l | awk -F/ '{print $5}'`
do
mkdir $HOMEDIR/$VMNAME/
VMPATH=`vmware-cmd -l | grep $VMNAME | awk -F. '{print $1}'`
for VMDK in `grep scsi0:.*fileName $VMPATH.vmx | awk -F\" '{print $2}'`
do
vmkfstools -i $VMPATH.vmdk $HOMEDIR/$VMNAME/$VMDK -d 'thin' -a lsilogic
done

vmware-cmd -s unregister $VMPATH.vmx

sleep 2

cp $VMPATH.vmx $HOMEDIR/$VMNAME/$VMNAME.vmx.new

sed "s/scsi0:0.fileName.*/scsi0:0.fileName = \/vmfs\/volumes\/d42ddbc9-93beeb73\/${VMNAME}\/${VMNAME}.vmdk/" $HOMEDIR/$VMNAME/$VMNAME.vmx.new > $HOMEDIR/$VMNAME/$VMNAME.vmx.new2
sed "s/scsi0:1.fileName.*/scsi0:1.fileName = \/vmfs\/volumes\/d42ddbc9-93beeb73\/${VMNAME}\/${VMNAME}_1.vmdk/" $HOMEDIR/$VMNAME/$VMNAME.vmx.new2 > $HOMEDIR/$VMNAME/$VMNAME.vmx.new3
sed "s/scsi0:2.fileName.*/scsi0:2.fileName = \/vmfs\/volumes\/d42ddbc9-93beeb73\/${VMNAME}\/${VMNAME}_2.vmdk/" $HOMEDIR/$VMNAME/$VMNAME.vmx.new3 > $HOMEDIR/$VMNAME/$VMNAME.vmx

rm -rf $HOMEDIR/$VMNAME/*.new*
chmod +x $HOMEDIR/$VMNAME/$VMNAME.vmx
vmware-cmd -s register $HOMEDIR/$VMNAME/$VMNAME.vmx
done

HP Onboard Administrator LDAP authentication search context issue

2009-02-09T12:36:00.000-08:00

This was an annoying problem today, so I'm posting it in case it helps anyone else.

I was trying to get the OA for our blades connecting to our AD. I setup the LDAP like the manual said, but no joy. Some research on the HP forums said that if the user you want to connect as is in a different OU than the group they're a member of, both need to be configured as search contexts in the OA config. My config page looks like this:

Directory Server Address: dc.domain.com

Directory Server SSL Port: 636

Search Context 1: OU=AdminGroups,OU=Admin,DC=domain,DC=com

Search Context 2: OU=Admins,OU=Admin,DC=domain,DC=com

And then the group setup is like so:

CN=ILO-Admin,OU=AdminGroups,OU=Admin,DC=domain,DC=com

So the group above is listed in the search context 1, but my admin account is in a different OU, which is search context 2. Bah.

Script to locate incorrect path to disk on Netapp

2009-01-29T07:50:00.000-08:00

One of the storage guys sent me an email indicating some of the disk paths were incorrect on some of our newer servers - specifically they were sending data over partner paths to disk (we have 2 heads in a cluster on that filer, so unless its in failover mode, its less efficient to send data via the partner path to disk).

So I wrote a quick script to check the paths on all hosts against the list of erroring paths he sent me:

rm -rf /root/lunerrors.txt
touch /root/lunerrors.txt
for host in `cat horhosts.txt`
do
echo $host > lunerrors.txt
for path in `cat errorpaths.txt`
do
ssh $host /usr/sbin/esxcfg-mpath -l | grep $path > lunerrors.txt
done
done

Mac address conflicts

2009-01-20T10:04:00.001-08:00

So it turns out some VMs have been assigned static MAC addresses already, so my script duplicated some MAC addresses in use. Not a huge deal (the network team bitched about some errors) until the following happened:

The SCCM guys assigned an image package via MAC address to those MAC addresses, and didn't restrict it to XP images. So the SCCM box re-imaged some windows 2003 boxes. Thank goodness we have vRanger in production, so the VMs were restored pretty quickly (save one, which errored out, and Vizioncore gave us a beta version which restored it).

I modified the script to use a range outside the first 256 addresses. Make sure you aren't conflicting if you use it, and make sure your imaging guys don't leave the package wide-open like ours did :).

2008-12-18T13:03:00.001-08:00

I wrote this script to create VMs, since we're making 350 for the remote access project. It used some code that a co-worked found on the internet, but I modified it appropriately so it would loop and create more than one.

#!/bin/bash
declare -i hexcount=0x0
for vmcount in $(seq 1 5)
do
VMDIR=vdi$vmcount
VMNAME=vdi$vmcount
VMMAC=00:50:56:00:01:$(printf %02x $hexcount)
NFSVOL="cgy-vdi-nfs01"

#Begin pasted script code
#Creates a new Virtual Machine

VMOS="winxppro"
VMDSIZE="20g"
VMMEMSIZE="1024"

mkdir /vmfs/volumes/$NFSVOL/$VMDIR
exec 6>&1
exec 1>/vmfs/volumes/NFSVOL/$VMDIR/$VMNAME.vmx

# write the configuration
echo #!/usr/bin/vmware
echo config.version = '"'8'"'
echo virtualHW.version = '"'4'"'
echo memsize = '"'$VMMEMSIZE'"'
echo floppy0.present = '"'TRUE'"'
echo usb.present = '"'FALSE'"'
echo displayName = '"'$VMNAME'"'
echo guestOS = '"'$VMOS'"'
echo ide0:0.present = '"'TRUE'"'
echo ide0:0.fileName = '"'/vmfs/volumes/vdi-scsi-local/BootableX86_12092008.iso'"'
echo ide0:0.deviceType = '"'cdrom-image'"'
echo ide:0.startConnected = '"'TRUE'"'
echo floppy0.startConnected = '"'FALSE'"'
echo floppy0.fileName = '"'/dev/fd0'"'
echo Ethernet0.present = '"'TRUE'"'
echo Ethernet0.networkName = '"'vmnts222'"'
echo Ethernet0.addressType = '"'static'"'
echo Ethernet0.address = '"'$VMMAC'"'
echo scsi0.present = '"'TRUE'"'
echo scsi0:1.present = '"'TRUE'"'
echo scsi0:1.filename = '"'$VMNAME.vmdk'"'
echo scsi0:1.writeThrough = '"'TRUE'"'
echo scsi0.virtualDev = '"'lsilogic'"'
echo
# close file
exec 1>&-

# make stdout a copy of FD 6 (reset stdout), and close FD6
exec 1>&6
exec 6>&-

chmod 755 /vmfs/volumes/$NFSVOL/$VMDIR/$VMNAME.vmx
# Create Disk & Register the .vmx configuration

#Creates 300mb disk (can be modified for larger disk sizes)
cd /vmfs/volumes/$NFSVOL/$VMDIR
vmkfstools -c $VMDSIZE $VMNAME.vmdk -a lsilogic

#Register VM
vmware-cmd -s register /vmfs/volumes/$NFSVOL/$VMDIR/$VMNAME.vmx

((hexcount=hexcount+1))
done

AMD cpu mask script

2008-10-27T14:06:00.000-07:00

I wrote this script when we got a HP DL 585 G2 server to add to our dev/test cluster, which is mostly G1s. It checks all the vmx files on a server, and modifies them to have the correct CPU masking for G1/G2 server vmotions to be successful.

#!/bin/bash

PATH=$PATH:/bin:/usr/bin

for i in `vmware-cmd -l`
do
cp -f --reply=yes $i $i.back
if [ `grep -c "cpuid.80000001.edx =" $i` = 0 ]
then
echo 'cpuid.80000001.edx = "-----------0--------H-----------"' >> $i
else
cp -f --reply=yes $i $i.sed1
sed -e 's/cpuid.80000001.edx = "--------------------H-----------"/cpuid.80000001.edx = "-----------0--------H-----------"/g' $i.sed1 > $i.sed2
sed -e 's/cpuid.80000001.edx = "-----------0--------------------"/cpuid.80000001.edx = "-----------0--------H-----------"/g' $i.sed2 > $i
fi
if [ `grep -c cpuid.80000001.edx.amd $i` = 0 ]
then
echo 'cpuid.80000001.edx.amd = "----0R-----0--------H------T----"' >> $i
else
cp -f --reply=yes $i $i.sed1
sed -e "s/cpuid.80000001.edx.amd = \"-----R--------------H------T----\"/cpuid.80000001.edx.amd = \"----0R-----0--------H------T----\"/g" $i.sed1 > $i.sed2
sed -e "s/cpuid.80000001.edx.amd = \"-----R-----0--------H------T----\"/cpuid.80000001.edx.amd = \"----0R-----0--------H------T----\"/g" $i.sed2 > $i.sed3
sed -e "s/cpuid.80000001.edx.amd = \"-----------0--------------------\"/cpuid.80000001.edx.amd = \"----0R-----0--------H------T----\"/g" $i.sed3 > $i
fi
done

More useful scripts

2008-09-24T13:27:00.000-07:00

Wrote a couple handy scripts lately. First one checks for ISOs mounted on VMs, because if they are, you can't VMotion them (and then often have to shut the guest OS down - annoying!)

#! /bin/bash
rm -rf /root/isocheck.txt
touch /root/isocheck.txt
for host in `cat isohosts.txt`
do
echo $host
for vmx in `ssh $host vmware-cmd -l`
do
echo $vmx >> isocheck.txt
ssh $host grep -i 'ide0:0.startConnected\ \=\ \"true\"' $vmx >> isocheck.txt
done
done

I have MIME-Lite installed on this ESX box, which is a simple perl-based SMTP agent, it emails me this file afterwards.

The second script sets up all the options to authenticate accounts on an ESX server user a domain account (in this example, esxrangerservice). You need only do "useradd username" after this to allow new users to connect.

#!/bin/bash
/usr/sbin/esxcfg-firewall -o 88,tcp,out,KerberosClient
/usr/sbin/esxcfg-firewall -o 464,tcp,out,KerberosPasswordChange
/usr/sbin/esxcfg-auth --enablead --addomain=yourdomain.com --addc=yourDC.yourdomain.com
/usr/sbin/esxcfg-auth --enablekrb5 --krb5realm=yourdomain.com --krb5kdc=yourDC.yourdomain.com
cp /etc/pam.d/vmware-authd /etc/pam.d/vmware-authd.old
/usr/sbin/useradd esxrangerservice
echo "#%PAM-1.0" > /etc/pam.d/vmware-authd
echo "auth sufficient pam_unix_auth.so shadow nullok" >> /etc/pam.d/vmware-authd
echo "auth required pam_stack.so service=system-auth" >> /etc/pam.d/vmware-authd
echo "account required pam_stack.so service=system-auth" >> /etc/pam.d/vmware-authd

Scripted ESX build with 3.5 not working

2008-09-23T09:23:00.000-07:00

I've been scripting our builds for a while now, but haven't done one since our 3.5 upgrade - I'm just going through a test one now before we roll out some new hosts, and my script process which was working great before doesn't seem to produce a working config anymore.

The script code for routing looks like this - there is some switch config done as well, but since the output is below of esxcfg-vswitch and such, i left it out.

#Service Console
#Add service console IP address
echo Adding service console IP address
/usr/sbin/esxcfg-vswif -a -i 172.31.8.168 -n 255.255.255.0 -p "Service Console" vswif1
sleep 5

#Kernel Network
echo Adding Kernel Network Address
/usr/sbin/esxcfg-vmknic -a -i 172.31.8.169 -n 255.255.255.0 VMotion
sleep 5

#Service Console Routing
#Add esx route to VMkernel
echo Adding default route to Service Console
/usr/sbin/esxcfg-route 172.31.8.1
sleep 3

#Routing
#add networking info to /etc/sysconfig/network
echo adding route
cat > /etc/sysconfig/network << EOF2
NETWORKING=yes
HOSTNAME=esxt01.domain.com
GATEWAY=172.31.8.1
GATEWAYDEV=vswif1
EOF2
sleep 3

Here's the contents of every file and command I could think of thats even remotely related.

/etc/sysconfig/network-scripts/ifcfg-vswif1
DEVICE=vswif1
MACADDR=00:50:56:48:9e:17
PORTGROUP="Service Console"
BOOTPROTO=static
BROADCAST=172.31.8.255
IPADDR=172.31.8.168
NETMASK=255.255.255.0
ONBOOT=yes

/etc/sysconfig/network
NETWORKING=yes
HOSTNAME=esxt01.domain.com
GATEWAY=172.31.8.1
GATEWAYDEV=vswif1

/etc/hosts

127.0.0.1 localhost.localdomain localhost
172.31.0.251 nts200.domain.com nts200
172.31.0.248 nts203.domain.com nts203
172.31.5.148 nts334.domain.com nts334
172.31.8.168 esxt01.domain.com esxt01

esxcfg-vmknic -l
Interface Port Group IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled
vmk0 VMotion 172.31.8.169 255.255.255.0 172.31.8.255 00:50:56:78:d6:51 1500 disabled true

esxcfg-vswif -l
Name Port Group IP Address Netmask Broadcast Enabled DHCP
vswif1 Service Console 172.31.8.168 255.255.255.0 172.31.8.255 true false

esxcfg-vswitch -l
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
Production 64 3 64 1500 vmnic1

PortGroup Name VLAN ID Used Ports Uplinks
vmnts08 8 0 vmnic1

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
Management 64 5 64 1500 vmnic0

PortGroup Name VLAN ID Used Ports Uplinks
VMotion 8 1 vmnic0
Service Console 8 1 vmnic0

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
TestPrivateNetwork64 1 64 1500

PortGroup Name VLAN ID Used Ports Uplinks
TestPrivateNetwork 0 0

esxcfg-route -l
VMkernel Routes:
Network Netmask Gateway
172.31.8.0 255.255.255.0 Local Subnet
default 0.0.0.0 172.31.8.1

the route command produces the following, with a big delay (10-15 seconds) before it displays the third line with the default route on it.

kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
172.31.8.0 * 255.255.255.0 U 0 0 0 vswif1
169.254.0.0 * 255.255.0.0 U 0 0 0 vswif1
default 172.31.8.1 0.0.0.0 UG 0 0 0 vswif1

I just get a "Destination Host Unreachable" when I try to ping anything, which tells me it doesn't know how to route to the network. whiskey tango foxtrot.

esxRanger Backups to DMZ machines

2008-09-05T10:38:00.000-07:00

I have setup ranger to run all our VM backups - its working pretty well, but a recent upgrade caused machines in our DMZ to do this:

Error In Backup Operations! Error: An error occurred during backup operations. The Archive Created Appears to be Invalid. Failed to read Status File.

As it turns out, the upgrade needs a whole bunch more ports to work through to the DMZ now. Since thats a terrible idea, I turned on the "-encryptdata" option, which means everything goes over port 22 now - much better idea for security, although slower.

Full error log is below.

:: vRanger PRO CLI Backup Commencing ::
Version: 3.2.3.3
"C:\Program Files\vizioncore\esxRanger Professional\esxRangerProCli.exe" esxd03.domain.com /vmfs/volumes/vmdmzprod01/dmz17/dmz17.vmx -copylocal "Z:\cgy-dmz-thur" "-drives:all" -mailonerror -vmnotes -noquiesce -diffratio 50 -maxfullage 5 -retendays 14 -description "cgy dmz thursday full" -zipname "[config]" -vmkey vm-658
[20080905_110907]
Sending log on error only.
The TarBall name will be: [dmz17].
VirtualCenter VM Key: vm-658.
Acquiring VM Lock. This may take many minutes!
VM Lock Acquired. Backup Initialization Commencing. Please Wait...
9/5/2008 11:09:28 AM: Backing up the VM 'dmz17'.
9/5/2008 11:09:28 AM: Beginning Testing Phase.
9/5/2008 11:09:28 AM: Testing Destination Path.
9/5/2008 11:09:33 AM: Snapshotting VM.
9/5/2008 11:09:35 AM: Starting Server Component.
9/5/2008 11:09:37 AM: Performing Backup.
9/5/2008 11:19:42 AM: An error occurred during backup operations:
The Archive Created Appears to be Invalid. Failed to read Status File.
9/5/2008 11:19:42 AM: Removing Backup Snapshot.
9/5/2008 11:20:16 AM: Setting Final Backup Data.
9/5/2008 11:20:16 AM: Writing Backup Information.
9/5/2008 11:20:17 AM: Writing VM Note.
Error In Backup Operations! Error: An error occurred during backup operations. The Archive Created Appears to be Invalid. Failed to read Status File.
9/5/2008 11:20:21 AM: Disconnecting Open Connections.
9/5/2008 11:20:21 AM: Waiting for Disconnection.
9/5/2008 11:20:22 AM: Done.
9/5/2008 11:20:22 AM: Disconnection Sequence Complete.
Pausing for 30 seconds...
You can safely close this window now.

I guess people think IT is like a toaster...

2008-09-02T09:18:00.001-07:00

This woman sent this email to all of IS. Managers, programmers, everyone...

That being said, it made me think about why she would send it. Anyone who can use email must get spam, so if she opens one of the attachments, she would probably see that. Its a common method of getting around email filters - and one that normally gets filtered by our message filters.

I guess she just assumes because it isn't really to her, that its an error, not some person on the internet being shady/clever (it is a pretty clever way of doing it frankly). When she pushes the bread down, she expects toast back up, not a bagel.

I am receiving tons of these notifications about e-mails that I have not sent. Can you please look into for me.
-----Original Message-----
From: postmaster@smtp01.2s1n.com [mailto:postmaster@smtp01.2s1n.com]Sent: Monday, September 01, 2008 11:55 AM
To: *********
Subject: Delivery Status Notification (Delay)

This is an automatically generated Delivery Status Notification.
THIS IS A WARNING MESSAGE ONLY.
YOU DO NOT NEED TO RESEND YOUR MESSAGE.
Delivery to the following recipients has been delayed.
webmaster@progressive-equip.com

Back from Vacation!

2008-08-25T10:43:00.000-07:00

Most of August was vacation time! Enjoyed the Shuswap lake in BC where my parents have a house - very relaxing. The kids spent most of the summer there too, and they can swim and waterski now too.

Vmware snapshots 160Gb!

2008-07-31T11:11:00.000-07:00

One of our field ESX servers had a snapshot from Ranger that didn't get deleted, and grew to 160Gb, freezing the VM and making people at the site unable to log in. I set the snapshot to delete, but its been more than 24 hours in deleting.

Turns out the only way to view a snapshot being deleted is if the VM is on, you can run esxtop, then press "e" and enter the ID of the process for the VM. This will expand the process for that VM, and you check for SnapshotVMXCombo process for that group ID. This means the snapshot process is still running.

If the VM is off, there's apparently no way to tell if its still deleting or not. wow.

2008-07-30T20:00:00.000-07:00

This set of scripts saved my ass 100% just now. We've had a few issues with VMware servers being disconnected due to network incidents - not just one but the whole farm goes down (they all think they are isolated, and in appropriate fashion shut down their VMs, expecting one of the other HA cluster members to power them up - however all the hosts are isolated, so there isn't a cluster up for the VMs to be powered up on).

Anyway, following the 2nd time that happened, I wrote some scripts, first to log what VMs are running on a host each night, and the second to power those VMs up (I wrote a third which also gracefully shuts down all the VMs on a host, its handy, I posted that one third).

The reason it saved me is because I had just had to hard reboot an completely unresponsive ESX host, and there are a couple servers that weren't powered on for various reasons, and I forgot to write down which ones beforehand. Yay for pre-emptive scripting!

vmstate script:
#! /bin/bash
echo This script exports the VMs that are running to a text file for later startup/shutdown operations
rm -rf /root/vmlist
rm -rf /root/vmonlist
rm -rf /root/vmofflist
touch /root/vmlist
touch /root/vmonlist
touch /root/vmofflist
ON="on"
for vm in $( vmware-cmd -l );
do
echo $vm >> vmlist
done
for vm2 in $( cat /root/vmlist );
do
state=$( vmware-cmd -q $vm2 getstate );
if [ "$state" = "$ON" ]
then echo $vm2 >> vmonlist
else echo $vm2 >> vmofflist
fi
done

vmstart script:
#! /bin/bash
echo This script starts all VMs listed in /root/vmonlist
echo If you are recovering from an incident, this list was generated at 5:30 PM
echo If you are unsure, please quit and ask someone else
OPTIONS="Proceed Quit"
select opt in $OPTIONS;
do
if [ "$opt" = "Quit" ]; then
exit
elif [ "$opt" = "Proceed" ]; then
for vm in $( cat /root/vmonlist );
do
vmware-cmd -q $vm start
done
exit
else
echo bad option
fi
done

vmstop script
#! /bin/bash
echo This script will shutdown all running Virtual Machines
echo ARE YOU SURE YOU WANT TO DO THIS?
OPTIONS="YES NO"
select opt in $OPTIONS;
do
if [ "$opt" = "NO" ]; then
exit
elif [ "$opt" = "YES" ]; then
for vm in $( cat /root/vmonlist );
do
echo shutting down $vm
vmware-cmd -q $vm stop trysoft
done
echo Waiting 5 minutes, then forcing shutdown
sleep 5m
for vm in $( cat /root/vmonlist );
do
vmware-cmd -q $vm stop hard
done
else
echo bad option
fi
done

I run a cron job at 5:30 every night to spit out the powered-on VMs into the file. Thats mostly because VMs move around mostly during the day, while a couple of the junior admins might be relocating or powering up new ones. Hope this helps someone!

2008-07-30T10:35:00.000-07:00

I bought a Wii Fit yesterday, and am going to track my progress on wiifits.blogspot.com - I'm going to try to convince stevie to do it as well.

I hate email users

2008-07-28T09:38:00.000-07:00

We got this string of emails forwarded to our company. I have removed the people on the to: list because I have no evidence that they forwarded it, unlike the people in the from fields.

Normally this sort of thing wouldn't concern me, we've locked down our top-level distribution lists to remove the possibility of anyone forwarding this to mass quantities of people. However some --enterprising-- (read: we've locked down the lists to keep you from doing this) user put EVERY OTHER MAILING LIST in a single email, and forwarded it to the WHOLE COMPANY. And then OTHER USERS STARTED RE-FORWARDING IT TO THE WHOLE COMPANY. And then OTHER USERS STARTED REPLYING TO THE WHOLE COMPANY TO PLEASE STOP FORWARDING THIS. And then finally, the last guy sent "ditto", which reminds us of course, of the dilbert cartoon about this exact subject.

The help desk "spoke directly to, and reminded each user of our email acceptable user policy". Glad I wasn't doing it, because there would have been some crying going on, and I probably would have got fired.

From: Goddard, Shando [mailto:SGoddard@petro-canada.ca]
Sent: Monday, July 07, 2008 4:50 PM
To: removed
Subject: FW: I DON'T KNOW HOW IT WORKS, BUT IT DOES.

--------------------------------------------------------------------------------
From: Appleton, Lee
Sent: Monday, July 07, 2008 7:10 AM
To: removed
Subject: FW: I DON'T KNOW HOW IT WORKS, BUT IT DOES.

Best Regards,

D.L. (Lee) Appleton

--------------------------------------------------------------------------------
From: McKinnon, Jim
Sent: Sunday, July 06, 2008 9:02 PM
To: removed
Subject: FW: I DON'T KNOW HOW IT WORKS, BUT IT DOES.

--------------------------------------------------------------------------------
From: Robert Lawson [mailto:palawson@telusplanet.net]
Sent: Sunday, July 06, 2008 2:34 PM
To: removed
Subject: FW: I DON'T KNOW HOW IT WORKS, BUT IT DOES.

Best of luck to everyone.

-----Original Message-----
From: Lawson, Robert [mailto:BoLawson@petro-canada.ca]
Sent: Sunday, July 06, 2008 2:43 PM
To: Lawson, Robert
Subject: FW: I DON'T KNOW HOW IT WORKS, BUT IT DOES.

--------------------------------------------------------------------------------

From: Campbell, Dennis
Sent: Thursday, July 03, 2008 8:41 AM
To: removed
Subject: FW: I DON'T KNOW HOW IT WORKS, BUT IT DOES.

This is a just in case it does work.

--------------------------------------------------------------------------------

From: Marty Price [mailto:Marty.Price@Halliburton.com]
Sent: Thursday, July 03, 2008 6:39 AM
To: removed
Subject: FW: I DON'T KNOW HOW IT WORKS, BUT IT DOES.

Blackberry post!

2008-07-28T09:01:00.001-07:00

My first mobile post. My new BB is easier to type on, full keyboard
and all. My old one suffered a "shuswap water" treatment and stopped
functioning, surprise surprise.

Anyway, fragapalooza is coming up, paul brad brent igor and laz are
all attending, should be fun. Brad sent me a link to his tech blog
www.bradstechblog.com and I remembered that SCCM makes baby jesus cry.

--
Sent from Gmail for mobile | mobile.google.com

Ack I should do more regular updates

2008-07-28T08:41:00.000-07:00

I figured out how to post to the blog remotely using my blackberry, so hopefully more posts will come, and more frequently.

We upgraded our netapp filers in 7.2.4 last week, and one of our LUNs had duplicate name mapping - how the hell does that happen? Anyway, all the VMs on that LUN had to be moved after the outage, thankfully nothing serious happened, but it just proves why we have jobs - IT stuff will always be ahead of the knowledge curve of normal users.

Sure they're catching up to IT workers in knowledge, but we'll always be ahead because IT will continue to become more complicated. Users can do a lot of the things for themselves that administrators used to have to do 10 years ago, but now we have more complicated tasks (like managing VMware infrastructure).

Stevie went to Kate's stag party on Saturday night, had to pick her drunk ass up at a Bar :) Hopefully Ken's will be just as much fun. Wait didn't they do these things the first time they got married? I'm glad I didn't know them the first time, because if I'd bought them a gift then, I would have stolen it back and re-gifted it to them.

Got into the Warhammer Online beta - can't discuss further, NDA etc...

I wonder how many phone calls he got

2007-10-29T06:58:00.000-07:00

This email arrived to "Information Systems - All" today (contains every IS worked, and IS management :)"

From: Don xxxxxxx
Sent: Saturday, October 27, 2007 10:10 AM
To: Information Systems - All
Subject: Please call me - Thanks

Bioshock

2007-08-22T14:51:00.000-07:00

So I've been reading the forum angst about Bioshock. 2 main issues (thoughts follow): Widescreen support is fake & Game can only be "installed" on 2 PCs at once.

The forums are full of rants about the widescreen being a jury-rigged afterthought. I say why do all the people using widescreen think its their god-given right to a wider aspect ratio? In counter-strike source, the wider screen lets you see people who can't see you - so why should they do the same in Bioshock? Why let the people who have purchased a widescreen monitor get possibly gameplay-altering advantages? These people need to STFU.

The game also apparently installs "securom" with itself (even the STEAM version! say it isn't so!). This will track your installations with a master server, and only let you have 2 going on at once. If you uninstall, you get credit back apparently.

I don't know what planet 2k games plays on, but I'm still installing games once in a while that are almost 10 years old. I installed quake3 a while ago for a LAN party, and I had my cdkey so no problem, but if the game had required me to talk to a probably-defunct auth server, I would have been screwed.

Someone on the forums suggested a boycott (which I'm going to participate in). Another then suggested " "Boycotting such a fine game due to activation woes thought is not the right thing to do." I think boycotting is absolutely the RIGHT thing to do. The problem with an internet age with no morales is that the same people also have no principles. These whiney teenagers have no character to stand up for what they believe in. Resisting corporate control is an excellent thing to rebel against, not just so you can pirate stuff, but so your entertainment isn't under the constant scrutiny of corporate shills.

So, my wife will be happy I'm not buying it - she hates when I get new obsessions.

First Post!

2007-06-12T13:39:00.001-07:00

Unlike the blathering of forum trolls, this actually means something this time. Having done reasonably well in english at higher levels of education, I haven't really written anything since then, and It think its time to adjust that.

My friends and co-workers joking call me the angry guy, but I don't think I"m any more angry than other IT workers, I'm just less polite about saying something about it. So hopefully this will serve as an outlet for me to talk about the things they do, so that I can leave that work angst at work, and enjoy my family time more.

Anyway, I work IT at an energy company. The difference is the guys I work with are actually pretty smart, instead of the usual IT trolls. So work is tolerable, but lots of the others... well the stories will follow.