Resurrection!

Back from the Dead!

I haven't posted here since I left Canadian Natural. A lot has happened since there, the quick recap:

I left CNRL to be the team lead at Canadian Pacific for their vblock implementation. It was so busy that I kind of dropped this blog off the edge of the world, and haven't picked it back up. Anyway, the vblock there was interesting. It was original intended for vCloud director and VDI for staff and IT, but as is often the case, SAP showed up and consumed all available resources. 

After CP I designed a pseudo-flexpod (all the parts, none of the overhead) for Autopro Automation for a big oil and gas client, and then implemented it. A lot of documentation went into that. A LOT. That and designing the whole project in advance on emulators and from memory/old documenation due to engineers wanting all their documentation up front was interesting. 

Then I took a solutions architect role for Hatsize Education. It wasn't a good fit, writing bash scripts 8 hours a day. 

Now I'm back at CP, Converged Infrastructure Specialist for their Positive Train Control initiative. I also recently completed my CCNA-DC certification, and renewed my VCP at 5.5. Very interesting so far, hopefully I'll have more postings in the future.

Personal notes - I bought a new house! 

Max. 

VCP 5 exam thoughts

Just passed my VCP 5 this morning. Was pretty tough for me as there were quite a few questions pertaining to vDSwitches, and my experience set is almost entirely Nexus 1000v based, so I"m sure I got those wrong.

Almost no configuration maximum questions (the only one I remember is the maximum LUNs presentable to single host) lots of vswitch scenario, policy questions. Quite a few on memory limits, reservations, and lots of CPU performance troubleshooting. A number of questions on the DCUI and what you can change/modify there, and what items are downloadable off the ESXi host homepage.

All around a tough but fair exam I think. Pretty obvious you couldn't just bootcamp it, you need pretty intimate familiarity with the UIs.

2012 New Year, New Job

So I changed to LongView Systems from CNRL at the beginning of the year, had 6 years at CNRL, thats a long time for me, it was fun, lots of great learning, but in the end it was just too static to keep me interested and engaged.

Joining the cloud practice at Longview should be really interesting. Alread on-site at a client and about to migrate from an f-block (reference architecture for a vblock as a POC) to a couple vblocks.

More OTV, Jumbo Frames and Vmware fun!

So we're moving datacenters. Primary and backup both moving in the same short timeframe. Fun.

Our new VMware design relies heavily on OTV from Cisco. 4 blades in each datacenter running the same cluster of virtual machines. We had it tested and working with the secondary DC moved to its new location, but over the weekend we fired up the new primary datacenter and moved the Nexus 7ks to it, while keeping the 6509s in our old primary DC. No VC-ESX communications worked after that.

Now, last week we discovered that OTV adds some packet overhead to communications (we knew it did but didn't realize the repercussions). Vmware secure communications to the VC server are pretty close to the maximum size already (1500 by default). When we tried to add a host to the cluster when the VC was connected to a vlan using OTV, the host would send SHA thumbprint info, but the communication would timeout after that. Thats because OTV adds 70 bytes or so. Pings even work normally, but using the size option (-s, -l depending on client) we found that pings of 1430 size worked, and 1431 size didn't.

So after discovering this we played around with resizing the MTU on vmware and the VC, but decided rather that the switches all should have their MTU fixed. The network team fixed the MTU size on the 7ks, but the 6509s will unfortunately cause OSPF errors if the MTU isn't the same on all the switches. means a big outage, so we're scheduling that.

So, why did it work during testing? Because the 7ks could talk directly to each other without a router (6509) prior to the move, and afterwards they couldn't. Doh.

So, why then couldn't the VC server, which was hosted in the backup DC, not even communicate with the ESX host it resided on? Because of OTV domain ownership. The 7ks in the primary datacenter own all the OTV vlans, and because the 7ks couldn't talk to each other anymore directly, the OTV vlans in the backup DC are broken until the 6509 reboots. Big d'oh.

VMware HA and Cisco OTV

So we're moving to 2 new data-centers in the next little while, from our existing 2. However the new ones will eventually be active/active, once we get Ontap 8.x whichever one supports metronet clusters.

So in designing our new HA architecture, I had to take into account the fact that we're using HP blades and OTV from cisco, which allows us to have a flat network and have VMs that are portable across the 2 data-centers.

Which seems like a great idea. We'll have cisco 7000 10G switches connecting the ESX hosts in each location, and the 6509s will handle the OTV domain ownership since they control the links between locations as well. Links are redundant, ESX clusters have capacity in both locations, and eventually we'll have metronet clusters for the filers, so storage will be available in both places.

HOWEVER, I was talking about the OTV vlans to our network team today. Seems that in the event of a datacenter outage, an OTV domain owned by the switch in the failed datacenter would be unavailable everywhere, even on the 7000s in the opposite DC. So the HA response for ESX would be to shut everything down, because its heartbeats would fail everywhere.

The solution is having another service console network for the whole cluster, and have its OTV domain be owned by the 6509 in its DC. That way in the event of a DC failure, those hosts could still communicate on their secondary SC network, and they wouldn't fail, and as well they would properly power on the VMs from the other DC, which is what we want.

And we have to be sure we add hosts in a specific order to the HA cluster. The first 5 are HA masters, so we need at least 1 in each location, preferably 2 - so adding 3 from one DC and then 2 from the other to the HA cluster is important.

So I'm writing this down here to help me remind me to think this out. I think the internet will be the downfall of democracy.

I watched a video from the TED conference with a speaker (he was familiar, but I forget who he was, a journalist) who was taking about science denial, and how everyone is entitled to an opinion, but not their own facts. I think the internet is great, but it does a terrible job (good job, but terrible effect) of empowering people who have an opinion counter to science. Vaccines are a great example of this (although I think we seem to have a handle on that one), but now theres a furor over high-fructose corn syrup. Our soda in Canada has sugar, not HFCS in it, and we still get fat from drinking it. I think its a victim attitude.

Maybe I shouldn't say the internet, maybe I should say facebook will be the downfall, although its existence is necessary for facebook to exist. People with some political agenda think product X is evil, with no scientific background whatsoever, make facebook page to convince others its evil, and those people think its right! no science involved whatsoever.

Anyway, when we stop listening to experts, our system is going to fall into chaos, with only the loudest or most popular voices being listened to. Sean Penn will be president! Or something equally as bad.

While enabling mailboxes in Symantec Enterprise Vault (software that archives most of you mail except the last 30 days, put in place because we're too nice to tell users to FUCKING DELETE THEIR SHITTY 15 year old emails) this error popped up. Funny.

If it looks like I'm not doing anything, it's because I did it right the first time

I can't stand make-work projects, and I also can't stand being bored. That means I end up writing a lot of scripts to make my life easier, rather than do something meaningless that someone else will look at twice and then forget about.

I also hate people who get panicked. Calm down and think about the situation before you make a snap decision.

NFS is great.... sorta

NFS is great technology when it comes to VMware - thin provisioning by default, no HBAs to mess with, dynamic storage allocation, etc.... however there is one pitfall that I've just noticed on our test lab.

With HBAs, when you ran out of space in a presented LUN, only VMs that had snapshots attached would crash, because they were writing a redo log, and no space was available for that.

Now, with NFS and thin-provisioned VMs (and even thin-provisioned VMs on VMFS) all the VMs have redo logs essentially, so if you run out of space on a volume, all the VMs in that volume are going to crash (which could be a large number fyi).

Thankfully, dynamic allocation of NFS volumes should prevent this, given sufficient warning on your storage.

VMworld 2009

Just got back from VMworld 2009 in San Francisco. Conference was great - good sessions, great labs (although I hear they had some trouble on Monday) and good people. Overall a great value, I'll be going again next year.

Max.

VMware storage migration script

We're moving from SAN attached to NFS storage on our vmware environment, (and vSphere4 isn't out yet) so I wrote a script that will migrate VMs from current location to the new one (I had to hard-code the new location in, since when using sed the path with vmfs/ messed up sed due to the s/).

Warning: if you have VMs with 2 disks with the same name (if they were created in different directories) then this won't work. It will work for up to 3 disks, and should be easy to make support more.

#!/bin/bash
HOMEDIR=/vmfs/volumes/d42ddbc9-93beeb73
for VMNAME in `vmware-cmd -l | awk -F/ '{print $5}'`
do
mkdir $HOMEDIR/$VMNAME/
VMPATH=`vmware-cmd -l | grep $VMNAME | awk -F. '{print $1}'`
for VMDK in `grep scsi0:.*fileName $VMPATH.vmx | awk -F\" '{print $2}'`
do
vmkfstools -i $VMPATH.vmdk $HOMEDIR/$VMNAME/$VMDK -d 'thin' -a lsilogic
done

vmware-cmd -s unregister $VMPATH.vmx

sleep 2

cp $VMPATH.vmx $HOMEDIR/$VMNAME/$VMNAME.vmx.new

sed "s/scsi0:0.fileName.*/scsi0:0.fileName = \/vmfs\/volumes\/d42ddbc9-93beeb73\/${VMNAME}\/${VMNAME}.vmdk/" $HOMEDIR/$VMNAME/$VMNAME.vmx.new > $HOMEDIR/$VMNAME/$VMNAME.vmx.new2
sed "s/scsi0:1.fileName.*/scsi0:1.fileName = \/vmfs\/volumes\/d42ddbc9-93beeb73\/${VMNAME}\/${VMNAME}_1.vmdk/" $HOMEDIR/$VMNAME/$VMNAME.vmx.new2 > $HOMEDIR/$VMNAME/$VMNAME.vmx.new3
sed "s/scsi0:2.fileName.*/scsi0:2.fileName = \/vmfs\/volumes\/d42ddbc9-93beeb73\/${VMNAME}\/${VMNAME}_2.vmdk/" $HOMEDIR/$VMNAME/$VMNAME.vmx.new3 > $HOMEDIR/$VMNAME/$VMNAME.vmx

rm -rf $HOMEDIR/$VMNAME/*.new*
chmod +x $HOMEDIR/$VMNAME/$VMNAME.vmx
vmware-cmd -s register $HOMEDIR/$VMNAME/$VMNAME.vmx
done

HP Onboard Administrator LDAP authentication search context issue

This was an annoying problem today, so I'm posting it in case it helps anyone else.

I was trying to get the OA for our blades connecting to our AD. I setup the LDAP like the manual said, but no joy. Some research on the HP forums said that if the user you want to connect as is in a different OU than the group they're a member of, both need to be configured as search contexts in the OA config. My config page looks like this:

Directory Server Address: dc.domain.com

Directory Server SSL Port: 636

Search Context 1: OU=AdminGroups,OU=Admin,DC=domain,DC=com

Search Context 2: OU=Admins,OU=Admin,DC=domain,DC=com

And then the group setup is like so:

CN=ILO-Admin,OU=AdminGroups,OU=Admin,DC=domain,DC=com

So the group above is listed in the search context 1, but my admin account is in a different OU, which is search context 2. Bah.

Script to locate incorrect path to disk on Netapp

One of the storage guys sent me an email indicating some of the disk paths were incorrect on some of our newer servers - specifically they were sending data over partner paths to disk (we have 2 heads in a cluster on that filer, so unless its in failover mode, its less efficient to send data via the partner path to disk).

So I wrote a quick script to check the paths on all hosts against the list of erroring paths he sent me:

rm -rf /root/lunerrors.txt
touch /root/lunerrors.txt
for host in `cat horhosts.txt`
do
echo $host > lunerrors.txt
for path in `cat errorpaths.txt`
do
ssh $host /usr/sbin/esxcfg-mpath -l | grep $path > lunerrors.txt
done
done

Mac address conflicts

So it turns out some VMs have been assigned static MAC addresses already, so my script duplicated some MAC addresses in use. Not a huge deal (the network team bitched about some errors) until the following happened:

The SCCM guys assigned an image package via MAC address to those MAC addresses, and didn't restrict it to XP images. So the SCCM box re-imaged some windows 2003 boxes. Thank goodness we have vRanger in production, so the VMs were restored pretty quickly (save one, which errored out, and Vizioncore gave us a beta version which restored it).

I modified the script to use a range outside the first 256 addresses. Make sure you aren't conflicting if you use it, and make sure your imaging guys don't leave the package wide-open like ours did :).

I wrote this script to create VMs, since we're making 350 for the remote access project. It used some code that a co-worked found on the internet, but I modified it appropriately so it would loop and create more than one.

#!/bin/bash
declare -i hexcount=0x0
for vmcount in $(seq 1 5)
do
VMDIR=vdi$vmcount
VMNAME=vdi$vmcount
VMMAC=00:50:56:00:01:$(printf %02x $hexcount)
NFSVOL="cgy-vdi-nfs01"

#Begin pasted script code
#Creates a new Virtual Machine

VMOS="winxppro"
VMDSIZE="20g"
VMMEMSIZE="1024"

mkdir /vmfs/volumes/$NFSVOL/$VMDIR
exec 6>&1
exec 1>/vmfs/volumes/NFSVOL/$VMDIR/$VMNAME.vmx

# write the configuration
echo #!/usr/bin/vmware
echo config.version = '"'8'"'
echo virtualHW.version = '"'4'"'
echo memsize = '"'$VMMEMSIZE'"'
echo floppy0.present = '"'TRUE'"'
echo usb.present = '"'FALSE'"'
echo displayName = '"'$VMNAME'"'
echo guestOS = '"'$VMOS'"'
echo ide0:0.present = '"'TRUE'"'
echo ide0:0.fileName = '"'/vmfs/volumes/vdi-scsi-local/BootableX86_12092008.iso'"'
echo ide0:0.deviceType = '"'cdrom-image'"'
echo ide:0.startConnected = '"'TRUE'"'
echo floppy0.startConnected = '"'FALSE'"'
echo floppy0.fileName = '"'/dev/fd0'"'
echo Ethernet0.present = '"'TRUE'"'
echo Ethernet0.networkName = '"'vmnts222'"'
echo Ethernet0.addressType = '"'static'"'
echo Ethernet0.address = '"'$VMMAC'"'
echo scsi0.present = '"'TRUE'"'
echo scsi0:1.present = '"'TRUE'"'
echo scsi0:1.filename = '"'$VMNAME.vmdk'"'
echo scsi0:1.writeThrough = '"'TRUE'"'
echo scsi0.virtualDev = '"'lsilogic'"'
echo
# close file
exec 1>&-

# make stdout a copy of FD 6 (reset stdout), and close FD6
exec 1>&6
exec 6>&-

chmod 755 /vmfs/volumes/$NFSVOL/$VMDIR/$VMNAME.vmx
# Create Disk & Register the .vmx configuration

#Creates 300mb disk (can be modified for larger disk sizes)
cd /vmfs/volumes/$NFSVOL/$VMDIR
vmkfstools -c $VMDSIZE $VMNAME.vmdk -a lsilogic

#Register VM
vmware-cmd -s register /vmfs/volumes/$NFSVOL/$VMDIR/$VMNAME.vmx

((hexcount=hexcount+1))
done

AMD cpu mask script

I wrote this script when we got a HP DL 585 G2 server to add to our dev/test cluster, which is mostly G1s. It checks all the vmx files on a server, and modifies them to have the correct CPU masking for G1/G2 server vmotions to be successful.

#!/bin/bash

PATH=$PATH:/bin:/usr/bin

for i in `vmware-cmd -l`
do
cp -f --reply=yes $i $i.back
if [ `grep -c "cpuid.80000001.edx =" $i` = 0 ]
then
echo 'cpuid.80000001.edx = "-----------0--------H-----------"' >> $i
else
cp -f --reply=yes $i $i.sed1
sed -e 's/cpuid.80000001.edx = "--------------------H-----------"/cpuid.80000001.edx = "-----------0--------H-----------"/g' $i.sed1 > $i.sed2
sed -e 's/cpuid.80000001.edx = "-----------0--------------------"/cpuid.80000001.edx = "-----------0--------H-----------"/g' $i.sed2 > $i
fi
if [ `grep -c cpuid.80000001.edx.amd $i` = 0 ]
then
echo 'cpuid.80000001.edx.amd = "----0R-----0--------H------T----"' >> $i
else
cp -f --reply=yes $i $i.sed1
sed -e "s/cpuid.80000001.edx.amd = \"-----R--------------H------T----\"/cpuid.80000001.edx.amd = \"----0R-----0--------H------T----\"/g" $i.sed1 > $i.sed2
sed -e "s/cpuid.80000001.edx.amd = \"-----R-----0--------H------T----\"/cpuid.80000001.edx.amd = \"----0R-----0--------H------T----\"/g" $i.sed2 > $i.sed3
sed -e "s/cpuid.80000001.edx.amd = \"-----------0--------------------\"/cpuid.80000001.edx.amd = \"----0R-----0--------H------T----\"/g" $i.sed3 > $i
fi
done

More useful scripts

Wrote a couple handy scripts lately. First one checks for ISOs mounted on VMs, because if they are, you can't VMotion them (and then often have to shut the guest OS down - annoying!)

#! /bin/bash
rm -rf /root/isocheck.txt
touch /root/isocheck.txt
for host in `cat isohosts.txt`
do
echo $host
for vmx in `ssh $host vmware-cmd -l`
do
echo $vmx >> isocheck.txt
ssh $host grep -i 'ide0:0.startConnected\ \=\ \"true\"' $vmx >> isocheck.txt
done
done


I have MIME-Lite installed on this ESX box, which is a simple perl-based SMTP agent, it emails me this file afterwards.

The second script sets up all the options to authenticate accounts on an ESX server user a domain account (in this example, esxrangerservice). You need only do "useradd username" after this to allow new users to connect.

#!/bin/bash
/usr/sbin/esxcfg-firewall -o 88,tcp,out,KerberosClient
/usr/sbin/esxcfg-firewall -o 464,tcp,out,KerberosPasswordChange
/usr/sbin/esxcfg-auth --enablead --addomain=yourdomain.com --addc=yourDC.yourdomain.com
/usr/sbin/esxcfg-auth --enablekrb5 --krb5realm=yourdomain.com --krb5kdc=yourDC.yourdomain.com
cp /etc/pam.d/vmware-authd /etc/pam.d/vmware-authd.old
/usr/sbin/useradd esxrangerservice
echo "#%PAM-1.0" > /etc/pam.d/vmware-authd
echo "auth sufficient pam_unix_auth.so shadow nullok" >> /etc/pam.d/vmware-authd
echo "auth required pam_stack.so service=system-auth" >> /etc/pam.d/vmware-authd
echo "account required pam_stack.so service=system-auth" >> /etc/pam.d/vmware-authd

Scripted ESX build with 3.5 not working

I've been scripting our builds for a while now, but haven't done one since our 3.5 upgrade - I'm just going through a test one now before we roll out some new hosts, and my script process which was working great before doesn't seem to produce a working config anymore.

The script code for routing looks like this - there is some switch config done as well, but since the output is below of esxcfg-vswitch and such, i left it out.

#Service Console
#Add service console IP address
echo Adding service console IP address
/usr/sbin/esxcfg-vswif -a -i 172.31.8.168 -n 255.255.255.0 -p "Service Console" vswif1
sleep 5

#Kernel Network
echo Adding Kernel Network Address
/usr/sbin/esxcfg-vmknic -a -i 172.31.8.169 -n 255.255.255.0 VMotion
sleep 5

#Service Console Routing
#Add esx route to VMkernel
echo Adding default route to Service Console
/usr/sbin/esxcfg-route 172.31.8.1
sleep 3

#Routing
#add networking info to /etc/sysconfig/network
echo adding route
cat > /etc/sysconfig/network << EOF2
NETWORKING=yes
HOSTNAME=esxt01.domain.com
GATEWAY=172.31.8.1
GATEWAYDEV=vswif1
EOF2
sleep 3

Here's the contents of every file and command I could think of thats even remotely related.

/etc/sysconfig/network-scripts/ifcfg-vswif1
DEVICE=vswif1
MACADDR=00:50:56:48:9e:17
PORTGROUP="Service Console"
BOOTPROTO=static
BROADCAST=172.31.8.255
IPADDR=172.31.8.168
NETMASK=255.255.255.0
ONBOOT=yes

/etc/sysconfig/network
NETWORKING=yes
HOSTNAME=esxt01.domain.com
GATEWAY=172.31.8.1
GATEWAYDEV=vswif1

/etc/hosts

127.0.0.1 localhost.localdomain localhost
172.31.0.251 nts200.domain.com nts200
172.31.0.248 nts203.domain.com nts203
172.31.5.148 nts334.domain.com nts334
172.31.8.168 esxt01.domain.com esxt01

esxcfg-vmknic -l
Interface Port Group IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled
vmk0 VMotion 172.31.8.169 255.255.255.0 172.31.8.255 00:50:56:78:d6:51 1500 disabled true

esxcfg-vswif -l
Name Port Group IP Address Netmask Broadcast Enabled DHCP
vswif1 Service Console 172.31.8.168 255.255.255.0 172.31.8.255 true false

esxcfg-vswitch -l
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
Production 64 3 64 1500 vmnic1

PortGroup Name VLAN ID Used Ports Uplinks
vmnts08 8 0 vmnic1

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
Management 64 5 64 1500 vmnic0

PortGroup Name VLAN ID Used Ports Uplinks
VMotion 8 1 vmnic0
Service Console 8 1 vmnic0

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
TestPrivateNetwork64 1 64 1500

PortGroup Name VLAN ID Used Ports Uplinks
TestPrivateNetwork 0 0

esxcfg-route -l
VMkernel Routes:
Network Netmask Gateway
172.31.8.0 255.255.255.0 Local Subnet
default 0.0.0.0 172.31.8.1

the route command produces the following, with a big delay (10-15 seconds) before it displays the third line with the default route on it.

kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
172.31.8.0 * 255.255.255.0 U 0 0 0 vswif1
169.254.0.0 * 255.255.0.0 U 0 0 0 vswif1
default 172.31.8.1 0.0.0.0 UG 0 0 0 vswif1

I just get a "Destination Host Unreachable" when I try to ping anything, which tells me it doesn't know how to route to the network. whiskey tango foxtrot.

esxRanger Backups to DMZ machines

I have setup ranger to run all our VM backups - its working pretty well, but a recent upgrade caused machines in our DMZ to do this:

Error In Backup Operations! Error: An error occurred during backup operations. The Archive Created Appears to be Invalid. Failed to read Status File.

As it turns out, the upgrade needs a whole bunch more ports to work through to the DMZ now. Since thats a terrible idea, I turned on the "-encryptdata" option, which means everything goes over port 22 now - much better idea for security, although slower.

Full error log is below.

:: vRanger PRO CLI Backup Commencing ::
Version: 3.2.3.3
"C:\Program Files\vizioncore\esxRanger Professional\esxRangerProCli.exe" esxd03.domain.com /vmfs/volumes/vmdmzprod01/dmz17/dmz17.vmx -copylocal "Z:\cgy-dmz-thur" "-drives:all" -mailonerror -vmnotes -noquiesce -diffratio 50 -maxfullage 5 -retendays 14 -description "cgy dmz thursday full" -zipname "[config]" -vmkey vm-658
[20080905_110907]
Sending log on error only.
The TarBall name will be: [dmz17].
VirtualCenter VM Key: vm-658.
Acquiring VM Lock. This may take many minutes!
VM Lock Acquired. Backup Initialization Commencing. Please Wait...
9/5/2008 11:09:28 AM: Backing up the VM 'dmz17'.
9/5/2008 11:09:28 AM: Beginning Testing Phase.
9/5/2008 11:09:28 AM: Testing Destination Path.
9/5/2008 11:09:33 AM: Snapshotting VM.
9/5/2008 11:09:35 AM: Starting Server Component.
9/5/2008 11:09:37 AM: Performing Backup.
9/5/2008 11:19:42 AM: An error occurred during backup operations:
The Archive Created Appears to be Invalid. Failed to read Status File.
9/5/2008 11:19:42 AM: Removing Backup Snapshot.
9/5/2008 11:20:16 AM: Setting Final Backup Data.
9/5/2008 11:20:16 AM: Writing Backup Information.
9/5/2008 11:20:17 AM: Writing VM Note.
Error In Backup Operations! Error: An error occurred during backup operations. The Archive Created Appears to be Invalid. Failed to read Status File.
9/5/2008 11:20:21 AM: Disconnecting Open Connections.
9/5/2008 11:20:21 AM: Waiting for Disconnection.
9/5/2008 11:20:22 AM: Done.
9/5/2008 11:20:22 AM: Disconnection Sequence Complete.
Pausing for 30 seconds...
You can safely close this window now.

I guess people think IT is like a toaster...

This woman sent this email to all of IS. Managers, programmers, everyone...

That being said, it made me think about why she would send it. Anyone who can use email must get spam, so if she opens one of the attachments, she would probably see that. Its a common method of getting around email filters - and one that normally gets filtered by our message filters.

I guess she just assumes because it isn't really to her, that its an error, not some person on the internet being shady/clever (it is a pretty clever way of doing it frankly). When she pushes the bread down, she expects toast back up, not a bagel.



I am receiving tons of these notifications about e-mails that I have not sent. Can you please look into for me.
-----Original Message-----
From: postmaster@smtp01.2s1n.com [mailto:postmaster@smtp01.2s1n.com]Sent: Monday, September 01, 2008 11:55 AM
To: *********
Subject: Delivery Status Notification (Delay)

This is an automatically generated Delivery Status Notification.
THIS IS A WARNING MESSAGE ONLY.
YOU DO NOT NEED TO RESEND YOUR MESSAGE.
Delivery to the following recipients has been delayed.
webmaster@progressive-equip.com