Browsed by
Tag: vmware

VMware Explore back to Vegas!

VMware Explore back to Vegas!

Today, VMware announced that Explore 2023 will be held in Las Vegas.

The conference (previously VMworld) was held in 2022 in San Francisco at the Moscone Center. 2020 and 2021 were virtual events, and 2019 was also in San Francisco. I did not attend 2018 in Las Vegas, but 2016 & 2017 were also there. 2015 and prior were in SF.

I have attended VMworld/Explore as a customer in 2012, 2015, 2016, & 2017. I attended as VMware Hands On Labs Support Staff in 2019, (2020), & 2022.

Personally, I have enjoyed the events in Las Vegas more than SF. I enjoy the event being in a single hotel/conference center, which leads to better networking opportunities, instead of attendees being spread all over the city. It also seems, anecdotally, that more customers are able to attend in Vegas, with travel/lodging costs being cheaper.

Regardless of where the event is held, it is a MUST ATTEND if you are a IT professional working with VMware or related products in any way. I hope to connect with you at the 2023 event in Vegas!

VMSA-2022-0033 Critical VMware Security Advisory

VMSA-2022-0033 Critical VMware Security Advisory

On 13 December 2022 VMware released a security advisory for VMware ESXi, Workstation, and Fusion.

The VMSA carries a rating of Moderate with a CVSSv3 score of 5.9 for VMware ESXi 8.0 and 7.0

It carries a rating of Critical with a CVSSv3 score of 9.3 for Fusion 12.x and Workstation 16.x

Fusion 13.x and Workstation 17.x are Unaffected.

Upgrade ESXi 8.x to ESXi 8.0a Build 20842819 (or later)

Upgrade ESXi 7.x to 7.0U3i Build 20842708 (or later)

Upgrade Fusion to 12.2.5, or 13.x. Similarly, upgrade Workstation to 16.2.5, or 17.x

More details: https://www.vmware.com/security/advisories/VMSA-2022-0033.html

Pro Tip: Be sure to sign up for VMware Security Advisory emails by clicking the signup link on the VMSA page. This way you can be notified immediately when a new VMSA is released.

VMSA-2022-0027 – Critical VMware Security Advisory

VMSA-2022-0027 – Critical VMware Security Advisory

On October 25 2022, VMware released VMware Security Advisory VMSA-2022-0027 affecting VMware Cloud Foundation 3.x.

Resources:

VMSA-2021-0027: https://www.vmware.com/security/advisories/VMSA-2022-0027.html

The affected products do NOT include VCF 4.x, only versions earlier than 3.11. Most of these products are EOGS (End of General Support) and should be removed from production.

vSphere 8!

vSphere 8!

vSphere 8 is GA!

VMware has a great landing page that goes into details on all the new features and functionality.

The new DPU functionality, with the ability to offload network services, is a great advancement. How will this change how you architect future datacenters? When combined with the vSAN 8 Express Storage Architecture, there are many opportunities for optimization with next-gen hardware.

VMSA-2021-0028: Critical VMware Security Advisory (Multiple Products; Apache Log4j Vulnerability)

VMSA-2021-0028: Critical VMware Security Advisory (Multiple Products; Apache Log4j Vulnerability)

On Friday, December 10, 2021, VMware released VMware Security Advisory VMSA-2021-0028 affecting numerous VMware products including vCenter Server. This advisory is re: Apache Log4j vulnerability CVE-2021-44228 with a CVSSv3 score of 10 out of 10.

Resources:

VMSA-2021-0028: https://www.vmware.com/security/advisories/VMSA-2021-0028.html
FAQ: https://via.vmw.com/vmsa-2021-0028-faq
Unaffected VMware products can be referred to on the Knowledge Base article: https://kb.vmware.com/s/article/87068
VMware Blog “Investigating the Log4j Vulnerability”: https://blogs.vmware.com/security/2021/12/investigating-cve-2021-44228-log4shell-vulnerability.html

The key takeaway is that IMMEDIATE ACTION IS REQUIRED. Workaround should be applied to all running products immediately.

This is being actively exploited in the wild, and should be treated as an emergency change with highest priority.

This vulnerability is in the open-source Apache Log4j Java logging library, which is used in numerous software packages. This is not a VMware-specific issue.

Additional background information:
Tech Solvency Log4Shell log4j vulnerability (CVE-2021-44228) – cheat-sheet reference guide

Birmingham AL VMUG – October 2021

Birmingham AL VMUG – October 2021

#vCommunity – Get involved on Twitter:

@ShanFitz, @vMiklm, @DerrickSkipwith, @BhamVMUG, @myVMUG, @vExpert, @VMware, @VMworld

#vExpert

VMworld session replays: vmware.com/vmworld

Recommended sessions:

Multi-Cloud:

MCL1833 – 10 Things You Need to Know About Project Monterey

MCL1853 – 60 Minutes of Non-Uniform Memory Access (NUMA) 3rd Edition

MCL3222 – VMware Cloud on AWS Outposts: Bring VMware Cloud to Your Data Center

MCL1635 – Extreme Performance Series: Performance Best Practices (vSphere 7 Performance Best Practices Whitepaper)

MCL1277 – A Big Update on vRealize Operations (also recommend: vROps VM Performance Dashboard Deepdive https://youtu.be/9BH7cEJAlu0)

MCL1202 – VMware DRaaS

MCL2768S – Dell APEX Cloud Services with VMware Cloud

MCL1453 – Introducing VMware Project Capitola

MCL2019 – What’s New: Skyline Pro

App Modernization:

APP1482 – Explore Tanzu Community Edition
(How-to blog for Tanzu CE on MacOS: https://www.samakroyd.com/2021/10/06/tanzu-community-edition-on-macos/)

APP2183 – Introduction to Kubernetes for the vSphere Admin (also: kube.academy)

End User Services

EUS1289 – VDI Nerdfest 2021: Demos That Make Admins Drool

Security

SEC1287 – Mount a Robust Defense in Depth Strategy Against Ransomware

vSphere Security Configuration guide

VMSA-2021-0020: Critical vCenter Server Security Advisory

VMSA-2021-0020: Critical vCenter Server Security Advisory

On Tuesday, September 21, 2021, VMware released VMware Security Advisory VMSA-2021-0020 affecting vCenter Server. This VMSA contains 1 Critical advisory and 18 Important advisories.

Resources:

VMSA-2021-0020: https://www.vmware.com/security/advisories/VMSA-2021-0020.html
Workaround*: VMware KB 85717: https://kb.vmware.com/kb/85717
FAQ: https://via.vmw.com/vmsa-2021-0020-faq
VMware Blog “VMSA-2021-0020: What You Need to Know”: https://via.vmw.com/vmsa-2021-0020-blog
VMware Communities: https://via.vmw.com/vmsa-2021-0020-community

*Workaround provided only for 1 Critical vulnerability. Additional 18 lower vulnerabilities require patch to be applied. Workaround should only be used as a temporary measure until patching can be completed. Refer to FAQ for additional information.

The key information to share is that IMMEDIATE ACTION IS REQUIRED. Apply the vCenter patch version now, or apply the workaround (and understand the impact to functionality) if you cannot patch immediately.

VMware has released a very detailed blog and FAQ for this VMSA, which should help clarify questions that are sure to arise. These resources are linked above.

Updated vSphere Security Configuration Guides

Updated vSphere Security Configuration Guides

As a follow-up to the previous post, VMware Security expert Bob Plankers has just published an update to the vSphere Security Configuration guides. The changes are detailed in a blog article here: 

https://blogs.vmware.com/vsphere/2021/02/evolving-the-vmware-vsphere-security-configuration-guides.html

One specific item of interest around OpenSLP/CIM service (ref. VMSA-2019-0022, VMSA-2020-0023):

“Added and updated guidance for disabling SLP and CIM service daemons on ESXi. Security advisories are often good opportunities to assess the state of things, and most customers do not use these protocols. No VMware products use these protocols, either. We now have good methods and guidance for disabling them.”

Furthermore, while not detailed in the blog, I understand that slpd service is disabled by default in ESXi going forward.

You can download the vSphere Security Configuration guides at https://core.vmware.com/security-configuration-guide

Birmingham AL VMUG Meeting – December 2018

Birmingham AL VMUG Meeting – December 2018

Presentation resources:

VMware Product Interoperability Matrices

vCenter Server 6.7 Update 1 Release Notes

ESXi 6.7 Update 1 Release Notes

vSAN 6.7 Update 1 Release Notes

Upgrade Considerations for VMware vSphere 6.7 (VMware vSphere Blog)

What’s New in Performance – VMware vSphere 6.7 (PDF)

VMware Design Studio

vCommunity – Get involved on Twitter:

@vMiklm, @BhamVMUG, @myVMUG, @vExpert, @VMware

#vExpert

Video:

vROps Sizing Tool

vROps Sizing Tool

A frequent question when deploying vRealize Operations relates to correctly sizing the environment.  An undersized vROps deployment will not perform optimally, and an oversized environment does not best utilize resources.  VMware has a great new tool to make sizing a vROps installation painless:

https://vropssizer.vmware.com/sizing-wizard/

Simply select the version of vROps, input the number of vCenters, hosts, clusters, VMs, etc., make the appropriate selections for data retention, and a sizing recommendation is provided:

I hope you’ll find this helpful for your next vROps deployment.  You can also check your current deployment against this tool to determine if your vROps install may be under- or over-provisioned.

Find vCenter PSC

Find vCenter PSC

Often it is necessary to determine the PSC a vCenter server is pointed to.  Is it an external PSC or an internal PSC?

In vSphere client, look at the vCenter Advanced Settings. Find the setting:

config.vpxd.sso.admin.uri

This URL will contain the hostname of the PSC.

Intel L1 Terminal Fault Security Vulnerability

Intel L1 Terminal Fault Security Vulnerability

The following resources should be helpful in understanding the Intel L1 Terminal Fault vulnerability announced today.

VMware KB 55636:  VMware Overview of ‘L1 Terminal Fault’ (L1TF) Speculative-Execution vulnerabilities in Intel processors: CVE-2018-3646, CVE-2018-3620, and CVE-2018-3615 (55636)

Patching for CVE-2018-3646 “L1TF – VMM” could have performance impacts in customer virtualization environments.  Refer to VMware KB 55806 for further details.

Overview video from Intel:

More detailed video from RedHat:

License groups in vROps 6.6

License groups in vROps 6.6

How to Remove “License Is Invalid” Watermark in vRealize Operations Manager

I have on good authority, although not publicly documented, that a code change happened in vROps 6.6 to resolve a bug where the “License is invalid” watermark was not showing up even when environments were not licensed. That is to say, no matter whether previous versions had valid license keys or not, the watermark never appeared. As a result of fixing this bug, some environments may now be seeing the watermark due to the incorrect construction of License Groups.

This is only applicable to using valid license keys in a supported manner. Nothing in this article is discussing bypassing valid product licensing for vROps. We carefully reviewed our VMware ELA with our AE and TAM to ensure we had valid entitlements.

The vROps environment I admin uses a combination of per-VM Advanced licenses and per-CPU vCloud Suite Advanced licenses. This is supported, as evidenced by the vRealize Operations 6.6.1 Release Notes:

You can deploy single choice and suite licenses together. License counting for individual license keys is handled through licensing groups. You can mix editions or licensing models …

However, the vROps documentation is extremely vague on how these license groups should or could be constructed.

We worked with VMware Support some time ago to establish license groups for each respective set of keys. However, upon upgrade to 6.6, we noted the “License is invalid” watermark appearing on many objects in the environment. License Keys were no longer associated with the License Groups. The dynamic criteria we used previously to create license groups was no longer valid in vROps 6.6, for reasons that are still not clear. After extensive work with VMware Support through BCS again, we have arrived at a solution that removes the watermark from all objects within our environment. I will try to elaborate on the general process here, as every environment will have unique requirements. Credit for this solution ultimately goes to Kyle P. at VMware Premier Support, and I hope a VMware KB eventually appears.

Criteria:
– License Group 1 “vCloud Suite” Host CPU-based licenses applied to Object type “Host System” for a given set of clusters.
– License Group 2 “VM Licenses” VM-based licenses applied to Object type “Virtual Machine” for all other VMs EXCEPT those on Hosts licensed with License Group 1.
– Use “dynamic” membership criteria, not static “Always include/exclude” lists to avoid manual maintenance of License Groups. This is an important distinction.

Important notes & considerations:
– When you apply a CPU license to a group containing Hosts, the VMs on the Hosts will still show “License is invalid” watermark. When you apply a VM license key to Virtual Machines, the Hosts on which those VMs run will still show the “License is invalid” watermark. I am convinced this is a bug in vROps, but I’m not getting much traction with the PR associated with my SR since a “workaround” is in place. So, the workaround is this: while the license is applied to the respective Object type of each License key, the related objects (parent or children) are also going to have to be included in membership for the License Group.

Let’s look at the example License Group 1 “vCloud Suite” first:

(I’m assuming if this article is relevant to you, that you know where to find License Groups under Administration – Management – Licensing & how to create/edit them.)


Object Type = Host System
Relationship – Descendant of – contains – [CLUSTER NAME] (a partial string can be used for wildcard matching) (If multiple clusters are needed that can’t be added with a wildcard, click “Add another criteria set” to create an OR construct and replicate the same criteria.)

This should make sense so far. We’re creating a License Group for CPU License Keys with membership of Hosts (which are descendants of the named cluster(s)).

Now here’s the part that is not so obvious: We also need to add additional OR criteria to select the Object Type “Virtual Machine” for all the VMs that live on these clusters. The license will not apply to the VMs for license utilization count, but the watermark will persist unless the VMs are added via this criteria.


Object Type = Virtual Machine
Relationship – Descendant of – contains – [CLUSTER NAME] (a partial string can be used for wildcard matching) – use the same criteria to select clusters as above.

Now, License Group 2 “VM Licenses”

Remember from our Criteria above, this license groups will include all VMs that do not live on the Host Systems licensed with License Group 1.


Object Type = Virtual Machine
Relationship – Descendant of – not contains – [CLUSTER NAME] (same string used in License Group 1 above)
For additional clusters, you’ll need to “Add” to use AND logic.

This selects all VMs in the environment that do NOT live on these clusters. Hopefully this makes sense. Now here again is the part that is does not follow logic: the Host Systems that host these VMs will still display “License is invalid” watermark until we add Host Systems to this license group. Again, I believe this is a flaw in vROps licensing, as the license group applying to object type Virtual Machine should not have to contain objects of type Host System. Despite that, in extensive testing, this is the workaround we’ve found to remove the watermark.

“Add another criteria set” (making an OR statement), Object Type = Host System.

That’s it. Note it can take a few minutes for the licensing to update. You can potentially speed this up by clicking Refresh under License Keys:

This was the solution we developed to meet our criteria as listed above. It should be understood that almost every environment will be different, and your criteria may be different. Hopefully the concepts behind this article can be extended to additional use cases to resolve the “License is invalid” watermark in vROps. If there’s anything I can clarify or any way I can help, contact me via Twitter @vMiklm.

vRealize Operations Certificate Installation Error

vRealize Operations Certificate Installation Error

There is a good bit of information available for installing certificates for vROps 6.x. I encountered an issue that required a VMware Support case and uncovered some changes in vRealize Operations Manager 6.3 and 6.4.

My environment was using an internally signed SHA1 (128-bit) certificate. This was being updated to a 256-bit certificate.

The process to generate a certificate for vROps is more complex than typical, and is poorly documented by VMware. There are some articles with better information on how to create and install vROps certificates:
http://www.kanap.net/2015/02/install-a-custom-certificate-for-vrealize-operations-manager/
http://virtuallyeuc.com/vrealize-operations-manager-6-0-certificate-creation/

After carefully following the process to create the CSR, having the certificate signed, and building the .pem file, vROps refused to accept the new cert:

Operation failed. If the error persists, contact VMware support.

Review casa.log [/var/log/casa_logs/casa.log on your master vROps node] and look for something similar to:

2016-12-06 17:45:44,102 [x2000UIE] [ajp-nio-127.0.0.1-8011-exec-7] INFO support.subprocess.GeneralCommand:255 - Command '/usr/lib/vmware-vcopssuite/python/bin/python /usr/lib/vmware-casa/bin/vropsCertificateTool.py -i /storage/db/tmp/uploaded_cert.tmp --no_describe --json --level NONE' threw exception: CommandLineExitException: key=general.failure; args=1,; cause=
2016-12-06 17:45:44,103 [x2000UIE] [ajp-nio-127.0.0.1-8011-exec-7] INFO casa.security.SecurityService:946 - validateCertificateFile script's STDOUT:
{"errors": [], "exitCode": 0, "warnings": [{"warning_args": "/C=US/ST=State/L=City/O=Org/OU=Domain/CN=CA", "warning_code": "WARN-2", "warning_message": "Issuer and subject names match (/C=US/ST=State/L=City/O=Org/OU=Domain/CN=CA), but the issuer key does not. This is likely the wrong issuer certificate."}]}

So here’s what happened:
Starting in vROps 6.3, a change was made to how vROps handles the upload of a certificate.  The service used for the handling of the chain file has a file limit of 8192 bytes.
– My certificate chain (.pem file) was 10.7 KB, due to the fact that we have 2 intermediate certificates (Root Cert->Intermediate 1->Intermediate 2->Server Cert)
– vROps was essentially truncating the file, causing it to interpret the first intermediate certificate as the root, causing the “likely the wrong issuer certificate” error.
– I confirmed the issue exists on 6.3 and 6.4; the TSE was not able to replicate on 6.2.1
The vRealize Operations Manager supports custom security certificates with key length up to 8192 bits. An error is displayed when you try to upload a security certificate generated with a stronger key length beyond 8192 bits.

Solutions:
1) Use a certificate with only one intermediate, which results in a .pem file size of ~7 KB
– I had the option to request a GlobalSign certificate at minimal cost; we were in the process of requesting this anyway to rule out our internal CA as the problem. It is of course understood that this will not be an option for many users.
2) “The workaround is to edit the ‘maxInMemorySize’ value for the spring service which handles the upload of the cert. This should prevent the use of the temp file which in turn will bypass the part of the code that reads the temp file with the 10k limit.” – Provided by VMware TSE. I did not pursue this workaround as we already had the new GlobalSign certificate ready to test on the same day this workaround was proposed. I have asked for additional information on this workaround and will update if it is provided.

Hopefully this information helps diagnose failures in certificate installation in vRealize Operations 6.3 or 6.4. I’m not sure why 8 KB was the file size limit chosen, but it is too small for many environments with multiple intermediate CAs. Hopefully this will be resolved in the next release.

Update:
1) To edit ‘maxInMemorySize’ value for the spring service to allow a larger certificate upload, the following information was provided by VMware support:
a. Stop casa.
service vmware-casa stop

b. Edit /usr/lib/vmware-casa/casa-webapp/webapps/casa/WEB-INF/casa-servlet.xml
Locate this text around line 256:
<bean id="multipartResolver" class="org.springframework.web.multipart.commons.CommonsMultipartResolver">
<property name="uploadTempDir" value="file:///${application.data?/storage}/db/tmp"/>
</bean>

Add the following within the <bean> definition:
<property name="maxInMemorySize" value="30720"/>

c. Restart casa.
service vmware-casa start

2) I have been advised that a fix for this issue will be included in the next release, although I cannot confirm that and have no information on the vROps release schedule.

I want to publicly acknowledge VMware Global Support Services (GSS) Business Critical Support (BCS) and the excellent experience working with them to diagnose and resolve this issue, as well as VMware Technical Account Manager (TAM) Services. While not cheap, these service offerings provide high value.

* Update 3/3/2017
vRealize Operations 6.5 was released on 3/2/2017. The 8192 bit certificate size limit is still reflected in the documentation .

VCSA – Moving Behind A Firewall

VCSA – Moving Behind A Firewall

We have a situation where we have to deploy VMware vCenter Server Appliance (VCSA), then move it to a secured network behind a firewall.

I have not (yet) had success changing the hostname, including the FQDN, so make sure you can create proper DNS entries on both sides of the firewall and can do the initial deployment with the intended final FQDN. Our VCSA’s FQDN is of the format “vcenterserver.securednet.companynet.org” so this was achievable (the segmented domain is a subdomain of the company domain)

Deploy the appliance, then access the VCSA console and press F2, then login as root. At this point you may also need to Edit the VM Settings and change it to the new network (behind the firewall, in my case).
– Configure Management Network
– Set IP Configuration, IPv6 Configuration, DNS Configuration, Custom DNS Suffixes as required.
– (Esc) Exit, (Y) Yes Apply changes and restart management network

At this point, I got the following 2 screens. IPv6 is understandable; we disabled it. The DNS error appears even though we have validated the DNS entries/servers.

Now, to access the shell console. Login as root.

Command> shell.set --enabled True
Command> shell
# vi /etc/sysconfig/networking/devices/ifcfg-eth0

Quick vi reminder: hit (Insert) to edit the line, then (Esc), :wq! to save and exit.

Make sure all the IP information is correct in this file. I had to update the broadcast & gateway for sure. Then,

# vi /etc/hosts

Fix the entry here to reflect the new IP.

Validate settings:

# ifconfig eth0
# route -n

At this point, we were missing the default gateway in the route table, even thought it was defined in the ifcfg-eth0 file and in the IP Configuration screen of the console. Also the Broadcast address still reflects the old network, but this is less of a problem.

To temporarily fix the default gateway, do # route add default gateway XXX.YYY.ZZZ.1, using the correct gateway IP obviously.

To permanently resolve the gateway issue, log into the web console (https://vcenterserver:443/vsphere-client/) as Administrator@vsphere.local or whatever you defined as your SSO domain.
– Go to System Configuration under Administration in the main panel
– Nodes (in left column, under System Configuration)
– Select your vCenter server under Nodes
– “Edit” in the upper right corner of the main panel
– Expand nic0 & define Default gateway
– Ok
– Reboot to validate configuration persists

VMware: Multi-NIC vMotion Configuration with PowerShell

VMware: Multi-NIC vMotion Configuration with PowerShell

Changed configuration of numerous VMware clusters to utilize multi-NIC vMotion. As with all my PS, it’s quick & dirty and cobbled from a couple of example scripts. Assumes existing vmkernel port group named “vMotion”

<#
.SYNOPSIS
   Configure multi-NIC vMotion
.DESCRIPTION
   Miklm.com 08/19/2015
   Credit to examples from http://www.yellow-bricks.com/2011/09/17/multiple-nic-vmotion-in-vsphere-5/#comment-28153 & http://hostilecoding.blogspot.com/2014/01/vmware-virtual-standard-switch.html
#>

$viserver = "vCenter.local"
$cluster = "CLUSTERNAME"
$vswitchName = "vSwitch1"
$VMotionVLan = "908"
$vSwitch2Nics = "vmnic1","vmnic3"
$VMotionPortGroupName1 = "vMotion-01"
$VMotionPortGroupName2 = "vMotion-02"

Connect-VIServer $viserver

Foreach ($esxhost in (Get-Cluster -name $cluster | Get-VMHost))
{
  $vswitch = Get-VirtualSwitch -VMHost $esxhost -Name $vswitchName

  # Remove the original "vMotion" VMkernel Port group
  Remove-VMHostNetworkAdapter -Nic (Get-VMHostNetworkAdapter -VMHost $esxhost | where {$_.PortGroupName -eq "vMotion"}) -Confirm:$false
  Remove-VirtualPortGroup -VirtualPortGroup (Get-VirtualPortGroup -Name "vMotion" -vmhost $esxhost) -Confirm:$false

  # Configure virtual switch for Multi-NIC vMotion
  $vportgroup1 = New-VirtualPortGroup -VirtualSwitch $vswitch -Name $VMotionPortGroupName1 -VLanId $VMotionVLan
  $vportgroup2 = New-VirtualPortGroup -VirtualSwitch $vswitch -Name $VMotionPortGroupName2 -VLanId $VMotionVLan

  $vNic1 = New-VMHostNetworkAdapter -VMHost $esxhost -VirtualSwitch $vswitch -PortGroup $vportgroup1 -VMotionEnabled:$true
  $vNic2 = New-VMHostNetworkAdapter -VMHost $esxhost -VirtualSwitch $vswitch -PortGroup $vportgroup2 -VMotionEnabled:$true

  $vportgroup1 | Get-NicTeamingPolicy | Set-NicTeamingPolicy -MakeNicActive $vSwitch2Nics[0] -MakeNicStandBy $vSwitch2Nics[1]
  $vportgroup2 | Get-NicTeamingPolicy | Set-NicTeamingPolicy -MakeNicActive $vSwitch2Nics[1] -MakeNicStandBy $vSwitch2Nics[0]
}

 

Cleaning up ARP tables – VMware ESXi 5.1

Cleaning up ARP tables – VMware ESXi 5.1

We had a recent situation where a virtual appliance (Avamar proxy) was mistakenly deployed configured with the IP address of the gateway. This caused the hosts to be unreachable from vCenter. After finding the offending VM and shutting it down from the console, the ARP tables either had to be cleaned up (you could also allow them to time out but that could take up to 20 minutes)

ESXi 5.1
esxcli network ip neighbor list
vsish -e set /net/tcpip/v4/neighbor del IPADDRESS

New functionality added in 5.5 to remove ARP entries through esxcli

Find a VM by MAC in vCenter with PowerCLI
Get-vm | Select Name, @{N=“Network“;E={$_ | Get-networkAdapter | ? {$_.macaddress -eq“00:50:56:A4:22:F4“}}} |Where {$_.Network-ne “”}

Also you may need to clean up your Windows vCenter server from the command prompt
arp -a
arp -d IPADDRESS

Or,
netsh interface ip delete arpcache
Also a good idea to:
ipconfig /flushdns