Browsed by
Tag: vrops

VMSA-2021-0028: Critical VMware Security Advisory (Multiple Products; Apache Log4j Vulnerability)

VMSA-2021-0028: Critical VMware Security Advisory (Multiple Products; Apache Log4j Vulnerability)

On Friday, December 10, 2021, VMware released VMware Security Advisory VMSA-2021-0028 affecting numerous VMware products including vCenter Server. This advisory is re: Apache Log4j vulnerability CVE-2021-44228 with a CVSSv3 score of 10 out of 10.


Unaffected VMware products can be referred to on the Knowledge Base article:
VMware Blog “Investigating the Log4j Vulnerability”:

The key takeaway is that IMMEDIATE ACTION IS REQUIRED. Workaround should be applied to all running products immediately.

This is being actively exploited in the wild, and should be treated as an emergency change with highest priority.

This vulnerability is in the open-source Apache Log4j Java logging library, which is used in numerous software packages. This is not a VMware-specific issue.

Additional background information:
Tech Solvency Log4Shell log4j vulnerability (CVE-2021-44228) – cheat-sheet reference guide

vROps Sizing Tool

vROps Sizing Tool

A frequent question when deploying vRealize Operations relates to correctly sizing the environment.  An undersized vROps deployment will not perform optimally, and an oversized environment does not best utilize resources.  VMware has a great new tool to make sizing a vROps installation painless:

Simply select the version of vROps, input the number of vCenters, hosts, clusters, VMs, etc., make the appropriate selections for data retention, and a sizing recommendation is provided:

I hope you’ll find this helpful for your next vROps deployment.  You can also check your current deployment against this tool to determine if your vROps install may be under- or over-provisioned.

License groups in vROps 6.6

License groups in vROps 6.6

How to Remove “License Is Invalid” Watermark in vRealize Operations Manager

I have on good authority, although not publicly documented, that a code change happened in vROps 6.6 to resolve a bug where the “License is invalid” watermark was not showing up even when environments were not licensed. That is to say, no matter whether previous versions had valid license keys or not, the watermark never appeared. As a result of fixing this bug, some environments may now be seeing the watermark due to the incorrect construction of License Groups.

This is only applicable to using valid license keys in a supported manner. Nothing in this article is discussing bypassing valid product licensing for vROps. We carefully reviewed our VMware ELA with our AE and TAM to ensure we had valid entitlements.

The vROps environment I admin uses a combination of per-VM Advanced licenses and per-CPU vCloud Suite Advanced licenses. This is supported, as evidenced by the vRealize Operations 6.6.1 Release Notes:

You can deploy single choice and suite licenses together. License counting for individual license keys is handled through licensing groups. You can mix editions or licensing models …

However, the vROps documentation is extremely vague on how these license groups should or could be constructed.

We worked with VMware Support some time ago to establish license groups for each respective set of keys. However, upon upgrade to 6.6, we noted the “License is invalid” watermark appearing on many objects in the environment. License Keys were no longer associated with the License Groups. The dynamic criteria we used previously to create license groups was no longer valid in vROps 6.6, for reasons that are still not clear. After extensive work with VMware Support through BCS again, we have arrived at a solution that removes the watermark from all objects within our environment. I will try to elaborate on the general process here, as every environment will have unique requirements. Credit for this solution ultimately goes to Kyle P. at VMware Premier Support, and I hope a VMware KB eventually appears.

– License Group 1 “vCloud Suite” Host CPU-based licenses applied to Object type “Host System” for a given set of clusters.
– License Group 2 “VM Licenses” VM-based licenses applied to Object type “Virtual Machine” for all other VMs EXCEPT those on Hosts licensed with License Group 1.
– Use “dynamic” membership criteria, not static “Always include/exclude” lists to avoid manual maintenance of License Groups. This is an important distinction.

Important notes & considerations:
– When you apply a CPU license to a group containing Hosts, the VMs on the Hosts will still show “License is invalid” watermark. When you apply a VM license key to Virtual Machines, the Hosts on which those VMs run will still show the “License is invalid” watermark. I am convinced this is a bug in vROps, but I’m not getting much traction with the PR associated with my SR since a “workaround” is in place. So, the workaround is this: while the license is applied to the respective Object type of each License key, the related objects (parent or children) are also going to have to be included in membership for the License Group.

Let’s look at the example License Group 1 “vCloud Suite” first:

(I’m assuming if this article is relevant to you, that you know where to find License Groups under Administration – Management – Licensing & how to create/edit them.)

Object Type = Host System
Relationship – Descendant of – contains – [CLUSTER NAME] (a partial string can be used for wildcard matching) (If multiple clusters are needed that can’t be added with a wildcard, click “Add another criteria set” to create an OR construct and replicate the same criteria.)

This should make sense so far. We’re creating a License Group for CPU License Keys with membership of Hosts (which are descendants of the named cluster(s)).

Now here’s the part that is not so obvious: We also need to add additional OR criteria to select the Object Type “Virtual Machine” for all the VMs that live on these clusters. The license will not apply to the VMs for license utilization count, but the watermark will persist unless the VMs are added via this criteria.

Object Type = Virtual Machine
Relationship – Descendant of – contains – [CLUSTER NAME] (a partial string can be used for wildcard matching) – use the same criteria to select clusters as above.

Now, License Group 2 “VM Licenses”

Remember from our Criteria above, this license groups will include all VMs that do not live on the Host Systems licensed with License Group 1.

Object Type = Virtual Machine
Relationship – Descendant of – not contains – [CLUSTER NAME] (same string used in License Group 1 above)
For additional clusters, you’ll need to “Add” to use AND logic.

This selects all VMs in the environment that do NOT live on these clusters. Hopefully this makes sense. Now here again is the part that is does not follow logic: the Host Systems that host these VMs will still display “License is invalid” watermark until we add Host Systems to this license group. Again, I believe this is a flaw in vROps licensing, as the license group applying to object type Virtual Machine should not have to contain objects of type Host System. Despite that, in extensive testing, this is the workaround we’ve found to remove the watermark.

“Add another criteria set” (making an OR statement), Object Type = Host System.

That’s it. Note it can take a few minutes for the licensing to update. You can potentially speed this up by clicking Refresh under License Keys:

This was the solution we developed to meet our criteria as listed above. It should be understood that almost every environment will be different, and your criteria may be different. Hopefully the concepts behind this article can be extended to additional use cases to resolve the “License is invalid” watermark in vROps. If there’s anything I can clarify or any way I can help, contact me via Twitter @vMiklm.

vRealize Operations Certificate Installation Error

vRealize Operations Certificate Installation Error

There is a good bit of information available for installing certificates for vROps 6.x. I encountered an issue that required a VMware Support case and uncovered some changes in vRealize Operations Manager 6.3 and 6.4.

My environment was using an internally signed SHA1 (128-bit) certificate. This was being updated to a 256-bit certificate.

The process to generate a certificate for vROps is more complex than typical, and is poorly documented by VMware. There are some articles with better information on how to create and install vROps certificates:

After carefully following the process to create the CSR, having the certificate signed, and building the .pem file, vROps refused to accept the new cert:

Operation failed. If the error persists, contact VMware support.

Review casa.log [/var/log/casa_logs/casa.log on your master vROps node] and look for something similar to:

2016-12-06 17:45:44,102 [x2000UIE] [ajp-nio-] INFO support.subprocess.GeneralCommand:255 - Command '/usr/lib/vmware-vcopssuite/python/bin/python /usr/lib/vmware-casa/bin/ -i /storage/db/tmp/uploaded_cert.tmp --no_describe --json --level NONE' threw exception: CommandLineExitException: key=general.failure; args=1,; cause=
2016-12-06 17:45:44,103 [x2000UIE] [ajp-nio-] INFO - validateCertificateFile script's STDOUT:
{"errors": [], "exitCode": 0, "warnings": [{"warning_args": "/C=US/ST=State/L=City/O=Org/OU=Domain/CN=CA", "warning_code": "WARN-2", "warning_message": "Issuer and subject names match (/C=US/ST=State/L=City/O=Org/OU=Domain/CN=CA), but the issuer key does not. This is likely the wrong issuer certificate."}]}

So here’s what happened:
Starting in vROps 6.3, a change was made to how vROps handles the upload of a certificate.  The service used for the handling of the chain file has a file limit of 8192 bytes.
– My certificate chain (.pem file) was 10.7 KB, due to the fact that we have 2 intermediate certificates (Root Cert->Intermediate 1->Intermediate 2->Server Cert)
– vROps was essentially truncating the file, causing it to interpret the first intermediate certificate as the root, causing the “likely the wrong issuer certificate” error.
– I confirmed the issue exists on 6.3 and 6.4; the TSE was not able to replicate on 6.2.1
The vRealize Operations Manager supports custom security certificates with key length up to 8192 bits. An error is displayed when you try to upload a security certificate generated with a stronger key length beyond 8192 bits.

1) Use a certificate with only one intermediate, which results in a .pem file size of ~7 KB
– I had the option to request a GlobalSign certificate at minimal cost; we were in the process of requesting this anyway to rule out our internal CA as the problem. It is of course understood that this will not be an option for many users.
2) “The workaround is to edit the ‘maxInMemorySize’ value for the spring service which handles the upload of the cert. This should prevent the use of the temp file which in turn will bypass the part of the code that reads the temp file with the 10k limit.” – Provided by VMware TSE. I did not pursue this workaround as we already had the new GlobalSign certificate ready to test on the same day this workaround was proposed. I have asked for additional information on this workaround and will update if it is provided.

Hopefully this information helps diagnose failures in certificate installation in vRealize Operations 6.3 or 6.4. I’m not sure why 8 KB was the file size limit chosen, but it is too small for many environments with multiple intermediate CAs. Hopefully this will be resolved in the next release.

1) To edit ‘maxInMemorySize’ value for the spring service to allow a larger certificate upload, the following information was provided by VMware support:
a. Stop casa.
service vmware-casa stop

b. Edit /usr/lib/vmware-casa/casa-webapp/webapps/casa/WEB-INF/casa-servlet.xml
Locate this text around line 256:
<bean id="multipartResolver" class="org.springframework.web.multipart.commons.CommonsMultipartResolver">
<property name="uploadTempDir" value="file:///${}/db/tmp"/>

Add the following within the <bean> definition:
<property name="maxInMemorySize" value="30720"/>

c. Restart casa.
service vmware-casa start

2) I have been advised that a fix for this issue will be included in the next release, although I cannot confirm that and have no information on the vROps release schedule.

I want to publicly acknowledge VMware Global Support Services (GSS) Business Critical Support (BCS) and the excellent experience working with them to diagnose and resolve this issue, as well as VMware Technical Account Manager (TAM) Services. While not cheap, these service offerings provide high value.

* Update 3/3/2017
vRealize Operations 6.5 was released on 3/2/2017. The 8192 bit certificate size limit is still reflected in the documentation .