MIKLM.COM

There is a good bit of information available for installing certificates for vROps 6.x. I encountered an issue that required a VMware Support case and uncovered some changes in vRealize Operations Manager 6.3 and 6.4.

My environment was using an internally signed SHA1 (128-bit) certificate. This was being updated to a 256-bit certificate.

The process to generate a certificate for vROps is more complex than typical, and is poorly documented by VMware. There are some articles with better information on how to create and install vROps certificates:
http://www.kanap.net/2015/02/install-a-custom-certificate-for-vrealize-operations-manager/
http://virtuallyeuc.com/vrealize-operations-manager-6-0-certificate-creation/

After carefully following the process to create the CSR, having the certificate signed, and building the .pem file, vROps refused to accept the new cert:

Operation failed. If the error persists, contact VMware support.

Review casa.log [/var/log/casa_logs/casa.log on your master vROps node] and look for something similar to:

2016-12-06 17:45:44,102 [x2000UIE] [ajp-nio-127.0.0.1-8011-exec-7] INFO support.subprocess.GeneralCommand:255 - Command '/usr/lib/vmware-vcopssuite/python/bin/python /usr/lib/vmware-casa/bin/vropsCertificateTool.py -i /storage/db/tmp/uploaded_cert.tmp --no_describe --json --level NONE' threw exception: CommandLineExitException: key=general.failure; args=1,; cause=
2016-12-06 17:45:44,103 [x2000UIE] [ajp-nio-127.0.0.1-8011-exec-7] INFO casa.security.SecurityService:946 - validateCertificateFile script's STDOUT:
{"errors": [], "exitCode": 0, "warnings": [{"warning_args": "/C=US/ST=State/L=City/O=Org/OU=Domain/CN=CA", "warning_code": "WARN-2", "warning_message": "Issuer and subject names match (/C=US/ST=State/L=City/O=Org/OU=Domain/CN=CA), but the issuer key does not. This is likely the wrong issuer certificate."}]}

So here’s what happened:
Starting in vROps 6.3, a change was made to how vROps handles the upload of a certificate.  The service used for the handling of the chain file has a file limit of 8192 bytes.
– My certificate chain (.pem file) was 10.7 KB, due to the fact that we have 2 intermediate certificates (Root Cert->Intermediate 1->Intermediate 2->Server Cert)
– vROps was essentially truncating the file, causing it to interpret the first intermediate certificate as the root, causing the “likely the wrong issuer certificate” error.
– I confirmed the issue exists on 6.3 and 6.4; the TSE was not able to replicate on 6.2.1
– The vRealize Operations Manager supports custom security certificates with key length up to 8192 bits. An error is displayed when you try to upload a security certificate generated with a stronger key length beyond 8192 bits.

Solutions:
1) Use a certificate with only one intermediate, which results in a .pem file size of ~7 KB
– I had the option to request a GlobalSign certificate at minimal cost; we were in the process of requesting this anyway to rule out our internal CA as the problem. It is of course understood that this will not be an option for many users.
2) “The workaround is to edit the ‘maxInMemorySize’ value for the spring service which handles the upload of the cert. This should prevent the use of the temp file which in turn will bypass the part of the code that reads the temp file with the 10k limit.” – Provided by VMware TSE. I did not pursue this workaround as we already had the new GlobalSign certificate ready to test on the same day this workaround was proposed. I have asked for additional information on this workaround and will update if it is provided.

Hopefully this information helps diagnose failures in certificate installation in vRealize Operations 6.3 or 6.4. I’m not sure why 8 KB was the file size limit chosen, but it is too small for many environments with multiple intermediate CAs. Hopefully this will be resolved in the next release.

Update:
1) To edit ‘maxInMemorySize’ value for the spring service to allow a larger certificate upload, the following information was provided by VMware support:
a. Stop casa.
service vmware-casa stop

b. Edit /usr/lib/vmware-casa/casa-webapp/webapps/casa/WEB-INF/casa-servlet.xml
Locate this text around line 256:
<bean id="multipartResolver" class="org.springframework.web.multipart.commons.CommonsMultipartResolver">
<property name="uploadTempDir" value="file:///${application.data?/storage}/db/tmp"/>
</bean>

Add the following within the <bean> definition:
<property name="maxInMemorySize" value="30720"/>

c. Restart casa.
service vmware-casa start

2) I have been advised that a fix for this issue will be included in the next release, although I cannot confirm that and have no information on the vROps release schedule.

I want to publicly acknowledge VMware Global Support Services (GSS) Business Critical Support (BCS) and the excellent experience working with them to diagnose and resolve this issue, as well as VMware Technical Account Manager (TAM) Services. While not cheap, these service offerings provide high value.

* Update 3/3/2017
vRealize Operations 6.5 was released on 3/2/2017. The 8192 bit certificate size limit is still reflected in the documentation .