Browsed by
Category: Tech

vRealize Operations Certificate Installation Error

vRealize Operations Certificate Installation Error

There is a good bit of information available for installing certificates for vROps 6.x. I encountered an issue that required a VMware Support case and uncovered some changes in vRealize Operations Manager 6.3 and 6.4.

My environment was using an internally signed SHA1 (128-bit) certificate. This was being updated to a 256-bit certificate.

The process to generate a certificate for vROps is more complex than typical, and is poorly documented by VMware. There are some articles with better information on how to create and install vROps certificates:
http://www.kanap.net/2015/02/install-a-custom-certificate-for-vrealize-operations-manager/
http://virtuallyeuc.com/vrealize-operations-manager-6-0-certificate-creation/

After carefully following the process to create the CSR, having the certificate signed, and building the .pem file, vROps refused to accept the new cert:

Operation failed. If the error persists, contact VMware support.

Review casa.log [/var/log/casa_logs/casa.log on your master vROps node] and look for something similar to:

2016-12-06 17:45:44,102 [x2000UIE] [ajp-nio-127.0.0.1-8011-exec-7] INFO support.subprocess.GeneralCommand:255 - Command '/usr/lib/vmware-vcopssuite/python/bin/python /usr/lib/vmware-casa/bin/vropsCertificateTool.py -i /storage/db/tmp/uploaded_cert.tmp --no_describe --json --level NONE' threw exception: CommandLineExitException: key=general.failure; args=1,; cause=
2016-12-06 17:45:44,103 [x2000UIE] [ajp-nio-127.0.0.1-8011-exec-7] INFO casa.security.SecurityService:946 - validateCertificateFile script's STDOUT:
{"errors": [], "exitCode": 0, "warnings": [{"warning_args": "/C=US/ST=State/L=City/O=Org/OU=Domain/CN=CA", "warning_code": "WARN-2", "warning_message": "Issuer and subject names match (/C=US/ST=State/L=City/O=Org/OU=Domain/CN=CA), but the issuer key does not. This is likely the wrong issuer certificate."}]}

So here’s what happened:
Starting in vROps 6.3, a change was made to how vROps handles the upload of a certificate.  The service used for the handling of the chain file has a file limit of 8192 bytes.
– My certificate chain (.pem file) was 10.7 KB, due to the fact that we have 2 intermediate certificates (Root Cert->Intermediate 1->Intermediate 2->Server Cert)
– vROps was essentially truncating the file, causing it to interpret the first intermediate certificate as the root, causing the “likely the wrong issuer certificate” error.
– I confirmed the issue exists on 6.3 and 6.4; the TSE was not able to replicate on 6.2.1
The vRealize Operations Manager supports custom security certificates with key length up to 8192 bits. An error is displayed when you try to upload a security certificate generated with a stronger key length beyond 8192 bits.

Solutions:
1) Use a certificate with only one intermediate, which results in a .pem file size of ~7 KB
– I had the option to request a GlobalSign certificate at minimal cost; we were in the process of requesting this anyway to rule out our internal CA as the problem. It is of course understood that this will not be an option for many users.
2) “The workaround is to edit the ‘maxInMemorySize’ value for the spring service which handles the upload of the cert. This should prevent the use of the temp file which in turn will bypass the part of the code that reads the temp file with the 10k limit.” – Provided by VMware TSE. I did not pursue this workaround as we already had the new GlobalSign certificate ready to test on the same day this workaround was proposed. I have asked for additional information on this workaround and will update if it is provided.

Hopefully this information helps diagnose failures in certificate installation in vRealize Operations 6.3 or 6.4. I’m not sure why 8 KB was the file size limit chosen, but it is too small for many environments with multiple intermediate CAs. Hopefully this will be resolved in the next release.

Update:
1) To edit ‘maxInMemorySize’ value for the spring service to allow a larger certificate upload, the following information was provided by VMware support:
a. Stop casa.
service vmware-casa stop

b. Edit /usr/lib/vmware-casa/casa-webapp/webapps/casa/WEB-INF/casa-servlet.xml
Locate this text around line 256:
<bean id="multipartResolver" class="org.springframework.web.multipart.commons.CommonsMultipartResolver">
<property name="uploadTempDir" value="file:///${application.data?/storage}/db/tmp"/>
</bean>

Add the following within the <bean> definition:
<property name="maxInMemorySize" value="30720"/>

c. Restart casa.
service vmware-casa start

2) I have been advised that a fix for this issue will be included in the next release, although I cannot confirm that and have no information on the vROps release schedule.

I want to publicly acknowledge VMware Global Support Services (GSS) Business Critical Support (BCS) and the excellent experience working with them to diagnose and resolve this issue, as well as VMware Technical Account Manager (TAM) Services. While not cheap, these service offerings provide high value.

* Update 3/3/2017
vRealize Operations 6.5 was released on 3/2/2017. The 8192 bit certificate size limit is still reflected in the documentation .

VCSA – Moving Behind A Firewall

VCSA – Moving Behind A Firewall

We have a situation where we have to deploy VCSA, then move it to a secured network behind a firewall.

I have not (yet) had success changing the hostname, including the FQDN, so make sure you can create proper DNS entries on both sides of the firewall and can do the initial deployment with the intended final FQDN. Our VCSA’s FQDN is of the format “vcenterserver.securednet.companynet.org” so this was achievable (the segmented domain is a subdomain of the company domain)

Deploy the appliance, then access the VCSA console and press F2, then login as root. At this point you may also need to Edit the VM Settings and change it to the new network (behind the firewall, in my case).
– Configure Management Network
– Set IP Configuration, IPv6 Configuration, DNS Configuration, Custom DNS Suffixes as required.
– (Esc) Exit, (Y) Yes Apply changes and restart management network

At this point, I got the following 2 screens. IPv6 is understandable; we disabled it. The DNS error appears even though we have validated the DNS entries/servers.

Now, to access the shell console. Login as root.

Command> shell.set --enabled True
Command> shell
# vi /etc/sysconfig/networking/devices/ifcfg-eth0

Quick vi reminder: hit (Insert) to edit the line, then (Esc), :wq! to save and exit.

Make sure all the IP information is correct in this file. I had to update the broadcast & gateway for sure. Then,

# vi /etc/hosts

Fix the entry here to reflect the new IP.

Validate settings:

# ifconfig eth0
# route -n

At this point, we were missing the default gateway in the route table, even thought it was defined in the ifcfg-eth0 file and in the IP Configuration screen of the console. Also the Broadcast address still reflects the old network, but this is less of a problem.

To temporarily fix the default gateway, do # route add default gateway XXX.YYY.ZZZ.1, using the correct gateway IP obviously.

To permanently resolve the gateway issue, log into the web console (https://vcenterserver:443/vsphere-client/) as Administrator@vsphere.local or whatever you defined as your SSO domain.
– Go to System Configuration under Administration in the main panel
– Nodes (in left column, under System Configuration)
– Select your vCenter server under Nodes
– “Edit” in the upper right corner of the main panel
– Expand nic0 & define Default gateway
– Ok
– Reboot to validate configuration persists

Cleaning up ARP tables – VMware ESXi 5.1

Cleaning up ARP tables – VMware ESXi 5.1

We had a recent situation where a virtual appliance (Avamar proxy) was mistakenly deployed configured with the IP address of the gateway. This caused the hosts to be unreachable from vCenter. After finding the offending VM and shutting it down from the console, the ARP tables either had to be cleaned up (you could also allow them to time out but that could take up to 20 minutes)

ESXi 5.1
esxcli network ip neighbor list
vsish -e set /net/tcpip/v4/neighbor del IPADDRESS

New functionality added in 5.5 to remove ARP entries through esxcli

Find a VM by MAC in vCenter with PowerCLI
Get-vm | Select Name, @{N=“Network“;E={$_ | Get-networkAdapter | ? {$_.macaddress -eq“00:50:56:A4:22:F4“}}} |Where {$_.Network-ne “”}

Also you may need to clean up your Windows vCenter server from the command prompt
arp -a
arp -d IPADDRESS

Or,
netsh interface ip delete arpcache
Also a good idea to:
ipconfig /flushdns

World’s Greatest… Radio Scanners

World’s Greatest… Radio Scanners

Article on Gizmodo.com by Alex Roy, famous Gumballer from Team Polizei, reviewing radio scanners to help avoid the highway patrol.

Personally, I have all the THP frequencies programmed in my handheld scanner, but I never pick up anything from them. I’m using frequencies from an old list however, so they may have gone to a digital trunked system. I still find a Valentine One to be the single most important driving aid. That investment alone has saved me thousands.

Using Gmail as a Spam Filter

Using Gmail as a Spam Filter

Article on MBoffin.com: Using Gmail as a Spam Filter

I set up an adaptation of this last night for my domain here. So far, after 12 hours, I’ve received 73 messages, 61 of which were SPAM, and Gmail caught 57 of those. It also has claimed 3 false positives, which is a problem.

There are some downfalls to using Gmail like this. Mainly, if Gmail flags a message as Spam and it isn’t (a false positive), when you move the message back to your Inbox it will not forward the message to your mail account. The message is stuck in your Gmail account, with no way to move it to your mail server.

On the up side, it has cleaned my mailbox up enough that I can use the push service from Verizon to get email on my Treo without getting floods of Spam. I’m going to continue to evaluate this, and I’ll keep you posted.

*Update* After one week, I must say I’m pretty impressed. After getting three false positives the first day, I was worried that may be a problem. However, Gmail hasn’t nipped any more legit messages since then, so I’ll just assume that was an anomally. 511 messages have been flagged as SPAM the past week, and Gmail is letting around 5 messages per day slip through. That’s an acceptable ratio to me. The experiment continues…

Articles by Janis Ian on the music industry and the Internet

Articles by Janis Ian on the music industry and the Internet

I found a link to a pair of articles by a singer/songwriter named Janis Ian. These are very well-written, and while already almost two years old, the login within holds up well. Basically, Janis takes the position that downloading music online creates EXPOSURE for artists, which leads to more CD sales. She also pretty much blames the state of the industry on the RIAA itself.

THE INTERNET DEBACLE – AN ALTERNATIVE VIEW
FALLOUT – a follow up to The Internet Debacle

I still feel that the RIAA and MPAA continue to set themselves up for massive destruction as they continue to accuse all their customers of being crooks, cheats, and theives.

2003 e-mail stats

2003 e-mail stats

Courtesy of Steve’s Sublimemail service, here are the email stats for my domain in 2003:

total messages: 70,733
flagged spam: 39,293
percentage spam: 56%
avg daily spam: 108
false positives: 0

I’d say Steve is catching about 75% of my spam before it ever hits my inbox. Of course, he reminds me, it would be higher if I’d submit the messages that make it past the filters. So, if the Mozilla MailNews developers would get on the stick and implement the Redirect option, it would make it a lot easier.

IBM Reinvents e-mail

IBM Reinvents e-mail

Reinventing Email :: Collaborative User Experience Group :: IBM Research

“The Collaborative User Experience (CUE) team in IBM Research has spent nearly a decade studying email. Not only has email become one of the most pervasive and successful collaborative tools available, it has also become a key component of IBM’s Lotus Software offerings. In many ways, email can be seen as a victim of its own success – users increasingly suffer from overload and interruptions as well as use email in a manner for which it was not intended.”

Slow to catch the IMAP bandwagon

Slow to catch the IMAP bandwagon

So, just tonight have I discovered why IMAP is so superior to POP3. While I’ve used it a bit before, I’ve never really had the need to get my mail on multiple machines all the same before. For the past few months I’ve needed to read @miklm.com mail from home, work, and on my laptop, so I resorted to webmail for the task. Now that I’m using Mozilla Thunderbird, I’ve discovered that IMAP will allow me to keep folders, filters, and such all the same on all accounts. I’m such an idiot for not doing this YEARS ago, when it was all the rage. Critch told me all along…

Legislature won’t vote on cable theft this year

Legislature won’t vote on cable theft this year

Legislature won’t vote on cable theft this year
By KATHY CARLSON
Staff Writer, The Tennessean

A bill pitting telecommunications and entertainment companies against some of their customers won’t come up for a vote in the General Assembly this year, its sponsors said yesterday.

Backers said the bill was needed to update state law on the theft of cable and other telecommunications services.

Opponents – many of them computer professionals and enthusiasts who mobilized via the Internet – said no new law was needed and the measure as originally written threatened privacy and civil liberties.

entire story

Does MS finally get it?

Does MS finally get it?

Q&A: Windows Server 2003 kernel guru
Looks like they are serious this time around about not making another lame-duck server product. From what I’ve seen in limited testing of Windows Server 2003 Enterprise Edition, it certainly looks like a huge improvement.

In other news, @miklm.com mail is working again, as I transferred it to the iGiles mail server. I’m still debating whether I’m better off getting one newer server to replace the two aging HP boxes, or whether to just repair regan.

Server drive failure

Server drive failure

The [primary&&only] SCSI drive in regan.miklm.net failed this afternoon, so it looks like @miklm.com mail will be offline until I get some sort of solution rigged up.

So… if your email to me bounces, just resend later, or email me @igiles.net.