In the process of building a full VMware SDDC infrastructure for a coming customer demo on private/hybrid cloud with VMware vCloud foundation having lost my existing VxRaiL lab to a another POC, I faced an issue with vRealize Automation and deployment of Software components that needs to be documented.
The infrastructure consists of a minimal vRa 7.4 and vRa IaaS Server 2016 machine deployment which is very straight forward. Everything seems to work fine until I wanted to start applying software components to some blueprints. Requests always failed when software components were added with error message “ InstallSoftwareWorkflow. Install software work item timeout “. No other error messages existed anywhere, and believe me, I checked all solution component logs.
I wont waste your time with all the effort put into troubleshooting which eventually led to a full redeployment with same results. Eventually I narrowed it down to the software component agent installed on the image template. The vRa provided PowerShell script was used to prepare the template for software components and everything went completely fine with no errors yet when I started digging, I found that one service was missing “ VMware vRealize Automation Software Service Agent bootstrap service “ which is responsible for calling software component deployment after the machine has finished customizing. Use the _debug_deployment custom property to make sure that vRa does not automatically revert the request and delete the VM.
Cleanup and re-installation of software component agent using script did not fix the issue and still the service was not being created. Maybe its a Server 2016 update or patch issue so I created an Server 2012 R2 fresh machine with exactly same results. During the script installation, I have decided to use a domain user for the service, so as a last attempt I went for a local user which should be auto created and configured by vRa.
When using the local user option I could see that the service is now created at last but using Local System not the default “Darwin” user that should be created. Turns out that since 7.3, Darwin user is NO more created and the service uses Local System although VMware documentation doesn’t seem to point this out as of yet. Testing seems to work out fine now but we still want to know why wouldn’t it work with a domain user.
After a bit more digging, I found out that my template was not domain joined which I didn’t think would be an issue, but turns out that if domainUser is to be used then the machine must be domain joined so that the service can be created successfully. When installing software components, the script did not point any errors nor did it complaint about domain membership, even when the service did not get created, no errors showed up so that was what took a day of troubleshooting, poor documentation and logging …
- If a domainUser is to be used for Software Component service installation then the template needs to be domain joined.
- When a local account default user is to be used for Software Component service installation then Local System is used and Darwin account is not created anymore.
- VMware needs to add a notification to the Software Components script installation when trying to use a domain account without the machine being domain joined.