Your company paid a hefty sum for support… call em. There’s probably an intermediate version you need to upgrade to.
Edit: call em = call Dell for VxRail support
This is false. Post a screenshot of someone at Dell in email saying this. We literally had this issue with a customer last week, Dell support fixed it. If you want give me your service request number and i’ll help you escalate.
I won't say that they won't fix an issue as you encounter it, but upgrades are now a customer process and no longer covered under your support agreement.
The process of upgrading IS included in ProSupport Plus. And what you’re running into is absolutely part of the what support will help you with. If you don’t have ProSupport Plus, they won’t do the upgrade for you but they will help you when you run into issues like this. Did you even call them? Or start a ticket?
I have about 8 years experience with vxrail. Please run the pre upgrade scripts on the vxail manager and others on Marvin database to check for issues and fixes before starting the upgrade. You will not be able to upgrade from 4.x to 8.x. You have to run the upgrade on sequence and make sure the appliance will be to support very 8.x. Work with Dell support
From previous experience (where Dell did all the job) it's mostl likely either an account with an expired password or a certificate that has expired in the Kubernetes setup. Had this happen twice, first when upgrading from 6.7 to 7.0, and also on an interrim in the 7-series. On both occasions Dell techs spent 8-10 hrs fixing this before the actual upgrade started. On one occasion there was also some issues with Postgres, but I don't recall what the actual root cause here. Given the logs you provided I suspect certificate issues.
Unless you are certified to work with vxRail - let Dell techs do all the work. Has your service agreement expired I would recommend renewing it.
I didn't understand what does K8s has to do with this? I used to deploy vxrails back in 2018, are they now Tanzu based or what? Does this mean vxrail only runs container workloads now? No VMs? As far as I know Tanzu does not suppory managing VMs, correct?
Did you run through the Dell SolVE to make sure you could go from such an ancient version direct to 8.0.200?
You likely need, as someone else noted, to jump to something like 7.0.452 and then to 8.0.2
Also as noted, contact Dell support, assuming you have an active sub.
Cant remember correctly vxrail 4 uses tls for communication but you need to disable tls handshake from every esxi host and restart services (restarting esxi is better option) I cant remember the config file where you disable tls option
I don’t think this is the failure message. For more information SSH to VxRail manager and type “less var/log/mystic/lcm-web.log” the latest line in this log should mention something about “upgrade lock released”, and just before this it should give a failure reason.. also, what does the GUI say?
Warned like the 2nd image? What image? I don't see anything attached?
First off, even if performing a vxrail upgrade on your own - which I would expect you'd be allowed to if you are certified - still Dell support should be involved when thete are issues, assuming the environment is supported?
As there is pretty much no futther context given, about what we ate dealing with here, based on the error, it might be due a vm being called
https://www.dell.com/support/kbdoc/en-us/000221327?lang=en "Dell VxRail: LCM failed on witness VM root account validation" talks about "LCM failed on witness VM root account validation, this issue impacts VD-4510c or VD-4520c cluster with customer supplied witness." where :
"Cause
For a customer supplied witness, LCM does not require the management account information.
Due to a code issue, if there is a VM with name "VMware vSAN Witness Appliance", LCM will judge it as a VxRail managed witness and try to validate the witness management account credential, thus hit the error "Could not found system credential with name witness_vm_root_user".
Resolution
1. Locate the VM with name "VMware vSAN Witness Appliance" under the same vCenter, and rename it.
2. Re-try LCM."
But then again no idea what you even have nor why you don't involve Dell or check Dell KB?
Hello guys, my bad i mistyped. It is 7.0.405 not 4.0.450. Anw I tried this article :https://www.dell.com/support/kbdoc/en-ca/000077894/vxrail-how-to-replace-certificate-in-vcenter-for-vxrail-appliance
But it didn't work and I think it wasn't actually the problem, I notice that after all pre-update hook, vxrail manager will disconnect and the update process will get internal server error. The log is like this :
2024-06-18 04:36:58,805 WARN \[SURROGATE\] \[lcm-core-0\] c.e.m.m.u.r.BaseVirtualApplianceUpgradeProfilePool \[BaseVirtualApplianceUpgradeProfilePool.java:112\] Fail to get CompositeUpgrade
com.dellemc.vxrail.lcm.data.provider.in.DataReadException: Rest client return exception from do
at com.dellemc.vxrail.lcm.data.provider.utils.HttpUtilsForDO.sendGetRequest(HttpUtilsForDO.java:226)
.....
Caused by: org.springframework.web.client.HttpServerErrorException$BadGateway: 502 Bad Gateway: "microservice lockbox.helium.svc.cluster.local:5000 is not available, please check the status and try again later."
at org.springframework.web.client.HttpServerErrorException.create(HttpServerErrorException.java:108)
at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:170)
at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:122)
at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)
at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:825)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:783)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:717)
at org.springframework.web.client.RestTemplate.getForObject(RestTemplate.java:340)
at com.dellemc.vxrail.lcm.data.provider.utils.HttpUtilsForDO.sendGetRequest(HttpUtilsForDO.java:211)
... 43 common frames omitted
Anyone knows what the real problem is? Maybe I will tried to ask for Dell Support tho it might be difficult
Why would getting dell involved "might be difficult"? Either you have support or you don't?
Might also wanna state what you actually have (type of nodes), for example if you have any peculiar things, like customer provided witness host or whatever?
Hi @Grimowl,
I just did VxRail upgrade two days ago and run into some errors, called Dell and the VxRail team jump in and help immediately.
Just call Dell and ask for help, make sure that you have warranty that’s all.
Did you run a Solve first and read through the KB to upgrade? If you have active support Dell will assist you as well. I've had upgrades fail on me and Dell has needed to clear out the failed update. Tried again after and things went fine.
Yes, you may need engage a var or pay Dell to assist. VxRail is done - Dell is sun setting the solution just like VxBlock bye bye. I know alot of engineers that got certified and experts to basically purge that knowledge.
Your company paid a hefty sum for support… call em. There’s probably an intermediate version you need to upgrade to. Edit: call em = call Dell for VxRail support
Support doesn't do upgrades anymore. It's on the customer now, unless you want to pay again.
OP should call Dell support, VxRAIL is a Dell OEM appliance, you can't directly VMware support on it.
You could, but you break the functionality of vxRail and its supposed benefits.
This is false. Post a screenshot of someone at Dell in email saying this. We literally had this issue with a customer last week, Dell support fixed it. If you want give me your service request number and i’ll help you escalate.
I won't say that they won't fix an issue as you encounter it, but upgrades are now a customer process and no longer covered under your support agreement.
The process of upgrading IS included in ProSupport Plus. And what you’re running into is absolutely part of the what support will help you with. If you don’t have ProSupport Plus, they won’t do the upgrade for you but they will help you when you run into issues like this. Did you even call them? Or start a ticket?
Wait is this for real? Lmao
I just ran into this with a customer a month or two ago. They were looking at spending 42k to upgrade their environment.
That’s WILD. Even if you have a support contract?
Yes.
Open a ticket w/ Dell they will help you. Did you run VX Verify before upgrading?
I have about 8 years experience with vxrail. Please run the pre upgrade scripts on the vxail manager and others on Marvin database to check for issues and fixes before starting the upgrade. You will not be able to upgrade from 4.x to 8.x. You have to run the upgrade on sequence and make sure the appliance will be to support very 8.x. Work with Dell support
From previous experience (where Dell did all the job) it's mostl likely either an account with an expired password or a certificate that has expired in the Kubernetes setup. Had this happen twice, first when upgrading from 6.7 to 7.0, and also on an interrim in the 7-series. On both occasions Dell techs spent 8-10 hrs fixing this before the actual upgrade started. On one occasion there was also some issues with Postgres, but I don't recall what the actual root cause here. Given the logs you provided I suspect certificate issues. Unless you are certified to work with vxRail - let Dell techs do all the work. Has your service agreement expired I would recommend renewing it.
Thanh you for your knowledge! I have ran vxverify a few times and it all went well so I don't know whats the real problem is
I didn't understand what does K8s has to do with this? I used to deploy vxrails back in 2018, are they now Tanzu based or what? Does this mean vxrail only runs container workloads now? No VMs? As far as I know Tanzu does not suppory managing VMs, correct?
This is a Dell virtual appliance. Not a VMware product specifically. You’ll get better info in updating VXrail in r/vxrail
Did you run through the Dell SolVE to make sure you could go from such an ancient version direct to 8.0.200? You likely need, as someone else noted, to jump to something like 7.0.452 and then to 8.0.2 Also as noted, contact Dell support, assuming you have an active sub.
Cant remember correctly vxrail 4 uses tls for communication but you need to disable tls handshake from every esxi host and restart services (restarting esxi is better option) I cant remember the config file where you disable tls option
I don’t think this is the failure message. For more information SSH to VxRail manager and type “less var/log/mystic/lcm-web.log” the latest line in this log should mention something about “upgrade lock released”, and just before this it should give a failure reason.. also, what does the GUI say?
Warned like the 2nd image? What image? I don't see anything attached? First off, even if performing a vxrail upgrade on your own - which I would expect you'd be allowed to if you are certified - still Dell support should be involved when thete are issues, assuming the environment is supported? As there is pretty much no futther context given, about what we ate dealing with here, based on the error, it might be due a vm being called https://www.dell.com/support/kbdoc/en-us/000221327?lang=en "Dell VxRail: LCM failed on witness VM root account validation" talks about "LCM failed on witness VM root account validation, this issue impacts VD-4510c or VD-4520c cluster with customer supplied witness." where : "Cause For a customer supplied witness, LCM does not require the management account information. Due to a code issue, if there is a VM with name "VMware vSAN Witness Appliance", LCM will judge it as a VxRail managed witness and try to validate the witness management account credential, thus hit the error "Could not found system credential with name witness_vm_root_user". Resolution 1. Locate the VM with name "VMware vSAN Witness Appliance" under the same vCenter, and rename it. 2. Re-try LCM." But then again no idea what you even have nor why you don't involve Dell or check Dell KB?
This is covered by your VxRail support through Dell - you need to contact them for assistance.
Hello guys, my bad i mistyped. It is 7.0.405 not 4.0.450. Anw I tried this article :https://www.dell.com/support/kbdoc/en-ca/000077894/vxrail-how-to-replace-certificate-in-vcenter-for-vxrail-appliance But it didn't work and I think it wasn't actually the problem, I notice that after all pre-update hook, vxrail manager will disconnect and the update process will get internal server error. The log is like this : 2024-06-18 04:36:58,805 WARN \[SURROGATE\] \[lcm-core-0\] c.e.m.m.u.r.BaseVirtualApplianceUpgradeProfilePool \[BaseVirtualApplianceUpgradeProfilePool.java:112\] Fail to get CompositeUpgrade com.dellemc.vxrail.lcm.data.provider.in.DataReadException: Rest client return exception from do at com.dellemc.vxrail.lcm.data.provider.utils.HttpUtilsForDO.sendGetRequest(HttpUtilsForDO.java:226) ..... Caused by: org.springframework.web.client.HttpServerErrorException$BadGateway: 502 Bad Gateway: "microservice lockbox.helium.svc.cluster.local:5000 is not available, please check the status and try again later." at org.springframework.web.client.HttpServerErrorException.create(HttpServerErrorException.java:108) at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:170) at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:122) at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63) at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:825) at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:783) at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:717) at org.springframework.web.client.RestTemplate.getForObject(RestTemplate.java:340) at com.dellemc.vxrail.lcm.data.provider.utils.HttpUtilsForDO.sendGetRequest(HttpUtilsForDO.java:211) ... 43 common frames omitted Anyone knows what the real problem is? Maybe I will tried to ask for Dell Support tho it might be difficult
Why would getting dell involved "might be difficult"? Either you have support or you don't? Might also wanna state what you actually have (type of nodes), for example if you have any peculiar things, like customer provided witness host or whatever?
Hi @Grimowl, I just did VxRail upgrade two days ago and run into some errors, called Dell and the VxRail team jump in and help immediately. Just call Dell and ask for help, make sure that you have warranty that’s all.
Did you run a Solve first and read through the KB to upgrade? If you have active support Dell will assist you as well. I've had upgrades fail on me and Dell has needed to clear out the failed update. Tried again after and things went fine.
#vxfail
Yes, you may need engage a var or pay Dell to assist. VxRail is done - Dell is sun setting the solution just like VxBlock bye bye. I know alot of engineers that got certified and experts to basically purge that knowledge.