Shutting Down an AHV Cluster for Maintenance or Relocation

Description

You might need to shut down an AHV cluster to perform maintenance or tasks such as relocating hardware.
To shut down all hosts in an AHV cluster, perform the following steps.
Note: Upgrade to the most recent version of NCC before proceeding with the following steps.

Solution

Shutting down an AHV cluster​

To shut down all hosts in an AHV cluster, perform the following:

1. Prior to the scheduled shutdown, SSH to a Controller VM and run ncc health_checks run_all.  If there are any errors or failures, contact Nutanix Support.
2. Shut down all the user VMs in the Nutanix cluster.
3. Stop all AFS cluster VMs if applicable
4. Stop the Nutanix cluster.
5. Shut down each node in the cluster.
6. After completing maintenance or other tasks, power on the nodes and start the cluster.

Shutting down guest VMs​

Guest VMs can be powered off in 3 ways:

• Power off the guest VM from within the guest OS.
• From Prism, go to VM > Table. 
  • Select each guest VM and clcik the Power Off Actions.
• If there are a large number of VMs to shut down, perform the following tasks to power them off by using aCLI:
  • SSH into any Controller VM in the cluster.
  • Shut down all non Controller VMs.
    • nutanix@CVM$ for i in `acli vm.list power_state=on | awk '{print $1}' | grep -v ^VM$` ; do acli vm.shutdown $i ; done
  • Confirm that all VMs are powered off:
    •  vm.list power_state=on
  • If any VMs are on, consider powering them off through the guest OS.  To force power-off through AHV use the following:
    •  vm.off

 Shutting down AFS (file server) VMs

• Please see KB-4429 for version specific details.

(Data Protection) Ensure there are no on-going replications if running Async DR Protection Domains:

• For Async DR: Ensure no schedules or replications are running by executing the following command from any Controller VM. If there are on-going replications, wait until they are complete before proceeding.
  nutanix@cvm$ ncli pd ls-repl-status

Stopping the Nutanix Cluster

Log on to any Controller VM using SSH with the Nutanix credentials and run the following command to stop the Nutanix cluster:

• nutanix@CVM$ cluster stop

Confirm using that the command has stopped the services successfully before continuing.

Shutting down the Controller VMs

Perform the following on each AHV host.

1. virsh list
2. virsh shutdown

Shutting down each node in the cluster

Perform the following steps for each node in the cluster you have stopped.

1. Log into the IPMI web console on each node.
2. Under Remote Control > Power Control, select Power Off Server - Orderly Shutdown to gracefully shut down the node.

      3. Confirm that the AHV hosts are no longer powered on or pingable.

Powering on the nodes and cluster after a shutdown

Perform these steps for each node in the cluster.

1. If the host is powered off, turn it on by pressing the power button on the front of the block for the host.
2. ​Log in to each host's IPMI interface.  Confirm that the host is powered on. 

Start the cluster

1. All Controller VMs start automatically after the node powers on.  Wait approximately 5 minutes after the last node is powered on to allow services to begin.
2. Log on to any one Controller VM in the cluster with SSH using Nutanix credentials.  
3. Start the Nutanix cluster by issuing the following command: 
   nutanix@cvm$ cluster start
4. Confirm that the cluster services are running:
   nutanix@cvm$ cluster status

5. Power on the guest VMs.  
   • From the Prism web console, navigate to Home > VM and select the Table view.  
6. If a large number of VMs need to be powered on, type the following aCLI command to power on the VMs:
   • nutanix@cvm$ for i in `acli vm.list power_state=off | awk '{print $1}' | grep -v ^VM$` ; do acli vm.on $i; done​
7. Confirm that all VMs are powered on.