13. Restore failed Contrail node

This guide describes how to replace the failed Contrail all-in-one node (with all Contrail roles assigned) in a multi-node environment.

If your Contrail node has been crashed, follow the steps to fix the issue:

1. Remove failed node from Cassandra cluster (on working contrail node)

   1. Obtain Host-ID of the failed Cassandra node:

      nodetool status
      

   2. Remove the failed node:

      nodetool removenode <Host-ID>
      

2. Deprovision analytics, control, database, and config components of the failed node from contrail db.

   1. Obtain IP address of Contrail API endpoint (Managment VIP):

      hiera management_vip
      

      Example of system response:

      10.109.1.3
      

   2. Obtain Neutron service password:

      hiera neutron_config | grep admin_password
      

      Example of system response:

      "keystone"=>{"admin_password"=>"VerySecurePassword!"},
      

   3. Deprovision contrail-config:

      /opt/contrail/utils/provision_config_node.py \
      --api_server_ip <Managment VIP> \
      --api_server_port 8082 \
      --oper del \
      --host_name node-294.domain.tld \
      --host_ip 172.21.129.193 \
      --admin_user neutron \
      --admin_tenant_name services \
      --admin_password <Neutron password>
      

   4. Deprovision contrail-analytics:

      /opt/contrail/utils/provision_analytics_node.py \
      --api_server_ip <Managment VIP> \
      --api_server_port 8082 \
      --oper del \
      --host_name node-294.domain.tld \
      --host_ip 172.21.129.193 \
      --admin_user neutron \
      --admin_tenant_name services \
      --admin_password <Neutron password>
      

   5. Deprovision contrail-control:

      /opt/contrail/utils/provision_control.py \
      --api_server_ip <Managment VIP> \
      --api_server_port 8082 \
      --oper del \
      --host_name  node-294.domain.tld \
      --host_ip 172.21.129.193 \
      --router_asn 64512 \
      --admin_user neutron \
      --admin_tenant_name services \
      --admin_password <Neutron password>
      

   6. Deprovision contrail-database:

      /opt/contrail/utils/provision_database_node.py \
      --api_server_ip <Managment VIP> \
      --api_server_port 8082 \
      --oper del \
      --host_name node-294.domain.tld \
      --host_ip 172.21.129.193 \
      --admin_user neutron \
      --admin_tenant_name services \
      --admin_password <Neutron password>
      

3. Add a new node with Contrail roles and deploy it with Fuel