#Migration of Shoot controlplane to another Seed

#Overall info

Migration at the end is making backup on source Seed and restore it on target Seed. In other words it is redeploying the same Shoot object with restored ETCD database.

Beside ETCD database needs to be also some secrets migrated:

- ca
- ca-front-proxy
- static-token
- ca-kubelet
- ca-metrics-server
- etcd-encryption-secret
- kube-aggregator
- kube-apiserver-basic-auth
- kube-apiserver
- service-account-key
- ssh-keypair

Other secrets can be regenerated from them.

Gardenlet deploys custom resources in the Source Seed cluster during Shoot reconciliation which are reconciled by extension controllers. The state of these controllers and any additional resources they create is independent of the gardenlet and must also be migrated to the Destination Seed. Following is a list of custom resources, and the state which is generated by them that has to be migrated.

• BackupBucket: nothing relevant for migration
• BackupEntry: nothing relevant for migration
• ControlPlane: nothing relevant for migration
• DNSProvider/DNSEntry: nothing relevant for migration
• Extensions: migration of state needs to be handled individually
• Infrastructure: terraform state
• Network: nothing relevant for migration
• OperatingSystemConfig: nothing relevant for migration
• Worker: Machine-Controller-Manager related objects: machineclasses, machinedeployments, machinesets, machines

#Migration workflow

• Starting migration
  • Migration can only be started after a Shoot cluster has been successfully created so that the status.seed field in the Shoot resource has been set.
  • The Shoot resource's field spec.seedName="new-seed" is edited to hold the name of the Destination Seed and reconciliation is automatically triggered
  • The Garden Controller Manager checks if the equality between spec.seedName and status.seed, detects that they are different and triggers migration.
• The Garden Controller Manager waits for the Destination Seed to be ready.
• Shoot's API server is stopped.
• Backup the Shoot's ETCD.
• Extension resources in the Source Seed are annotated with gardener.cloud/operation=migrate
• Scale Down the Shoot's control plane in the Source Seed.
• The gardenlet in the Destination Seed fetches the state of extension resources from the ShootState resource in the Garden cluster.
• Normal reconciliation flow is resumed in the Destination Seed. Extension resources are annotated with gardener.cloud/operation=restore to instruct the extension controllers to reconstruct their state.
• The Shoot's namespace in Source Seed is deleted.

#Migration of Shoot by editing Shoot manifest manualy

Shoot name: sts-01
Gardener project name: dev
Name of seed namespace for Shoot: Shoot--dev--sts-01
Name of source seed cluster: fsd2-0
Name of destination seed cluster: fsd2-1
Name of Garden cluster: fgd2

Migration could be initiated through editing the Shoot object from the Gardener cluster.

kubectl edit Shoot/mcm1 -n garden-dev --kubeconfig <kubeconfig of fgd2 garden cluster>

apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
metadata:
  name: sts-01
  ...
spec:
 seedName: fsd2-0 --> fsd2-1
  ...

#Migration of shoot via OSC Dashboard

The second option of how to modify a Shoot object is through the OSC Dashboard (admin permission for Gardener cluster needed).

#Migration Specification and Progress

Let's assume that some prerequisites are done and start with migration process:

#Prerequisites

• Target Seed already exists
• Credential specified in yaml of the Shoot field credentialsBindingName exists on target Seed
• Seed persists on the same Gardener cluster

1. Select proper destination Seed and confirm action by writing the Shoot name in confirmation dialog.

2. After pressing SAVE button, redeployment / migration of Shoot start.

3. After migration of Shoot, we can see and check the Shoot specification for the Shoot. The Shoot was successfully migrated.

#Validation and further checks after migration of Shoot

During the deployment of the Shoot, various objects have been created by deployer also on api cluster, therefore after migration of the Shoot, from source to target Seed, we need to validate some objects which can be related to the old Shoot and were not recreated, or stayed with wrong naming logic (objects are namespaced in Kubernetes).

Findings:

NOTE: fsd2-1 - source Seed, fsd2-0 - target Seed

• namespace - during the initial deployment of the Seed, new namespaces are not created automatically on api cluster, therefore are not deleted automatically, either.

• machines - still in old namespace

  • before migration:

• after migration:

• secrets - not deleted or recreated. Secrets (see below) created during initial deployment of the Shoot, were created in Shoot namespace on api cluster, but during the migration of the Shoot were not deleted or recreated in new namespace.

• Shoot yaml

After the migration of the Shoot between the Seeds, some information in not updated in the Shoot yaml.

  credentialsBindingName: fsd2-1-creds
  seedName: fsd2-0
  seedSelector:
    matchLabels:
      purpose: dev-fsd2-1

• check OIDC kubeconfigs after migration.