Install and Configure Avi Multi-Cluster Kubernetes Operator

Overview

The Avi Multi-Cluster Kubernetes Operator (AMKO) facilitates application deployments across multiple OpenShift/Kubernetes clusters.

AMKO

AMKO is aware of the following object types:

  • Kubernetes:
    • Ingress
    • Service type load balancer
  • OpenShift:
    • Routes
    • Service type load balancer

AMKO is a Kubernetes pod that runs in a namespace called avi-system.

Installation of AMKO is done via Helm which applies the following permissions via a Kubernetes service account for GSLBConfig and GlobalDeploymentPolicy objects:

  • CREATE
  • GET
  • LIST
  • READ
  • UPDATE

AMKO also requires permissions to read the ingress and service objects for all the member clusters. For this, a separate kubeconfig file is created with all the required permissions from all the clusters and is passed to AMKO via an OpenShift/Kubernetes secret object.

To know more, refer to the following articles:

Requirements

To know the minimum version requirements to use AMKO in Kubernetes clusters and OpenShift clusters, see Compatibility Guide.

Pre-Installation Checklist

Ensure that the following tasks are completed before commencing AMKO installation:

  • At least one OpenShift/ Kubernetes cluster

  • At least one Avi Controller

  • AMKO assumes that it has connectivity to all the member clusters’ OpenShift/Kubernetes API servers. Without this, AMKO will not be able to watch over the Kubernetes and OpenShift resources in the member clusters.

  • Before deploying AMKO, one of the clusters have to be designated as the leader and rest of the clusters as followers. AMKO has to be deployed on all clusters (wherever federation is required). This is to ensure that the leader cluster’s AMKO would federate the AMKO config objects like GSLBConfig and GlobalDeploymentPolicy objects to all follower clusters. Refer to the Federation article for more details on the specifics of federation and how to recover from a disaster recovery scenario.

  • On all clusters, create the namespace avi-system.


kubectl create ns avi-system
  • Create a kubeconfig file with the permissions to read the service and the ingress/route objects for all the member clusters. To know more refer Creating a kubeconfig file for multi cluster access. Name this file gslb-members and generate a secret with the kubeconfig file in cluster-amko as shown below:

kubectl create secret generic gslb-config-secret --from-file gslb-members -n avi-system

This has to be done for all the member clusters wherever AMKO is going to be deployed.

The permissions provided in the kubeconfig file for all the clusters must have at least the permissions to [get, list, watch] on:

  • Kubernetes ingress and service type load balancers.
  • OpenShift routes and service type load balancers.

AMKO also needs permissisons to [create, get, list, watch, update, delete] on:

  • GSLBConfig object
  • GlobalDeploymentPolicy object

Installing AMKO

Note: Only Helm version 3.0 is supported.

To install AMKO via Helm,

  1. Create the avi-system namespace:

    
     $ kubectl create ns avi-system
     

    Note: Helm version 3.8 and later is required to proceed with helm installation.

  2. Search the available charts for AMKO:
    
    $ helm show chart oci://projects.registry.vmware.com/ako/helm-charts/amko --version 1.10.1
    Pulled: projects.registry.vmware.com/ako/helm-charts/amko:1.10.1
    Digest: sha256:xyxyxxyxyx
    apiVersion: v2
    appVersion: 1.10.1
    description: A helm chart for Avi Kubernetes Operator
    name: amko
    type: application
    version: 1.10.1
     
  3. Configure the parameters using values.yaml from this repository to provide values related to the Avi configuration. To get the values.yaml for a release, run the following command:

    
     helm show values oci://projects.registry.vmware.com/ako/helm-charts/amko --version 1.10.1 > values.yaml
     
  4. To configure federation via values.yaml:
    • Set configs.federation.currentClusterIsLeader to true for the leader cluster. For all follower clusters, set this to false.
    • Set configs.federation.currentCluster to the current cluster context.
    • Add the member clusters to configs.federation.memberClusters.
  5. Install AMKO:

    
     $ helm install --generate-name oci://projects.registry.vmware.com/ako/helm-charts/amko --version 1.10.1 -f /path/to/values.yaml  --set configs.gsllbLeaderController=<leader_controller_ip> --namespace=avi-system
     
  6. Verify the installation:
    
    $ helm list -n avi-system
       
    NAME           	NAMESPACE 	REVISION	UPDATED                                	STATUS  	CHART                 	APP VERSION
    amko-1598451370	avi-system	1       	2020-08-26 14:16:21.889538175 +0000 UTC	deployed	amko-1.10.1	            1.10.1
    

Configuring AMKO

AMKO is configured via two CRDs - GSLBConfig and GlobalDeploymentPolicy. The Helm based installation procedure will automatically create these two in the specified namespaces.

During AMKO installation, set the required parameters via values.yaml. These parameters are translated to the GSLBConfig and the GlobalDeploymentPolicy objects.

Parameter Description Default
configs.controllerVersion Leader site’s controller version 20.1.4
configs.federation.image.repository Image repository for AMKO federator projects.registry.vmware.com/ako/amko-federator
configs.federation.image.pullPolicy Image pull policy for AMKO federator IfNotPresent
configs.federation.currentCluster Current cluster context (required) Not Applicable
configs.federation.currentClusterIsLeader Set to _true_ if current cluster is the leader (required) False
configs.gslbLeaderController GSLB leader site URL Not Applicable
configs.federation.memberClusters Member clusters on which federation should be done Not Applicable
gslbLeaderCredentials.username GSLB leader controller username admin
gslbLeaderCredentials.password GSLB leader controller password
configs.memberClusters.clusterContext Kubernetes member cluster context for GSLB cluster1-admin and cluster2-admin
configs.refreshInterval Interval in seconds after which the in-memory caches are refreshed 1800 seconds
configs.logLevel Log level to be used by AMKO to print the type of logs. The supported values are `INFO`, `DEBUG`, `WARN` and `ERROR` `INFO`
configs.useCustomGlobalFqdn Select the GslbService FQDN mode for AMKO. If set to true, AMKO observes the HostRules to look for mapping between local and global FQDNs False
gdpConfig.appSelector.label{.key,.value} Selection criteria for applications, label key and value are provided Nil
configs.domainNames Domain names supported in the GSLB configuration foo.com
gdpConfig.appSelector.label{.key,.value} Selection criteria for applications, label key and value are provided Not Applicable
gdpConfig.namespaceSelector.label{.key,.value} Selection criteria for namespaces, label key and value are provided Not Applicable
gdpConfig.matchClusters List of clusters (names must match the names in configs.memberClusters) from where the objects will be selected Not Applicable
gdpConfig.trafficSplit List of weights for clusters (names must match the names in configs.memberClusters), each weight must range from 1 to 20 Not Applicable
gdpConfig.ttl Time To Live, ranges from 1-86400 seconds Nil
gdpConfig.healthMonitorRefs List of health monitor references to be applied on all GSLB Services Nil
gdpConfig.healthMonitorTemplate Health monitor template using which a user can customize the health monitor settings such as Client Request Header and Response Code Nil
gdpConfig.sitePersistenceRef Reference for a federated application persistence profile created on the Avi Controller Nil
gdpConfig.poolAlgorithmSettings Pool algorithm settings to be used by the GslbServices for traffic distribution across pool members.
For more information, click here.
GSLB_ALGORITHM_ROUND_ROBIN
gdpConfig.downResponse Type of response to the client query when the GSLB service is DOWN Nil

With the AMKO installation based on the parameters configured above, Helm creates the objects GSLBConfig and GlobalDeploymentPolicy.
If appSelector and namespaceSelector were not specified via the Helm installation (values.yaml), the respective fields will be empty in the resulting GlobalDeploymentPolicy object.
This means that by default, no objects are selected.
You can edit this GlobalDeploymentPolicy object and add the required criteria for app or cluster selection.

To know more about the CRDs used to configure the GSLB services in the GSLB leader site and override specific GslbService properties , click here.

Editing the Runtime Parameters of AMKO

The GDP object can be edited at runtime to change the application selection parameters, traffic split and the applicable clusters. AMKO will recognize these changes and will update the GSLBServices accordingly.

Changing only logLevel is permissible at runtime via an edit of the GSLBConfig. For all other changes to the GSLBConfig, the AMKO pod has to be restarted.

Choosing a Mode of GslbService FQDN

There can be different requirements to either use local FQDNs as the GslbService FQDNs, or use a different FQDN as the Global FQDN. Refer to Deriving GSLB Service FQDNs to choose a mode fit for the use-case and enable accordingly.

GSLB Service Properties

Certain GSLB Service properties can be set and modified at runtime. If these are set through the GDP object, they are applied to all the Gslb Services. To override specific properties, use a GSLBHostRule object for a GslbService.

Properties Configured Via
TTL GDP, GSLBHostRule
Site Persistence GDP, GSLBHostRule
Custom Health Monitors GDP, GSLBHostRule
Third Party Members GSLBHostRule
Traffic Split GDP, GSLBHostRule
Pool Algorithm Settings GDP, GSLBHostRule
Down Response GDP, GSLBHostRule
Public IP Address GSLBHostRule

To set the properties, follow the steps provided in the Deriving GSLB Service FQDNs article.

Uninstall AMKO using Helm

To uninstall AMKO, use the command:


helm uninstall -n avi-system <amko-release-name>

To remove the already created GSLB services, remove the GDP object first. This will remove all the GSLB services selected via the GDP object.


kubectl delete gdp -n avi-system global-gdp

Also, delete the avi-system namespace:


kubectl delete ns avi-system

Installing AMKO Offline Using Helm

Pre-requisites for Installation

  • The Docker image downloaded from the Avi Portal
  • A private container registry to upload the AMKO Docker image
  • Helm version 3.0 or higher installed

Installing AMKO

To install AMKO offline using Helm,

  1. Create the GSLB members with the member clusters configuration and generate a secret with the kubeconfig file in cluster-amko as shown below:
    
     kubectl create secret generic gslb-config-secret --from-file gslb-members -n avi-system 
     
  2. Extract the .tar file to get the AMKO installation directory with the helm and docker images.
    
     tar -zxvf amko_cpr_sample.tar.gz
     amko/
     amko/install_docs.txt
     amko/amko-1.10.1-docker.tar.gz
     amko/amko-1.10.1-helm.tgz
     amko/amko-federator-1.10.1-docker.tar.gz
    
  3. Change the working directory to this path: cd amko/.

  4. Load the docker image in one of your machines.
    
     sudo docker load < amko-1.10.1-docker.tar.gz
     
  5. Push the docker image to your private registry. For more information, click here.

  6. Extract the AMKO Helm package. This will create a sub-directory amko/amko which contains the Helm charts for AMKO (amko/chart.yaml crds templates values.yaml).

  7. Update the helm values.yaml with the required AMKO configuration (Controller IP/credentials, docker registry information etc).

  8. Create the namespace avi-system on the OpenShift/Kubernetes cluster.
    
      kubectl create namespace avi-system
      
  9. Install AMKO using the updated helm charts.
    
     helm install ./amko --generate-name --namespace=avi-system
     

Uninstall AMKO via Helm

To uninstall AMKO using Helm,



helm uninstall -n avi-system <amko-release-name>

To remove the GSLB services created, remove the GDP object first.
This will remove all the GSLB services selected via the GDP object.


kubectl delete gdp -n avi-system global-gdp

Delete the avi-system namespace:


kubectl delete ns avi-system

Upgrading AMKO Using Helm

To upgrade AMKO from an older release,

  1. Update the helm repo with the following command:
    
    helm repo update amko
    
  2. Helm does not upgrade the CRDs during a release upgrade. Before you upgrade a release, run the following command to download and upgrade the CRDs:
    
    helm template oci://projects.registry.vmware.com/ako/helm-charts/amko --version 1.10.1 --include-crds --output-dir <output_dir>
    

    This will save the helm files to an output directory which will contain the CRDs corresponding to the AMKO version.

  3. Install the CRDs using:
    
     kubectl apply -f <output_dir>/amko/crds/
    
  4. Get the values.yaml for the AMKO version 1.10.1 and edit the values as per the requirement.
    
    helm show values oci://projects.registry.vmware.com/ako/helm-charts/amko --version 1.10.1 > values.yaml
    
  5. Upgrade the helm chart:
    
    helm upgrade amko-1598451370 oci://projects.registry.vmware.com/ako/helm-charts/amko -f /path/to/values.yaml --version 1.10.1 --set configs.gslbLeaderController=<IP or Hostname> --set gslbLeaderCredentials.password=<username> --set gslbLeaderCredentials.username=<username> --namespace=avi-system
    

AMKO Default Values

A GSLBService created by AMKO has the following default values:

GSLBService Fields Default Value
Pool load balancing algorithm Round robin
TTL Provided by the DNS service (the default value is 30s)
Health Monitor(s) Refer to the GS Health Monitors section

GSLB Service Health Monitors

AMKO creates health monitors to be used for site recovery. The object specific health monitors are as follows:

  • Service Type Load Balancer: For GSLB services serving service type load balancer, the health monitor is of System-GSLB-TCP or System-GSLB-UDP depending on the service type.

  • Insecure Ingresses and Insecure Routes: A custom health monitor of type System-GSLB-HTTP is created for each path present in the ingress/route definition and is added to the GSLB service.

  • Secure Ingresses and Secure Routes (excluding passthrough routes): A custom health monitor of type System-GSLB-HTTPS is created for each path present in the ingress/route definition and added to the GSLB service.

  • Secure Passthrough Routes: A custom health monitor of type System-GSLB-TCP is created and shared across all such GSLB Services serving passthrough routes.

  • Custom Health Monitors: One or more federated health monitor(s) can be created on Avi, and the ref(s) for them can be specified in the GSLBHostRule or GDP object. In this case, the GSLB Services will be updated to use the referred health monitors instead of the usual path-based health monitors.

Document Revision History

Date Change Summary
February 03, 2023 Updated the AMKO Installation Guide for Version 1.9.1
December 01, 2022 Updated the AMKO Installation Guide for Version 1.8.2
October 10, 2022 Updated the AMKO Installation Guide for Version 1.8.1
June 08, 2022 Updated the AMKO Installation Guide for Version 1.7.1
January 25, 2022 Updated the AMKO Installation Guide for Version 1.6.1
July 29, 2021 Updated the AMKO Installation Guide for Version 1.4.2
May 04, 2021 Updated the AMKO Installation Guide for Version 1.4.1
October 1, 2020 Published the AMKO Installation Guide for Version 1.2.1
July 21, 2020 Published the AMKO Installation Guide (Tech Preview)