Contents

AKS - Part 4 - Create AKS - Azure CLI

Before start it

Before creating a new AKS you might like to know the things you can change after creating the cluster and the things you can’t.

CAN

  • IP authorized to access Kubernetes API
  • Attach and detach Azure Container Register
  • Enable or disable cluster autoscaler
  • Add or Remove pools
  • When using Standard Load Balancer you can add or remove outbound IPs and ports
  • Integrate an existing cluster with AD and/or the integration credentials
  • AKS Service Principal
  • Rotate Kubernetes API certificates
  • Upgrade Kubernetes version
  • Enable or Disable add-on
  • SSH keys when using VirtualMachineScaleSets

CAN NOT

  • AKS Subscription or Resource Group
  • SSH keys when using AvailabilitySet
  • Enable or Disable RBAc
  • Kubernetes API DNS and IP
  • Docker bridge, Pod and Service CIDR
  • Integrate managed identity
  • Enable private cluster
  • Load Balancer SKU
  • Network Plugin
  • Network Policy
  • VM Size of an existing nodepool
  • Enable or Disable paid SLA

If you need to change any of those listed above you have to delete your AKS and create a new one

How to create a new AKS

Azure offers you different ways to create a resource and there are also third-part tools you can use.

Tool Pro Cons
Azure Portal Easy to use Pretty hard to automate it; Low customization
Azure CLI Full customization and automation Learning Curve
Terraform Full customization and automation Learning Curve

I rather use terraform since I can use the same tool later to work with AWS or any other cloud provider

Some decisions to make

The very first decision you have to make is related to the Network plugin and the Load Balancer.

  • Load Balancer

If you need to customize outbound IPs and fine tune your outbound traffic you must use a Standard Load Balancer. The Basic Load Balancer does not support outbound IP configuration.

  • Network Plugin

If you plan to use Azure Network Policy you must use Azure CNI as Network Plugin

  • Cluster Auto Scaler

If you enable cluster Auto Scaler please mind that the min-counter and the max-couter need to have valid values based on the number of nodes your cluster type supports.

Important

Docker bridge address, pod CIDR and Service CIDR can not overlap each other

Azure CLI

In order to understand the AZ CLI commands and figure out what else can you do with it just check the Azure CLI doc page

Steps to follow

  1. Export some Environment variables
  2. Create VNET Resource Group
  3. Create VNET and SUBNET
  4. Create AKS Resource Group
  5. Create AKS Service Principal
  6. Use the output to export some other env variables
  7. Get SUBNET ID
  8. Add AKS SP Owner of the VNET RG
  9. Get Kubernetes Credentials

Commands to execute

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
set -x -g SUBSCRIPTION "..."            # REPLACE IT
set -x -g REGION "eastus"

set -x -g ACR_NANE "rainbowcontainer"   # REPLACE IT
set -x -g AKS_SP_ACCOUNT "sp-aks"
set -x -g AKS_RG "rg-aks-celebrate"
set -x -g AKS_NAME "aks-celebrate"

set -x -g VNET_NAME "vnet-aks"
set -x -g VNET_RG "rg-vnet-aks"
set -x -g SUBNET_NAME "subnet-aks"

# Create VNET Resource Groups
az group create \
  --name $VNET_RG \
  --location $REGION \
  --subscription $SUBSCRIPTION

# Create VNET and SUBNET
az network vnet create \
  --name $VNET_NAME \
  --resource-group $VNET_RG \
  --address-prefixes 192.0.0.0/8 \
  --subnet-name $SUBNET_NAME \
  --subnet-prefixes 192.10.0.0/16 \
  --location $REGION \
  --subscription $SUBSCRIPTION

# Create AKS Resource Group
az group create \
  --location $REGION \
  --name $AKS_RG \
  --subscription $SUBSCRIPTION

# Create AKS Service Principal
az ad sp create-for-rbac \
  --skip-assignment \
  -n $AKS_SP_ACCOUNT

# Use the output to export some other env variables

set -g -x AKS_SP_ID "..."             # REPLACE IT
set -g -x AKS_SP_PASS "..."           # REPLACE IT

# Add AKS SP Owner of the VNET RG
az role assignment create \
  --role "Owner" \
  --assignee $AKS_SP_ID \
  --resource-group $VNET_RG

# Get SUBNET ID

set -x -g VNET_ID (az network vnet show --resource-group $VNET_RG --name $VNET_NAME --query id -o tsv)
set -x -g SUBNET_ID (az network vnet subnet show --resource-group $VNET_RG --vnet-name $VNET_NAME --name $SUBNET_NAME --query id -o tsv)

# Create AKS
az aks create \
  --location $REGION \
  --subscription $SUBSCRIPTION \
  --resource-group $AKS_RG \
  --name $AKS_NAME \
  --ssh-key-value $HOME/.ssh/id_rsa.pub \
  --service-principal $AKS_SP_ID \
  --client-secret $AKS_SP_PASS \
  --network-plugin kubenet \
  --load-balancer-sku standard \
  --outbound-type loadBalancer \
  --vnet-subnet-id $SUBNET_ID \
  --pod-cidr 10.244.0.0/16 \
  --service-cidr 10.0.0.0/16 \
  --dns-service-ip 10.0.0.10 \
  --docker-bridge-address 172.17.0.1/16 \
  --node-vm-size Standard_B2s \
  --enable-cluster-autoscaler \
  --max-count 5 \
  --min-count 2 \
  --node-count 2 \
  --attach-acr $ACR_NAME \
  --tags 'ENV=DEV'

# Get Kubernetes Credentials
az aks get-credentials \
  -n $AKS_NAME \
  -g $AKS_RG \
  --subscription $SUBSCRIPTION \
  --admin

We created a cluster with the capability of auto scale. The minimum amount of nodes is 2 and the maximum is 5. AKS Auto scaler will scale the cluster considering the number of pods that are “evicted”.

When a node in a Kubernetes cluster is running out of memory or disk, it activates a flag signaling that it is under pressure. This blocks any new allocation in the node and starts the eviction process. At that moment, kubelet starts to reclaim resources, killing containers and declaring pods as failed until the resource usage is under the eviction threshold again.

AKS Auto Scaler does not consider CPU and memory to scale up your cluster. It will thou use hardware metrics to scale your cluster down.