Azure Kubernetes Service (AKS) cluster deployment with Terraform
Troubleshooting an AKS deployment with Terraform
Lab guide covering environment selection, prerequisites, Azure CLI checks, deployment validation, common errors, and best practices.
Troubleshooting an AKS deployment with Terraform
This guide documents the issues encountered while deploying an Azure Kubernetes Service (AKS) cluster and Azure Container Registry (ACR) with Terraform. The examples use PowerShell and Azure CLI unless marked Bash. Replace placeholder values with your own; do not publish subscription IDs, credentials, kubeconfigs, or personal account information in a blog post.
Using Bash instead of PowerShell
Azure CLI commands (az ...) and Terraform commands are the same in Bash and PowerShell. The shell-specific parts are variable assignment, environment variables, and line continuation:
| Task | PowerShell | Bash |
|---|---|---|
| Assign a variable | $subscriptionId = "..." | subscription_id="..." |
| Set environment variable | $env:ARM_SUBSCRIPTION_ID = $subscriptionId | export ARM_SUBSCRIPTION_ID="$subscription_id" |
| Continue a command on next line | Backtick: ` | Backslash: \ |
| Reference variable | $subscriptionId | $subscription_id |
For example, Bash sign-in and subscription selection:
az login --use-device-code
subscription_id="<full-subscription-guid>"
az account set --subscription "$subscription_id"
az account show --output table
export ARM_SUBSCRIPTION_ID="$subscription_id"If using Azure Cloud Shell, select Bash when opening the shell. The check-regions.sh script is Bash; run it in Bash, WSL, or Git Bash, not PowerShell. In Bash, use / in paths. In PowerShell on Windows, use Windows paths and the PowerShell forms above.
Lab note: The Terraform configuration uses AzureRM 5.9.0. Region presets for test and preprod are examples, not guarantees of current AKS, VM SKU, or quota availability. Azure offerings and quotas change; query your own subscription and target region before deploying.
What this lab deploys
- An Azure resource group
- An ACR registry
- An AKS cluster with a system-assigned managed identity
- An
AcrPullrole assignment granting the cluster's kubelet identity access to ACR - A local
kubeconfigfile written by Terraform
The default dev configuration uses one Standard_D2s_v5 node in southindia. Test and preprod default to two nodes each. These are small lab settings, not production availability recommendations. Each environment has its own configuration file and Terraform state workspace.
Select an environment
Choose an environment using both its matching Terraform workspace and its matching variable file. The default workspace is kept for the existing dev deployment; test and preprod use separate workspaces and therefore separate state.
# Dev: retain/use the existing default workspace and state.
terraform workspace select default
terraform plan -var-file=terraform.tfvars -out=dev.tfplan
terraform apply dev.tfplan
# Test: create this workspace once, then select it on later runs.
terraform workspace new test
terraform plan -var-file=test.tfvars -out=test.tfplan
terraform apply test.tfplan
# Preprod: create this workspace once, then select it on later runs.
terraform workspace new preprod
terraform plan -var-file=preprod.tfvars -out=preprod.tfplan
terraform apply preprod.tfplanIf a workspace already exists, use terraform workspace select test or terraform workspace select preprod instead of terraform workspace new. Before each plan, check terraform workspace show and make sure the selected workspace matches the environment in the variable file. Terraform includes a precondition that rejects a mismatch. The default workspace is accepted only for dev to avoid accidentally moving the existing dev state.
The current presets set dev to one node and test/preprod to two nodes. Override system_node_counts only when quota and capacity have been checked; two Standard_D2s_v5 nodes require at least 4 vCPUs, not including headroom for upgrades. The sample test.tfvars and preprod.tfvars use candidate regions from the lab's region-check script. Verify each region, supported Kubernetes version, VM size, quotas, and globally unique ACR name, then edit the matching file before planning. replacewithuniqueacrtest and replacewithuniqueacrpreprod are placeholders and must be replaced.
Terraform CLI workspaces separate state but are not a security boundary. For team or production deployments, configure a secured remote backend with separate state keys/containers and access controls for each environment.
Prerequisites
Tools
Install and verify:
az version
terraform version
kubectl version --clientUse a current Terraform CLI and Azure CLI. Install or update kubectl using your preferred package manager or Azure CLI:
az aks install-cliaz version prints installed Azure CLI component versions; terraform version prints the Terraform CLI version; kubectl version --client checks the Kubernetes client without contacting a cluster. az aks install-cli downloads kubectl and, where supported, kubelogin. Run it once per machine or when you need to update these tools. Azure CLI selects binaries for the operating system where it is running.
Azure identity and permissions
Sign in with the intended Microsoft Entra ID identity and select the target subscription:
az login --use-device-code
$subscriptionId = "<full-subscription-guid>"
az account set --subscription $subscriptionId
az account show --output tableaz login --use-device-code starts interactive sign-in and prints a one-time code and URL; complete sign-in in a browser using the intended user. This is useful when the normal browser handoff does not let you choose the correct account. az account set selects the subscription for later Azure CLI commands. az account show displays the active subscription and tenant; confirm these before running Terraform. The Azure CLI account is separate from the identity passed as --assignee in a role assignment.
az role assignment create grants a role to an existing identity; it does not create that identity or set its password. Confirm the account's actual sign-in name with the tenant administrator if sign-in selects the wrong account.
The deployment identity needs permission to create the resources in this lab. In particular, Terraform creates an AcrPull role assignment for the AKS kubelet identity. A plain Contributor assignment does not include Microsoft.Authorization/roleAssignments/write; an administrator must grant the deployer Owner or User Access Administrator, or otherwise provide that specific permission at the appropriate scope.
If an administrator needs to grant Contributor access to an existing user, use the full subscription GUID:
az role assignment create `
--assignee "<user-upn-or-object-id>" `
--role "Contributor" `
--scope "/subscriptions/$subscriptionId"This assignment alone does not grant permission to create role assignments. The equivalent Bash command uses a backslash to continue lines:
az role assignment create \
--assignee "<user-upn-or-object-id>" \
--role "Contributor" \
--scope "/subscriptions/$subscription_id"Terraform inputs and state
Review the selected environment's .tfvars file before applying. Verify the resource group, region, cluster name, Kubernetes version, selected node count, VM size, and globally unique ACR name. Never commit credentials or sensitive Terraform state. The generated kubeconfig is ignored by Git and grants access to the cluster; keep it private.
Initialize the providers and inspect the lock file:
terraform init
terraform providersterraform init initializes the working directory, downloads the providers selected by version constraints and the lock file, and configures the backend. terraform providers shows which providers the configuration requires and where they are used.
When intentionally upgrading a provider, update the version constraint, then run terraform init -upgrade and review the .terraform.lock.hcl changes. Major provider upgrades can require configuration migrations.
Pre-deployment checks
Confirm the selected subscription and region
az account show --query "{name:name,id:id,tenantId:tenantId}" --output table
az account list-locations --query "[?name=='southindia'].{name:name,displayName:displayName}" --output tableaz account show confirms the active subscription name, full ID, and tenant ID. --query uses JMESPath to select fields and --output table makes the result easier to read. az account list-locations lists regions available to the selected subscription; the query filters the result to South India. To check another region, replace southindia. Seeing a region here does not confirm that AKS, a particular VM size, or sufficient quota is available there. Make sure the selected subscription is the one with the quota and access you expect.
Check supported AKS versions
Query the selected environment's target region instead of assuming a Kubernetes patch release is available. For example:
az aks get-versions --location southindia --output tableaz aks get-versions asks Azure for AKS Kubernetes versions offered in the specified region. --location must match the environment being deployed. The table shows available minor versions; use --output json to inspect patch versions and upgrade paths. A version available in one region may not be available in another, and the list can change over time.
To see the full patch and upgrade data:
az aks get-versions --location southindia --output jsonThe JSON response includes each minor release and its patchVersions map. Choose an exact patch present in that map for kubernetes_version; do not infer a patch from the newest upstream Kubernetes release.
Set that environment's kubernetes_version to a version listed for its region. In the original South India deployment, 1.36.5 was rejected; the query returned 1.36.4, so the dev input was changed to 1.36.4. Recheck each region's list when reproducing the lab.
Check VM size availability and quota
Check the target size in the selected region:
az vm list-skus `
--location southindia `
--resource-type virtualMachines `
--size Standard_D2s_v5 `
--all `
--output tableaz vm list-skus asks Azure Resource Manager which VM SKUs are offered in the region for this subscription. --resource-type virtualMachines limits the query to VM sizes. Use the exact value of node_vm_size from the selected .tfvars file with --size; --all requests all matching sizes. Inspect the entire restrictions field: NotAvailableForSubscription means that size is not usable by this subscription, even if the SKU exists in Azure. No restriction is a useful indicator, not a guarantee that AKS creation or capacity will succeed. If Azure returns an allowed-size list during cluster creation, choose a permitted size from that list.
For Bash, use backslashes instead of PowerShell backticks:
az vm list-skus \
--location southindia \
--resource-type virtualMachines \
--size Standard_D2s_v5 \
--all \
--output tableReview regional and VM-family quota before deploying each environment:
az vm list-usage --location southindia --output tableaz vm list-usage reports usage and limits for quotas in the selected region, including regional and VM-family vCPU quotas. Check both: the effective capacity is constrained by all applicable quotas, not just the general regional core limit. Compare remaining quota (limit - used) with the total vCPUs needed by all requested nodes. Allow headroom for cluster operations and upgrades, which may temporarily use surge capacity. If quota is insufficient, request a quota increase or use a smaller dev/test configuration only when the available quota supports it.
Check resource provider registration
This lab uses resource_provider_registrations = "none" to avoid AzureRM trying to register its broad provider list automatically. That setting means required resource providers must already be registered. Check the providers you need and ask a subscription administrator to register any that are missing:
az provider show --namespace Microsoft.ContainerService --query registrationState --output tsv
az provider show --namespace Microsoft.ContainerRegistry --query registrationState --output tsv
az provider show --namespace Microsoft.Authorization --query registrationState --output tsvaz provider show reads the registration state of one Azure resource provider namespace in the active subscription. --query registrationState --output tsv prints just the state (for example Registered, NotRegistered, or Registering). Check the namespaces used by the Terraform resources. Provider registration can take time and may require an administrator.
For example, an authorized administrator can register a required provider and wait for completion:
az provider register --namespace Microsoft.ContainerService --waitaz provider register starts registration for that namespace in the active subscription. --wait keeps the command running until registration completes. Register only providers required by the deployment and only with authorization to register them. Verify the result again with az provider show.
Register only providers needed by the resources being deployed. A Microsoft.MixedReality message during AzureRM 3.x initialization does not mean this AKS lab uses Mixed Reality; it can occur because that older provider attempted to inspect/register its predefined provider list.
Validate and deploy
Run local checks before making Azure changes. For an environment, first select its workspace and matching variable file as described above:
terraform fmt -check -recursive
terraform validateProvide the subscription ID required by AzureRM 4.x and later, then create and review a plan:
$env:ARM_SUBSCRIPTION_ID = $subscriptionId
terraform plan -var-file=terraform.tfvars -out=dev.tfplanIn Bash, export the variable instead; the Terraform commands themselves are unchanged:
export ARM_SUBSCRIPTION_ID="$subscription_id"
terraform plan -var-file=terraform.tfvars -out=dev.tfplanARM_SUBSCRIPTION_ID supplies the subscription ID required by AzureRM 4.x and later. Set it to the same full GUID selected with az account set. The -var-file argument loads that environment's inputs; -out saves the plan so the reviewed plan, rather than a newly calculated one, is applied.
Apply only after reviewing the plan:
terraform apply dev.tfplanterraform plan -out=dev.tfplan compares the selected configuration and state, then saves the proposed changes to a plan file. Review the proposed resources and changes before applying. terraform apply dev.tfplan applies that exact saved plan; do not switch workspace or edit inputs between these commands. The plan file can contain sensitive values, so keep it out of source control and delete it securely when no longer needed.
After deployment, verify Azure reports a succeeded cluster and retrieve fresh credentials:
az aks show `
--resource-group "<resource-group>" `
--name "<cluster-name>" `
--query provisioningState `
--output tsv
az aks get-credentials `
--resource-group "<resource-group>" `
--name "<cluster-name>" `
--overwrite-existing
kubectl get nodes
kubectl get pods --all-namespacesaz aks show reads the cluster by resource group and name; the query prints its provisioning state, which should be Succeeded. az aks get-credentials retrieves connection information and merges it into the local kubeconfig. --overwrite-existing replaces a same-named existing context, which is useful after recreating a cluster. Keep kubeconfig private. kubectl get nodes verifies API connectivity and node registration; kubectl get pods --all-namespaces checks cluster system and workload pods. If these fail, confirm the active subscription, credentials, network access, and local system clock before changing TLS settings.
The equivalent Bash Azure CLI commands use \ for line continuation:
az aks show \
--resource-group "<resource-group>" \
--name "<cluster-name>" \
--query provisioningState \
--output tsv
az aks get-credentials \
--resource-group "<resource-group>" \
--name "<cluster-name>" \
--overwrite-existing
kubectl get nodes
kubectl get pods --all-namespacesIssues encountered and resolutions
1. AzureRM version and provider configuration
Symptom: The configuration pinned AzureRM 3.62.1; its old resource provider registration behavior attempted to inspect a broad set of providers, and the configuration's original required_providers block was not nested inside terraform.
Resolution: Nest required_providers inside terraform, upgrade to the AzureRM version selected for the lab, and refresh the lock file. AzureRM 5.x removed skip_provider_registration; resource_provider_registrations = "none" is the current setting when provider registration is managed outside Terraform.
terraform {
required_providers {
azurerm = {
source = "hashicorp/azurerm"
version = "5.9.0"
}
}
}
provider "azurerm" {
features {}
resource_provider_registrations = "none"
}Validation: terraform init -upgrade, then terraform validate. Review the provider's upgrade guides when crossing major versions.
2. AzureRM 5.x AKS schema changes
Symptom: Validation reported a missing node_provisioning_profile block and rejected enable_auto_scaling.
Resolution: Add the required profile with mode = "Manual" for this manually managed node-pool lab, and use the current auto_scaling_enabled argument in default_node_pool.
Validation: Run terraform validate after provider upgrades; passing validation confirms the configuration matches the installed schema, not that Azure will accept every regional setting.
3. Resource provider registration error
Symptom: AzureRM 3.x reported that Microsoft.MixedReality was not returned while checking resource providers.
Resolution: The lab needs AKS and ACR providers, not Mixed Reality. The current lab disables automatic provider registration and checks/registers only the namespaces its resources require. Registration may require administrator permissions.
4. Unsupported availability zones
Symptom: AKS returned AvailabilityZoneNotSupported for South India and said no zones were supported for the node pool.
Resolution: Remove zones = [1, 2, 3] from the default node pool. Do not specify zones unless Azure reports that the selected region, VM SKU, and subscription support them.
5. Unsupported Kubernetes patch version
Symptom: AKS rejected 1.36.5 with K8sVersionNotSupported.
Resolution: Query az aks get-versions --location <region> and choose a listed patch. The lab used 1.36.4 based on the South India results at the time.
6. VM size not allowed in the subscription
Symptom: AKS rejected Standard_DS2_v2 and returned an allowed-size list for South India.
Resolution: Choose a size from the list returned for the subscription and region. The lab switched to Standard_D2s_v5; verify current availability and quota before relying on that choice.
7. Insufficient regional vCPU quota
Symptom: AKS reported 2 regional vCPUs remaining but requested 4 for the two-node pool.
Resolution: The lab was reduced to one Standard_D2s_v5 node (2 vCPUs) for dev/test. This reduces availability and is not a production recommendation. Request a quota increase before scaling out or deploying a production cluster, and leave room for temporary upgrade/surge nodes.
8. Kubernetes client reports the API certificate is not yet valid
Symptom: kubectl get nodes failed TLS verification with x509: certificate has expired or is not yet valid, and gave the server certificate's validity timestamp.
Checks and resolution:
- actual UTC time. Check Windows time and synchronization:
``powershell Get-Date [DateTime]::UtcNow w32tm /query /status ``
- disable TLS verification.
- time has passed, refresh kubeconfig and retry:
``powershell az aks get-credentials --resource-group "<resource-group>" --name "<cluster-name>" --overwrite-existing kubectl get nodes ```
- according to a correctly synchronized client, collect the UTC timestamp and error details and contact the Azure administrator or Azure Support.
In the reported example the certificate start time was 03:43:11Z, which is 09:13:11 in India Standard Time (UTC+05:30). Compare against the actual machine time at the time of the failure; do not assume waiting is the fix if the local clock is incorrect or the certificate remains invalid after that time.
Deployment best practices
- Pin provider versions and commit
.terraform.lock.hclso team members use the same provider build. - Read provider major-version upgrade guides and run
terraform validatebefore planning. - Check region-specific AKS versions, VM SKU restrictions, zones, and quota before deployment.
- Keep secrets, credentials, state files, and kubeconfig out of source control. Use an appropriately secured remote Terraform backend for shared or production state.
- Give the Terraform identity only the permissions it needs. Creating role assignments requires permissions beyond the normal Contributor role.
- Register only the Azure resource providers required by the deployment.
- Always review
terraform planbeforeterraform apply. - Treat a one-node dev cluster as disposable and not highly available. A two-node test/preprod system pool is still a lab configuration; size production pools and quota for resilience, workload separation, and maintenance/upgrade surge.
- Avoid manually deleting resources managed by Terraform. If a create operation fails after some resources succeeded, inspect both Azure and Terraform state, then rerun a plan before retrying.
- Keep region-check scripts aligned with the actual lab. The repository's
check-regions.shcurrently checks a different region list andStandard_B2; it does not validate the lab's South IndiaStandard_D2s_v5configuration. It is a Bash script, so run it in WSL, Git Bash, or another Bash environment, and update its inputs before using it for this lab.
Reader discussion
What was the most useful pre-deployment check in your AKS lab: verifying the Kubernetes version, VM SKU availability, or regional quota? What other failure would you add to this troubleshooting checklist?