1 - Deploy with Customized Images

Deploy a cluster with customized CCM or CNM images.

Switch to the project root directory and run the following command to build both CCM and CNM images:

make image

If you want to build only one of them, try make build-ccm-image or ARCH=amd64 make build-node-image-linux.

To push the images to your own image registry, you can specify the registry and image tag while building:

IMAGE_REGISTRY=<image registry name> IMAGE_TAG=<tag name> make image

After building, you can push them to your image registry by make push.

Please follow here to build multi-arch image

2 - Dependency Management

Manage Cloud Provider Azure dependencies using go modules.

cloud-provider-azure uses go modules for Go dependency management.

Usage

Run make update-dependencies whenever vendored dependencies change. This takes a minute to complete.

Run make update-mocks whenever implementations for pkg/azureclients change.

Updating dependencies

New dependencies causes golang to recompute the minor version used for each major version of each dependency. And golang automatically removes dependencies that nothing imports any more.

To upgrade to the latest version for all direct and indirect dependencies of the current module:

  • run go get -u <package> to use the latest minor or patch releases
  • run go get -u=patch <package> to use the latest patch releases
  • run go get <package>@VERSION to use the specified version

You can also manually editing go.mod and update the versions in require and replace parts.

Because of staging in Kubernetes, manually go.mod updating is required for Kubernetes and its staging packages. In cloud-provider-azure, their versions are set in replace part, e.g.

replace (
    ...
    k8s.io/kubernetes => k8s.io/kubernetes v0.0.0-20190815230911-4e7fd98763aa
)

To update their versions, you need switch to $GOPATH/src/k8s.io/kubernetes, checkout to the version you want upgrade to, and finally run the following commands to get the go modules expected version:

commit=$(TZ=UTC git --no-pager show --quiet --abbrev=12 --date='format-local:%Y%m%d%H%M%S' --format="%cd-%h")
echo "v0.0.0-$commit"

After this, replace all kubernetes and staging versions (e.g. v0.0.0-20190815230911-4e7fd98763aa in above example) in go.mod.

Always run hack/update-dependencies.sh after changing go.mod by any of these methods (or adding new imports).

See golang’s go.mod, Using Go Modules and Kubernetes Go modules docs for more details.

Updating mocks

mockgen v1.6.0 is used to generate mocks.

mockgen -copyright_file=<copyright file> -source=<azureclient source> -package=<mock package>

3 - E2E tests

E2E tests guidance.

3.1 - Azure E2E tests

Azure E2E tests guidance.

Overview

Here provides some E2E tests only specific to Azure provider.

Prerequisite

Deploy a Kubernetes cluster with Azure CCM

Refer step 1-3 in e2e-tests for deploying the Kubernetes cluster.

Setup Azure credentials

export AZURE_TENANT_ID=<tenant-id>                    # the tenant ID
export AZURE_SUBSCRIPTION_ID=<subscription-id>        # the subscription ID
export AZURE_CLIENT_ID=<service-principal-id>         # the service principal ID
export AZURE_CLIENT_SECRET=<service-principal-secret> # the service principal secret
export AZURE_ENVIRONMENT=<AzurePublicCloud>           # the cloud environment (optional, default is AzurePublicCloud)
export AZURE_LOCATION=<location>                      # the location
export AZURE_LOADBALANCER_SKU=<loadbalancer-sku>      # the sku of load balancer (optional, default is basic)

Setup KUBECONFIG

  • Locate your kubeconfig and set it as env variable export KUBECONFIG=<kubeconfig> or cp <kubeconfig> ~/.kube/config

  • Test it via kubectl version

Run tests

  • Run default tests

    The following command ensures gingko v2 is installed and then runs default tests.

    make test-ccm-e2e

  • Run specific tests

    go test -v ./tests/e2e/ -timeout 0 -ginkgo.focus <focus-keyword> --ginkgo.skip <skip-keyword>

After a long time test, a JUnit report will be generated in a directory named by the cluster name

3.2 - Kubernetes E2E tests

Kubernetes E2E tests guidance.

Prerequisite

  • An azure service principal

    Please follow this guide for creating an azure service principal The service principal should either have:

    • Contributor permission of a subscription
    • Contributor permission of a resource group. In this case, please create the resource group first
  • Docker daemon enabled

How to run Kubernetes e2e tests locally

  1. Prepare dependency project
  • kubectl

    Kubectl allows you to run command against Kubernetes cluster, which is also used for deploying CSI plugins. You can follow here to install kubectl. e.g. on Linux

    curl -LO https://dl.k8s.io/release/$(curl -sL https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl
    chmod +x kubectl
    sudo mv kubectl /usr/local/bin/
    
  1. Build docker images azure-cloud-controller-manager, azure-cloud-node-manager and push them to your image repository.

    git clone https://github.com/kubernetes-sigs/cloud-provider-azure $GOPATH/src/sigs.k8s.io/cloud-provider-azure
    cd $GOPATH/src/sigs.k8s.io/cloud-provider-azure
    export IMAGE_REGISTRY=<your-registry>
    export IMAGE_TAG=<tag>
    make image # build all images of different ARCHs and OSes
    make push # push all images of different ARCHs and OSes to your registry. Or manually `docker push`
    
  2. Deploy a Kubernetes cluster with the above azure-cloud-controller-manager and azure-cloud-node-manager images.

    To deploy a cluster, export all the required environmental variables first and then invoke make deploy-cluster. Please notice that cluster-api-provider-azure is used to provision the management and workload clusters. To learn more about this provisioner, you can refer to its quick-start doc.

    export AZURE_SUBSCRIPTION_ID=<subscription-id>
    export AZURE_TENANT_ID=<tenant-id>
    export AZURE_CLIENT_ID=<client-id>
    export AZURE_CLIENT_SECRET=<client-secret>
    export CLUSTER_NAME=<cluster-name>
    export AZURE_RESOURCE_GROUP=<resource-group>
    export AZURE_CLOUD_CONTROLLER_MANAGER_IMG=<cloud-controller-manager-image>
    export AZURE_CLOUD_NODE_MANAGER_IMG=<cloud-node-manager-image>
    
    make deploy-cluster
    

    To connect the cluster:

    export KUBECONFIG=$GOPATH/src/sigs.k8s.io/cloud-provider-azure/$CLUSTER_NAME-kubeconfig
    kubectl cluster-info
    

    To check out more of the deployed cluster , replace kubectl cluster-info with other kubectl commands. To further debug and diagnose cluster problems, use kubectl cluster-info dump

  3. Run Kubernetes E2E tests

    make test-e2e-capz
    

4 - Deploy clusters

Deploy Kubernetes clusters

Cluster API Provider Azure (CAPZ)

A management cluster is needed to deploy workload clusters. Please follow the instructions.

After the management cluster is provisioned, please run the following command in the root directory of the repo.

make deploy-workload-cluster

Customizations are supported by environment variables:

Environment variablesrequireddescriptiondefault
AZURE_SUBSCRIPTION_IDtruesubscription ID
AZURE_TENANT_IDtruetenant ID
AZURE_CLIENT_IDtrueclient ID with permission
CLUSTER_NAMEtruename of the cluster
AZURE_RESOURCE_GROUPtruename of the resource group to be deployed (auto generated if not existed)
MANAGEMENT_CLUSTER_NAMEtruename of the management cluster
KINDfalsewhether the management cluster is provisioned by kindtrue
WORKLOAD_CLUSTER_TEMPLATEfalsepath to the cluster-api templatetests/k8s-azure-manifest/cluster-api/vmss-multi-nodepool.yaml
CUSTOMIZED_CLOUD_CONFIG_TEMPLATEfalsecustomized cloud provider configs
AZURE_CLUSTER_IDENTITY_SECRET_NAMEfalsename of the cluster identity secretcluster-identity-secret
AZURE_CLUSTER_IDENTITY_SECRET_NAMESPACEfalsenamespace of the cluster identity secretdefault
CLUSTER_IDENTITY_NAMEfalsename of the AzureClusterIdentity CRDcluster-identity
CONTROL_PLANE_MACHINE_COUNTfalsenumber of the control plane nodes1
WORKER_MACHINE_COUNTfalsenumber of the worker nodes2
AZURE_CONTROL_PLANE_MACHINE_TYPEfalseVM SKU of the control plane nodesStandard_D4s_v3
AZURE_NODE_MACHINE_TYPEfalseVM SKU of the worker nodesStandard_D2s_v3
AZURE_LOCATIONfalseregion of the cluster resourceswestus2
AZURE_CLOUD_CONTROLLER_MANAGER_IMG_REGISTRYfalseimage registry of the cloud-controller-managermcr.microsoft.com/oss/kubernetes
AZURE_CLOUD_NODE_MANAGER_IMG_REGISTRYfalseimage registry of the cloud-node-managermcr.microsoft.com/oss/kubernetes
AZURE_CLOUD_CONTROLLER_MANAGER_IMG_NAMEfalseimage name of the cloud-controller-managerazure-cloud-controller-manager
AZURE_CLOUD_NODE_MANAGER_IMG_NAMEfalseimage name of the cloud-node-managerazure-cloud-node-manager
AZURE_CLOUD_CONTROLLER_MANAGER_IMG_TAGfalseimage tag of the cloud-controller-managerv1.28.4
AZURE_CLOUD_NODE_MANAGER_IMG_TAGfalseimage tag of the cloud-node-managerv1.28.4
KUBERNETES_VERSIONfalseKubernetes components versionv1.28.0
AZURE_LOADBALANCER_SKUfalseLoadBalancer SKU, Standard or BasicStandard
LB_BACKEND_POOL_CONFIG_TYPEfalseLoadBalancer backend pool configuration type, nodeIPConfiguration, nodeIP or podIPnodeIPConfiguration
PUT_VMSS_VM_BATCH_SIZEfalseBatch size when updating VMSS VM concurrently0
AZURE_SSH_PUBLIC_KEYfalseSSH public key to connecet to the VMs""

To completely remove the cluster, specify the “${CLUSTER_NAME}” and run:

make delete-workload-cluster

5 - Design Docs and KEPs

Design Docs and KEPs related to this project.

This is the staging area of the design docs prior to or under development. Once the feature is done, the corresponding design doc would be moved to Topics.

5.1 - Azure Private Link Service Integration

Azure PLS Integration Design Document.

This feature is now generally available. The page has been moved to (topics).

5.2 - Basic to Standard Load Balancer Migration

Support migration from Basic Load Balancer to Standard Load Balancer.

Background

On September 30, 2025, Basic Load Balancer will be retired. This design document describes the planned migration path from Basic Load Balancer to Standard Load Balancer.

Design

The following User stands for human users, or any cluster provisioning tools (e.g., AKS).

  1. User must change the Load Balancer sku manually in the configuration file from Basic to Standard.

  2. If the basic Load Balancer is not removed, it will be deleted during the migration.

  3. All IPs of existing Load Balancer typed services will remain unchanged after migration, but the public IP sku will be changed to Standard.

  4. All nodes will join the new Standard Load Balancer backend pool after migration.

  5. User must manually configure the Standard Load Balancer for outbound traffic after migration.

  6. User must manually restart the cloud-controller-manager to trigger the migration process.

Workflow

This proposed migration process may introduce downtime.

  1. Introduce a new function that runs 1 time per pod restart to decouple the nodes and remove the basic Load Balancer if needed.

  2. Check the service ingress IP address, if it is an internal service, create a corresponding frontend IP configuration with the IP address in reconcileFrontendIPConfigs.

  3. If it is an external service, link the public IP ID to the frontend IP configuration, and update the public IP sku to Standard in ensurePublicIPExists.

  4. Could keep other code path unchanged (to be validated).

5.3 - Refining Local Service Backend Pool Updater

Retry and error handling design for the local service backend pool updater.

Refining Local Service Backend Pool Updater

Purpose

The local service backend pool updater batches EndpointSlice-driven IP changes for local Services and applies them asynchronously to Azure Load Balancer backend pools. Today, when an updater batch observes an error from Get or CreateOrUpdate, it drops the batch after emitting a failure event, except for backend pool 404, which is treated as stale work and skipped quietly.

This design adds bounded retry for errors that are worth retrying at the updater layer. It does not duplicate Azure SDK retries for the status codes already handled by the SDK retry policy. The retry behavior is local to the updater, reuses the existing updater tick loop, and avoids unbounded requeue.

Scope

In scope:

  • Retry failed local-service backend-pool update operations when the error is classified as retriable.
  • Add retry state to queued backend-pool update operations.
  • Add a new retry-count configuration field for this updater.
  • Emit retry and failure events with bounded event volume.
  • Preserve current stale 404 behavior.
  • Add unit tests for retry, exhaustion, non-retry, and stale cases.

Out of scope:

  • Redesigning multiple standard load balancer selection.
  • Changing main Service reconciliation retries.
  • Changing Azure SDK retry policy.
  • Changing cleanup behavior for local service backend pools.

Configuration

Add a new config field near LoadBalancerBackendPoolUpdateIntervalInSeconds:

// LoadBalancerBackendPoolUpdateRetryCount is the number of retries for retriable
// local-service backend-pool update failures. Defaults to 3.
LoadBalancerBackendPoolUpdateRetryCount *int `json:"loadBalancerBackendPoolUpdateRetryCount,omitempty" yaml:"loadBalancerBackendPoolUpdateRetryCount,omitempty"`

Add a default constant near DefaultLoadBalancerBackendPoolUpdateIntervalInSeconds:

DefaultLoadBalancerBackendPoolUpdateRetryCount = 3

Semantics:

  • nil uses the default value 3.
  • Explicit 0 disables updater-level retry and preserves today’s drop-after-failure behavior for retriable errors.
  • Negative values are normalized to 0.
  • The count means extra retries after the first failed attempt. With the default 3, a retriable error can be observed up to four times: one initial attempt plus three requeued retry attempts.

Use a pointer so the config loader can distinguish “unset” from “explicitly set to 0”. Normalization happens next to the existing LoadBalancerBackendPoolUpdateIntervalInSeconds defaulting in pkg/provider/azure.go: if the pointer is nil, use DefaultLoadBalancerBackendPoolUpdateRetryCount; if the pointer is non-nil and negative, clamp it to 0.

The omitempty tag is intentional with the pointer field: a nil pointer is omitted and therefore defaults, while a non-nil pointer to 0 is preserved across marshal/unmarshal and keeps retry disabled.

Retry Model

Use requeue-on-retriable-error. Each loadBalancerBackendPoolUpdateOperation carries retry metadata:

  • retryCount int, tracking how many retryable failures have already been consumed
  • nextEligibleAt time.Time, tracking when a throttled operation is eligible to be processed again

On each updater tick:

  1. Snapshot the queued operations under updater.lock.
  2. Clear the live queue while holding the lock.
  3. Release the lock before filtering, grouping, or calling Azure.
  4. Filter operations using the existing relevance checks:
    • the Service is still in localServiceNameToServiceInfoMap
    • the Service still points to the same load balancer
  5. Group otherwise relevant operations by loadBalancerName/backendPoolName.
  6. If any operation in a group has nextEligibleAt in the future, requeue the whole group. Do not call ARM, emit an event, record a metric, or increment retry state for skipped groups.
  7. Process each eligible group once.
  8. On retriable failure of a group, increment each constituent operation’s retryCount uniformly.
  9. Requeue operations whose retry budget remains.
  10. Drop exhausted operations and emit LoadBalancerBackendPoolUpdateFailed once per distinct Service in the group.

Retries do not carry over the in-memory backend pool object. The next tick re-issues Get for the group before re-applying queued operations, which is required for CreateOrUpdate conflicts and stale etags.

The updater must re-acquire the lock only when appending retry operations back to the live queue. New EndpointSlice events can continue adding fresh operations while a previous snapshot is doing Azure calls.

Merge Rule

Fresh operations and requeued operations for the same loadBalancerName/backendPoolName merge on the next updater tick through the existing grouping logic. Each operation keeps its own retryCount and nextEligibleAt; fresh operations start at retryCount=0 and no nextEligibleAt.

The group’s eligibility is determined by the most restrictive member: if any operation in the merged group has a future nextEligibleAt, the whole group is preserved without ARM calls. When an eligible merged group fails, the ARM error applies to the group, not to an individual operation. Per-operation error attribution is not possible at this layer, so every constituent operation consumes one retry. Exhausted operations are dropped and reported, while operations with remaining budget are requeued.

The throttling delay is group-level. If one operation for lbName/backendPoolName is parked by nextEligibleAt, all operations in that same group are preserved until the group becomes eligible. This avoids splitting add/remove operations for the same backend pool and preserves batching order. Unrelated groups continue processing normally.

Before requeueing or emitting retry/failure events, re-check that each operation is still relevant. If the Service is gone or now maps to a different load balancer, drop the stale operation quietly.

Error Classification

Classify updater errors into three categories:

  • stale: drop quietly
  • retriable: emit retrying event and requeue while budget remains
  • terminal: emit failed event and drop

Rules:

  • ARM resource-not-found errors as detected by errutils.CheckResourceExistsFromAzcoreError are stale.
  • ARM wire 429 TooManyRequests responses are retriable.
    • Classify them by checking errors.As(err, &respErr) for *azcore.ResponseError with StatusCode == http.StatusTooManyRequests.
    • Extract Retry-After from respErr.RawResponse.Header when RawResponse is non-nil.
    • Parse Retry-After the same way retryrepectthrottled.ThrottlingPolicy.processThrottlePolicy does: integer seconds or RFC1123 time.
    • If the response has no parseable Retry-After, fall back to the next updater tick.
  • Local throttle-gate retryrepectthrottled.ErrTooManyRequest is retriable.
    • This path is detected with errors.Is(err, retryrepectthrottled.ErrTooManyRequest) when no *azcore.ResponseError.RawResponse is available.
    • The local throttle gate returns the sentinel error without a response, header, or exported gate timestamp, so the updater cannot compute the policy’s gate expiry.
    • Fall back to the next updater tick for this path.
    • The updater does not read ThrottlingPolicy.RetryAfterReader or RetryAfterWriter directly. The policy instance is not exposed through the backend-pool client factory.
  • ARM 409 Conflict and 412 PreconditionFailed responses are retriable; no special payload handling is needed because the next updater tick already runs through a fresh Get before the next CreateOrUpdate.
  • ARM response statuses in retryrepectthrottled.GetRetriableStatusCode() are terminal for this updater. The Azure SDK already retries those statuses inside the individual ARM call, so if the updater sees one of those errors, the SDK retry budget has already been exhausted. The updater emits LoadBalancerBackendPoolUpdateFailed and drops the operation instead of adding a second retry layer for the same condition.
  • Non-ARM errors can opt into retry through a small wrapper/helper, such as:
func newRetriableBackendPoolUpdateError(err error) error
func isRetriableBackendPoolUpdateError(err error) bool

Keep this helper local to pkg/provider/azure_local_services.go unless another caller needs the same marker.

  • context.Canceled and context.DeadlineExceeded are terminal when ctx.Err() is non-nil.
  • Other timeout-like non-ARM errors are terminal unless explicitly wrapped by the local retriable-error helper.
  • All other errors are terminal.

The helper should use errors.As and errors.Is so wrapped errors remain classifiable.

The call site does not need generated backend-pool client wrappers to return raw *http.Response values. It should inspect the returned error directly:

var respErr *azcore.ResponseError
if errors.As(err, &respErr) && respErr.RawResponse != nil {
    retryAfter := respErr.RawResponse.Header.Get(retryrepectthrottled.HeaderRetryAfter)
    // parse retryAfter for 429 handling
}

Event Behavior

Use distinct events for retry and final failure:

  • LoadBalancerBackendPoolUpdateRetrying: emitted on every retryable failed attempt that will be requeued.
  • LoadBalancerBackendPoolUpdateFailed: emitted when a terminal error occurs or the retry budget is exhausted.
  • LoadBalancerBackendPoolUpdated: keep existing success behavior.
  • ARM resource-not-found: no event.

The new LoadBalancerBackendPoolUpdateRetrying reason should be added as a constant in pkg/consts/, consistent with the repository convention for shared constants. If the implementation touches the existing LoadBalancerBackendPoolUpdateFailed or LoadBalancerBackendPoolUpdated literals, move those reasons to pkg/consts/ in the same narrow change.

Events should be emitted once per distinct Service in a backend-pool group, not once per raw add/remove operation. This avoids duplicate retry events when multiple operations for the same Service are batched together.

This intentionally fixes the current notify() behavior that reports only the first operation in a batch because of its break after the first loop iteration. Add a regression test so future changes preserve one event per distinct Service in the group.

When an operation is waiting for nextEligibleAt, the updater should not emit another retrying event on each skipped tick. The retrying event belongs to the failed attempt that scheduled the retry.

Locking And Queue Safety

Keep updater.lock, but narrow it to queue access:

  • lock to snapshot and clear updater.operations
  • unlock while doing filtering, grouping, and Azure calls
  • lock to append retry operations
  • keep addOperation and removeOperation protected by the same lock

This prevents slow Azure calls and retry paths from blocking EndpointSlice event handlers that need to enqueue newer operations.

The removeOperation(serviceName) method cannot remove operations already in a processing snapshot. To avoid stale retry behavior, the processing path must re-check service relevance before requeueing and before sending retry/failure events.

Parked operations waiting for nextEligibleAt live in updater.operations, so removeOperation(serviceName) can remove them while they are parked. If removal races with the short snapshot-to-requeue window, the next tick’s relevance check drops the stale operation quietly.

The relevance re-check depends on the Service cleanup path also removing the Service from localServiceNameToServiceInfoMap. The implementation must preserve that invariant: a deleted Service should either be removed from the live queue by removeOperation(serviceName) or be rejected by the map-based relevance check before a retry is requeued or reported.

New retry-path logging should use pkg/log contextual loggers. Do not add new direct klog calls while touching this file.

Cancellation And Shutdown

If the updater context is canceled, the run() loop exits and parked in-memory operations are discarded with the process. The updater should not emit retry or failure events for operations that are only abandoned because the controller is shutting down.

If an in-flight ARM call returns context.Canceled or context.DeadlineExceeded after the updater context is done, treat it as terminal shutdown behavior and do not requeue. If process() observes cancellation before re-appending a snapshot, it should drop the snapshot instead of preserving retry state for a loop that is stopping.

Metrics

The updater metric should describe terminal outcomes, not intermediate retry attempts.

  • Requeued attempts do not call ObserveOperationWithResult(false).
  • Queue-preservation ticks while waiting for nextEligibleAt do not record a metric.
  • Success records one successful observation when LoadBalancerBackendPoolUpdated is emitted.
  • Terminal failure records one failed observation when LoadBalancerBackendPoolUpdateFailed is emitted.
  • Stale resource-not-found and stale Service/LB drops record no observation, matching the existing quiet-skip behavior.

This prevents a retried-then-succeeded operation from appearing as multiple failed operations followed by one success.

Retry Timing

Updater-level retry uses the existing LoadBalancerBackendPoolUpdateIntervalInSeconds tick; this design does not add a separate sleep loop inside process(). Without Retry-After, default retry count 3 and default interval 30s means a continuously failing retriable condition emits retrying events on the first three failed attempts and emits the final failed event on the fourth failed attempt, roughly 90 seconds after the first observed failure plus ARM call latency. Depending on where the first failure lands relative to the updater tick and how long each ARM call takes, the wall-clock time from the original EndpointSlice change to final failure can be close to or above two minutes.

For 429 throttling, Retry-After overrides the next normal updater tick by setting nextEligibleAt. Ticks before nextEligibleAt only preserve the operation in the queue after re-checking Service/LB relevance; they do not call ARM, emit retrying events, or consume retry budget. LoadBalancerBackendPoolUpdateRetryCount bounds failed processing attempts, not elapsed wall-clock time, so a long Retry-After can delay final success or failure beyond the normal interval-based timing.

During sustained Azure throttling, operators can reduce LoadBalancerBackendPoolUpdateRetryCount or increase LoadBalancerBackendPoolUpdateIntervalInSeconds to avoid retry amplification when Retry-After is unavailable.

Tests

Add focused unit tests in pkg/provider/azure_local_services_test.go:

  1. ARM wire 429 from Get with a parseable Retry-After in azcore.ResponseError.RawResponse sets nextEligibleAt; ticks before that time requeue quietly without ARM calls, retry events, retry-count increments, or metrics.
  2. ARM wire 429 from CreateOrUpdate follows the same Retry-After and requeue behavior.
  3. ARM wire 429 without a parseable Retry-After requeues, emits LoadBalancerBackendPoolUpdateRetrying, then succeeds on the next eligible tick.
  4. Local-gate retryrepectthrottled.ErrTooManyRequest without a raw response falls back to the next updater tick and does not panic on nil response.
  5. ARM 409 or 412 from CreateOrUpdate requeues, emits LoadBalancerBackendPoolUpdateRetrying, then succeeds on the next tick after a fresh Get.
  6. If any operation in a lbName/backendPoolName group is waiting for nextEligibleAt, the whole group is preserved and no same-group operation is processed early.
  7. A fresh operation merged with a requeued operation keeps its own retry counter; on group failure, all operations in the group consume one retry.
  8. Retry budget exhaustion emits LoadBalancerBackendPoolUpdateFailed and leaves the queue empty.
  9. Statuses from retryrepectthrottled.GetRetriableStatusCode() do not requeue and emit LoadBalancerBackendPoolUpdateFailed.
  10. Other non-retriable errors do not requeue and emit LoadBalancerBackendPoolUpdateFailed.
  11. ARM resource-not-found does not requeue and emits no event.
  12. Stale service or changed load balancer before requeue is dropped quietly.
  13. A Service that disappears while its operation is parked behind nextEligibleAt is dropped quietly on the next relevance check.
  14. Multiple operations for multiple Services in a backend-pool group emit one retry/failure event per distinct Service, covering the current notify() break-after-first-operation bug.
  15. removeOperation(serviceName) removes parked operations whose nextEligibleAt is in the future.
  16. Updater shutdown with parked operations does not emit retry/failure events and does not requeue.
  17. Explicit LoadBalancerBackendPoolUpdateRetryCount: 0 remains non-nil through config load and disables updater retry.
  18. A retried-then-succeeded operation records exactly one successful metric observation and no failed metric observations.

Use a fake event recorder where needed to assert retrying and failed event reasons precisely.

Expected Behavior

With default retry count 3, a retriable updater failure behaves as:

  1. First failed attempt: emit LoadBalancerBackendPoolUpdateRetrying, requeue.
  2. Second failed attempt: emit LoadBalancerBackendPoolUpdateRetrying, requeue.
  3. Third failed attempt: emit LoadBalancerBackendPoolUpdateRetrying, requeue.
  4. Fourth failed attempt: emit LoadBalancerBackendPoolUpdateFailed, drop.

If a later retry succeeds, the updater emits LoadBalancerBackendPoolUpdated and drops the completed operations.

If the Service becomes stale during retry, the updater drops the operation quietly instead of emitting retry or failure events.

If an ARM call returns a status from retryrepectthrottled.GetRetriableStatusCode() after the SDK retry policy has already run, the updater emits LoadBalancerBackendPoolUpdateFailed immediately and drops the operation.

If a 429 response provides Retry-After, the updater respects that value before the next processing attempt. Intermediate updater ticks before nextEligibleAt are queue-preservation ticks, not retry attempts.

6 - Image building

Image building.

multi-arch image

Currently, only Linux multi-arch cloud-node-manager image is supported as a result of customer requests and windows limitations. Supported Linux archs are defined by ALL_ARCH.linux in Makefile, and Windows os versions are by ALL_OSVERSIONS.windows.

Windows multi-arch image limitation

Images nanoserver and servercore are referenced to build a Windows image, but as current officially published servercore images does not support non-amd64 image, and only Windows server 1809 has the support of non-amd64 for nanoserver, amd64 is the only supported arch for a range of Windows OS version so far. This issue is tracked here

hand-on examples

To build and publish the multi-arch image for node manager

IMAGE_REGISTRY=<registry> make build-all-node-images
IMAGE_REGISTRY=<registry> make push-multi-arch-node-manager-image

To build a specific Linux arch image for node manager

IMAGE_REGISTRY=<registry> ARCH=amd64 make build-node-image-linux

To build specific Windows OS and arch image for node manager

IMAGE_REGISTRY=<registry> OUTPUT_TYPE=registry ARCH=amd64 WINDOWS_OSVERSION=1809 build-node-image-windows

The OUTPUT_TYPE registry here means the built image will be published to the registry, this is necessary to build a Windows image from a Linux working environment. An alternative is to export the image tarball to a local destination, like OUTPUT_TYPE=docker,dest=dstdir/azure-cloud-node-manager.tar. For more info about docker buildx output type, please check out here

7 - Future Plans

Future Plans.

To be completed.