Skip to content

Operations

Prerequisites

Toolbox container (primary interface)

The toolbox image (ghcr.io/polarsquad/krops-toolbox) carries krops-bootstrap plus every tool used by bootstrap, pivot, and teardown. It intentionally omits the development-only Go toolchain and the Zarf CLI; a pinned Python and uv are present only as a runtime backend for azure-cli and gcloud. The host needs the repository checkout and a running Docker engine or Podman 5.5+. Lifecycle runs also download the pinned go and zarf tools through mise at runtime (the toolbox container is ephemeral, so this repeats on every run); outbound access to their release hosts is needed. This is unintended and tracked in #601.

The toolbox image is published to ghcr.io/polarsquad/krops-toolbox for Linux amd64 and arm64 as X.Y.Z, X.Y, and stable latest, signed with GitHub OIDC and carrying a Syft SPDX JSON SBOM attestation. The toolbox-release workflow publishes those tags from a matching v* tag. Use a published tag, or build the current checkout only for unreleased changes:

docker build -f bootstrap-rs/Dockerfile -t krops-toolbox:dev .
export TOOLBOX_IMAGE=krops-toolbox:dev
mkdir -p .kube

A complete raw Docker run for local-host is:

docker run --rm -it \
  -v "$PWD:/workspace" -w /workspace \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v "$PWD/.kube:/root/.kube" \
  -e KUBECONFIG=/workspace/.kube/kind.yaml \
  "$TOOLBOX_IMAGE" local-host

This form assumes the standard /var/run/docker.sock daemon socket. Use the wrapper below for Docker contexts or Podman installations with a different host socket.

The lifecycle wrapper also creates $HOME/.krops-azure and mounts it at AZURE_CONFIG_DIR in the container (default /root/.azure). It forwards that resolved path after loading .env, which takes precedence over shell values. This preserves the Azure CLI login from subscription preparation across bootstrap and pivot. Raw Azure runs must mount the same cache themselves; see Azure setup.

The AWS environment also needs the Git source, PAT, age key, and AWS credentials inside the container. Source the repository .env so shell quotes are removed, then pass values by name rather than putting secrets in argv:

cp .env.example .env
$EDITOR .env
set -a
. ./.env
set +a

docker run --rm -it \
  -v "$PWD:/workspace" -w /workspace \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v "$PWD/.kube:/root/.kube" \
  -e KUBECONFIG=/workspace/.kube/kind.yaml \
  -e GIT_REPO_URL -e GITHUB_TOKEN -e GITHUB_USER \
  -e AGE_KEY_FILE -e AGE_PUBLIC_KEY \
  -e AWS_REGION -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY \
  -e AWS_SESSION_TOKEN -e AWS_PROFILE \
  "$TOOLBOX_IMAGE"

Generate the age key and re-encrypt secrets for a new fork before the AWS run; see Secret management. If credentials come from an AWS shared configuration instead of environment variables, also mount that configuration under /root/.aws and pass AWS_PROFILE. The local-talos environment needs the same Git source, PAT, and age key, but no AWS credentials.

Podman socket locations differ across rootful Linux, rootless Linux, and podman machine. Use the checked-in wrapper to resolve the host mount and the socket path seen by the daemon:

TOOLBOX_IMAGE="$TOOLBOX_IMAGE" scripts/toolbox-run.sh bootstrap local-host
CONTAINER_ENGINE=podman TOOLBOX_IMAGE="$TOOLBOX_IMAGE" \
  scripts/toolbox-run.sh bootstrap local-host

The wrapper is the lifecycle entry point for every environment:

TOOLBOX_IMAGE="$TOOLBOX_IMAGE" scripts/toolbox-run.sh bootstrap aws

KROPS_PROFILE (or the positional profile argument after the lifecycle verb, which reaches krops-bootstrap) selects the environment. It loads .env before detecting the engine or resolving a socket for it, so a .env-selected CONTAINER_ENGINE takes effect from the start (issue #257). It also refreshes remote-registry TOOLBOX_IMAGE references before every run: it pulls the image so the mutable :latest tag is revalidated against the registry and a fresh release reaches hosts that already pulled it (issue #645). A failed pull never blocks a lifecycle operation; the wrapper warns and continues with the locally cached image, so offline runs and registry outages keep working. Local builds (any bare name:tag such as krops-toolbox:dev) have no registry source and are used as-is. For the supported single-line dotenv syntax, it follows mise's env_file, so scripts/toolbox-run.sh and mise run bootstrap hand the container the same values:

  • .env wins over the process environment: a NAME=value prefix on the command line (such as CONTAINER_ENGINE=podman above) only applies when .env does not set NAME. The helper tasks follow the same rule inside the container (see Helper tasks in the toolbox).
  • Line syntax: an optional export prefix, whitespace trimmed around keys and values, single and double quotes, escaped spaces/quotes/backslashes, \n / \r / \t escapes, and a # after whitespace starting a comment in an unquoted value. $VAR and ${VAR} expand inherited variables or earlier assignments outside single quotes; unset references become empty. Escaped dollars stay literal. Repeated assignments use the last value. Lines without = or with an invalid key are skipped. Backslash escapes inside single quotes (whose mise semantics vary by version), multiline values, and parameter operators such as ${VAR:-default} are outside the wrapper's supported syntax; use simple assignments instead. Unsupported or malformed parsed values are skipped with a warning that names only the key.

Engine detection tries docker info first; if that fails (for example because the current user cannot access the socket), it silently falls back to podman info. On Linux, see Host-side access after a toolbox local-host run (Linux) for socket permission steps.

tests/test-toolbox-run-env-precedence.py pins the .env-precedence behavior above and, when mise is installed, checks it against mise itself in isolated temporary state. CI requires that cross-check. Parsing never evaluates shell commands. The wrapper then passes only a fixed set of values into the container, built by build_env_args from the TOOLBOX_ENV_SPEC list, split into two groups: mutable (forwarded from whatever .env or the shell set) and immutable (a value the wrapper computes or hardcodes itself; never an operator override, even from .env). See .env.example for the full, per-environment list of both groups; the krops-bootstrap CLI's own knobs (KROPS_PROFILE, REGISTRY_PORT, ...) are covered separately in bootstrap-cli.md ("Bootstrap and pivot controls").

If an operator value is set for an immutable key, build_env_args prints a warning to stderr instead of silently discarding it (this is what let #357 happen for CLOUDSDK_CONFIG).

It does not pass BOOTSTRAP_CONFIG, REGISTRY_READY_RETRIES, LOCAL_RECONCILE_TIMEOUT, MGMT_KUBECONFIG, MGMT_READY_TIMEOUT, MGMT_POLL_INTERVAL, BOOTSTRAP_KUBECONTEXT, or the teardown controls AWS_ONLY, FORCE_KIND_DELETE, CLUSTER_DELETE_TIMEOUT, and PROVIDER_DELETE_TIMEOUT. Use a raw container run with explicit -e entries when overriding those values.

Inside the toolbox:

  • The entrypoint sets KROPS_TOOLBOX=1 and execs krops-bootstrap, which owns engine detection, the daemon-side ENGINE_SOCK used by kind's socket mount, and kind-network attach/detach (issue #256).
  • Bootstrap joins the kind network explicitly after creating (or reusing) the cluster; recreate, pivot, and teardown detach before deleting the bootstrap cluster; teardown re-joins first if it needs the internal API endpoint.
  • Kind's internal API endpoint and krops-registry:5000 then resolve by name.
  • Host-only CAPD endpoint rewrites are skipped because the recorded endpoints already resolve on that network.
  • On macOS the persisted .kube/krops-mgmt.yaml keeps the kind network address, which Docker Desktop does not route; see Host-side access after a toolbox local-host run (macOS) for the host rewrite.
  • KUBECONFIG must name one writable file. The documented invocation uses /workspace/.kube/kind.yaml, and the CLI replaces it with kind's internal kubeconfig after creation.
  • /root/.kube maps to the checkout's .kube/, so the exported management kubeconfig persists on the host as .kube/krops-mgmt.yaml.

Helper tasks in the toolbox

The one-off and helper steps (cloud preparation, SOPS key work, kubeconfig exports, oci-push) are mise tasks defined in mise.toml and the mise.<env>.toml layers. They run in the same toolbox image as the lifecycle, by pointing the container's entrypoint at mise and running the task from the mounted checkout. There is no wrapper verb for them; each environment page shows its commands and they all follow one of two shapes.

Repo-only tasks (sops-*, aws-credentials) run as your own user so the files they write are owned by you:

export TOOLBOX_IMAGE=ghcr.io/polarsquad/krops-toolbox:latest   # or krops-toolbox:dev
docker run --rm -it --user "$(id -u):$(id -g)" -e HOME=/tmp \
  -v "$PWD:/workspace" -w /workspace \
  -e MISE_AUTO_INSTALL=0 \
  --entrypoint mise "$TOOLBOX_IMAGE" run <task> [args]

Cluster tasks (mgmt-kubeconfig, kubeconfigs, oci-push, the cloud *-bootstrap and *-federate tasks) run as root, with the persisted kubeconfig directory and the engine socket mounted like the lifecycle run:

docker run --rm -it \
  -v "$PWD:/workspace" -w /workspace \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v "$PWD/.kube:/root/.kube" \
  -e KUBECONFIG=/workspace/.kube/krops-mgmt.yaml \
  -e MISE_AUTO_INSTALL=0 \
  --entrypoint mise "$TOOLBOX_IMAGE" -E <env> run <task>

Rules that apply to every helper run:

  • --entrypoint mise bypasses toolbox-entrypoint.sh, so KROPS_TOOLBOX is not set. The local-host kubeconfig tasks need it (-e KROPS_TOOLBOX=1) together with --network kind: they then keep the CAPD-recorded kind-network endpoint instead of rewriting it to 127.0.0.1, which inside a container is the container itself. A kubeconfig exported that way works from later toolbox runs on the kind network. On macOS, the kind-network address is not routable from the host (use the macOS section below for the port-mapped form). On Linux with a rootful engine, it works from the host after fixing file ownership (see the Linux section).
  • MISE_AUTO_INSTALL=0 is required. The mounted mise.toml pins dev-only tools (zarf, go) that the image does not carry; without the flag mise tries to install them, and as a non-root user that fails with Permission denied under /usr/local/share/mise.
  • mise loads /workspace/.env (env_file in mise.toml) and its values override the process environment. Cloud credentials for aws-bootstrap, aws-credentials, azure-bootstrap, and gcp-bootstrap come from .env first; to run with other credentials, pass -e MISE_ENV_FILE=/dev/null and the variables by name (-e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN -e AWS_REGION). A bare -e NAME forwards the host value without placing it in argv.
  • Interactive logins (az login --use-device-code, gcloud auth login --no-launch-browser) need -it. The gcloud session persists in the checkout's gitignored .gcloud/ (-e CLOUDSDK_CONFIG=/workspace/.gcloud); the Azure session persists through -v "$PWD/.azure:/root/.azure" (add .azure/ to your global gitignore or keep it outside the checkout with another host path).
  • Podman: replace docker with podman and the socket source with the one scripts/toolbox-run.sh resolves (podman info --format '{{.Host.RemoteSocket.Path}}').

Host-side on purpose: mise run validate (repository development, see Validation) and mise -E local-host run podinfo-port-forward (the browser is on the host; a toolbox form is shown with the local-host chain below).

Host mise version. Host mise is only needed for mise run validate and mise -E local-host run podinfo-port-forward; everything else runs inside the toolbox, whose mise is pinned in the image. mise.toml sets min_version; distribution packages often lag behind it, causing the mise shell hook to error on every cd into the checkout. To use those two tasks, install a current mise (see https://mise.jdx.dev; or mise self-update for installer-based installs). To silence the hook on a host that does not need those tasks, ignore the checkout's config:

# In your shell rc (portable; works on all mise versions):
export MISE_IGNORED_CONFIG_PATHS="/abs/path/to/krops"

# Or in ~/.config/mise/miserc.toml (current mise only):
ignored_config_paths = ["/abs/path/to/krops"]

Note: mise settings add ignored_config_paths <path> is rejected on current mise; that setting must be written to ~/.config/mise/miserc.toml or set via the env var, not via the CLI. Ignoring the checkout also disables validate and podinfo-port-forward from within it.

Verifying a toolbox release

After the first release is published, replace X.Y.Z in these commands with the matching Cargo and Git tag version:

IMAGE=ghcr.io/polarsquad/krops-toolbox:X.Y.Z
IDENTITY=https://github.com/polarsquad/krops/.github/workflows/toolbox-release.yml@refs/tags/vX.Y.Z

cosign verify \
  --certificate-identity "$IDENTITY" \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  "$IMAGE"

cosign verify-attestation \
  --type spdxjson \
  --certificate-identity "$IDENTITY" \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  "$IMAGE"

E2E AWS account

Designation: Account 120392301094 was designated on 2026-09-20 (#239) as the single account for all krops end-to-end testing (live acceptance #143, scheduled e2e #185). It is a Control Tower-managed Organizations member account.

Owner and escalation contact: joseph.shriner@polarsquad.com

Shared sandbox warning: This is a shared account with non-krops resources (training EKS clusters, Terraform VPCs, workshop buckets). Every krops tool filters on default_*, krops-*, and CAPA ownership tags. Do not run account-wide cleanup commands; use the teardown path which scopes deletions to krops-owned resources.

Runbook index:

What Where
Spend ceiling and alert contact E2E account budget
Service quotas and first-run errors AWS service quotas
CI access (GitHub Actions OIDC) CI access in aws-iam.md
Incident and credential revocation E2E account incident and credential revocation in aws-iam.md
Cluster teardown and leftover cleanup Teardown
Orphan reporting issue #380
Resource tagging standard issue #381

When a budget alert fires:

  1. Check for live krops clusters: 
    aws eks list-clusters --region eu-north-1
aws eks list-clusters --region eu-west-1
  2. Use the AWS Cost Explorer console (group by service) to distinguish krops spend from other shared-account spend.
  3. If the spend is from leftover krops resources, clean them up through the teardown path.
  4. If the spend is from non-krops resources, escalate to the account owner (joseph.shriner@polarsquad.com).
  5. To adjust the ceiling, update the budget in the AWS Budgets console and update the budget paragraph in docs/operations.md in the same PR.

AWS service quotas (common first-run blockers)

Quota Code Needed Why
EC2-VPC Elastic IPs (per region) L-0263D0A3 ≥ 6 free in eu-north-1, ≥ 3 free in eu-west-1 One EIP per NAT gateway (3 AZs): two clusters in eu-north-1 (management + workload), one in eu-west-1
VPCs per region L-F678F1CE ≥ 2 free in eu-north-1, ≥ 1 free in eu-west-1 One VPC per cluster: two clusters in a full run in eu-north-1, one cluster in eu-west-1. In e2e account 120392301094, as of 2026-09-20 (#382), the eu-north-1 total quota was 8 (5 in use, 3 free) and the eu-west-1 total quota was 5 (3 in use, 2 free). This is a dated snapshot, not guaranteed headroom; usage in the shared account drifts, so re-check it before every run.

Both quotas default to 5 per region. The EIP quota stalls the run mid-way through the second eu-north-1 cluster (6 EIPs needed); insufficient free VPC capacity fails the run at VPC creation. Before every run, query the configured quota and current usage for both resources in both target regions:

for region in eu-north-1 eu-west-1; do
  echo "$region: Elastic IP quota and usage"
  aws service-quotas get-service-quota \
    --service-code ec2 \
    --quota-code L-0263D0A3 \
    --region "$region" \
    --query 'Quota.Value'
  aws ec2 describe-addresses \
    --region "$region" \
    --query 'length(Addresses)'

  echo "$region: VPC quota and usage"
  aws service-quotas get-service-quota \
    --service-code vpc \
    --quota-code L-F678F1CE \
    --region "$region" \
    --query 'Quota.Value'
  aws ec2 describe-vpcs \
    --region "$region" \
    --query 'length(Vpcs)'
done

Subtract usage from the configured quota to calculate free capacity. EIPs require at least 6 free in eu-north-1 and 3 free in eu-west-1; request an increase in each region where that capacity is unavailable. VPCs require at least 2 free in eu-north-1 and 1 free in eu-west-1; request a VPC increase only in a region where its required free capacity is unavailable.

For quota-increase requests, --desired-value is the new total regional quota, not the amount of additional capacity. Set it to at least the current usage plus the required free slots. To increase the Elastic IP quota where needed:

region=eu-north-1 # Set the region that lacks the required free capacity.
required_free=6   # Use 6 for eu-north-1 or 3 for eu-west-1.
current_usage="$(aws ec2 describe-addresses --region "$region" --query 'length(Addresses)' --output text)"
desired_value=$((current_usage + required_free))
aws service-quotas request-service-quota-increase --service-code ec2 --quota-code L-0263D0A3 --desired-value "$desired_value" --region "$region"

To increase the VPCs-per-region quota where needed (service code is vpc, not ec2):

region=eu-north-1 # Set the region that lacks the required free capacity.
required_free=2   # Use 2 for eu-north-1 or 1 for eu-west-1.
current_usage="$(aws ec2 describe-vpcs --region "$region" --query 'length(Vpcs)' --output text)"
desired_value=$((current_usage + required_free))
aws service-quotas request-service-quota-increase --service-code vpc --quota-code L-F678F1CE --desired-value "$desired_value" --region "$region"

Bootstrap now enforces the EIP quota at runtime during preflight checks before any provisioning begins. The credentials used must have servicequotas:GetServiceQuota and ec2:DescribeAddresses permissions.

E2E account budget

The e2e AWS account 120392301094 carries a monthly cost budget krops-e2e-monthly with a $200 ceiling. Notifications publish to the SNS topic krops-e2e-budget-alerts (us-east-1) at 80% forecasted and 100% actual spend, and the topic's email subscription delivers them to joseph.shriner@polarsquad.com, the escalation path for budget alerts. The email subscription only activates after the SNS confirmation email is accepted.

E2E resource tagging standard

Every e2e AWS resource created by a bootstrap or pivot run carries four tags for lifecycle tracking and cost attribution (#381):

Tag key ConfigMap key Default Meaning
krops.io/run-id KROPS_RUN_ID {profile}-{timestamp} (auto-generated) Bootstrap/pivot invocation identifier (e.g. gha-123 for GitHub Actions run, or {profile}-{timestamp} auto-generated)
krops.io/revision KROPS_REVISION unknown Git branch HEAD SHA; extracted from GitHub API during bootstrap, else the KROPS_OCI_GIT_SHA checkout SHA the toolbox wrapper forwards
krops.io/expires-at KROPS_EXPIRES_AT[^expires-at-derived] {now + 24h} Resource expiry target (RFC3339 timestamp or literal never)
krops.io/run-kind KROPS_RUN_KIND manual Lifecycle context (manual, scheduled, emergency, etc.)

[^expires-at-derived]: The krops.io/expires-at value is derived from KROPS_RUN_TTL (default 24h); use KROPS_RUN_TTL=none for no expiry.

Tag application

Tags are written imperatively to the krops-run ConfigMap in flux-system during bootstrap (before Flux instance installation). Flux then uses the ConfigMap values via postBuild.substituteFrom to apply them to:

  • CAPA control planes (AWSManagedControlPlane spec): VPC, subnets, NAT gateways, Elastic IPs, and node groups inherit the control plane's tags automatically.
  • ACK-managed resources: S3 Bucket, RDS DBInstance, IAM Role, and IAM User CRs apply tags directly in their specs.

Not tagged: - CloudFormation stacks backing CAPA (account-global, persistent); would need stack-level parameters outside the current scope. - kind bootstrap cluster (ephemeral, purely local); lifecycle tags are meaningless for a temporary local fixture.

Rerun behavior

Three paths handle reruns differently:

Rust CLI (krops-bootstrap): When KROPS_RUN_ID is set and non-empty, and the krops-run ConfigMap exists in kind's flux-system, all four values are reused without change (rerun-safe). If the ConfigMap is gone but KROPS_RUN_ID was set, a fresh ConfigMap is written with KROPS_RUN_ID from the environment and expires-at recalculated from KROPS_RUN_TTL. Only an absent ConfigMap permits this fallback; API failures, malformed JSON, and missing or empty data keys stop bootstrap. With KROPS_RUN_ID unset or empty, a fresh run-id is generated (profile + timestamp) and the ConfigMap is overwritten.

bootstrap.sh: Always rewrites the krops-run ConfigMap from environment variables. To preserve the same identity across reruns, export KROPS_RUN_ID (and optionally KROPS_RUN_TTL) before calling bootstrap.

Pivot (pivot.sh and the CLI's pivot phase): Never generates tag values. Copies the four data keys from the kind cluster's krops-run ConfigMap using the BOOTSTRAP_KUBECONTEXT (default kind-mgmt); fails if the ConfigMap is missing or any required data key is missing, empty, or invalid. API read failures also stop the pivot.

TTL and expiry

  • Default: resources tagged with a 24-hour expiry (krops.io/expires-at={now + 24h}).
  • Override with KROPS_RUN_TTL=<duration>: parses durations like 2h, 30m, 3600s, or bare decimal seconds (3600). Leading zeros are decimal; surrounding whitespace is trimmed. Empty or unset values default to 24h. Negative, fractional, compound, malformed, overflowing durations and expiry dates beyond year 9999 are rejected.
  • KROPS_RUN_TTL=none tags with krops.io/expires-at=never (no auto-expiry).

Post-pivot seeding

The krops-run ConfigMap created in the kind bootstrap cluster is seeded to the management cluster before reconciliation starts: install the Flux Operator and bootstrap secrets, apply validated run metadata, then install the FluxInstance and wait for readiness. The shell pivot reads and validates source metadata before target preparation; Rust reads it after installing the operator and before installing the FluxInstance. Failed reads or writes prevent activation in both paths.

Rust uses the discovered branch SHA as the default revision, falling back to the KROPS_OCI_GIT_SHA checkout SHA the toolbox wrapper forwards, then unknown; shell defaults to unknown. Both honor KROPS_REVISION.

Lookup and cleanup

Find resources by run-id: 

aws resourcegroupstaggingapi get-resources --region <region> \
  --tag-filters Key=krops.io/run-id,Values=<run-id>
aws iam list-role-tags --role-name <name>
aws s3api get-bucket-tagging --bucket <name>

Group orphaned resources by krops.io/run-id and flag any where krops.io/expires-at is in the past for manual teardown.

E2E account orphan report

Daily at 05:41 UTC (or manual dispatch) the aws-orphan-report workflow discovers unowned AWS resources. The discover job runs krops-bootstrap orphans aws to scan:

  • EKS clusters and nodegroups
  • RDS instances
  • VPCs with CAPA ownership tags
  • NAT gateways and Elastic IPs
  • S3 buckets matching cluster name patterns

Resources older than ORPHAN_MIN_AGE_HOURS (default 6 hours) are flagged as orphans. The ORPHAN_REPORT_JSON environment variable is set to capture the report as JSON, and the markdown report is posted to the job summary.

Not scanned: IAM roles/users (usually free) and CloudFormation bootstrap stacks (also free). See the teardown section below for cleanup.

When orphans are found or the discovery fails, the report-status job opens or comments on the tracking issue aws-e2e: orphaned resources detected in the e2e account. The issue is closed when the next run finds no orphans.

The report is read-only; no resources are deleted. To clean up orphans, run:

AWS_ONLY=1 mise run teardown aws

local-talos prerequisites

In addition to the PAT and age key shared with the AWS environment (local-talos syncs its configuration from GitHub, so bootstrap seeds the same flux-github-pat and sops-age secrets), the local-talos environment needs:

  • A reachable Tinkerbell stack on the machine's network, and a Hardware resource describing the target machine. The stack is operator-owned infrastructure this repository does not deploy. Minimal setup:
  • Install the tink stack (Smee serving DHCP/PXE/iPXE, plus the Tinkerbell API) on a helper Kubernetes cluster that can reach the machine's L2 network.
  • Create the Hardware CR naming the machine's MAC address and reserved IP; its name must match the hardwareName in mgmt/local-talos/clusters/management/cluster.yaml (talos-mgmt-01 as checked in).
  • Customize the installer image with the hardware.tinkerbell.org/installer-image annotation on the Tinkerbell Hardware CR: the pinned CAPT fork (v0.7.1) mirrors it into the InfraMachine's status.installerImage (#156), which CABPT (>= v0.7.6, now v0.9.0) injects as machine.install.image. The cluster definition itself deliberately pins no installer image, so without the annotation installs use the default Image Factory schematic and CABPT treats the absent field as "no override".

  • Set the machine to PXE-boot from the network Smee serves.

    Installer image handoff status: the annotation-mirror path above is the mechanism in use and was validated by the #105 hardware acceptance run (done, closed, ISO boot); the only open item is the PXE/Tinkerbell-Workflow provisioning transport run (issue #225). Verify the image the machine actually receives until that run. Fork retirement is tracked in issue #266, blocked on upstream PR tinkerbell/cluster-api-provider-tinkerbell#604, which still carries the installer-image mirror the fork adds. - Two site-specific values in mgmt/local-talos/clusters/management/cluster.yaml before the first run: spec.controlPlaneEndpoint.host (the machine's stable IP) and the TinkerbellMachineTemplate hardwareName (the Hardware CR name).

talosctl (pinned in mise.local-talos.toml) is an operator convenience for the machine itself, not a lifecycle dependency; install it on the host with mise -E local-talos install if you want it.

Configuration

Copy the env template and fill it in. Both the lifecycle wrapper (scripts/toolbox-run.sh) and the helper mise tasks (through mise's env_file, inside the toolbox) load .env automatically, and it is gitignored:

cp .env.example .env
$EDITOR .env

The Flux Operator chart is pulled anonymously for all environments. The GitHub PAT is needed for the AWS and local-talos environments (both sync from GitHub); AWS credentials and AWS_REGION are only needed with the AWS environment.

Repository-owned lifecycle configuration lives in bootstrap.toml. It defines the environment names, sync paths, management clusters, imperative chart versions, provider manifests, and teardown targets. krops-bootstrap reads it from the working directory unless BOOTSTRAP_CONFIG selects another path. Runtime environment variables take precedence over configurable defaults, and mise run validate cross-checks the file against the manifests.

Bootstrap

Every environment boots through the same lifecycle wrapper; the profile selects the environment (the default is aws from bootstrap.toml):

TOOLBOX_IMAGE="$TOOLBOX_IMAGE" scripts/toolbox-run.sh bootstrap            # aws
TOOLBOX_IMAGE="$TOOLBOX_IMAGE" scripts/toolbox-run.sh bootstrap local-host
TOOLBOX_IMAGE="$TOOLBOX_IMAGE" scripts/toolbox-run.sh bootstrap local-talos
TOOLBOX_IMAGE="$TOOLBOX_IMAGE" scripts/toolbox-run.sh bootstrap azure      # after azure-bootstrap
TOOLBOX_IMAGE="$TOOLBOX_IMAGE" scripts/toolbox-run.sh bootstrap gcp        # after gcp-bootstrap

Azure: see azure.md for the subscription prep step that precedes the azure wrapper run (azure-bootstrap registers the providers — including the Arc ones — and creates the shared resource group and the krops-capz user-assigned identity with its role grants; nothing it prints is secret). After the kind cluster is created, bootstrap-rs runs the arc-federate mise task, which Arc-connects kind with an OIDC issuer for CAPZ/ASO workload identity (issue #236).

GCP: see gcp.md for the project prep step that precedes the gcp wrapper run (gcp-bootstrap enables the APIs and creates the krops-capg / krops-reader service accounts and the krops workload identity pool; nothing it prints is secret). After the kind cluster is created, bootstrap-rs runs the wif-federate mise task, which registers the kind cluster's OIDC provider and the impersonation bindings for CAPG and Config Connector.

Before the first AWS bootstrap, generate an age key for SOPS. See Secret management for the host and toolbox-only setups.

This initial imperative phase performs these steps:

  1. Creates the mgmt kind cluster.
  2. Installs the Flux Operator (Helm).
  3. Creates the flux-github-pat secret (for Git access) and the sops-age secret (the age private key Flux uses to decrypt SOPS-encrypted secrets) in both the Flux namespace (flux-system) and default (where the ResourceSet remote-apply Kustomizations SOPS-decrypt the per-region pull secret).
  4. Installs a FluxInstance that syncs mgmt/aws/ and hands off to GitOps.
  5. Pivots: moves the CAPI inventory into the self-managed management cluster and deletes the kind cluster (see Pivot recovery).

Everything downstream (providers, EKS clusters, workload Flux instances, the ACK controllers, IAM roles, and S3 buckets) reconciles from Git with no further manual steps.

The local-host environment performs the cluster, Flux Operator, and FluxInstance steps in the mgmt management cluster, but does not create GitHub or SOPS secrets. Instead, it bootstraps a local Docker Registry container (registry:2) running on the host machine (accessible at localhost:5001 by default), publishes the mgmt/local-host/ and workload/local-host/ folders as the initial krops:latest OCI artifact, and configures Flux to reconcile that path from the artifact. Flux then installs the CAPI core, kubeadm, and Docker infrastructure providers and creates local-workload, a one-control-plane/one-worker Kubernetes cluster in containers. The management cluster then installs a Flux Operator and FluxInstance on local-workload; that instance reconciles workload/local-host/ from the same OCI artifact. CAPD is intended for local development and testing, not production.

Together, these stages make local-host an end-to-end environment: one command bootstraps the management control plane, publishes and reconciles the OCI configuration, provisions a workload cluster through CAPI, installs a distinct Flux control plane on that cluster, and reconciles a reachable Podinfo workload. It exercises the complete cluster-to-workload GitOps lifecycle locally; only the AWS-specific infrastructure and ACK resources are outside its scope.

The local-talos environment performs the same kind bootstrap and Flux handoff, but syncs from GitHub like AWS and pivots onto operator-provided bare metal: CAPT PXE-boots the machine named by the Tinkerbell Hardware resource, installs Talos with the image resolved from the Hardware's hardware.tinkerbell.org/installer-image annotation, and the single-node Talos cluster takes over as the management cluster. The management-ready wait defaults to 30 minutes (mgmt-ready-timeout in bootstrap.toml) to cover the PXE install and first Talos boot. Scope fence: management-only. There is no workload cluster and no CNI addon; Talos ships flannel. Verify after the pivot with the persisted .kube/krops-mgmt.yaml and kubectl get nodes: one Ready node on the committed endpoint, schedulable for the full management plane (allowSchedulingOnControlPlanes).

OCI Registry (local-host environment only): - Provides a local container registry for development workflows - Enables developers to build and push OCI artifacts from git checkouts - Flux syncs and deploys the OCI artifact without external dependencies - Configurable via REGISTRY_PORT env var (defaults to 5001) - Idempotent: restarts if stopped, no action needed if already running

scripts/toolbox-run.sh resolves current SHA, branch (or detached), and origin URL on the host after loading .env, then forwards them as KROPS_OCI_GIT_SHA, KROPS_OCI_GIT_REF, and KROPS_OCI_SOURCE_URL. These names are reserved for checkout metadata; do not set them in .env, which mise loads again inside the container. The raw helper below supplies all three explicitly. OCI publication uses mise's configuration root and never follows a linked worktree's host-only .git pointer when metadata is supplied. Native runs can derive metadata from Git; a partial or empty supplied tuple fails with an error. Without an origin, the source is a file:// URL for the checkout.

Workflow: 

# Republish the local management and workload folders after making changes
# Resolve metadata on the host, including when this checkout is a linked worktree.
export KROPS_OCI_GIT_SHA="$(git rev-parse HEAD)"
export KROPS_OCI_GIT_REF="$(git branch --show-current)"
export KROPS_OCI_GIT_REF="${KROPS_OCI_GIT_REF:-detached}"
export KROPS_OCI_SOURCE_URL="$(git config --get remote.origin.url || true)"
export KROPS_OCI_SOURCE_URL="${KROPS_OCI_SOURCE_URL:-file://$PWD}"
docker run --rm -it \
  -v "$PWD:/workspace" -w /workspace \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v "$PWD/.kube:/root/.kube" \
  -e KUBECONFIG=/workspace/.kube/krops-mgmt.yaml \
  -e MISE_AUTO_INSTALL=0 --network kind -e REGISTRY_HOST=krops-registry -e REGISTRY_PORT=5000 \
  -e KROPS_OCI_GIT_SHA -e KROPS_OCI_GIT_REF -e KROPS_OCI_SOURCE_URL \
  --entrypoint mise "$TOOLBOX_IMAGE" -E local-host run oci-push

# Optional overrides: add -e OCI_REPOSITORY=my-config -e OCI_TAG=v1 to the same run.

# FluxInstance pulls and reconciles the artifact's mgmt/local-host kustomization.
# The bootstrap configures kind's containerd to mirror localhost:5001 to the
# registry's in-cluster endpoint, krops-registry:5000, which is the endpoint
# the toolbox run above pushes to over the kind network.

The artifact contains only mgmt/local-host/ and workload/local-host/, preserving those directory paths when the artifact is pulled. Keeping the source scope narrow also prevents local credentials and age private keys elsewhere in the repository from being packaged.

The AWS and local-talos environments add the GitHub/SOPS secrets and configure the FluxInstance to sync mgmt/aws/ or mgmt/local-talos/ respectively.

Watch reconciliation after a toolbox run with the persisted management kubeconfig:

export KUBECONFIG="$PWD/.kube/krops-mgmt.yaml"
flux get kustomizations --watch

After a toolbox local-host run on macOS the persisted file carries the kind Docker network address, which Docker Desktop does not route; export the host copy from Host-side access after a toolbox local-host run (macOS) instead.

For the local-host environment, export the CAPD workload kubeconfig after docker-workload-cluster reports Ready. The toolbox run keeps the kind-network endpoint, so the file is read back through the toolbox too:

docker run --rm -it \
  -v "$PWD:/workspace" -w /workspace \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v "$PWD/.kube:/root/.kube" \
  -e KUBECONFIG=/workspace/.kube/krops-mgmt.yaml \
  -e MISE_AUTO_INSTALL=0 --network kind -e KROPS_TOOLBOX=1 \
  --entrypoint mise "$TOOLBOX_IMAGE" -E local-host run kubeconfigs
docker run --rm --network kind -v "$PWD:/workspace" -w /workspace \
  --entrypoint kubectl "$TOOLBOX_IMAGE" --kubeconfig local-workload.kubeconfig get nodes

The host task form of kubeconfigs still reads the persisted management kubeconfig before it can export the workload one, so on macOS it must run against the host copy from Host-side access after a toolbox local-host run (macOS).

The local workload Flux instance installs Podinfo from its OCI Helm chart. Open it in a host browser by running the port-forward in a separate terminal. This is the one helper that stays a host task on purpose (the browser is on the host); it needs a host kubectl and a kubeconfig with the 127.0.0.1 endpoint, which mise -E local-host run kubeconfigs on the host produces:

mise -E local-host run podinfo-port-forward

On an engine-only host, publish the port from a toolbox run on the kind network instead:

docker run --rm -it --network kind -p 9898:9898 \
  -v "$PWD:/workspace" -w /workspace -e MISE_AUTO_INSTALL=0 \
  --entrypoint kubectl "$TOOLBOX_IMAGE" --kubeconfig local-workload.kubeconfig \
  port-forward --namespace podinfo --address 0.0.0.0 service/podinfo 9898:9898

The workload uses Kubernetes v1.36.4. A CAPI ClusterResourceSet installs a pinned Kindnet daemon as its CNI before the Flux addons are delivered. The management cluster needs access to the container-engine socket, which the bootstrap mounts automatically.

Local-host bootstrap first waits for the management flux-apps Kustomization without printing transient Unknown status rows. After flux-apps becomes Ready, it connects to local-workload, streams the workload Flux controller error logs, and returns after the workload root Kustomization becomes Ready. Filtering the workload stream to errors avoids showing normal startup retries and advisory messages as apparent failures. Each readiness wait defaults to 15 minutes and can be changed with LOCAL_RECONCILE_TIMEOUT in a raw container or fallback native run. The current wrapper does not forward that override.

EKS clusters typically take 15–25 minutes to come up; node groups and the downstream app chain follow a few minutes after.

Host-side access after a toolbox local-host run (macOS)

The pivot exports the management kubeconfig with the API server address CAPD recorded: the local-management-lb container IP on the kind Docker network. Native local-host runs rewrite that address to the localhost port, but toolbox runs skip the rewrite (should_rewrite_capd_endpoint in bootstrap-rs/src/main.rs) because the address resolves on the kind network the toolbox joins. Docker Desktop on macOS runs the engine in a VM and does not route the kind network from the host, so host commands against the persisted .kube/krops-mgmt.yaml time out. Rewrite a host copy to the port kind publishes on localhost:

cp .kube/krops-mgmt.yaml .kube/krops-mgmt.host.yaml
PORT=$(docker port local-management-lb 6443/tcp | head -1 | sed 's/.*://')
kubectl config set-cluster local-management --server="https://127.0.0.1:${PORT}" \
  --kubeconfig .kube/krops-mgmt.host.yaml

Use the host copy for every host command that talks to the management cluster:

export KUBECONFIG="$PWD/.kube/krops-mgmt.host.yaml"
flux get kustomizations --watch

The host form of the kubeconfigs task reads the management kubeconfig before it can export the workload one, so run it with the host copy exported; without KROPS_TOOLBOX it rewrites the workload endpoint to 127.0.0.1, and podinfo-port-forward then needs no further changes:

mise -E local-host run podinfo-port-forward

The published port belongs to the local-management-lb container and stays the same while that container exists. It changes when the container is recreated, for example after teardown plus bootstrap. Re-run the docker port and kubectl config set-cluster lines whenever docker port local-management-lb 6443/tcp reports a different port than the kubeconfig carries.

With Podman on macOS the same limitation applies inside the podman machine VM; use podman port in place of docker port.

On Linux with a rootful engine, the host routes to the kind network directly and no kubeconfig rewrite is needed. See Host-side access after a toolbox local-host run (Linux) for file ownership and Docker socket access.

The unaffected alternative is a one-off toolbox container attached to the kind network, where the recorded address resolves with no rewrite:

docker run --rm -it --network kind \
  -v "$PWD/.kube:/root/.kube" \
  -e KUBECONFIG=/root/.kube/krops-mgmt.yaml \
  --entrypoint flux \
  "$TOOLBOX_IMAGE" get kustomizations --watch

Use --entrypoint kubectl for kubectl commands. The mise helper tasks are host forms; run them against the rewritten copy.

Teardown is unaffected: it reads the management kubeconfig from inside the toolbox, which is attached to the kind network.

Host-side access after a toolbox local-host run (Linux)

Scope. This section applies to native Docker Engine or rootful Podman on Linux. With Docker Desktop for Linux (which runs the engine in a VM), follow the macOS section instead. Rootless engines keep container networks in their own namespace, so the kind address does not route from the host there either.

Docker socket access

If the current user cannot access /var/run/docker.sock, docker info fails and scripts/toolbox-run.sh silently falls back to podman info, which then fails on the rootful Podman socket (/run/podman/podman.sock).

Fix: add your user to the docker group and log out and back in:

sudo usermod -aG docker "$USER"
newgrp docker  # or log out and back in
docker info    # confirm it works without sudo

Note: docker group membership is root-equivalent (the daemon runs as root and accepts commands from group members).

As an alternative, pin the engine in .env so a socket problem fails on Docker instead of falling back silently:

CONTAINER_ENGINE=docker

File ownership

The toolbox runs as root; on a rootful engine its writes into the checkout are root-owned. Affected files: .kube/kind.yaml, .kube/krops-mgmt.yaml (mode 0600, so host kubectl gets a permission denied), and local-workload.kubeconfig if you have run the kubeconfigs cluster task.

Fix:

sudo chown -R "$(id -u):$(id -g)" .kube
sudo chown "$(id -u):$(id -g)" local-workload.kubeconfig 2>/dev/null || true

Repeat when a later lifecycle run recreates those files (teardown and re-bootstrap, for example). Rewrites of an existing file keep your ownership.

This is not needed on Docker Desktop for Linux or rootless engines.

Management cluster access

After fixing ownership, the management kubeconfig works directly from the host:

export KUBECONFIG="$PWD/.kube/krops-mgmt.yaml"
flux get kustomizations --watch

Workload cluster access

Linux hosts route the kind network directly. Export the CAPD workload kubeconfig with its kind-network endpoint and use it from host kubectl without a port mapping:

docker run --rm --network kind -v "$PWD:/workspace" -w /workspace \
  -e KUBECONFIG=/workspace/.kube/krops-mgmt.yaml \
  --entrypoint clusterctl "$TOOLBOX_IMAGE" get kubeconfig local-workload -n default > local-workload.kubeconfig
kubectl --kubeconfig local-workload.kubeconfig port-forward -n podinfo service/podinfo 9898:9898

The redirect runs in the host shell, so the file is owned by you. Use local-workload.kubeconfig (gitignored; the default for mise -E local-host run podinfo-port-forward). The workload.kubeconfig name from some earlier examples is not gitignored; avoid it.

mise -E local-host run podinfo-port-forward then works unchanged once ownership and kubeconfig export are done.

Verifying the full chain

For the local-host end-to-end chain:

TOOLBOX_IMAGE="$TOOLBOX_IMAGE" scripts/toolbox-run.sh bootstrap local-host
docker run --rm -it \
  -v "$PWD:/workspace" -w /workspace \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v "$PWD/.kube:/root/.kube" \
  -e KUBECONFIG=/workspace/.kube/krops-mgmt.yaml \
  -e MISE_AUTO_INSTALL=0 --network kind -e KROPS_TOOLBOX=1 \
  --entrypoint mise "$TOOLBOX_IMAGE" -E local-host run kubeconfigs
docker run --rm --network kind -v "$PWD:/workspace" -w /workspace \
  --entrypoint flux "$TOOLBOX_IMAGE" --kubeconfig local-workload.kubeconfig get all --all-namespaces
# then the port-forward from the section above, and browse to http://localhost:9898

The toolbox form above needs no rewrite: the runs stay on the kind network. Running those steps as host commands instead needs the host copy from Host-side access after a toolbox local-host run (macOS) in place of .kube/krops-mgmt.yaml.

Bootstrap does not return until the management and workload root Kustomizations are Ready. The final port-forward verifies that the workload Flux instance successfully delivered the application.

For the AWS chain:

# Management cluster after a toolbox run
export KUBECONFIG="$PWD/.kube/krops-mgmt.yaml"
kubectl get kustomizations -n flux-system            # all Ready (incl. ack-controllers, workload-resources)
kubectl get clusters.cluster.x-k8s.io -A             # Provisioned
kubectl get buckets.s3.services.k8s.aws -n ack-system
kubectl get dbinstances.rds.services.k8s.aws -n ack-system
kubectl get roles.iam.services.k8s.aws -n ack-system

# Workload clusters: export kubeconfigs first (see the AWS page for the
# toolbox run of `kubeconfigs`), then
#   export KUBECONFIG=.kube/krops-workloads.yaml
#   kubectl config use-context eu-north-1-workload   (or eu-west-1-workload)
kubectl get kustomizations -n flux-system            # root only; workload/base is empty since #346

# AWS
aws s3api get-bucket-encryption    --bucket krops-<account>-eu-north-1-workload-data
aws s3api get-public-access-block  --bucket krops-<account>-eu-north-1-workload-data

Pivot recovery

Bootstrap ends with a pivot: the CAPI inventory moves from the local mgmt kind cluster into the self-managed management cluster, and the kind cluster is deleted. scripts/toolbox-run.sh bootstrap runs the pivot by default (BOOTSTRAP_PIVOT=0 opts out). scripts/toolbox-run.sh pivot starts the same rerun-safe CLI and resumes through the pivot; there is no separate pivot subcommand. See the bootstrap CLI for its interface and controls.

clusterctl move is re-runnable: an object is deleted from the source kind cluster only after it was created on the target, so kind stays authoritative until the final kind deletion. On a clean first run the pivot also WAITS for two Flux-driven prerequisites instead of failing fast: the management Cluster definition (polled up to MGMT_READY_TIMEOUT, surfacing failed Kustomizations on timeout) and the first target nodes (up to 15m, tolerating a nodeless EKS start). A timeout in either prints the Kustomization or node state and is safe to re-run. If a pivot phase fails:

  1. Fix the reported cause.
  2. Re-run the pivot (scripts/toolbox-run.sh pivot, or rerun bootstrap) from a checkout of the revision you want self-managed, normally main. A fallback native run must use the bootstrap context (kind-mgmt; BOOTSTRAP_KUBECONTEXT overrides). Toolbox mode selects kind's internal kubeconfig automatically. The CLI reuses an existing healthy mgmt kind cluster on rerun.
  3. Set PIVOT_SKIP_DELETE=1 to keep the kind bootstrap cluster around for inspection once the pivot completes.

Note: never delete moved Cluster, AWSManaged*, MachinePool, or Dev* objects on the target cluster to work around a failure. The CAPI providers treat deletion as deprovisioning and destroy the real infrastructure (EKS clusters, VPCs, IAM roles; CAPD the local containers). Re-run the move instead. Only pure-config duplicates without a move hook (for example an identity created by hand during a failed install) are safe to delete.

The management kubeconfig is written to MGMT_KUBECONFIG, with context krops-mgmt. The toolbox mount makes its /root/.kube/krops-mgmt.yaml appear on the host as ./.kube/krops-mgmt.yaml; a fallback native run defaults to ~/.kube/krops-mgmt.yaml. After a toolbox local-host run on macOS the file carries the kind Docker network address; see Host-side access after a toolbox local-host run (macOS) for host-side use.

Teardown

The wrapper runs the Rust subcommand in the toolbox:

TOOLBOX_IMAGE="$TOOLBOX_IMAGE" scripts/toolbox-run.sh teardown            # aws
TOOLBOX_IMAGE="$TOOLBOX_IMAGE" scripts/toolbox-run.sh teardown local-host
TOOLBOX_IMAGE="$TOOLBOX_IMAGE" scripts/toolbox-run.sh teardown local-talos

Azure teardown is manual; the CLI prints the steps.

The retained ./teardown.sh reference path and a native krops-bootstrap teardown [PROFILE] run are the fallbacks. The positional profile selects an environment from bootstrap.toml. Teardown reads resource names and targets from bootstrap.toml and discovers where the CAPI controllers are running:

  • the mgmt kind cluster before pivot
  • the exported self-managed management kubeconfig after pivot
  • no reachable Kubernetes controller host, in which case the AWS environment falls back to the AWS orphan sweep. In the toolbox, discovery first attaches the container to the kind network so the pre-pivot kind endpoint (mgmt-control-plane:6443) resolves; a host run already reaches it. When the host is genuinely unreachable and the run was not AWS_ONLY=1, the run finishes the sweep but then reports failure (nonzero exit) so automation does not read a skipped k8s side as success. AWS_ONLY=1 is the explicit exit-0 recovery path (see orphan report above for daily read-only discovery)

The main controls keep the shell interface:

Variable Default Effect
AWS_ONLY 0 Literal 1 runs only the AWS orphan sweep; invalid with local-host and local-talos
FORCE_KIND_DELETE 0 Literal 1 overrides the final controller-host deletion guard
CLUSTER_DELETE_TIMEOUT 1200 seconds CAPI cluster deletion wait (aws workloads, local-talos management)
PROVIDER_DELETE_TIMEOUT 300 seconds CAPI provider deletion wait
MGMT_KUBECONFIG ~/.kube/krops-mgmt.yaml (in the toolbox that is the checkout's .kube/ mount) Post-pivot controller-host kubeconfig

The hard preflight depends on the mode:

Mode Required tools
local-host kind, kubectl; AWS_ONLY=1 is rejected
local-talos kind, kubectl; AWS_ONLY=1 is rejected
normal aws kind, helm, kubectl, xargs; a missing AWS CLI skips the orphan sweep
AWS_ONLY=1 AWS CLI only

A required-tool failure happens before mutation. The wrapper does not forward the four teardown controls in the table above; use a raw container invocation with explicit -e entries or a fallback native run for recovery overrides.

For local-host, teardown suspends the workload Kustomization, deletes the CAPD workload cluster, waits for its containers to disappear, removes either the pre-pivot kind cluster or the post-pivot self-managed management containers, and removes krops-registry last.

For local-talos, teardown suspends Flux and deletes every CAPI Cluster, the management cluster included: the deletion IS the release, and CAPT returns the machine's Hardware entry to the Tinkerbell pool. It waits for the deprovision to complete, then deletes the kind bootstrap cluster if the pivot has not run yet. The machine itself is never wiped: it keeps running Talos for the operator to re-use or PXE-boot fresh. There is no orphan sweep; the environment owns no cloud resources.

For aws, teardown suspends Flux, deletes every workload CAPI Cluster while leaving the management Cluster object alone, and waits before touching the controller host. It then runs a best-effort AWS sweep for both workload regions and the self-managed management cluster. The sweep removes nodegroups, EKS control planes, orphaned RDS instances, CAPA-tagged VPC resources in dependency order, versioned S3 buckets, CAPA and ACK IAM roles, the krops-reader user, and the clusterawsadm CloudFormation stack. It removes CAPI providers and bootstrap Helm releases when the controller host remains reachable.

The controller-host guard prevents removal while CAPI workload deletion is unconfirmed. Do not bypass it unless you accept orphaned infrastructure. If a management cluster is already unreachable, rerun with AWS_ONLY=1 to make the recovery mode explicit. A missing AWS CLI in normal AWS mode is reported and the orphan sweep is skipped; AWS_ONLY=1 requires the CLI and fails preflight without it.

ACK resources can survive if their workload cluster disappears before their custom resources finish deleting. The explicit sweep is what removes those orphans, including both workload regions and the self-managed management cluster.

Validation

Run the repository validation before pushing:

mise run validate

The task checks shell syntax for the retained lifecycle scripts, runs the bootstrap.toml manifest cross-check and the toolbox/doc tests, and builds every kustomize overlay under mgmt/, workload/, and virtualized-e2e/. It is repository development, so it stays a host task on purpose (it needs the pinned Python, uv, and kubectl). Contributors without a host toolchain can run the same task in the toolbox, redirecting the uv environment off the mount:

docker run --rm -v "$PWD:/workspace" -w /workspace \
  -e MISE_AUTO_INSTALL=0 -e UV_PROJECT_ENVIRONMENT=/tmp/krops-venv \
  --entrypoint mise "$TOOLBOX_IMAGE" run validate

.github/workflows/validate.yml separately builds every overlay, runs the Renovate air-gap digest and managed-pin coverage tests, cross-checks bootstrap.toml, and lints YAML on pushes to main and on pull requests. .github/workflows/bootstrap-rs.yml runs Rust format, clippy, build, and tests, then builds and smokes the toolbox image when its inputs change.

Managed EKS worker lifecycle tags

CAPA's ASG tags propagate to later workers; initial workers can launch before those ASG tags exist. The management worker-run-tags minute CronJob backfills or repairs the four lifecycle keys on currently discoverable managed workers, using each EKS cluster's own tags. It preserves other instance tags and does not replace pools. This is eventual tagging: API availability, the external IAM prerequisite in AWS IAM, and a successful Job are required. A worker that leaves its ASG before discovery can remain untagged. A cluster with missing or malformed metadata is skipped and the Job fails while independent clusters are still processed.

Lifecycle values are decoded from AWS JSON and passed through NUL-delimited fields without whitespace splitting or trimming. Tag writes use a JSON array, preserving supported CAPA ASCII punctuation and whitespace exactly, including leading/trailing space, tab, LF, FF, and CR. Vertical tab and non-ASCII values are rejected; lifecycle values must contain 1–256 ASCII bytes. Expiry must be never or an exact UTC YYYY-MM-DDTHH:MM:SSZ timestamp with a valid calendar date and clock time (years 0001–9999); invalid dates stop that cluster's writes while other clusters continue.

The Job uses AWS CLI pagination for nodegroups/ASGs, checks account and the three committed region/cluster pairs, deduplicates workers, then rechecks ASG membership and EKS ownership before applying drift-only changes. Disappearing or terminated workers are skipped. Jobs cannot overlap, have a 55-second execution deadline and no Kubernetes retries; the next minute retries failed work. At larger worker counts, monitor deadline failures before relying on the interval. IAM denials are errors, with no fallback to broader discovery.

Live acceptance after merge and the operator's IAM grant:

  1. Verify the effective static-principal policy includes the documented scoped reads and ownership/tag-key-restricted ec2:CreateTags permission.
  2. Observe a successful worker-run-tags Job on the management cluster.
  3. Discover an actual EC2 instance through each allowed EKS managed nodegroup's ASG. Use aws ec2 describe-instances --region <region> --instance-ids <id> to compare its four lifecycle tags with aws eks describe-cluster --region <region> --name <cluster>. Include an initial worker with missing tags and a new scale-out worker; verify unrelated tags remain unchanged.
  4. Repeat after a minute and verify unchanged instances receive no new writes in Job logs/CloudTrail. Check that all three clusters' metadata stays distinct.

Offline tests cover discovery and write guards; live acceptance has not been performed by this change. It does not establish tagging for orphaned workers, CloudFormation stacks, or other excluded resource kinds.