devnet/testnet: deploy device-health-oracle

[nikw9944]

1. Add a device-health-oracle dir like https://github.com/malbeclabs/infra/tree/main/k8s/shred-oracle and model it after that

2. Add the secret signing key for each environment in dz-{devnet,testnet} here: https://github.com/malbeclabs/infra/tree/main/k8s/secrets

3. There's some shared config in https://github.com/malbeclabs/infra/tree/main/k8s/config we can use if needed/appropriate

4. Let's make it auto-deploy to devnet on merge to the main branch of https://github.com/malbeclabs/doublezero/. There's an example of https://github.com/malbeclabs/lake/ doing it via https://github.com/malbeclabs/infra/blob/main/k8s/argocd/image-updater/lake-staging.yaml

Note: Please also read the relevant README.md files - k8s/shred-oracle/README.md, k8s/config/README.md, and k8s/secrets/README.md.

[nikw9944]

Work Plan (Revision 3)

Work Plan (Revision 3)

Work Plan: Issue #3420 — Deploy device-health-oracle to devnet/testnet

Summary

The device-health-oracle has a k8s deployment defined in the infra repo for devnet, but it uses a hardcoded image tag (ghcr.io/malbeclabs/doublezero-core:nikw-2669) instead of :edge, which means it never picks up new builds pushed on every commit to main. The pod hasn't been updated since 2026-03-24 despite continuous Docker image builds. The fix is threefold: (1) update the devnet deployment to use the :edge tag so it tracks main, (2) create a testnet deployment for the device-health-oracle, and (3) create the testnet device-health-oracle-env SOPS-encrypted secret.

Approach

Root cause analysis

The release.docker.core.yml workflow builds and pushes ghcr.io/malbeclabs/doublezero-core:edge on every push to main. The Dockerfile includes doublezero-device-health-oracle. However, the devnet deployment in infra/k8s/kustomize/core/overlays/devnet/device-health-oracle.yaml references ghcr.io/malbeclabs/doublezero-core:nikw-2669 — a branch-specific tag from initial development. This tag never gets updated.

Even after fixing the tag to :edge, Kubernetes won't automatically restart pods when the image behind an :edge tag changes. The other devnet telemetry deployments use :edge via kustomize patches, but they face the same issue — they only get updated when someone manually runs k8s/scripts/deploy-core.sh devnet or restarts the pods. There is no ArgoCD application for the core kustomize overlays (unlike the shred-oracle which has its own ArgoCD app with selfHeal: true).

Proposed changes

1. Fix devnet image tag (infra repo)
Update device-health-oracle.yaml to use :edge instead of :nikw-2669, consistent with how other devnet telemetry deployments reference images.

2. Create testnet deployment (infra repo)
Add a device-health-oracle.yaml to the testnet overlay, modeled on the devnet deployment but with --env testnet and testnet-specific PodMonitor labels.

3. Add testnet kustomization reference (infra repo)
Update k8s/kustomize/core/overlays/testnet/kustomization.yaml to include the new device-health-oracle.yaml resource and add an image patch to use :edge.

4. Create testnet SOPS-encrypted secret (infra repo)
Create the device-health-oracle-env.enc.yaml secret in the testnet overlay. This secret provides the environment variables the oracle needs at runtime:

• DZ_LEDGER_URL — testnet RPC endpoint
• DZ_SERVICEABILITY_PROGRAM_ID — testnet serviceability program ID
• DZ_TELEMETRY_PROGRAM_ID — testnet telemetry program ID
• DZ_SIGNER_KEYPAIR — signer keypair for health update transactions

The secret must be encrypted with SOPS using the appropriate AWS KMS key for the testnet namespace. Since SOPS encryption requires AWS credentials and access to the KMS key, the encrypted secret file will need to be created by the operator (or someone with AWS access) and added to the PR before merge. The PR should include a plaintext template showing the required keys so the operator knows exactly what to populate.

Out of scope

The broader issue of automated rollouts (pods not restarting when :edge is updated) affects all core k8s deployments, not just device-health-oracle. Solving this comprehensively (e.g., adding an ArgoCD application for core overlays, or a post-push workflow that does kubectl rollout restart) is a separate concern. This plan focuses on getting device-health-oracle correctly configured so that manual deploys via deploy-core.sh will pick up the latest image.

Files to Change

All changes are in the infra repo:

File	Change
k8s/kustomize/core/overlays/devnet/device-health-oracle.yaml	Change image tag from :nikw-2669 to :edge
k8s/kustomize/core/overlays/testnet/device-health-oracle.yaml	New file — deployment + PodMonitor for testnet, modeled on devnet version with --env testnet and testnet labels
k8s/kustomize/core/overlays/testnet/kustomization.yaml	Add device-health-oracle.yaml to resources list
k8s/kustomize/core/overlays/testnet/device-health-oracle-env.enc.yaml	New file — SOPS-encrypted secret containing DZ_LEDGER_URL, DZ_SERVICEABILITY_PROGRAM_ID, DZ_TELEMETRY_PROGRAM_ID, and DZ_SIGNER_KEYPAIR for the testnet environment. Operator to create/encrypt this file with the appropriate KMS key and add to the PR.

Risks & Considerations

1. SOPS encryption requires AWS access: The device-health-oracle-env.enc.yaml file must be encrypted with the correct KMS key. The implementation will prepare the secret structure, but the operator needs to encrypt it with sops --encrypt using the testnet KMS key. The operator has indicated they can add this to the PR manually.

2. No automatic rollout: Changing the tag to :edge alone does not cause running pods to restart. After merging, someone needs to manually apply via deploy-core.sh devnet (and deploy-core.sh testnet) or do a kubectl rollout restart. This is the same situation as other telemetry deployments. The PR description should note this as a post-merge step.

3. No deploy-core.devnet workflow: There is no deploy-core.devnet.yml workflow in the infra repo (there is one for testnet and mainnet-beta). Devnet core deploys appear to be manual only via deploy-core.sh. This is an existing gap but out of scope.

Testing Strategy

1. Kustomize build validation: Run kustomize build k8s/kustomize/core/overlays/devnet/ and kustomize build k8s/kustomize/core/overlays/testnet/ to verify the manifests render correctly without errors.
2. k8s-lint: The infra repo has a make k8s-lint target that runs on PRs — verify this passes.
3. Dry-run deploy: After merge, run deploy-core.sh devnet --dry-run to verify the diff shows only the image tag change for device-health-oracle.
4. Manual verification: After applying, verify the device-health-oracle pod is running with the :edge image and starts successfully.

Estimated Scope

• ~60 lines added (new testnet deployment YAML + secret template + kustomization.yaml edits)
• Well within the 500-line threshold

[nikw9944]

Work Log — waiting_review

Let me read the referenced files from the infra repo to understand the patterns.

Now let me read the key files directly from the infra repo to get exact content.

Let me check the Docker release workflow and the existing devnet deployment more carefully.

Now let me check the remaining pieces — the config components and the Docker image/workflow details.

Let me check for devnet config/secrets and ArgoCD structure.

Now let me check the entrypoint.sh for the device-health-oracle to understand what env vars it uses.

Let me check the config package for network configs and what program IDs are used for each env.

Now let me check the program ID constants and the Dockerfile for doublezero-core.

Good. Now I have all the information I need. Let me write the revised plan.

Revised plan saved to docs/work-plan-3420.md. Here's the summary of the new direction:

7 steps, all in the infra repo:

1. k8s/device-health-oracle/ — new standalone directory modeled exactly after k8s/shred-oracle/ with base/ deployment + devnet/ and testnet/ overlays
2. Secrets — device-health-oracle-env.enc.yaml (signer keypair) in both dz-devnet and dz-testnet
3. Shared config — new k8s/config/devnet/ component; add DZ_TELEMETRY_PROGRAM_ID to existing testnet config; add doublezero-core image entries
4. dz-devnet secrets dir — new directory (doesn't exist today) with shared-secrets, ghcr-creds, and device-health-oracle-env
5. ArgoCD Applications — device-health-oracle-devnet and device-health-oracle-testnet with auto-sync + self-heal
6. ArgoCD ImageUpdater — watches doublezero-core:edge digest, auto-deploys devnet on merge to main in doublezero
7. Cleanup — remove old k8s/kustomize/core/overlays/devnet/device-health-oracle* (separate PR after migration verified)

Key considerations flagged: the dz-devnet secrets directory is entirely new, liveness probe /readyz needs verification, and the migration should be staged (new deploy → verify → delete old) to avoid running two instances simultaneously.

STATUS: COMPLETE — Revised plan is written and ready for re-review.

--- Result: success | Session cost: $1.7430 | Turns: 40 | Tokens: 1897 in / 13925 out ---

[nikw9944]

Approved Work Plan (Revision 2)

Approved Work Plan (Revision 2)

Work Plan: Issue #3420 — Deploy device-health-oracle to devnet/testnet

Revision 2

Summary

Migrate the device-health-oracle from the legacy k8s/kustomize/core/ deployment pattern to a standalone ArgoCD-managed deployment following the shred-oracle pattern. Create a k8s/device-health-oracle/ directory with base/overlay structure, add SOPS-encrypted signer keypair secrets for devnet and testnet, wire up ArgoCD Applications, and configure an ArgoCD ImageUpdater to auto-deploy to devnet on every push to main in the doublezero repo.

Context

Current state: The device-health-oracle runs in devnet only, deployed via k8s/kustomize/core/overlays/devnet/device-health-oracle.yaml with a stale image tag (ghcr.io/malbeclabs/doublezero-core:nikw-2669). There is no testnet deployment.

Target state: A standalone k8s/device-health-oracle/ directory modeled after k8s/shred-oracle/, with:

• ArgoCD Applications for devnet and testnet (auto-sync, self-heal)
• ArgoCD ImageUpdater watching ghcr.io/malbeclabs/doublezero-core:edge to auto-deploy devnet on merge to main
• Per-environment SOPS-encrypted secrets for the signer keypair
• Shared config components for environment-specific values

Docker image: ghcr.io/malbeclabs/doublezero-core — built and pushed by release.docker.core.yml in the doublezero repo. On push to main, tags edge and sha-<short> are produced. The binary inside is doublezero-device-health-oracle.

Approach

1. Create k8s/device-health-oracle/ directory (infra repo)

Follow the shred-oracle pattern exactly:

k8s/device-health-oracle/
├── README.md
├── base/
│   ├── deployment.yaml
│   └── kustomization.yaml
├── devnet/
│   └── kustomization.yaml
└── testnet/
    └── kustomization.yaml

base/deployment.yaml — shared deployment spec:

• Image: ghcr.io/malbeclabs/doublezero-core:PLACEHOLDER (tag set by config component or ImageUpdater)
• Command: doublezero-device-health-oracle
• Args: --signer-keypair /etc/device-health-oracle/keypair.json --ledger-rpc-url $(DZ_LEDGER_URL) --serviceability-program-id $(DZ_SERVICEABILITY_PROGRAM_ID) --telemetry-program-id $(DZ_TELEMETRY_PROGRAM_ID)
• envFrom: configMapRef: shared-config + secretRef: shared-secrets (consistent with shred-oracle)
• Volume: keypair mounted from device-health-oracle-env secret
• Metrics port: 8080 (existing convention for this service)
• Grafana scrape annotations + reloader annotation
• Liveness probe on /readyz endpoint (port 8080)
• Resources: 50m/64Mi requests, 500m/256Mi limits (same as shred-oracle)
• imagePullSecrets: ghcr-creds
• Strategy: Recreate

Note: Unlike the existing devnet deployment which uses --env devnet (relying on built-in config), the new deployment uses explicit flags with values from shared-config/shared-secrets. This is consistent with how shred-oracle works and decouples the binary from environment awareness at deploy time.

base/kustomization.yaml — references deployment.yaml, adds app.kubernetes.io/part-of: device-health-oracle label.

devnet/kustomization.yaml — overlay:

• References ../base
• Includes ../../config/devnet component (new — see step 3)
• Sets namespace dz-devnet
• Adds env: devnet label

testnet/kustomization.yaml — overlay:

• References ../base
• Includes ../../config/testnet component
• Sets namespace dz-testnet
• Adds env: testnet label

2. Create secrets in k8s/secrets/ (infra repo)

k8s/secrets/dz-devnet/device-health-oracle-env.enc.yaml (new file):

• Secret named device-health-oracle-env in namespace dz-devnet
• Contains keypair.json — the signer keypair for devnet
• Encrypted with SOPS using the AWS KMS key

k8s/secrets/dz-testnet/device-health-oracle-env.enc.yaml (new file):

• Secret named device-health-oracle-env in namespace dz-testnet
• Contains keypair.json — the signer keypair for testnet
• Encrypted with SOPS using the AWS KMS key

k8s/secrets/dz-devnet/kustomization.yaml — this directory does not exist yet. Create it with references to device-health-oracle-env.enc.yaml, plus shared-secrets.enc.yaml and ghcr-creds.enc.yaml (which also need to be created for the dz-devnet namespace).

k8s/secrets/dz-testnet/kustomization.yaml — add device-health-oracle-env.enc.yaml to the resources list.

Operator action required: The actual encrypted secret files must be created by someone with AWS KMS access. The plan provides the plaintext template structure; the operator encrypts with sops -e.

3. Create/update shared config components (infra repo)

k8s/config/devnet/ (new directory) — create a devnet config component:

kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

resources:
  - shared-config.yaml

images:
  - name: ghcr.io/malbeclabs/doublezero-core
    newTag: "edge"

shared-config.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: shared-config
data:
  SOLANA_URL: "devnet"
  DZ_SERVICEABILITY_PROGRAM_ID: "GYhQDKuESrasNZGyhMJhGYFtbzNijYhcrN9poSqCQVah"
  DZ_TELEMETRY_PROGRAM_ID: "C9xqH76NSm11pBS6maNnY163tWHT8Govww47uyEmSnoG"

k8s/config/testnet/shared-config.yaml — add DZ_TELEMETRY_PROGRAM_ID:

DZ_TELEMETRY_PROGRAM_ID: "3KogTMmVxc5eUHtjZnwm136H5P8tvPwVu4ufbGPvM7p1"

k8s/config/testnet/kustomization.yaml — add doublezero-core image entry:

images:
  - name: ghcr.io/malbeclabs/doublezero-shreds
    newTag: "0.1.1"
  - name: ghcr.io/malbeclabs/doublezero-core
    newTag: "edge"

Note: The newTag for testnet doublezero-core should be set to whatever tag the operator wants for testnet. Using edge is one option (tracks main), or a pinned version tag for more control. The operator should decide — edge is used here as a placeholder.

4. Create k8s/secrets/dz-devnet/ directory (infra repo)

The dz-devnet secrets directory does not exist yet. It needs:

• kustomization.yaml — listing all secret resources
• shared-secrets.enc.yaml — containing DZ_LEDGER_URL for devnet (the DZ ledger RPC endpoint)
• ghcr-creds.enc.yaml — GHCR image pull credentials for this namespace
• device-health-oracle-env.enc.yaml — signer keypair (from step 2)

The shared-secrets and ghcr-creds secrets may already exist in the dz-devnet namespace from the core kustomize deployment. The operator should verify and either:

• Create new SOPS-encrypted copies for the standalone secrets directory, or
• Confirm they're already synced via another mechanism (e.g., Reflector annotations on ghcr-creds)

5. Create ArgoCD Applications (infra repo)

k8s/argocd/applications/devnet/device-health-oracle.yaml (new file + new devnet/ directory):

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: device-health-oracle-devnet
  namespace: argocd
  annotations:
    notifications.argoproj.io/subscribe.dz-on-change.dz_slack: "#tmp-notify"
spec:
  project: default
  source:
    repoURL: https://github.com/malbeclabs/infra.git
    targetRevision: main
    path: k8s/device-health-oracle/devnet
  destination:
    server: https://kubernetes.default.svc
    namespace: dz-devnet
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

k8s/argocd/applications/testnet/device-health-oracle.yaml (new file):

• Same pattern, name device-health-oracle-testnet, path k8s/device-health-oracle/testnet, namespace dz-testnet

6. Create ArgoCD ImageUpdater for devnet auto-deploy (infra repo)

k8s/argocd/image-updater/device-health-oracle-devnet.yaml (new file):

apiVersion: argocd-image-updater.argoproj.io/v1alpha1
kind: ImageUpdater
metadata:
  name: device-health-oracle-devnet
  namespace: argocd
spec:
  namespace: argocd
  applicationRefs:
    - namePattern: device-health-oracle-devnet
      images:
        - alias: doublezero-core
          imageName: ghcr.io/malbeclabs/doublezero-core:edge
          commonUpdateSettings:
            updateStrategy: digest
            pullSecret: pullsecret:argocd/ghcr-creds

This watches the :edge tag's digest. When release.docker.core.yml pushes a new doublezero-core:edge image on merge to main, the ImageUpdater detects the digest change and triggers an ArgoCD sync of the devnet Application — achieving auto-deploy on merge to main.

7. Remove old devnet deployment from core overlay (infra repo)

Remove the device-health-oracle from the legacy kustomize/core path:

• Delete k8s/kustomize/core/overlays/devnet/device-health-oracle.yaml
• Delete k8s/kustomize/core/overlays/devnet/device-health-oracle-env.enc.yaml
• Remove references from k8s/kustomize/core/overlays/devnet/kustomization.yaml

This should be done after the new ArgoCD-managed deployment is confirmed working, to avoid downtime. Can be a follow-up commit or separate PR.

Files to Change (all in infra repo)

File	Change
k8s/device-health-oracle/README.md	New — documents the deployment structure
k8s/device-health-oracle/base/deployment.yaml	New — shared deployment spec
k8s/device-health-oracle/base/kustomization.yaml	New — base kustomization
k8s/device-health-oracle/devnet/kustomization.yaml	New — devnet overlay
k8s/device-health-oracle/testnet/kustomization.yaml	New — testnet overlay
k8s/config/devnet/kustomization.yaml	New — devnet config component
k8s/config/devnet/shared-config.yaml	New — devnet shared ConfigMap
k8s/config/testnet/shared-config.yaml	Edit — add DZ_TELEMETRY_PROGRAM_ID
k8s/config/testnet/kustomization.yaml	Edit — add doublezero-core image entry
k8s/secrets/dz-devnet/kustomization.yaml	New — devnet secrets kustomization
k8s/secrets/dz-devnet/device-health-oracle-env.enc.yaml	New — signer keypair (operator encrypts)
k8s/secrets/dz-devnet/shared-secrets.enc.yaml	New — DZ ledger URL (operator encrypts)
k8s/secrets/dz-devnet/ghcr-creds.enc.yaml	New — GHCR pull creds (operator encrypts)
k8s/secrets/dz-testnet/device-health-oracle-env.enc.yaml	New — signer keypair (operator encrypts)
k8s/secrets/dz-testnet/kustomization.yaml	Edit — add device-health-oracle-env.enc.yaml
k8s/argocd/applications/devnet/device-health-oracle.yaml	New — ArgoCD Application
k8s/argocd/applications/testnet/device-health-oracle.yaml	New — ArgoCD Application
k8s/argocd/image-updater/device-health-oracle-devnet.yaml	New — auto-deploy on edge digest change
k8s/secrets/README.md	Edit — add dz-devnet section to current secrets list

Post-migration cleanup (separate PR)

File	Change
k8s/kustomize/core/overlays/devnet/device-health-oracle.yaml	Delete
k8s/kustomize/core/overlays/devnet/device-health-oracle-env.enc.yaml	Delete
k8s/kustomize/core/overlays/devnet/kustomization.yaml	Edit — remove device-health-oracle refs

Risks & Considerations

1. dz-devnet secrets directory is new. No other service currently uses standalone secrets for devnet. The shared-secrets and ghcr-creds secrets need to be created for this namespace. The operator should verify whether these already exist in the cluster (they may have been created by the core kustomize deployment) and decide whether to create SOPS-encrypted copies or rely on existing mechanisms.

2. Shared config changes affect shred-oracle. Adding DZ_TELEMETRY_PROGRAM_ID to k8s/config/testnet/shared-config.yaml and adding doublezero-core to the images transformer in k8s/config/testnet/kustomization.yaml will be picked up by shred-oracle's testnet overlay too. The additional ConfigMap key is harmless (shred-oracle ignores unknown env vars). The additional image entry is also harmless (kustomize only applies image transforms to containers that match the image name). However, this should be verified with a kustomize build of shred-oracle after the changes.

3. SOPS encryption requires AWS access. All .enc.yaml files must be encrypted by someone with the appropriate KMS key access. The plan provides plaintext templates; the operator encrypts them.

4. Liveness probe assumes /readyz endpoint. The device-health-oracle exposes metrics on :8080 at /metrics. Need to verify whether it implements /readyz. If not, either use a TCP check on port 8080, or omit the liveness probe initially and add it once the endpoint exists.

5. Migration window. The old core overlay deployment and the new ArgoCD deployment should not run simultaneously (two instances writing health updates). The cutover should be: deploy new → verify working → delete old. Consider doing this in two PRs with a manual verification step between.

6. ImageUpdater only for devnet. Testnet uses a pinned image tag in the config component, giving the operator explicit control over when to promote. The operator can later add an ImageUpdater for testnet if desired.

Testing Strategy

1. Kustomize build validation: Run kustomize build k8s/device-health-oracle/devnet/ and kustomize build k8s/device-health-oracle/testnet/ to verify manifests render correctly.
2. Verify no cross-contamination: Run kustomize build k8s/shred-oracle/testnet/ after shared-config changes to verify shred-oracle is unaffected.
3. ArgoCD dry-run: After applying the ArgoCD Application, verify it syncs successfully and the pod starts.
4. Post-deploy verification: Check that the device-health-oracle pod is running, metrics are being scraped, and health updates are being written onchain.
5. Auto-deploy test: After merging a change to main in doublezero, verify the ImageUpdater detects the new edge digest and triggers a sync.

Estimated Scope

• ~15 new files, ~3 edited files
• ~200 lines of YAML added (deployment, kustomizations, ArgoCD apps, ImageUpdater, config, README)
• Well within the 500-line threshold
• Post-migration cleanup is ~2 deleted files + 1 edit, done as a separate PR

[nikw9944]

Work Log — error

Failed to authenticate. API Error: 401 {"type":"error","error":{"type":"authentication_error","message":"OAuth token has expired. Please obtain a new token or refresh your existing token."},"request_id":"req_011CZdKLpqQH44p9yJNVoZpy"}

--- Result: success | Session cost: $0.0000 | Turns: 1 | Tokens: 0 in / 0 out ---

[nikw9944]

Work Plan (Revision 5)

Work Plan (Revision 5)

Work Plan: Issue #3420 — Deploy device-health-oracle to devnet/testnet

Revision 2

Summary

Migrate the device-health-oracle from the legacy k8s/kustomize/core/ deployment pattern to a standalone ArgoCD-managed deployment following the shred-oracle pattern. Create a k8s/device-health-oracle/ directory with base/overlay structure, add SOPS-encrypted signer keypair secrets for devnet and testnet, wire up ArgoCD Applications, and configure an ArgoCD ImageUpdater to auto-deploy to devnet on every push to main in the doublezero repo.

Context

Current state: The device-health-oracle runs in devnet only, deployed via k8s/kustomize/core/overlays/devnet/device-health-oracle.yaml with a stale image tag (ghcr.io/malbeclabs/doublezero-core:nikw-2669). There is no testnet deployment.

Target state: A standalone k8s/device-health-oracle/ directory modeled after k8s/shred-oracle/, with:

• ArgoCD Applications for devnet and testnet (auto-sync, self-heal)
• ArgoCD ImageUpdater watching ghcr.io/malbeclabs/doublezero-core:edge to auto-deploy devnet on merge to main
• Per-environment SOPS-encrypted secrets for the signer keypair
• Shared config components for environment-specific values

Docker image: ghcr.io/malbeclabs/doublezero-core — built and pushed by release.docker.core.yml in the doublezero repo. On push to main, tags edge and sha-<short> are produced. The binary inside is doublezero-device-health-oracle.

Approach

1. Create k8s/device-health-oracle/ directory (infra repo)

Follow the shred-oracle pattern exactly:

k8s/device-health-oracle/
├── README.md
├── base/
│   ├── deployment.yaml
│   └── kustomization.yaml
├── devnet/
│   └── kustomization.yaml
└── testnet/
    └── kustomization.yaml

base/deployment.yaml — shared deployment spec:

• Image: ghcr.io/malbeclabs/doublezero-core:PLACEHOLDER (tag set by config component or ImageUpdater)
• Command: doublezero-device-health-oracle
• Args: --signer-keypair /etc/device-health-oracle/keypair.json --ledger-rpc-url $(DZ_LEDGER_URL) --serviceability-program-id $(DZ_SERVICEABILITY_PROGRAM_ID) --telemetry-program-id $(DZ_TELEMETRY_PROGRAM_ID)
• envFrom: configMapRef: shared-config + secretRef: shared-secrets (consistent with shred-oracle)
• Volume: keypair mounted from device-health-oracle-env secret
• Metrics port: 8080 (existing convention for this service)
• Grafana scrape annotations + reloader annotation
• Liveness probe on /readyz endpoint (port 8080)
• Resources: 50m/64Mi requests, 500m/256Mi limits (same as shred-oracle)
• imagePullSecrets: ghcr-creds
• Strategy: Recreate

Note: Unlike the existing devnet deployment which uses --env devnet (relying on built-in config), the new deployment uses explicit flags with values from shared-config/shared-secrets. This is consistent with how shred-oracle works and decouples the binary from environment awareness at deploy time.

base/kustomization.yaml — references deployment.yaml, adds app.kubernetes.io/part-of: device-health-oracle label.

devnet/kustomization.yaml — overlay:

• References ../base
• Includes ../../config/devnet component (new — see step 3)
• Sets namespace dz-devnet
• Adds env: devnet label

testnet/kustomization.yaml — overlay:

• References ../base
• Includes ../../config/testnet component
• Sets namespace dz-testnet
• Adds env: testnet label

2. Create secrets in k8s/secrets/ (infra repo)

k8s/secrets/dz-devnet/device-health-oracle-env.enc.yaml (new file):

• Secret named device-health-oracle-env in namespace dz-devnet
• Contains keypair.json — the signer keypair for devnet
• Encrypted with SOPS using the AWS KMS key

k8s/secrets/dz-testnet/device-health-oracle-env.enc.yaml (new file):

• Secret named device-health-oracle-env in namespace dz-testnet
• Contains keypair.json — the signer keypair for testnet
• Encrypted with SOPS using the AWS KMS key

k8s/secrets/dz-devnet/kustomization.yaml — this directory does not exist yet. Create it with references to device-health-oracle-env.enc.yaml, plus shared-secrets.enc.yaml and ghcr-creds.enc.yaml (which also need to be created for the dz-devnet namespace).

k8s/secrets/dz-testnet/kustomization.yaml — add device-health-oracle-env.enc.yaml to the resources list.

Operator action required: The actual encrypted secret files must be created by someone with AWS KMS access. The plan provides the plaintext template structure; the operator encrypts with sops -e.

3. Create/update shared config components (infra repo)

k8s/config/devnet/ (new directory) — create a devnet config component:

kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

resources:
  - shared-config.yaml

images:
  - name: ghcr.io/malbeclabs/doublezero-core
    newTag: "edge"

shared-config.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: shared-config
data:
  SOLANA_URL: "devnet"
  DZ_SERVICEABILITY_PROGRAM_ID: "GYhQDKuESrasNZGyhMJhGYFtbzNijYhcrN9poSqCQVah"
  DZ_TELEMETRY_PROGRAM_ID: "C9xqH76NSm11pBS6maNnY163tWHT8Govww47uyEmSnoG"

k8s/config/testnet/shared-config.yaml — add DZ_TELEMETRY_PROGRAM_ID:

DZ_TELEMETRY_PROGRAM_ID: "3KogTMmVxc5eUHtjZnwm136H5P8tvPwVu4ufbGPvM7p1"

k8s/config/testnet/kustomization.yaml — add doublezero-core image entry:

images:
  - name: ghcr.io/malbeclabs/doublezero-shreds
    newTag: "0.1.1"
  - name: ghcr.io/malbeclabs/doublezero-core
    newTag: "edge"

Note: The newTag for testnet doublezero-core should be set to whatever tag the operator wants for testnet. Using edge is one option (tracks main), or a pinned version tag for more control. The operator should decide — edge is used here as a placeholder.

4. Create k8s/secrets/dz-devnet/ directory (infra repo)

The dz-devnet secrets directory does not exist yet. It needs:

• kustomization.yaml — listing all secret resources
• shared-secrets.enc.yaml — containing DZ_LEDGER_URL for devnet (the DZ ledger RPC endpoint)
• ghcr-creds.enc.yaml — GHCR image pull credentials for this namespace
• device-health-oracle-env.enc.yaml — signer keypair (from step 2)

The shared-secrets and ghcr-creds secrets may already exist in the dz-devnet namespace from the core kustomize deployment. The operator should verify and either:

• Create new SOPS-encrypted copies for the standalone secrets directory, or
• Confirm they're already synced via another mechanism (e.g., Reflector annotations on ghcr-creds)

5. Create ArgoCD Applications (infra repo)

k8s/argocd/applications/devnet/device-health-oracle.yaml (new file + new devnet/ directory):

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: device-health-oracle-devnet
  namespace: argocd
  annotations:
    notifications.argoproj.io/subscribe.dz-on-change.dz_slack: "#tmp-notify"
spec:
  project: default
  source:
    repoURL: https://github.com/malbeclabs/infra.git
    targetRevision: main
    path: k8s/device-health-oracle/devnet
  destination:
    server: https://kubernetes.default.svc
    namespace: dz-devnet
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

k8s/argocd/applications/testnet/device-health-oracle.yaml (new file):

• Same pattern, name device-health-oracle-testnet, path k8s/device-health-oracle/testnet, namespace dz-testnet

6. Create ArgoCD ImageUpdater for devnet auto-deploy (infra repo)

k8s/argocd/image-updater/device-health-oracle-devnet.yaml (new file):

apiVersion: argocd-image-updater.argoproj.io/v1alpha1
kind: ImageUpdater
metadata:
  name: device-health-oracle-devnet
  namespace: argocd
spec:
  namespace: argocd
  applicationRefs:
    - namePattern: device-health-oracle-devnet
      images:
        - alias: doublezero-core
          imageName: ghcr.io/malbeclabs/doublezero-core:edge
          commonUpdateSettings:
            updateStrategy: digest
            pullSecret: pullsecret:argocd/ghcr-creds

This watches the :edge tag's digest. When release.docker.core.yml pushes a new doublezero-core:edge image on merge to main, the ImageUpdater detects the digest change and triggers an ArgoCD sync of the devnet Application — achieving auto-deploy on merge to main.

7. Remove old devnet deployment from core overlay (infra repo)

Remove the device-health-oracle from the legacy kustomize/core path:

• Delete k8s/kustomize/core/overlays/devnet/device-health-oracle.yaml
• Delete k8s/kustomize/core/overlays/devnet/device-health-oracle-env.enc.yaml
• Remove references from k8s/kustomize/core/overlays/devnet/kustomization.yaml

This should be done after the new ArgoCD-managed deployment is confirmed working, to avoid downtime. Can be a follow-up commit or separate PR.

Files to Change (all in infra repo)

File	Change
k8s/device-health-oracle/README.md	New — documents the deployment structure
k8s/device-health-oracle/base/deployment.yaml	New — shared deployment spec
k8s/device-health-oracle/base/kustomization.yaml	New — base kustomization
k8s/device-health-oracle/devnet/kustomization.yaml	New — devnet overlay
k8s/device-health-oracle/testnet/kustomization.yaml	New — testnet overlay
k8s/config/devnet/kustomization.yaml	New — devnet config component
k8s/config/devnet/shared-config.yaml	New — devnet shared ConfigMap
k8s/config/testnet/shared-config.yaml	Edit — add DZ_TELEMETRY_PROGRAM_ID
k8s/config/testnet/kustomization.yaml	Edit — add doublezero-core image entry
k8s/secrets/dz-devnet/kustomization.yaml	New — devnet secrets kustomization
k8s/secrets/dz-devnet/device-health-oracle-env.enc.yaml	New — signer keypair (operator encrypts)
k8s/secrets/dz-devnet/shared-secrets.enc.yaml	New — DZ ledger URL (operator encrypts)
k8s/secrets/dz-devnet/ghcr-creds.enc.yaml	New — GHCR pull creds (operator encrypts)
k8s/secrets/dz-testnet/device-health-oracle-env.enc.yaml	New — signer keypair (operator encrypts)
k8s/secrets/dz-testnet/kustomization.yaml	Edit — add device-health-oracle-env.enc.yaml
k8s/argocd/applications/devnet/device-health-oracle.yaml	New — ArgoCD Application
k8s/argocd/applications/testnet/device-health-oracle.yaml	New — ArgoCD Application
k8s/argocd/image-updater/device-health-oracle-devnet.yaml	New — auto-deploy on edge digest change
k8s/secrets/README.md	Edit — add dz-devnet section to current secrets list

Post-migration cleanup (separate PR)

File	Change
k8s/kustomize/core/overlays/devnet/device-health-oracle.yaml	Delete
k8s/kustomize/core/overlays/devnet/device-health-oracle-env.enc.yaml	Delete
k8s/kustomize/core/overlays/devnet/kustomization.yaml	Edit — remove device-health-oracle refs

Risks & Considerations

1. dz-devnet secrets directory is new. No other service currently uses standalone secrets for devnet. The shared-secrets and ghcr-creds secrets need to be created for this namespace. The operator should verify whether these already exist in the cluster (they may have been created by the core kustomize deployment) and decide whether to create SOPS-encrypted copies or rely on existing mechanisms.

2. Shared config changes affect shred-oracle. Adding DZ_TELEMETRY_PROGRAM_ID to k8s/config/testnet/shared-config.yaml and adding doublezero-core to the images transformer in k8s/config/testnet/kustomization.yaml will be picked up by shred-oracle's testnet overlay too. The additional ConfigMap key is harmless (shred-oracle ignores unknown env vars). The additional image entry is also harmless (kustomize only applies image transforms to containers that match the image name). However, this should be verified with a kustomize build of shred-oracle after the changes.

3. SOPS encryption requires AWS access. All .enc.yaml files must be encrypted by someone with the appropriate KMS key access. The plan provides plaintext templates; the operator encrypts them.

4. Liveness probe assumes /readyz endpoint. The device-health-oracle exposes metrics on :8080 at /metrics. Need to verify whether it implements /readyz. If not, either use a TCP check on port 8080, or omit the liveness probe initially and add it once the endpoint exists.

5. Migration window. The old core overlay deployment and the new ArgoCD deployment should not run simultaneously (two instances writing health updates). The cutover should be: deploy new → verify working → delete old. Consider doing this in two PRs with a manual verification step between.

6. ImageUpdater only for devnet. Testnet uses a pinned image tag in the config component, giving the operator explicit control over when to promote. The operator can later add an ImageUpdater for testnet if desired.

Testing Strategy

1. Kustomize build validation: Run kustomize build k8s/device-health-oracle/devnet/ and kustomize build k8s/device-health-oracle/testnet/ to verify manifests render correctly.
2. Verify no cross-contamination: Run kustomize build k8s/shred-oracle/testnet/ after shared-config changes to verify shred-oracle is unaffected.
3. ArgoCD dry-run: After applying the ArgoCD Application, verify it syncs successfully and the pod starts.
4. Post-deploy verification: Check that the device-health-oracle pod is running, metrics are being scraped, and health updates are being written onchain.
5. Auto-deploy test: After merging a change to main in doublezero, verify the ImageUpdater detects the new edge digest and triggers a sync.

Estimated Scope

• ~15 new files, ~3 edited files
• ~200 lines of YAML added (deployment, kustomizations, ArgoCD apps, ImageUpdater, config, README)
• Well within the 500-line threshold
• Post-migration cleanup is ~2 deleted files + 1 edit, done as a separate PR

[nikw9944]

Work Plan (Revision 3)

Work Plan (Revision 3)

Work Plan: Issue #3420 — Deploy device-health-oracle to devnet/testnet

Revision 2

Summary

Migrate the device-health-oracle from the legacy k8s/kustomize/core/ deployment pattern to a standalone ArgoCD-managed deployment following the shred-oracle pattern. Create a k8s/device-health-oracle/ directory with base/overlay structure, add SOPS-encrypted signer keypair secrets for devnet and testnet, wire up ArgoCD Applications, and configure an ArgoCD ImageUpdater to auto-deploy to devnet on every push to main in the doublezero repo.

Context

Current state: The device-health-oracle runs in devnet only, deployed via k8s/kustomize/core/overlays/devnet/device-health-oracle.yaml with a stale image tag (ghcr.io/malbeclabs/doublezero-core:nikw-2669). There is no testnet deployment.

Target state: A standalone k8s/device-health-oracle/ directory modeled after k8s/shred-oracle/, with:

• ArgoCD Applications for devnet and testnet (auto-sync, self-heal)
• ArgoCD ImageUpdater watching ghcr.io/malbeclabs/doublezero-core:edge to auto-deploy devnet on merge to main
• Per-environment SOPS-encrypted secrets for the signer keypair
• Shared config components for environment-specific values

Docker image: ghcr.io/malbeclabs/doublezero-core — built and pushed by release.docker.core.yml in the doublezero repo. On push to main, tags edge and sha-<short> are produced. The binary inside is doublezero-device-health-oracle.

Approach

1. Create k8s/device-health-oracle/ directory (infra repo)

Follow the shred-oracle pattern exactly:

k8s/device-health-oracle/
├── README.md
├── base/
│   ├── deployment.yaml
│   └── kustomization.yaml
├── devnet/
│   └── kustomization.yaml
└── testnet/
    └── kustomization.yaml

base/deployment.yaml — shared deployment spec:

• Image: ghcr.io/malbeclabs/doublezero-core:PLACEHOLDER (tag set by config component or ImageUpdater)
• Command: doublezero-device-health-oracle
• Args: --signer-keypair /etc/device-health-oracle/keypair.json --ledger-rpc-url $(DZ_LEDGER_URL) --serviceability-program-id $(DZ_SERVICEABILITY_PROGRAM_ID) --telemetry-program-id $(DZ_TELEMETRY_PROGRAM_ID)
• envFrom: configMapRef: shared-config + secretRef: shared-secrets (consistent with shred-oracle)
• Volume: keypair mounted from device-health-oracle-env secret
• Metrics port: 8080 (existing convention for this service)
• Grafana scrape annotations + reloader annotation
• Liveness probe on /readyz endpoint (port 8080)
• Resources: 50m/64Mi requests, 500m/256Mi limits (same as shred-oracle)
• imagePullSecrets: ghcr-creds
• Strategy: Recreate

Note: Unlike the existing devnet deployment which uses --env devnet (relying on built-in config), the new deployment uses explicit flags with values from shared-config/shared-secrets. This is consistent with how shred-oracle works and decouples the binary from environment awareness at deploy time.

base/kustomization.yaml — references deployment.yaml, adds app.kubernetes.io/part-of: device-health-oracle label.

devnet/kustomization.yaml — overlay:

• References ../base
• Includes ../../config/devnet component (new — see step 3)
• Sets namespace dz-devnet
• Adds env: devnet label

testnet/kustomization.yaml — overlay:

• References ../base
• Includes ../../config/testnet component
• Sets namespace dz-testnet
• Adds env: testnet label

2. Create secrets in k8s/secrets/ (infra repo)

k8s/secrets/dz-devnet/device-health-oracle-env.enc.yaml (new file):

• Secret named device-health-oracle-env in namespace dz-devnet
• Contains keypair.json — the signer keypair for devnet
• Encrypted with SOPS using the AWS KMS key

k8s/secrets/dz-testnet/device-health-oracle-env.enc.yaml (new file):

• Secret named device-health-oracle-env in namespace dz-testnet
• Contains keypair.json — the signer keypair for testnet
• Encrypted with SOPS using the AWS KMS key

k8s/secrets/dz-devnet/kustomization.yaml — this directory does not exist yet. Create it with references to device-health-oracle-env.enc.yaml, plus shared-secrets.enc.yaml and ghcr-creds.enc.yaml (which also need to be created for the dz-devnet namespace).

k8s/secrets/dz-testnet/kustomization.yaml — add device-health-oracle-env.enc.yaml to the resources list.

Operator action required: The actual encrypted secret files must be created by someone with AWS KMS access. The plan provides the plaintext template structure; the operator encrypts with sops -e.

3. Create/update shared config components (infra repo)

k8s/config/devnet/ (new directory) — create a devnet config component:

kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

resources:
  - shared-config.yaml

images:
  - name: ghcr.io/malbeclabs/doublezero-core
    newTag: "edge"

shared-config.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: shared-config
data:
  SOLANA_URL: "devnet"
  DZ_SERVICEABILITY_PROGRAM_ID: "GYhQDKuESrasNZGyhMJhGYFtbzNijYhcrN9poSqCQVah"
  DZ_TELEMETRY_PROGRAM_ID: "C9xqH76NSm11pBS6maNnY163tWHT8Govww47uyEmSnoG"

k8s/config/testnet/shared-config.yaml — add DZ_TELEMETRY_PROGRAM_ID:

DZ_TELEMETRY_PROGRAM_ID: "3KogTMmVxc5eUHtjZnwm136H5P8tvPwVu4ufbGPvM7p1"

k8s/config/testnet/kustomization.yaml — add doublezero-core image entry:

images:
  - name: ghcr.io/malbeclabs/doublezero-shreds
    newTag: "0.1.1"
  - name: ghcr.io/malbeclabs/doublezero-core
    newTag: "edge"

Note: The newTag for testnet doublezero-core should be set to whatever tag the operator wants for testnet. Using edge is one option (tracks main), or a pinned version tag for more control. The operator should decide — edge is used here as a placeholder.

4. Create k8s/secrets/dz-devnet/ directory (infra repo)

The dz-devnet secrets directory does not exist yet. It needs:

• kustomization.yaml — listing all secret resources
• shared-secrets.enc.yaml — containing DZ_LEDGER_URL for devnet (the DZ ledger RPC endpoint)
• ghcr-creds.enc.yaml — GHCR image pull credentials for this namespace
• device-health-oracle-env.enc.yaml — signer keypair (from step 2)

The shared-secrets and ghcr-creds secrets may already exist in the dz-devnet namespace from the core kustomize deployment. The operator should verify and either:

• Create new SOPS-encrypted copies for the standalone secrets directory, or
• Confirm they're already synced via another mechanism (e.g., Reflector annotations on ghcr-creds)

5. Create ArgoCD Applications (infra repo)

k8s/argocd/applications/devnet/device-health-oracle.yaml (new file + new devnet/ directory):

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: device-health-oracle-devnet
  namespace: argocd
  annotations:
    notifications.argoproj.io/subscribe.dz-on-change.dz_slack: "#tmp-notify"
spec:
  project: default
  source:
    repoURL: https://github.com/malbeclabs/infra.git
    targetRevision: main
    path: k8s/device-health-oracle/devnet
  destination:
    server: https://kubernetes.default.svc
    namespace: dz-devnet
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

k8s/argocd/applications/testnet/device-health-oracle.yaml (new file):

• Same pattern, name device-health-oracle-testnet, path k8s/device-health-oracle/testnet, namespace dz-testnet

6. Create ArgoCD ImageUpdater for devnet auto-deploy (infra repo)

k8s/argocd/image-updater/device-health-oracle-devnet.yaml (new file):

apiVersion: argocd-image-updater.argoproj.io/v1alpha1
kind: ImageUpdater
metadata:
  name: device-health-oracle-devnet
  namespace: argocd
spec:
  namespace: argocd
  applicationRefs:
    - namePattern: device-health-oracle-devnet
      images:
        - alias: doublezero-core
          imageName: ghcr.io/malbeclabs/doublezero-core:edge
          commonUpdateSettings:
            updateStrategy: digest
            pullSecret: pullsecret:argocd/ghcr-creds

This watches the :edge tag's digest. When release.docker.core.yml pushes a new doublezero-core:edge image on merge to main, the ImageUpdater detects the digest change and triggers an ArgoCD sync of the devnet Application — achieving auto-deploy on merge to main.

7. Remove old devnet deployment from core overlay (infra repo)

Remove the device-health-oracle from the legacy kustomize/core path:

• Delete k8s/kustomize/core/overlays/devnet/device-health-oracle.yaml
• Delete k8s/kustomize/core/overlays/devnet/device-health-oracle-env.enc.yaml
• Remove references from k8s/kustomize/core/overlays/devnet/kustomization.yaml

This should be done after the new ArgoCD-managed deployment is confirmed working, to avoid downtime. Can be a follow-up commit or separate PR.

Files to Change (all in infra repo)

File	Change
k8s/device-health-oracle/README.md	New — documents the deployment structure
k8s/device-health-oracle/base/deployment.yaml	New — shared deployment spec
k8s/device-health-oracle/base/kustomization.yaml	New — base kustomization
k8s/device-health-oracle/devnet/kustomization.yaml	New — devnet overlay
k8s/device-health-oracle/testnet/kustomization.yaml	New — testnet overlay
k8s/config/devnet/kustomization.yaml	New — devnet config component
k8s/config/devnet/shared-config.yaml	New — devnet shared ConfigMap
k8s/config/testnet/shared-config.yaml	Edit — add DZ_TELEMETRY_PROGRAM_ID
k8s/config/testnet/kustomization.yaml	Edit — add doublezero-core image entry
k8s/secrets/dz-devnet/kustomization.yaml	New — devnet secrets kustomization
k8s/secrets/dz-devnet/device-health-oracle-env.enc.yaml	New — signer keypair (operator encrypts)
k8s/secrets/dz-devnet/shared-secrets.enc.yaml	New — DZ ledger URL (operator encrypts)
k8s/secrets/dz-devnet/ghcr-creds.enc.yaml	New — GHCR pull creds (operator encrypts)
k8s/secrets/dz-testnet/device-health-oracle-env.enc.yaml	New — signer keypair (operator encrypts)
k8s/secrets/dz-testnet/kustomization.yaml	Edit — add device-health-oracle-env.enc.yaml
k8s/argocd/applications/devnet/device-health-oracle.yaml	New — ArgoCD Application
k8s/argocd/applications/testnet/device-health-oracle.yaml	New — ArgoCD Application
k8s/argocd/image-updater/device-health-oracle-devnet.yaml	New — auto-deploy on edge digest change
k8s/secrets/README.md	Edit — add dz-devnet section to current secrets list

Post-migration cleanup (separate PR)

File	Change
k8s/kustomize/core/overlays/devnet/device-health-oracle.yaml	Delete
k8s/kustomize/core/overlays/devnet/device-health-oracle-env.enc.yaml	Delete
k8s/kustomize/core/overlays/devnet/kustomization.yaml	Edit — remove device-health-oracle refs

Risks & Considerations

1. dz-devnet secrets directory is new. No other service currently uses standalone secrets for devnet. The shared-secrets and ghcr-creds secrets need to be created for this namespace. The operator should verify whether these already exist in the cluster (they may have been created by the core kustomize deployment) and decide whether to create SOPS-encrypted copies or rely on existing mechanisms.

2. Shared config changes affect shred-oracle. Adding DZ_TELEMETRY_PROGRAM_ID to k8s/config/testnet/shared-config.yaml and adding doublezero-core to the images transformer in k8s/config/testnet/kustomization.yaml will be picked up by shred-oracle's testnet overlay too. The additional ConfigMap key is harmless (shred-oracle ignores unknown env vars). The additional image entry is also harmless (kustomize only applies image transforms to containers that match the image name). However, this should be verified with a kustomize build of shred-oracle after the changes.

3. SOPS encryption requires AWS access. All .enc.yaml files must be encrypted by someone with the appropriate KMS key access. The plan provides plaintext templates; the operator encrypts them.

4. Liveness probe assumes /readyz endpoint. The device-health-oracle exposes metrics on :8080 at /metrics. Need to verify whether it implements /readyz. If not, either use a TCP check on port 8080, or omit the liveness probe initially and add it once the endpoint exists.

5. Migration window. The old core overlay deployment and the new ArgoCD deployment should not run simultaneously (two instances writing health updates). The cutover should be: deploy new → verify working → delete old. Consider doing this in two PRs with a manual verification step between.

6. ImageUpdater only for devnet. Testnet uses a pinned image tag in the config component, giving the operator explicit control over when to promote. The operator can later add an ImageUpdater for testnet if desired.

Testing Strategy

1. Kustomize build validation: Run kustomize build k8s/device-health-oracle/devnet/ and kustomize build k8s/device-health-oracle/testnet/ to verify manifests render correctly.
2. Verify no cross-contamination: Run kustomize build k8s/shred-oracle/testnet/ after shared-config changes to verify shred-oracle is unaffected.
3. ArgoCD dry-run: After applying the ArgoCD Application, verify it syncs successfully and the pod starts.
4. Post-deploy verification: Check that the device-health-oracle pod is running, metrics are being scraped, and health updates are being written onchain.
5. Auto-deploy test: After merging a change to main in doublezero, verify the ImageUpdater detects the new edge digest and triggers a sync.

Estimated Scope

• ~15 new files, ~3 edited files
• ~200 lines of YAML added (deployment, kustomizations, ArgoCD apps, ImageUpdater, config, README)
• Well within the 500-line threshold
• Post-migration cleanup is ~2 deleted files + 1 edit, done as a separate PR

[nikw9944]

Work Plan (Revision 4)

Work Plan (Revision 4)

Work Plan: Issue #3420 — Deploy device-health-oracle to devnet/testnet

Revision 2

Summary

Migrate the device-health-oracle from the legacy k8s/kustomize/core/ deployment pattern to a standalone ArgoCD-managed deployment following the shred-oracle pattern. Create a k8s/device-health-oracle/ directory with base/overlay structure, add SOPS-encrypted signer keypair secrets for devnet and testnet, wire up ArgoCD Applications, and configure an ArgoCD ImageUpdater to auto-deploy to devnet on every push to main in the doublezero repo.

Context

Current state: The device-health-oracle runs in devnet only, deployed via k8s/kustomize/core/overlays/devnet/device-health-oracle.yaml with a stale image tag (ghcr.io/malbeclabs/doublezero-core:nikw-2669). There is no testnet deployment.

Target state: A standalone k8s/device-health-oracle/ directory modeled after k8s/shred-oracle/, with:

• ArgoCD Applications for devnet and testnet (auto-sync, self-heal)
• ArgoCD ImageUpdater watching ghcr.io/malbeclabs/doublezero-core:edge to auto-deploy devnet on merge to main
• Per-environment SOPS-encrypted secrets for the signer keypair
• Shared config components for environment-specific values

Docker image: ghcr.io/malbeclabs/doublezero-core — built and pushed by release.docker.core.yml in the doublezero repo. On push to main, tags edge and sha-<short> are produced. The binary inside is doublezero-device-health-oracle.

Approach

1. Create k8s/device-health-oracle/ directory (infra repo)

Follow the shred-oracle pattern exactly:

k8s/device-health-oracle/
├── README.md
├── base/
│   ├── deployment.yaml
│   └── kustomization.yaml
├── devnet/
│   └── kustomization.yaml
└── testnet/
    └── kustomization.yaml

base/deployment.yaml — shared deployment spec:

• Image: ghcr.io/malbeclabs/doublezero-core:PLACEHOLDER (tag set by config component or ImageUpdater)
• Command: doublezero-device-health-oracle
• Args: --signer-keypair /etc/device-health-oracle/keypair.json --ledger-rpc-url $(DZ_LEDGER_URL) --serviceability-program-id $(DZ_SERVICEABILITY_PROGRAM_ID) --telemetry-program-id $(DZ_TELEMETRY_PROGRAM_ID)
• envFrom: configMapRef: shared-config + secretRef: shared-secrets (consistent with shred-oracle)
• Volume: keypair mounted from device-health-oracle-env secret
• Metrics port: 8080 (existing convention for this service)
• Grafana scrape annotations + reloader annotation
• Liveness probe on /readyz endpoint (port 8080)
• Resources: 50m/64Mi requests, 500m/256Mi limits (same as shred-oracle)
• imagePullSecrets: ghcr-creds
• Strategy: Recreate

Note: Unlike the existing devnet deployment which uses --env devnet (relying on built-in config), the new deployment uses explicit flags with values from shared-config/shared-secrets. This is consistent with how shred-oracle works and decouples the binary from environment awareness at deploy time.

base/kustomization.yaml — references deployment.yaml, adds app.kubernetes.io/part-of: device-health-oracle label.

devnet/kustomization.yaml — overlay:

• References ../base
• Includes ../../config/devnet component (new — see step 3)
• Sets namespace dz-devnet
• Adds env: devnet label

testnet/kustomization.yaml — overlay:

• References ../base
• Includes ../../config/testnet component
• Sets namespace dz-testnet
• Adds env: testnet label

2. Create secrets in k8s/secrets/ (infra repo)

k8s/secrets/dz-devnet/device-health-oracle-env.enc.yaml (new file):

• Secret named device-health-oracle-env in namespace dz-devnet
• Contains keypair.json — the signer keypair for devnet
• Encrypted with SOPS using the AWS KMS key

k8s/secrets/dz-testnet/device-health-oracle-env.enc.yaml (new file):

• Secret named device-health-oracle-env in namespace dz-testnet
• Contains keypair.json — the signer keypair for testnet
• Encrypted with SOPS using the AWS KMS key

k8s/secrets/dz-devnet/kustomization.yaml — this directory does not exist yet. Create it with references to device-health-oracle-env.enc.yaml, plus shared-secrets.enc.yaml and ghcr-creds.enc.yaml (which also need to be created for the dz-devnet namespace).

k8s/secrets/dz-testnet/kustomization.yaml — add device-health-oracle-env.enc.yaml to the resources list.

Operator action required: The actual encrypted secret files must be created by someone with AWS KMS access. The plan provides the plaintext template structure; the operator encrypts with sops -e.

3. Create/update shared config components (infra repo)

k8s/config/devnet/ (new directory) — create a devnet config component:

kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

resources:
  - shared-config.yaml

images:
  - name: ghcr.io/malbeclabs/doublezero-core
    newTag: "edge"

shared-config.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: shared-config
data:
  SOLANA_URL: "devnet"
  DZ_SERVICEABILITY_PROGRAM_ID: "GYhQDKuESrasNZGyhMJhGYFtbzNijYhcrN9poSqCQVah"
  DZ_TELEMETRY_PROGRAM_ID: "C9xqH76NSm11pBS6maNnY163tWHT8Govww47uyEmSnoG"

k8s/config/testnet/shared-config.yaml — add DZ_TELEMETRY_PROGRAM_ID:

DZ_TELEMETRY_PROGRAM_ID: "3KogTMmVxc5eUHtjZnwm136H5P8tvPwVu4ufbGPvM7p1"

k8s/config/testnet/kustomization.yaml — add doublezero-core image entry:

images:
  - name: ghcr.io/malbeclabs/doublezero-shreds
    newTag: "0.1.1"
  - name: ghcr.io/malbeclabs/doublezero-core
    newTag: "edge"

Note: The newTag for testnet doublezero-core should be set to whatever tag the operator wants for testnet. Using edge is one option (tracks main), or a pinned version tag for more control. The operator should decide — edge is used here as a placeholder.

4. Create k8s/secrets/dz-devnet/ directory (infra repo)

The dz-devnet secrets directory does not exist yet. It needs:

• kustomization.yaml — listing all secret resources
• shared-secrets.enc.yaml — containing DZ_LEDGER_URL for devnet (the DZ ledger RPC endpoint)
• ghcr-creds.enc.yaml — GHCR image pull credentials for this namespace
• device-health-oracle-env.enc.yaml — signer keypair (from step 2)

The shared-secrets and ghcr-creds secrets may already exist in the dz-devnet namespace from the core kustomize deployment. The operator should verify and either:

• Create new SOPS-encrypted copies for the standalone secrets directory, or
• Confirm they're already synced via another mechanism (e.g., Reflector annotations on ghcr-creds)

5. Create ArgoCD Applications (infra repo)

k8s/argocd/applications/devnet/device-health-oracle.yaml (new file + new devnet/ directory):

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: device-health-oracle-devnet
  namespace: argocd
  annotations:
    notifications.argoproj.io/subscribe.dz-on-change.dz_slack: "#tmp-notify"
spec:
  project: default
  source:
    repoURL: https://github.com/malbeclabs/infra.git
    targetRevision: main
    path: k8s/device-health-oracle/devnet
  destination:
    server: https://kubernetes.default.svc
    namespace: dz-devnet
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

k8s/argocd/applications/testnet/device-health-oracle.yaml (new file):

• Same pattern, name device-health-oracle-testnet, path k8s/device-health-oracle/testnet, namespace dz-testnet

6. Create ArgoCD ImageUpdater for devnet auto-deploy (infra repo)

k8s/argocd/image-updater/device-health-oracle-devnet.yaml (new file):

apiVersion: argocd-image-updater.argoproj.io/v1alpha1
kind: ImageUpdater
metadata:
  name: device-health-oracle-devnet
  namespace: argocd
spec:
  namespace: argocd
  applicationRefs:
    - namePattern: device-health-oracle-devnet
      images:
        - alias: doublezero-core
          imageName: ghcr.io/malbeclabs/doublezero-core:edge
          commonUpdateSettings:
            updateStrategy: digest
            pullSecret: pullsecret:argocd/ghcr-creds

This watches the :edge tag's digest. When release.docker.core.yml pushes a new doublezero-core:edge image on merge to main, the ImageUpdater detects the digest change and triggers an ArgoCD sync of the devnet Application — achieving auto-deploy on merge to main.

7. Remove old devnet deployment from core overlay (infra repo)

Remove the device-health-oracle from the legacy kustomize/core path:

• Delete k8s/kustomize/core/overlays/devnet/device-health-oracle.yaml
• Delete k8s/kustomize/core/overlays/devnet/device-health-oracle-env.enc.yaml
• Remove references from k8s/kustomize/core/overlays/devnet/kustomization.yaml

This should be done after the new ArgoCD-managed deployment is confirmed working, to avoid downtime. Can be a follow-up commit or separate PR.

Files to Change (all in infra repo)

File	Change
k8s/device-health-oracle/README.md	New — documents the deployment structure
k8s/device-health-oracle/base/deployment.yaml	New — shared deployment spec
k8s/device-health-oracle/base/kustomization.yaml	New — base kustomization
k8s/device-health-oracle/devnet/kustomization.yaml	New — devnet overlay
k8s/device-health-oracle/testnet/kustomization.yaml	New — testnet overlay
k8s/config/devnet/kustomization.yaml	New — devnet config component
k8s/config/devnet/shared-config.yaml	New — devnet shared ConfigMap
k8s/config/testnet/shared-config.yaml	Edit — add DZ_TELEMETRY_PROGRAM_ID
k8s/config/testnet/kustomization.yaml	Edit — add doublezero-core image entry
k8s/secrets/dz-devnet/kustomization.yaml	New — devnet secrets kustomization
k8s/secrets/dz-devnet/device-health-oracle-env.enc.yaml	New — signer keypair (operator encrypts)
k8s/secrets/dz-devnet/shared-secrets.enc.yaml	New — DZ ledger URL (operator encrypts)
k8s/secrets/dz-devnet/ghcr-creds.enc.yaml	New — GHCR pull creds (operator encrypts)
k8s/secrets/dz-testnet/device-health-oracle-env.enc.yaml	New — signer keypair (operator encrypts)
k8s/secrets/dz-testnet/kustomization.yaml	Edit — add device-health-oracle-env.enc.yaml
k8s/argocd/applications/devnet/device-health-oracle.yaml	New — ArgoCD Application
k8s/argocd/applications/testnet/device-health-oracle.yaml	New — ArgoCD Application
k8s/argocd/image-updater/device-health-oracle-devnet.yaml	New — auto-deploy on edge digest change
k8s/secrets/README.md	Edit — add dz-devnet section to current secrets list

Post-migration cleanup (separate PR)

File	Change
k8s/kustomize/core/overlays/devnet/device-health-oracle.yaml	Delete
k8s/kustomize/core/overlays/devnet/device-health-oracle-env.enc.yaml	Delete
k8s/kustomize/core/overlays/devnet/kustomization.yaml	Edit — remove device-health-oracle refs

Risks & Considerations

1. dz-devnet secrets directory is new. No other service currently uses standalone secrets for devnet. The shared-secrets and ghcr-creds secrets need to be created for this namespace. The operator should verify whether these already exist in the cluster (they may have been created by the core kustomize deployment) and decide whether to create SOPS-encrypted copies or rely on existing mechanisms.

2. Shared config changes affect shred-oracle. Adding DZ_TELEMETRY_PROGRAM_ID to k8s/config/testnet/shared-config.yaml and adding doublezero-core to the images transformer in k8s/config/testnet/kustomization.yaml will be picked up by shred-oracle's testnet overlay too. The additional ConfigMap key is harmless (shred-oracle ignores unknown env vars). The additional image entry is also harmless (kustomize only applies image transforms to containers that match the image name). However, this should be verified with a kustomize build of shred-oracle after the changes.

3. SOPS encryption requires AWS access. All .enc.yaml files must be encrypted by someone with the appropriate KMS key access. The plan provides plaintext templates; the operator encrypts them.

4. Liveness probe assumes /readyz endpoint. The device-health-oracle exposes metrics on :8080 at /metrics. Need to verify whether it implements /readyz. If not, either use a TCP check on port 8080, or omit the liveness probe initially and add it once the endpoint exists.

5. Migration window. The old core overlay deployment and the new ArgoCD deployment should not run simultaneously (two instances writing health updates). The cutover should be: deploy new → verify working → delete old. Consider doing this in two PRs with a manual verification step between.

6. ImageUpdater only for devnet. Testnet uses a pinned image tag in the config component, giving the operator explicit control over when to promote. The operator can later add an ImageUpdater for testnet if desired.

Testing Strategy

1. Kustomize build validation: Run kustomize build k8s/device-health-oracle/devnet/ and kustomize build k8s/device-health-oracle/testnet/ to verify manifests render correctly.
2. Verify no cross-contamination: Run kustomize build k8s/shred-oracle/testnet/ after shared-config changes to verify shred-oracle is unaffected.
3. ArgoCD dry-run: After applying the ArgoCD Application, verify it syncs successfully and the pod starts.
4. Post-deploy verification: Check that the device-health-oracle pod is running, metrics are being scraped, and health updates are being written onchain.
5. Auto-deploy test: After merging a change to main in doublezero, verify the ImageUpdater detects the new edge digest and triggers a sync.

Estimated Scope

• ~15 new files, ~3 edited files
• ~200 lines of YAML added (deployment, kustomizations, ArgoCD apps, ImageUpdater, config, README)
• Well within the 500-line threshold
• Post-migration cleanup is ~2 deleted files + 1 edit, done as a separate PR

[nikw9944]

Architecture review findings:

• Critical: 0
• High: 2
• Medium: 3
• Low: 4

Architecture Review

Critical

• None

High

• Testnet image tag is edge -- likely should be a pinned version. The diff for k8s/config/testnet/kustomization.yaml sets DEVICE_HEALTH_ORACLE_IMAGE to newTag: "edge". The stated intent is that testnet uses "manual tag updates" (no ImageUpdater), but the initial tag is edge, which is a rolling/mutable tag. Compare with the existing shred-oracle pattern in the same file which pins to "0.1.3". This should be set to a specific release tag (or at least documented as intentionally temporary) before merging, otherwise testnet will be stuck on whatever edge pointed to at deploy time -- confusing for operators who expect pinned versions in that environment.

  • Location: k8s/config/testnet/kustomization.yaml
  • Recommendation: Pin to a specific doublezero-core release tag or document as intentionally temporary

• SOLANA_URL in shared-config.yaml is unused by the device-health-oracle binary. The deployment uses --env $(DZ_ENV) which calls NetworkConfigForEnv(), which hardcodes the RPC URL per environment and only respects DZ_LEDGER_RPC_URL as an override (see config/env.go:188). The SOLANA_URL ConfigMap entry (set to "devnet" / "testnet") is not read by this binary. If the intent is to allow RPC URL override, the env var name is wrong. If SOLANA_URL exists for other services sharing this ConfigMap, that is fine, but the README or deployment should clarify that the oracle resolves its own RPC endpoint via --env and ignores SOLANA_URL. No functional bug here, but could mislead operators into thinking they can control the RPC URL via this ConfigMap.

  • Location: k8s/config/devnet/shared-config.yaml, k8s/config/testnet/shared-config.yaml
  • Recommendation: Document that the oracle derives RPC URL from --env flag and DZ_LEDGER_RPC_URL override

Medium

• DZ_SERVICEABILITY_PROGRAM_ID in shared-config is also unused by this binary when --env is set. The NetworkConfigForEnv() function resolves program IDs from compiled-in constants, not from environment variables. The shared ConfigMap entry exists for other services, but having it in the same ConfigMap that gets envFrom'd into this pod creates the false impression that the oracle reads it.

  • Location: k8s/config/devnet/shared-config.yaml
  • Recommendation: Consider documenting that the oracle derives all program IDs from the --env flag

• No liveness or readiness probes on the deployment. The annotation policy.skipReadinessProbe: "true" explicitly opts out, which is understandable for a worker-style (non-serving) pod. However, there is no liveness probe either. If the oracle's main loop deadlocks or panics in a goroutine without crashing the process, Kubernetes will not restart it.

  • Location: k8s/device-health-oracle/base/deployment.yaml
  • Recommendation: Consider adding a liveness probe hitting the metrics endpoint (:8080)

• No SLACK_WEBHOOK_URL passed through. The binary accepts --slack-webhook-url for alerting, but the deployment does not pass it. This means device health state-change alerts will silently not fire.

  • Location: k8s/device-health-oracle/base/deployment.yaml
  • Recommendation: If Slack alerts are desired, add secret reference for the webhook URL

Low

• Kustomize Component kind uses v1alpha1 API. While functional and consistent with existing patterns, the alpha API version is worth tracking as an API stability concern.

  • Location: k8s/config/devnet/kustomization.yaml

• imagePullPolicy: Always will cause unnecessary pulls for testnet once a pinned tag is set. Consider making this configurable per overlay.

  • Location: k8s/device-health-oracle/base/deployment.yaml

• Secret naming convention device-health-oracle-env uses -env suffix suggesting environment variables, but contains a keypair file. Minor naming inconsistency.

  • Location: k8s/device-health-oracle/base/deployment.yaml

• No terminationGracePeriodSeconds specified. The default 30s is likely fine, but since the binary uses signal.NotifyContext for graceful shutdown, an explicit value would be clearer.

  • Location: k8s/device-health-oracle/base/deployment.yaml

Summary

The migration cleanly follows the established shred-oracle pattern and the ArgoCD/Kustomize structure is sound. The two items that warrant attention before merge are the testnet edge tag (should be pinned) and the absence of a liveness probe for a long-running worker process.

[nikw9944]

Security review findings:

• Critical: 0
• High: 0
• Medium: 1
• Low: 6

Security Review

Critical

• None

High

• None

Medium

• ArgoCD project is default with no restrictions. Both ArgoCD Application manifests use project: default, which typically has no source/destination restrictions. Consider using a dedicated AppProject with explicit sourceRepos, destinations, and clusterResourceWhitelist constraints. (Consistent with existing patterns in the infra repo.)
  • Location: k8s/argocd/applications/devnet/device-health-oracle.yaml, k8s/argocd/applications/testnet/device-health-oracle.yaml
  • Recommendation: Verify that the default project has appropriate restrictions, or use a dedicated AppProject

Low

• CreateNamespace=true allows implicit namespace creation. Namespaces created this way lack default NetworkPolicy or ResourceQuota. If these namespaces already exist, this is a no-op.

  • Location: k8s/argocd/applications/devnet/device-health-oracle.yaml, k8s/argocd/applications/testnet/device-health-oracle.yaml
  • Recommendation: Remove if namespaces are pre-created, or ensure default policies exist

• No securityContext on the pod or container. Missing runAsNonRoot: true, readOnlyRootFilesystem: true, allowPrivilegeEscalation: false, and capabilities: { drop: [ALL] }.

  • Location: k8s/device-health-oracle/base/deployment.yaml
  • Recommendation: Add restrictive securityContext for defense-in-depth

• No liveness or readiness probes. A liveness probe on the metrics port would help detect hung processes.

  • Location: k8s/device-health-oracle/base/deployment.yaml
  • Recommendation: Add liveness probe on :8080

• Secret volume mount is well-configured. The keypair is mounted read-only with explicit items projection. No issues.

  • Location: k8s/device-health-oracle/base/deployment.yaml

• shared-secrets SecretRef exposes all keys as env vars. If shared-secrets grows to include credentials not needed by this service, they'll still be available in the container's environment.

  • Location: k8s/device-health-oracle/base/deployment.yaml
  • Recommendation: Consider using explicit env.valueFrom.secretKeyRef for only needed keys

• No image-updater config for testnet. Testnet image updates are manual, which is intentional but means testnet could run stale images if updates are forgotten.

  • Location: k8s/argocd/image-updater/
  • Recommendation: Ensure testnet image update process is documented and followed

Summary

The deployment follows established patterns and handles the signer keypair correctly via a read-only secret volume mount with SOPS/KMS encryption. The main areas for hardening are adding a restrictive securityContext to the container spec, scoping the ArgoCD project, and ensuring shared-secrets does not over-expose credentials via envFrom. No critical or high-severity issues were found.

[nikw9944]

Work Log — blocked

Work Done

Created standalone device-health-oracle k8s deployment in the infra repo following the shred-oracle pattern:

New files:

• k8s/device-health-oracle/ — base deployment + devnet/testnet overlays + README
• k8s/config/devnet/ — shared config component (first devnet config component)
• k8s/argocd/applications/devnet/device-health-oracle.yaml — ArgoCD app for devnet
• k8s/argocd/applications/testnet/device-health-oracle.yaml — ArgoCD app for testnet
• k8s/argocd/image-updater/device-health-oracle-devnet.yaml — auto-deploy on :edge tag
• k8s/secrets/dz-devnet/kustomization.yaml — devnet secrets directory

Modified files:

• k8s/config/testnet/kustomization.yaml — added DEVICE_HEALTH_ORACLE_IMAGE
• k8s/config/testnet/shared-config.yaml — added DZ_ENV
• k8s/secrets/dz-testnet/kustomization.yaml — added device-health-oracle-env reference
• k8s/secrets/README.md — documented dz-devnet directory
• k8s/kustomize/core/overlays/devnet/device-health-oracle.yaml — fixed stale image tag to :edge

Infra PR: https://github.com/malbeclabs/infra/pull/878

Blocked — Architecture Review High Findings

The architecture review identified 2 high-severity findings requiring operator guidance:

1. Testnet image tag is edge — The testnet DEVICE_HEALTH_ORACLE_IMAGE tag is set to "edge" (a rolling tag). The shred-oracle pattern pins testnet to a specific version like "0.1.3". Should we pin to a specific doublezero-core release tag, or is "edge" acceptable for initial testnet rollout?

2. SOLANA_URL in shared-config is unused by device-health-oracle — The oracle uses --env $(DZ_ENV) to resolve all config (program IDs, RPC URLs) internally via compiled-in constants, with DZ_LEDGER_RPC_URL as the only env var override. The SOLANA_URL and DZ_SERVICEABILITY_PROGRAM_ID entries in shared-config exist for other services but are injected (unused) into this pod via envFrom. This is informational/misleading but not a functional issue.

Manual Steps Required After Merge

1. Create SOPS-encrypted secrets: device-health-oracle-env.enc.yaml for both dz-devnet and dz-testnet
2. Create shared-secrets.enc.yaml and ghcr-creds.enc.yaml for the new dz-devnet namespace
3. Apply all secrets to the cluster before ArgoCD apps sync

[nikw9944]

Work Log — waiting_review

No changes on either branch yet — previous instances did research but errored before committing. Let me gather the reference materials.

Now I have a comprehensive understanding. Let me check a few more details before implementing.

Good. Now I have the full picture. Let me plan and implement.

Plan:

1. Create k8s/device-health-oracle/ with base + devnet/testnet overlays (following shred-oracle pattern)
2. Create k8s/config/devnet/ shared config component (doesn't exist yet)
3. Add DEVICE_HEALTH_ORACLE_IMAGE to devnet and testnet config components
4. Create ArgoCD applications for devnet and testnet
5. Create ImageUpdater for devnet auto-deploy on :edge tag
6. Add placeholder entries in secrets kustomizations (actual SOPS encryption is manual)
7. Add README.md

I'll use sub-agents to create the files in parallel.

Let me verify the created files and check program IDs.

The devnet serviceability program ID is wrong. The actual value from the doublezero config constants is GYhQDKuESrasNZGyhMJhGYFtbzNijYhcrN9poSqCQVah. Let me fix that, and verify all the other created files.

Good. Let me fix the devnet config to remove the unnecessary SHRED_ORACLE_IMAGE entry:

Now let me also add the secrets kustomization entries. I can't create SOPS-encrypted secrets, but I should add placeholder entries to the dz-devnet and dz-testnet kustomizations and document what needs to be done manually.

I need to set up the secrets references. Let me update the testnet kustomization and create the devnet secrets directory.

Now let me also update the secrets README to document the new dz-devnet directory and the device-health-oracle-env secret.

Now let me review the full diff and commit both repos. First, let me verify everything looks good.

Good. Let me also verify the doublezero repo has no changes (all changes are in infra).

No changes in doublezero repo — all changes are in infra. Let me now commit and get the diff ready for review.

Now let me get the diff for the reviews and run them in parallel.

Now let me launch the architecture and security reviews in parallel.

The architecture review has 2 high findings. Let me post both reviews to the issue, then report status.

Reviews posted. Now, the architecture review has 2 high severity findings. Per the workflow rules, I must report STATUS: BLOCKED.

Before reporting, let me push what I have so the work is not lost.

There's a previous instance's commit on the remote. Let me rebase.

The remote already has substantial changes from a previous instance. Let me reset to the remote and verify its state.

The remote branch only has the legacy kustomize/core overlay changes, not the standalone shred-oracle pattern. Let me verify.

The remote branch used the legacy kustomize/core pattern. The approved plan calls for the standalone shred-oracle pattern. Let me redo this properly — keep the devnet image tag fix but replace the testnet approach with standalone.

Now let me unstage the testnet kustomize/core files (wrong pattern), keep the devnet fix, and add my standalone files.

Now add all the standalone files.

Now update the testnet config and secrets files.

Now create the dz-devnet secrets kustomization and update the secrets README.

Now update the secrets README. I already read it earlier, let me apply the edits.

Now let me stage everything and commit.

Now push (force needed since we rewrote the branch).

Now let me check if there's an existing PR for infra.

No PR yet. Let me create one.

PR created at https://github.com/malbeclabs/infra/pull/878.

Now let me post the status. The architecture review had 2 high findings that require operator guidance.

STATUS: BLOCKED

Infra PR created: https://github.com/malbeclabs/infra/pull/878

The architecture review identified 2 high-severity findings requiring operator guidance:

1. Testnet image tag is edge — Should we pin to a specific doublezero-core release tag for testnet, or is "edge" acceptable for initial rollout? The shred-oracle pattern pins testnet to specific versions (e.g., "0.1.3").

2. SOLANA_URL in shared-config unused by this binary — The device-health-oracle resolves all config via --env flag internally. The shared-config entries (SOLANA_URL, DZ_SERVICEABILITY_PROGRAM_ID) are injected but unused by this specific pod. Not a functional bug, but could mislead operators. Should we document this or restructure the envFrom?

Additionally, manual steps are required after merge:

• Create SOPS-encrypted device-health-oracle-env.enc.yaml secrets for dz-devnet and dz-testnet
• Create shared-secrets.enc.yaml and ghcr-creds.enc.yaml for the new dz-devnet namespace
• Apply secrets to cluster before ArgoCD apps sync

--- Result: success | Session cost: $5.4293 | Turns: 105 | Tokens: 1650 in / 36431 out ---

[nikw9944]

Work Log — error

Failed to authenticate. API Error: 401 {"type":"error","error":{"type":"authentication_error","message":"OAuth token has expired. Please obtain a new token or refresh your existing token."},"request_id":"req_011CZnhntFc63rzFumRRqmNM"}

--- Result: success | Session cost: $0.0000 | Turns: 1 | Tokens: 0 in / 0 out ---

[nikw9944]

Work Plan (Revision 5)

Work Plan (Revision 5)

Work Plan: Issue #3420 — Deploy device-health-oracle to devnet/testnet

Revision 2

Summary

Migrate the device-health-oracle from the legacy k8s/kustomize/core/ deployment pattern to a standalone ArgoCD-managed deployment following the shred-oracle pattern. Create a k8s/device-health-oracle/ directory with base/overlay structure, add SOPS-encrypted signer keypair secrets for devnet and testnet, wire up ArgoCD Applications, and configure an ArgoCD ImageUpdater to auto-deploy to devnet on every push to main in the doublezero repo.

Context

Current state: The device-health-oracle runs in devnet only, deployed via k8s/kustomize/core/overlays/devnet/device-health-oracle.yaml with a stale image tag (ghcr.io/malbeclabs/doublezero-core:nikw-2669). There is no testnet deployment.

Target state: A standalone k8s/device-health-oracle/ directory modeled after k8s/shred-oracle/, with:

• ArgoCD Applications for devnet and testnet (auto-sync, self-heal)
• ArgoCD ImageUpdater watching ghcr.io/malbeclabs/doublezero-core:edge to auto-deploy devnet on merge to main
• Per-environment SOPS-encrypted secrets for the signer keypair
• Shared config components for environment-specific values

Docker image: ghcr.io/malbeclabs/doublezero-core — built and pushed by release.docker.core.yml in the doublezero repo. On push to main, tags edge and sha-<short> are produced. The binary inside is doublezero-device-health-oracle.

Approach

1. Create k8s/device-health-oracle/ directory (infra repo)

Follow the shred-oracle pattern exactly:

k8s/device-health-oracle/
├── README.md
├── base/
│   ├── deployment.yaml
│   └── kustomization.yaml
├── devnet/
│   └── kustomization.yaml
└── testnet/
    └── kustomization.yaml

base/deployment.yaml — shared deployment spec:

• Image: ghcr.io/malbeclabs/doublezero-core:PLACEHOLDER (tag set by config component or ImageUpdater)
• Command: doublezero-device-health-oracle
• Args: --signer-keypair /etc/device-health-oracle/keypair.json --ledger-rpc-url $(DZ_LEDGER_URL) --serviceability-program-id $(DZ_SERVICEABILITY_PROGRAM_ID) --telemetry-program-id $(DZ_TELEMETRY_PROGRAM_ID)
• envFrom: configMapRef: shared-config + secretRef: shared-secrets (consistent with shred-oracle)
• Volume: keypair mounted from device-health-oracle-env secret
• Metrics port: 8080 (existing convention for this service)
• Grafana scrape annotations + reloader annotation
• Liveness probe on /readyz endpoint (port 8080)
• Resources: 50m/64Mi requests, 500m/256Mi limits (same as shred-oracle)
• imagePullSecrets: ghcr-creds
• Strategy: Recreate

Note: Unlike the existing devnet deployment which uses --env devnet (relying on built-in config), the new deployment uses explicit flags with values from shared-config/shared-secrets. This is consistent with how shred-oracle works and decouples the binary from environment awareness at deploy time.

base/kustomization.yaml — references deployment.yaml, adds app.kubernetes.io/part-of: device-health-oracle label.

devnet/kustomization.yaml — overlay:

• References ../base
• Includes ../../config/devnet component (new — see step 3)
• Sets namespace dz-devnet
• Adds env: devnet label

testnet/kustomization.yaml — overlay:

• References ../base
• Includes ../../config/testnet component
• Sets namespace dz-testnet
• Adds env: testnet label

2. Create secrets in k8s/secrets/ (infra repo)

k8s/secrets/dz-devnet/device-health-oracle-env.enc.yaml (new file):

• Secret named device-health-oracle-env in namespace dz-devnet
• Contains keypair.json — the signer keypair for devnet
• Encrypted with SOPS using the AWS KMS key

k8s/secrets/dz-testnet/device-health-oracle-env.enc.yaml (new file):

• Secret named device-health-oracle-env in namespace dz-testnet
• Contains keypair.json — the signer keypair for testnet
• Encrypted with SOPS using the AWS KMS key

k8s/secrets/dz-devnet/kustomization.yaml — this directory does not exist yet. Create it with references to device-health-oracle-env.enc.yaml, plus shared-secrets.enc.yaml and ghcr-creds.enc.yaml (which also need to be created for the dz-devnet namespace).

k8s/secrets/dz-testnet/kustomization.yaml — add device-health-oracle-env.enc.yaml to the resources list.

Operator action required: The actual encrypted secret files must be created by someone with AWS KMS access. The plan provides the plaintext template structure; the operator encrypts with sops -e.

3. Create/update shared config components (infra repo)

k8s/config/devnet/ (new directory) — create a devnet config component:

kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

resources:
  - shared-config.yaml

images:
  - name: ghcr.io/malbeclabs/doublezero-core
    newTag: "edge"

shared-config.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: shared-config
data:
  SOLANA_URL: "devnet"
  DZ_SERVICEABILITY_PROGRAM_ID: "GYhQDKuESrasNZGyhMJhGYFtbzNijYhcrN9poSqCQVah"
  DZ_TELEMETRY_PROGRAM_ID: "C9xqH76NSm11pBS6maNnY163tWHT8Govww47uyEmSnoG"

k8s/config/testnet/shared-config.yaml — add DZ_TELEMETRY_PROGRAM_ID:

DZ_TELEMETRY_PROGRAM_ID: "3KogTMmVxc5eUHtjZnwm136H5P8tvPwVu4ufbGPvM7p1"

k8s/config/testnet/kustomization.yaml — add doublezero-core image entry:

images:
  - name: ghcr.io/malbeclabs/doublezero-shreds
    newTag: "0.1.1"
  - name: ghcr.io/malbeclabs/doublezero-core
    newTag: "edge"

Note: The newTag for testnet doublezero-core should be set to whatever tag the operator wants for testnet. Using edge is one option (tracks main), or a pinned version tag for more control. The operator should decide — edge is used here as a placeholder.

4. Create k8s/secrets/dz-devnet/ directory (infra repo)

The dz-devnet secrets directory does not exist yet. It needs:

• kustomization.yaml — listing all secret resources
• shared-secrets.enc.yaml — containing DZ_LEDGER_URL for devnet (the DZ ledger RPC endpoint)
• ghcr-creds.enc.yaml — GHCR image pull credentials for this namespace
• device-health-oracle-env.enc.yaml — signer keypair (from step 2)

The shared-secrets and ghcr-creds secrets may already exist in the dz-devnet namespace from the core kustomize deployment. The operator should verify and either:

• Create new SOPS-encrypted copies for the standalone secrets directory, or
• Confirm they're already synced via another mechanism (e.g., Reflector annotations on ghcr-creds)

5. Create ArgoCD Applications (infra repo)

k8s/argocd/applications/devnet/device-health-oracle.yaml (new file + new devnet/ directory):

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: device-health-oracle-devnet
  namespace: argocd
  annotations:
    notifications.argoproj.io/subscribe.dz-on-change.dz_slack: "#tmp-notify"
spec:
  project: default
  source:
    repoURL: https://github.com/malbeclabs/infra.git
    targetRevision: main
    path: k8s/device-health-oracle/devnet
  destination:
    server: https://kubernetes.default.svc
    namespace: dz-devnet
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

k8s/argocd/applications/testnet/device-health-oracle.yaml (new file):

• Same pattern, name device-health-oracle-testnet, path k8s/device-health-oracle/testnet, namespace dz-testnet

6. Create ArgoCD ImageUpdater for devnet auto-deploy (infra repo)

k8s/argocd/image-updater/device-health-oracle-devnet.yaml (new file):

apiVersion: argocd-image-updater.argoproj.io/v1alpha1
kind: ImageUpdater
metadata:
  name: device-health-oracle-devnet
  namespace: argocd
spec:
  namespace: argocd
  applicationRefs:
    - namePattern: device-health-oracle-devnet
      images:
        - alias: doublezero-core
          imageName: ghcr.io/malbeclabs/doublezero-core:edge
          commonUpdateSettings:
            updateStrategy: digest
            pullSecret: pullsecret:argocd/ghcr-creds

This watches the :edge tag's digest. When release.docker.core.yml pushes a new doublezero-core:edge image on merge to main, the ImageUpdater detects the digest change and triggers an ArgoCD sync of the devnet Application — achieving auto-deploy on merge to main.

7. Remove old devnet deployment from core overlay (infra repo)

Remove the device-health-oracle from the legacy kustomize/core path:

• Delete k8s/kustomize/core/overlays/devnet/device-health-oracle.yaml
• Delete k8s/kustomize/core/overlays/devnet/device-health-oracle-env.enc.yaml
• Remove references from k8s/kustomize/core/overlays/devnet/kustomization.yaml

This should be done after the new ArgoCD-managed deployment is confirmed working, to avoid downtime. Can be a follow-up commit or separate PR.

Files to Change (all in infra repo)

File	Change
k8s/device-health-oracle/README.md	New — documents the deployment structure
k8s/device-health-oracle/base/deployment.yaml	New — shared deployment spec
k8s/device-health-oracle/base/kustomization.yaml	New — base kustomization
k8s/device-health-oracle/devnet/kustomization.yaml	New — devnet overlay
k8s/device-health-oracle/testnet/kustomization.yaml	New — testnet overlay
k8s/config/devnet/kustomization.yaml	New — devnet config component
k8s/config/devnet/shared-config.yaml	New — devnet shared ConfigMap
k8s/config/testnet/shared-config.yaml	Edit — add DZ_TELEMETRY_PROGRAM_ID
k8s/config/testnet/kustomization.yaml	Edit — add doublezero-core image entry
k8s/secrets/dz-devnet/kustomization.yaml	New — devnet secrets kustomization
k8s/secrets/dz-devnet/device-health-oracle-env.enc.yaml	New — signer keypair (operator encrypts)
k8s/secrets/dz-devnet/shared-secrets.enc.yaml	New — DZ ledger URL (operator encrypts)
k8s/secrets/dz-devnet/ghcr-creds.enc.yaml	New — GHCR pull creds (operator encrypts)
k8s/secrets/dz-testnet/device-health-oracle-env.enc.yaml	New — signer keypair (operator encrypts)
k8s/secrets/dz-testnet/kustomization.yaml	Edit — add device-health-oracle-env.enc.yaml
k8s/argocd/applications/devnet/device-health-oracle.yaml	New — ArgoCD Application
k8s/argocd/applications/testnet/device-health-oracle.yaml	New — ArgoCD Application
k8s/argocd/image-updater/device-health-oracle-devnet.yaml	New — auto-deploy on edge digest change
k8s/secrets/README.md	Edit — add dz-devnet section to current secrets list

Post-migration cleanup (separate PR)

File	Change
k8s/kustomize/core/overlays/devnet/device-health-oracle.yaml	Delete
k8s/kustomize/core/overlays/devnet/device-health-oracle-env.enc.yaml	Delete
k8s/kustomize/core/overlays/devnet/kustomization.yaml	Edit — remove device-health-oracle refs

Risks & Considerations

1. dz-devnet secrets directory is new. No other service currently uses standalone secrets for devnet. The shared-secrets and ghcr-creds secrets need to be created for this namespace. The operator should verify whether these already exist in the cluster (they may have been created by the core kustomize deployment) and decide whether to create SOPS-encrypted copies or rely on existing mechanisms.

2. Shared config changes affect shred-oracle. Adding DZ_TELEMETRY_PROGRAM_ID to k8s/config/testnet/shared-config.yaml and adding doublezero-core to the images transformer in k8s/config/testnet/kustomization.yaml will be picked up by shred-oracle's testnet overlay too. The additional ConfigMap key is harmless (shred-oracle ignores unknown env vars). The additional image entry is also harmless (kustomize only applies image transforms to containers that match the image name). However, this should be verified with a kustomize build of shred-oracle after the changes.

3. SOPS encryption requires AWS access. All .enc.yaml files must be encrypted by someone with the appropriate KMS key access. The plan provides plaintext templates; the operator encrypts them.

4. Liveness probe assumes /readyz endpoint. The device-health-oracle exposes metrics on :8080 at /metrics. Need to verify whether it implements /readyz. If not, either use a TCP check on port 8080, or omit the liveness probe initially and add it once the endpoint exists.

5. Migration window. The old core overlay deployment and the new ArgoCD deployment should not run simultaneously (two instances writing health updates). The cutover should be: deploy new → verify working → delete old. Consider doing this in two PRs with a manual verification step between.

6. ImageUpdater only for devnet. Testnet uses a pinned image tag in the config component, giving the operator explicit control over when to promote. The operator can later add an ImageUpdater for testnet if desired.

Testing Strategy

1. Kustomize build validation: Run kustomize build k8s/device-health-oracle/devnet/ and kustomize build k8s/device-health-oracle/testnet/ to verify manifests render correctly.
2. Verify no cross-contamination: Run kustomize build k8s/shred-oracle/testnet/ after shared-config changes to verify shred-oracle is unaffected.
3. ArgoCD dry-run: After applying the ArgoCD Application, verify it syncs successfully and the pod starts.
4. Post-deploy verification: Check that the device-health-oracle pod is running, metrics are being scraped, and health updates are being written onchain.
5. Auto-deploy test: After merging a change to main in doublezero, verify the ImageUpdater detects the new edge digest and triggers a sync.

Estimated Scope

• ~15 new files, ~3 edited files
• ~200 lines of YAML added (deployment, kustomizations, ArgoCD apps, ImageUpdater, config, README)
• Well within the 500-line threshold
• Post-migration cleanup is ~2 deleted files + 1 edit, done as a separate PR