Kubernetes DCP Workers

One kubectl command · registered workers that scale with replicas · CPU or NVIDIA GPU

Before you start, you'll need two things from your Distributive account:

You also need kubectl access to the cluster with permission to create a namespace, a default StorageClass (most managed clusters have one), and outbound HTTPS from the nodes to the internet.

Running on a single machine instead? See Docker DCP Worker.

1 GPU setup (NVIDIA only, once per cluster)

Skip this step for CPU-only workers. For NVIDIA GPU nodes, install NVIDIA's GPU Operator with Helm. It installs the driver, the container toolkit, and the nvidia runtime class the GPU workers use: 

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && \
helm repo update && \
helm install --wait gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator --create-namespace

The operator then installs the driver on each GPU node, which takes several minutes. When it's done, every GPU node reports its GPUs. This should list a number for each GPU node: 

kubectl get nodes -o custom-columns='NODE:.metadata.name,GPUS:.status.allocatable.nvidia\.com/gpu'

On k3s, restart k3s on each GPU node once the operator's pods are running (sudo systemctl restart k3s) so its built-in containerd picks up the NVIDIA runtime; until then the GPUs don't appear.

If your cluster already provides NVIDIA GPUs some other way (for example a managed cloud GPU node pool), check that it has a RuntimeClass named nvidia: kubectl get runtimeclass.

2 Deploy the workers

Fill in your values and the command below updates. Nothing you type leaves this page.

Or copy the command as-is and replace the placeholder values before running it. 

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Namespace
metadata:
  name: dcp
  labels:
    pod-security.kubernetes.io/enforce: baseline
---
apiVersion: v1
kind: Secret
metadata:
  name: dcp-worker
  namespace: dcp
stringData:
  registration-key: "0xYOUR_REGISTRATION_KEY"
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: dcp-worker
  namespace: dcp
data:
  earnings-account: "0xYOUR_EARNINGS_ACCOUNT"
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: dcp-worker
  namespace: dcp
spec:
  serviceName: dcp-worker
  replicas: 2
  podManagementPolicy: Parallel
  selector:
    matchLabels: { app: dcp-worker }
  template:
    metadata:
      labels: { app: dcp-worker }
    spec:
      terminationGracePeriodSeconds: 300
      initContainers:
        - name: register
          image: distributivenetwork/dcp-worker
          command: ["/bin/sh", "-c"]
          args:
            - |
              [ -f /opt/dcp/.dcp/registered ] && exit 0
              /bin/sh /opt/dcp/bin/dcp-docker.sh --register="$REGISTRATION_KEY,$POD_NAME" &&
              touch /opt/dcp/.dcp/registered && sleep 65
          env:
            - name: REGISTRATION_KEY
              valueFrom: { secretKeyRef: { name: dcp-worker, key: registration-key } }
            - name: POD_NAME
              valueFrom: { fieldRef: { fieldPath: metadata.name } }
          volumeMounts:
            - { name: identity, mountPath: /opt/dcp/.dcp }
      containers:
        - name: worker
          image: distributivenetwork/dcp-worker
          args: ["--no-global", "--earnings-account=$(EARNINGS_ACCOUNT)"]
          env:
            - name: EARNINGS_ACCOUNT
              valueFrom: { configMapKeyRef: { name: dcp-worker, key: earnings-account } }
          resources:
            requests: { cpu: "2", memory: 2Gi }
            limits: { cpu: "4", memory: 4Gi }
          volumeMounts:
            - { name: identity, mountPath: /opt/dcp/.dcp }
  volumeClaimTemplates:
    - metadata:
        name: identity
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests: { storage: 64Mi }
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: dcp-worker-gpu
  namespace: dcp
spec:
  serviceName: dcp-worker-gpu
  replicas: 0
  podManagementPolicy: Parallel
  selector:
    matchLabels: { app: dcp-worker-gpu }
  template:
    metadata:
      labels: { app: dcp-worker-gpu }
    spec:
      runtimeClassName: nvidia
      terminationGracePeriodSeconds: 300
      initContainers:
        - name: register
          image: distributivenetwork/dcp-worker
          command: ["/bin/sh", "-c"]
          args:
            - |
              [ -f /opt/dcp/.dcp/registered ] && exit 0
              /bin/sh /opt/dcp/bin/dcp-docker.sh --register="$REGISTRATION_KEY,$POD_NAME" &&
              touch /opt/dcp/.dcp/registered && sleep 65
          env:
            - name: REGISTRATION_KEY
              valueFrom: { secretKeyRef: { name: dcp-worker, key: registration-key } }
            - name: POD_NAME
              valueFrom: { fieldRef: { fieldPath: metadata.name } }
          volumeMounts:
            - { name: identity, mountPath: /opt/dcp/.dcp }
      containers:
        - name: worker
          image: distributivenetwork/dcp-worker
          args: ["--no-global", "--earnings-account=$(EARNINGS_ACCOUNT)"]
          env:
            - name: EARNINGS_ACCOUNT
              valueFrom: { configMapKeyRef: { name: dcp-worker, key: earnings-account } }
          resources:
            requests: { cpu: "2", memory: 4Gi }
            limits: { cpu: "4", memory: 8Gi, nvidia.com/gpu: 1 }
          volumeMounts:
            - { name: identity, mountPath: /opt/dcp/.dcp }
  volumeClaimTemplates:
    - metadata:
        name: identity
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests: { storage: 64Mi }
EOF

What it creates, all in a new dcp namespace:

  1. A Secret and a ConfigMap holding your registration key and earnings account.
  2. A StatefulSet of CPU workers. Each pod gets its own small persistent volume holding its identity, so it stays the same registered worker through restarts, upgrades, and moves to another node.
  3. On first start, each pod registers itself (named after the pod, e.g. dcp-worker-0) and waits about a minute for the registration to take effect. Later restarts skip this.
  4. Each worker runs with --no-global, so it only runs jobs from your own compute groups, and sizes itself to its CPU limit (4 cores here).
  5. A StatefulSet of GPU workers, one NVIDIA GPU per pod, using the nvidia runtime class from step 1. WebGPU is on by default in the image.

3 Confirm it's running

kubectl -n dcp get pods

On first start each pod shows Init:0/1 for about two minutes while it registers, then Running. To watch a worker's log: 

kubectl -n dcp logs -f dcp-worker-0 -c worker

A healthy startup ends with:

* Starting Distributive Worker <worker-id>
. Configured Cores:    cpu: 4    gpu: 0
! Leaving the global compute group
* Ready.

A GPU worker (dcp-worker-gpu-0) shows WebGPU available with vendor: nvidia and gpu: 1. When a job from one of your compute groups is available you'll see Slice Completed lines.

4 Scaling and updating

Change the number of CPU workers (new pods register themselves):

kubectl -n dcp scale statefulset dcp-worker --replicas=4

Update all workers to the latest image, one pod at a time:

kubectl -n dcp rollout restart statefulset

Each pod gets up to five minutes to finish the slices it's running before it stops. Pods removed by scaling down keep their identity volume, so scaling back up brings back the same workers. To change the earnings account, edit the ConfigMap with kubectl -n dcp edit configmap dcp-worker, then run the restart command above.

5 Uninstalling

Stop and remove the workers, keeping their identities:

kubectl -n dcp delete statefulset --all

The identity volumes stay, so running the deploy command again brings back the same registered workers.

To remove everything, delete the namespace. This deletes the workers' identities too; deploying again later registers new workers. 

kubectl delete namespace dcp

6 Troubleshooting

Pod stuck in Init or Init:Error: registration is failing. See why with kubectl -n dcp logs dcp-worker-0 -c register. Check the key was copied completely and that it hasn't been used up or expired.

Pod stuck in Pending: run kubectl -n dcp describe pod dcp-worker-0. Usually the cluster has no default StorageClass for the identity volume, not enough free CPU or memory, or (for GPU workers) no free nvidia.com/gpu.

Pods rejected by Pod Security: the worker image starts as root and drops to an unprivileged user for the worker itself, so the namespace uses the baseline policy. Clusters that enforce restricted everywhere need an exception for the dcp namespace.

Running and Ready, but never gets work: check that a job is running in one of your compute groups, then restart the pod with kubectl -n dcp delete pod dcp-worker-0; it's recreated with the same identity.

GPU worker logs This GPU is not supported; disabling: the pod didn't get the NVIDIA graphics driver. Check that the pod spec has runtimeClassName: nvidia and that the GPU Operator's pods are all running: kubectl -n gpu-operator get pods.