Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,15 @@ Never use a guessed checksum or silently fall back to a source build.
Missing or invalid release pins prevent builds. `make versions` reports the
release tag and archive digest without requiring the distill-fs submodule.

The optional node-wide ChunkDB capacity is configured with standalone
`AKERNEL_CHUNK_DB_SIZE`, Helm `node.config.sandboxd.chunkDbSize` (under
`core` in the umbrella chart), or Terraform `chunk_db_size`. Guided profiles
accept `make config CHUNK_DB_SIZE=64GiB`. Empty values preserve the Linux
100 GiB default. Sandboxd passes the same capacity to every distill-fs user;
existing-cache resizing is unsupported. Preserve the
`# AKERNEL_CHUNK_DB_SIZE` marker in custom deployment templates. See
`deploy/README.md#shared-chunkdb-capacity` for cache replacement semantics.

Each component embeds its own semantic version: sandboxd uses
`version/VERSION`, while distill-fs uses its release package version in
`Cargo.toml`. AKernel does not inject parent-repository version metadata into
Expand Down
2 changes: 2 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ ACR_NAMESPACE ?=
MONITOR_STORAGE_CLASS ?=
INSTALL_MONITOR ?=
INSTALL_DRAGONFLY ?=
CHUNK_DB_SIZE ?=
ENABLE_RUNC ?=
SCHEDULE_PLACEMENT_POLICY ?=
GRAFANA_PUBLIC_ACCESS ?=
Expand Down Expand Up @@ -93,6 +94,7 @@ config:
if [[ -n "$(IMAGE_TAG)" ]]; then args+=(--image-tag "$(IMAGE_TAG)"); fi; \
if [[ -n "$(INSTALL_MONITOR)" ]]; then args+=(--install-monitor "$(INSTALL_MONITOR)"); fi; \
if [[ -n "$(INSTALL_DRAGONFLY)" ]]; then args+=(--install-dragonfly "$(INSTALL_DRAGONFLY)"); fi; \
if [[ -n "$(CHUNK_DB_SIZE)" ]]; then args+=(--chunk-db-size "$(CHUNK_DB_SIZE)"); fi; \
if [[ -n "$(ENABLE_RUNC)" ]]; then args+=(--enable-runc "$(ENABLE_RUNC)"); fi; \
if [[ -n "$(SCHEDULE_PLACEMENT_POLICY)" ]]; then args+=(--schedule-placement-policy "$(SCHEDULE_PLACEMENT_POLICY)"); fi; \
if [[ -n "$(GRAFANA_PUBLIC_ACCESS)" ]]; then args+=(--grafana-public-access "$(GRAFANA_PUBLIC_ACCESS)"); fi; \
Expand Down
6 changes: 3 additions & 3 deletions builder/distill-fs-versions.env
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,6 @@

# AKernel's distill-fs release dependency. Verify the published release archive
# before updating its version, URL, and SHA-256 together.
DISTILL_FS_RELEASE=v0.1.1
DISTILL_FS_AMD64_URL=https://github.com/inclusionAI/distill-fs/releases/download/v0.1.1/distill-fs-v0.1.1-linux-amd64.tar.gz
DISTILL_FS_AMD64_SHA256=b16e225e5b777f673bd98dd0968a15c190e333690356552b69a2b2624c3d5dcb
DISTILL_FS_RELEASE=v0.1.2
DISTILL_FS_AMD64_URL=https://github.com/inclusionAI/distill-fs/releases/download/v0.1.2/distill-fs-v0.1.2-linux-amd64.tar.gz
DISTILL_FS_AMD64_SHA256=d71620f5e9a36ae08602adea76f604cc113953cb0dd9d4f134aee16f0c3f6f62
8 changes: 8 additions & 0 deletions deploy/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -404,3 +404,11 @@ The all-in-one image downloads the static Linux/amd64 distill-fs release pinned
Publish and verify the distill-fs release before updating the AKernel version, URL, and checksum pin together. Missing or invalid pins stop `make build` before either image is built. There is no source-build fallback.

Validate installation against a downloaded candidate or release with `python3 builder/scripts/test-install-distill-fs.py /path/to/distill-fs-vX.Y.Z-linux-amd64.tar.gz` on Linux/amd64 with curl, jq, and binutils. This checks normal installation and rejects missing/invalid pins, corrupted archives, version/architecture mismatch, binary hash mismatch, and dynamically linked executables. The sandboxd pipeline and gitlink are independent of this dependency.

## Shared ChunkDB capacity

The node image's distill-fs v0.1.2 supports a configurable shared image-cache database capacity. The Linux default remains 100 GiB when omitted. Set `AKERNEL_CHUNK_DB_SIZE=64GiB` for standalone, `core.node.config.sandboxd.chunkDbSize=64GiB` in the umbrella Helm chart (`node.config.sandboxd.chunkDbSize` in the core chart), or `chunk_db_size = "64GiB"` in either cloud Terraform module. Guided profiles accept `make config CHUNK_DB_SIZE=64GiB`; this records the value in the generated Terraform configuration. Use a node image built with the matching sandboxd and distill-fs pins before enabling this option.

Whole bytes and integer `B`, `KiB`, `MiB`, `GiB`, or `TiB` values are supported. Sandboxd validates the minimum of 1 MiB, addressability, and host page alignment and supplies the same capacity to mounts, stats, GC, and recovered daemons. This is the LMDB map limit, not a memory reservation, a whole-cache disk quota, or a sandbox `storage_mb` limit.

Resizing an existing cache is unsupported. Same-Pod service restarts retain it; a replacement Pod with a changed hostname clears the image-manager root and creates a new cache with the configured capacity. Drain workloads before replacement. Standalone deployments must stop all users and select a fresh image-manager cache directory when changing capacity. Custom sandboxd configuration templates must retain the `# AKERNEL_CHUNK_DB_SIZE` marker when using the standalone or Helm override.
11 changes: 11 additions & 0 deletions deploy/akernel/charts/core/templates/node/configmap.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,17 @@ data:
{{- end }}
{{- $sandboxdConfig = replace $runcMarker "runc=\"/usr/local/bin/runc\"" $sandboxdConfig }}
{{- end }}
{{- $chunkDbSize := .Values.node.config.sandboxd.chunkDbSize | default "" }}
{{- if $chunkDbSize }}
{{- if not (regexMatch "^[0-9]+(B|KiB|MiB|GiB|TiB)?$" $chunkDbSize) }}
{{- fail "node.config.sandboxd.chunkDbSize must be whole bytes or an integer with B/KiB/MiB/GiB/TiB" }}
{{- end }}
{{- $chunkDbMarker := "# AKERNEL_CHUNK_DB_SIZE" }}
{{- if not (contains $chunkDbMarker $sandboxdConfig) }}
{{- fail "node.config.sandboxd.chunkDbSize requires the # AKERNEL_CHUNK_DB_SIZE marker in node.config.sandboxd.config" }}
{{- end }}
{{- $sandboxdConfig = replace $chunkDbMarker (printf "chunk_db_size=%s" ($chunkDbSize | quote)) $sandboxdConfig }}
{{- end }}
{{ $sandboxdConfig | indent 4 }}

registry.json: |
Expand Down
3 changes: 3 additions & 0 deletions deploy/akernel/charts/core/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -451,6 +451,8 @@ node:
# The node image must also be built with AKERNEL_ENABLE_RUNC=true.
# Sandboxd probes runc and runc-shim before advertising the runtime.
enableRunc: false
# Empty uses the distill-fs default (100 GiB on Linux).
chunkDbSize: ""
config: |-
rootDir="/home/akernel/sandboxd/root"
storeDir="/home/akernel/sandboxd/store"
Expand Down Expand Up @@ -519,6 +521,7 @@ node:
# configs live under /home/akernel/sandboxd/config/ so the wipe never
# hits the read-only bind-mounted config files.
[plugin.image]
# AKERNEL_CHUNK_DB_SIZE
Comment thread
tianyuzhou95 marked this conversation as resolved.
Resolved
root="/home/akernel/sandboxd/image_manager"
distill_fs_bin="/usr/local/bin/distill_fs"
oss_template="/home/akernel/sandboxd/config/oss.json"
Expand Down
12 changes: 12 additions & 0 deletions deploy/scripts/configure.sh
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ image_repository_override=""
image_tag_override=""
install_monitor_override=""
install_dragonfly_override=""
chunk_db_size_override=""
enable_runc_override=""
schedule_placement_policy_override=""
grafana_public_access_override=""
Expand Down Expand Up @@ -118,6 +119,10 @@ while [[ $# -gt 0 ]]; do
install_dragonfly_override="$2"
shift 2
;;
--chunk-db-size)
chunk_db_size_override="$2"
shift 2
;;
--enable-runc)
enable_runc_override="$2"
shift 2
Expand Down Expand Up @@ -238,6 +243,10 @@ esac
set_or_prompt image_tag "All-in-one image tag" "${default_tag}" "${image_tag_override}"
set_or_prompt install_monitor "Install monitor chart (true/false)" "true" "${install_monitor_override}"
set_or_prompt install_dragonfly "Install Dragonfly and dedicated node pools (true/false)" "false" "${install_dragonfly_override}"
set_or_prompt chunk_db_size "Shared ChunkDB capacity (empty uses distill-fs default)" "" "${chunk_db_size_override}"
if [[ -n "${chunk_db_size}" && ! "${chunk_db_size}" =~ ^[0-9]+(B|KiB|MiB|GiB|TiB)?$ ]]; then
die "CHUNK_DB_SIZE must be whole bytes or an integer with B/KiB/MiB/GiB/TiB"
fi
set_or_prompt enable_runc "Enable the optional runc runtime (true/false)" "false" "${enable_runc_override}"
set_or_prompt schedule_placement_policy \
"YuanRong schedule placement policy (binpack/spread)" "spread" \
Expand Down Expand Up @@ -373,6 +382,7 @@ grafana_public_access = ${grafana_public_access}
grafana_admin_password = "${grafana_admin_password}"

install_dragonfly = ${install_dragonfly}
chunk_db_size = "${chunk_db_size}"
enable_runc = ${enable_runc}
schedule_placement_policy = "${schedule_placement_policy}"
EOF
Expand Down Expand Up @@ -428,6 +438,7 @@ grafana_public_access = ${grafana_public_access}
grafana_admin_password = "${grafana_admin_password}"

install_dragonfly = ${install_dragonfly}
chunk_db_size = "${chunk_db_size}"
enable_runc = ${enable_runc}
schedule_placement_policy = "${schedule_placement_policy}"
EOF
Expand All @@ -450,6 +461,7 @@ IMAGE_TAG=${image_tag}
CORE_NAMESPACE=akernel
MONITOR_NAMESPACE=akernel-monitor
INSTALL_DRAGONFLY=${install_dragonfly}
CHUNK_DB_SIZE=${chunk_db_size}
AKERNEL_ENABLE_RUNC=${enable_runc}
SCHEDULE_PLACEMENT_POLICY=${schedule_placement_policy}
EOF
Expand Down
1 change: 1 addition & 0 deletions deploy/standalone/config/sandboxd_config.toml
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ kvm_device="/dev/kvm"
# on pod change; input configs live under /home/akernel/sandboxd/config/ so the
# wipe never hits them.
[plugin.image]
# AKERNEL_CHUNK_DB_SIZE
root="/home/akernel/sandboxd/image_manager"
distill_fs_bin="/usr/local/bin/distill_fs"
oss_template="/home/akernel/sandboxd/config/oss.json"
Expand Down
15 changes: 15 additions & 0 deletions deploy/standalone/start.sh
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ TOKEN_FILE="${DATA_DIR}/token"
SANDBOXD_CONFIG_FILE="${DATA_DIR}/sandboxd/config.toml"
AKERNEL_NAT_BACKEND="${AKERNEL_NAT_BACKEND:-iptables}"
AKERNEL_ENABLE_RUNC="${AKERNEL_ENABLE_RUNC:-false}"
AKERNEL_CHUNK_DB_SIZE="${AKERNEL_CHUNK_DB_SIZE:-}"
YR_IMAGE_PROCESS_CONFIG="${YR_IMAGE_PROCESS_CONFIG:-/run/akernel/yr-image-process.json}"
LITEBUS_DATA_KEY=""

Expand Down Expand Up @@ -268,6 +269,20 @@ configure_network() {
-e 's|^[[:space:]]*# AKERNEL_RUNTIME_RUNC[[:space:]]*$|runc="/usr/local/bin/runc"|'
)
fi
if [[ -n "${AKERNEL_CHUNK_DB_SIZE}" ]]; then
if [[ ! "${AKERNEL_CHUNK_DB_SIZE}" =~ ^[0-9]+(B|KiB|MiB|GiB|TiB)?$ ]]; then
log_error "AKERNEL_CHUNK_DB_SIZE must be whole bytes or an integer with B/KiB/MiB/GiB/TiB"
exit 1
fi
if ! grep -q '^[[:space:]]*# AKERNEL_CHUNK_DB_SIZE[[:space:]]*$' \
"${CONFIG_DIR}/sandboxd_config.toml"; then
log_error "AKERNEL_CHUNK_DB_SIZE requires the # AKERNEL_CHUNK_DB_SIZE marker in sandboxd_config.toml"
exit 1
fi
sed_args+=(
-e "s|^[[:space:]]*# AKERNEL_CHUNK_DB_SIZE[[:space:]]*$|chunk_db_size=\"${AKERNEL_CHUNK_DB_SIZE}\"|"
)
fi
sed "${sed_args[@]}" "${CONFIG_DIR}/sandboxd_config.toml" > "${config_tmp}"
mv "${config_tmp}" "${SANDBOXD_CONFIG_FILE}"

Expand Down
1 change: 1 addition & 0 deletions deploy/terraform/aliyun/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,7 @@ locals {
master_service_type = (var.master_public_access_8888 && !var.traefik_enabled) ? var.master_service_type : "ClusterIP"
traefik_enabled = var.traefik_enabled
sandboxd_nat_backend = var.sandboxd_nat_backend
chunk_db_size = var.chunk_db_size
enable_runc = var.enable_runc
node_secret_create = var.node_secret_create
node_home_use_csi_ephemeral = var.node_home_use_csi_ephemeral
Expand Down
2 changes: 2 additions & 0 deletions deploy/terraform/aliyun/values-akernel.yaml.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,7 @@ node:
cpuQuota: "100%"
sandboxd:
enableRunc: ${enable_runc}
chunkDbSize: ${jsonencode(chunk_db_size)}
config: |-
rootDir="/home/akernel/sandboxd/root"
storeDir="/home/akernel/sandboxd/store"
Expand Down Expand Up @@ -213,6 +214,7 @@ node:
# configs live under /home/akernel/sandboxd/config/ so the wipe never
# hits the read-only bind-mounted config files.
[plugin.image]
# AKERNEL_CHUNK_DB_SIZE
root="/home/akernel/sandboxd/image_manager"
distill_fs_bin="/usr/local/bin/distill_fs"
oss_template="/home/akernel/sandboxd/config/oss.json"
Expand Down
10 changes: 10 additions & 0 deletions deploy/terraform/aliyun/variables.tf
Original file line number Diff line number Diff line change
Expand Up @@ -966,3 +966,13 @@ variable "tempo_resources" {
})
default = {}
}

variable "chunk_db_size" {
type = string
description = "Shared distill-fs ChunkDB map capacity, e.g. 64GiB; empty uses the Linux default of 100 GiB. Existing caches must not be resized."
default = ""
validation {
condition = var.chunk_db_size == "" || can(regex("^[0-9]+(B|KiB|MiB|GiB|TiB)?$", var.chunk_db_size))
error_message = "chunk_db_size must be empty, whole bytes, or an integer with B/KiB/MiB/GiB/TiB."
}
}
1 change: 1 addition & 0 deletions deploy/terraform/huaweicloud/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,7 @@ locals {
master_service_annotations = merge(local.huaweicloud_master_elb_annotations, var.master_service_annotations)
master_service_loadbalancer_ip = var.master_public_access_8888 ? var.master_service_loadbalancer_ip : ""
sandboxd_nat_backend = var.sandboxd_nat_backend
chunk_db_size = var.chunk_db_size
enable_runc = var.enable_runc
node_secret_create = var.node_secret_create
node_home_use_csi_ephemeral = var.node_home_use_csi_ephemeral
Expand Down
2 changes: 2 additions & 0 deletions deploy/terraform/huaweicloud/values-akernel.yaml.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,7 @@ node:
cpuQuota: "100%"
sandboxd:
enableRunc: ${enable_runc}
chunkDbSize: ${jsonencode(chunk_db_size)}
config: |-
rootDir="/home/akernel/sandboxd/root"
storeDir="/home/akernel/sandboxd/store"
Expand Down Expand Up @@ -191,6 +192,7 @@ node:
# configs live under /home/akernel/sandboxd/config/ so the wipe never
# hits the read-only bind-mounted config files.
[plugin.image]
# AKERNEL_CHUNK_DB_SIZE
root="/home/akernel/sandboxd/image_manager"
distill_fs_bin="/usr/local/bin/distill_fs"
oss_template="/home/akernel/sandboxd/config/oss.json"
Expand Down
10 changes: 10 additions & 0 deletions deploy/terraform/huaweicloud/variables.tf
Original file line number Diff line number Diff line change
Expand Up @@ -1000,3 +1000,13 @@ variable "tempo_resources" {
})
default = {}
}

variable "chunk_db_size" {
type = string
description = "Shared distill-fs ChunkDB map capacity, e.g. 64GiB; empty uses the Linux default of 100 GiB. Existing caches must not be resized."
default = ""
validation {
condition = var.chunk_db_size == "" || can(regex("^[0-9]+(B|KiB|MiB|GiB|TiB)?$", var.chunk_db_size))
error_message = "chunk_db_size must be empty, whole bytes, or an integer with B/KiB/MiB/GiB/TiB."
}
}
Loading