VaultLayer examples — GPU training scripts that survive interruptions

Ready-to-run PyTorch and Hugging Face training scripts for VaultLayer, the BYOC (bring-your-own-cloud) AI training control plane. VaultLayer runs your existing training command on your own cloud or on elastic GPU capacity and adds the reliability layer — orchestration, automatic checkpointing, and resume-on-interruption — so a reclaimed spot GPU or crashed host never costs you the run.

Every script here checkpoints as it trains and resumes from the last saved step after an interruption, on the same machine or a different one.

Quickstart

pip install vaultlayer
vaultlayer init                  # one-time authentication
vl run python train_quick.py     # submit any training script

No SDK, no decorators, no YAML — vl run wraps the command you already use. Sign up at dashboard.vaultlayer.cloud/signup.

Real LLM fine-tunes (QLoRA)

Script	Model	VRAM	Duration	Notes
train_tinyllama_qlora.py	TinyLlama-1.1B	~3 GB	~10 min on A10G	Smoke test — fits any GPU
train_llama3_2_3b_qlora.py	Llama-3.2-3B-Instruct	~8 GB	~25–40 min on A10G	Gated model — pass --env HF_TOKEN=hf_...
train_qwen7b_qlora.py	Qwen2.5-7B-Instruct	~16 GB	~12–20 min on A100-40	Production-class 7B, no HF token needed
train_mistral7b_qlora.py	Mistral-7B-v0.1	~15 GB	~12–20 min on A100-40	Alternative 7B arch — gated, needs HF_TOKEN

vl run python train_tinyllama_qlora.py
vl run --gpu A10G --env HF_TOKEN=hf_... python train_llama3_2_3b_qlora.py
vl run --gpu A100_40 python train_qwen7b_qlora.py

Not sure which GPU a model needs? See how much GPU memory to fine-tune an LLM and QLoRA vs LoRA vs full fine-tuning.

Smaller tests

Script	Duration	What it shows
train_quick.py	~30 s	GPU detection + checkpoint pipeline smoke test
train_mnist.py	~2 min	Real CNN on MNIST (12 MB auto-download)
train_long.py	~10 min	Long run for testing interruption + auto-resume
demo_resume.py	—	End-to-end proof: interrupt a run mid-training, watch it resume from the last checkpoint
tinyllama_alpaca.py	~45 min	Full (non-QLoRA) fine-tune — heavier, kept for comparison

How resume-on-interruption works

Scripts save checkpoints in the standard Hugging Face checkpoint-<step>/ format to the directory VaultLayer provides, and resume with one argument:

import os
ckpt_dir = os.environ["VAULTLAYER_CHECKPOINT_DIR"]   # provided on the GPU node

# Hugging Face Trainer / TRL: save periodically, resume from the latest
args = TrainingArguments(output_dir=ckpt_dir, save_steps=200, ...)
trainer.train(resume_from_checkpoint=latest_checkpoint(ckpt_dir))

VaultLayer syncs that directory to durable storage as training runs and restores it onto the replacement machine after an interruption — the details are in how checkpoint & resume works. Plain-PyTorch integration is a three-line helper; see resume PyTorch training after a crash.

Learn more

• Docs — CLI reference
• BYOC GPU training — run on your own cloud
• Fault-tolerant training
• Spot GPU training that actually finishes
• FSDP vs DDP
• All guides

License

MIT. Free to fork and modify for your own workloads.

Contributing

PRs welcome. New scripts must:

• Read $VAULTLAYER_CHECKPOINT_DIR (not a hard-coded path).
• Save Hugging Face-style checkpoint-<step>/ directories (save_steps= or equivalent).
• Resume via trainer.train(resume_from_checkpoint=...) so interrupted runs continue cleanly.
• Include a header docstring with VRAM + duration + GPU guidance.
• Print with flush=True so progress streams live.

Questions: rahuljain@vaultlayer.cloud