This guide implements a single-developer Constitutional AI loop on a rented GPU using Mistral-7B-Instruct-v0.3, LLaMA-Factory, and local scripts, avoiding third-party APIs. The workflow begins locally by writing principles.txt (answer constraints), prompts.txt (training questions), and heldout.txt (evaluation questions), then uploading them to a CloudGPU instance via SCP. A 7B model in bfloat16 requires sufficient VRAM; the guide uses a PyTorch-enabled template or Raw GPU install. LLaMA-Factory is installed from source (hiyouga/LlamaFactory) with metrics requirements. The core loop (loop.py) runs three batched passes via transformers: generate answers (temperature 0.7, top_p 0.9, max_new_tokens 400), critique them against principles (max_new_tokens 200), and revise only flagged answers. Outputs are saved to loop_raw.jsonl and loop_revised.jsonl. A filter script (filter.py) removes empty, identical, over-length (>2500 chars), under-length (<20 chars), duplicate, or meta-leaking revisions, producing selfloop_sft.json in LLaMA-Factory's Alpaca format and a 30-row markdown sample for manual review. The dataset is registered in data/dataset_info.json. A LoRA adapter (rank 8, target all) is trained for 3 epochs (batch 1, grad_accum 8, lr 1e-4, cosine scheduler, warmup 0.1, bf16, cutoff_len 2048, mistral template) using llamafactory-cli train. Evaluation (compare.py) runs greedy decoding on heldout questions for both base and PeftModel-wrapped adapter, writing side-by-side comparisons to compare.md. The user judges adherence to principles; if the adapter is not clearly better, the loop repeats with revised principles or prompts. Finally, the adapter directory and comparison file are rsynced/scp'd locally before instance destruction stops billing. License compliance (Apache-2.0 for Mistral, naming restrictions for Llama/Gemma derivatives) and AI-generated labeling are emphasized throughout.
Read original
dev.to