This guide details a single-developer pipeline for aligning a language model using Direct Preference Optimization (DPO) on preference pairs generated by an LLM judge. The process begins locally: a user writes principles and questions, then uses a Python script with the OpenAI SDK to query DeepSeek's deepseek-v4.1-flash model via the CloudGPU API ( https://cloudgpu.app/v1 ). For each question, the model generates two answers at temperature 1.0, then judges them twice against the principles with swapped order; pairs are kept only if both verdicts agree on the winner, yielding a dpo_pairs.json dataset in LLaMA-Factory preference format. The user manually validates a sample of pairs before scaling up. Training runs on a rented CloudGPU instance (billed per minute, minimum one minute) via SSH. After uploading the dataset, the user installs LLaMA-Factory, registers the dataset with "ranking": true , and configures a LoRA DPO run (e.g., lora_rank: 8 , pref_beta: 0.1 , pref_loss: sigmoid , learning_rate: 5e-6 , 3 epochs, BF16) on a base model such as Mistral-7B-Instruct-v0.3 (template mistral ), Llama-3.2-3B-Instruct ( llama3 ), or Gemma-3-4B-IT ( gemma3 ). Post-training, the adapter is evaluated locally via llamafactory-cli chat on held-out questions; the round is discarded unless the adapter visibly follows principles better than the base. Finally, the adapter ( adapter_model.safetensors , adapter_config.json ) is copied off via rsync and the instance destroyed to halt billing. Compliance notes: DeepSeek Terms permit training on outputs (Section 4.2(3)) with AI-generated labeling (Section 3.1); base model licenses must be verified; CloudGPU AUP prohibits unauthorized datasets.