ComfyUI ControlNet Tutorial: Guided Image Generation with Edge Detection
ComfyUI ControlNet Tutorial: Guided Image Generation
Want to generate an image that follows the exact structure of a reference photo? That’s what ControlNet does — it extracts the edges, pose, or depth from an image and uses that as a guide for generation. This tutorial shows you how to set it up in ComfyUI.
What You’ll Build
A workflow that:
- Loads a reference image (e.g., a product photo, architectural sketch, or portrait)
- Extracts edge structure using Canny edge detection
- Generates a new image guided by those edges + your text prompt
- Runs 100% locally — no cloud, no API costs
Use cases: Product visualization, architectural rendering, brand-consistent content, style transfer.
Prerequisites
- ComfyUI installed (Desktop app or manual)
- Stable Diffusion checkpoint from Stability AI (e.g.,
sd_xl_base_1.0.safetensorsor any SDXL model) - ControlNet model (e.g.,
control-lora-canny-rank256.safetensors) — see the ControlNet collection on HuggingFace - ComfyUI ControlNet Aux node pack: install via ComfyUI Manager
flowchart LR
INPUT["Input Image"] --> CN["ControlNet<br/>Preprocessor"]
CN --> COND["Conditioning"]
PROMPT["Text Prompt"] --> COND
COND --> SAMPLER["KSampler"]
SAMPLER --> OUTPUT["Generated Image"]
style INPUT fill:#1E293B,color:#FAFAFA
style CN fill:#F5A623,color:#0B1628
style OUTPUT fill:#059669,color:#FAFAFAThe Workflow (11 Nodes)
Reference Image → Canny Edge Detection → ControlNet Apply
↓
Text Prompt → CLIP Encode ──────────────→ KSampler → VAE Decode → Save Image
↑
Checkpoint → Model + VAE ─────────────────────┘Node-by-Node Setup
| # | Node | Settings |
|---|---|---|
| 1 | CheckpointLoaderSimple | Select your SDXL or SD 1.5 checkpoint |
| 2 | ControlNetLoader | Load control-lora-canny-rank256.safetensors |
| 3 | LoadImage | Upload your reference image |
| 4 | CannyEdgePreprocessor | low_threshold: 100, high_threshold: 200, resolution: 1024 |
| 5 | CLIPTextEncode (positive) | Your prompt: “professional product photo, studio lighting” |
| 6 | CLIPTextEncode (negative) | “blurry, low quality, watermark, text” |
| 7 | ControlNetApplyAdvanced | strength: 0.8, start_percent: 0, end_percent: 1.0 |
| 8 | EmptyLatentImage | width: 1024, height: 1024, batch_size: 1 |
| 9 | KSampler | steps: 25, cfg: 7.5, sampler: euler_ancestral, scheduler: normal |
| 10 | VAEDecode | Connect to KSampler output |
| 11 | SaveImage | filename_prefix: “controlnet_“ |
Key Parameters
| Parameter | What it does | Recommended |
|---|---|---|
| strength (ControlNet) | How strictly to follow the edge guide | 0.7-0.9 for structure, 0.3-0.5 for loose guidance |
| low_threshold (Canny) | Minimum edge sensitivity | 100 (default) |
| high_threshold (Canny) | Maximum edge sensitivity | 200 (default) |
| cfg (KSampler) | How closely to follow the text prompt | 7-8 for balanced, 10+ for strict |
Tips for Best Results
- Start with strong ControlNet (0.8+) and reduce if results look rigid
- Use line art or architectural photos as references — clean edges work best
- Match resolution: if your reference is 1024x1024, set EmptyLatentImage to 1024x1024
- Experiment with ControlNet types: Canny (edges), Depth (3D), Pose (human), Scribble (sketches)
- SDXL models produce higher quality than SD 1.5 but need more VRAM (6+ GB)
Download the Workflow
Ready-to-import JSON file:
Download vorlux_controlnet_canny_guided.json →
Import in ComfyUI: Load Workflow → Upload File.
Hardware Requirements
| Hardware | VRAM | Speed | Quality |
|---|---|---|---|
| Mac Mini M4 (24GB) | 24GB unified | ~30s/image | Excellent |
| RTX 4060 Ti (16GB) | 16GB | ~15s/image | Excellent |
| RTX 3090 (24GB) | 24GB | ~10s/image | Excellent |
| Mac M3 Pro (32GB) | 32GB unified | ~25s/image | Excellent |
Common Issues and Fixes
Even experienced users run into problems with ControlNet. Here are the most frequent issues and their solutions:
| Problem | Cause | Fix |
|---|---|---|
| Output ignores reference image | ControlNet strength too low or node not connected properly | Verify the ControlNet Apply node is wired between the CLIP encode and KSampler. Set strength to 0.8+ as a starting point. |
| Output looks like a blurry copy | Strength too high combined with low CFG | Reduce ControlNet strength to 0.6-0.7 and raise CFG to 8-10 to give the text prompt more influence. |
| Black or blank output image | Mismatched model types (SD 1.5 ControlNet with SDXL checkpoint) | Ensure your ControlNet model matches your base checkpoint version. SDXL checkpoints need SDXL-compatible ControlNet models. |
| ”ControlNet model not found” error | Model file in wrong directory | Place .safetensors ControlNet files in ComfyUI/models/controlnet/, not in the checkpoints folder. |
| Canny edges too noisy or too sparse | Threshold values not tuned for your image | For photos with lots of detail, raise low_threshold to 150. For sketches or simple outlines, lower high_threshold to 150. Preview the Canny output node to check before generating. |
| Very slow generation (60s+) | Resolution mismatch or model loaded on CPU | Match EmptyLatentImage resolution to your reference image. Check ComfyUI console output to confirm the model loaded on GPU, not CPU. |
Tip: Always preview the Canny edge detection output before running the full generation. If the edges do not clearly capture the structure you want to preserve, adjust thresholds first. Generating with bad edges wastes compute time and produces unusable results.
Related resources
- 230 Workflows library — browse all workflows including 92 ComfyUI
- AI Models catalog — find Stable Diffusion and FLUX models
- Software stack — ComfyUI in our tech stack
- Contact — need help setting up ComfyUI?
Sources:
Related reading
- ComfyUI Batch Image Generation: Create 100 Product Images in Minutes
- Docebo MCP: Connect Your LMS to AI Assistants in 5 Minutes
- How to Deploy AI Locally in Your Business: Complete 2026 Guide
¿Listo para empezar?
VORLUX AI ayuda a empresas españolas y europeas a desplegar soluciones de IA que se quedan en tu hardware, bajo tu control. Ya necesites despliegue de IA en edge, integración LMS o consultoría de cumplimiento con la Ley de IA de la UE — podemos ayudarte.
Reserva una consulta gratuita para hablar de tu estrategia de IA, o explora nuestros servicios para ver cómo trabajamos.