ComfyUI: Bringing the Whole Midjourney Pipeline Back to Your GPU
Midjourney is great until you need an unfiltered, reproducible, scriptable image pipeline that runs offline. ComfyUI is that โ a node graph you own, on hardware you own, with no watermark and no waitlist. This is the guide to actually getting there, including the parts the splash screens skip.
Cloud image tools (Midjourney, DALLยทE, the hosted FLUX endpoints) are convenient but share a problem: your prompts, your reference images, and your outputs pass through someone else's servers. For concept art, product mockups, or anything sensitive, that is a liability. And the creative controls โ exact compositions, reused faces, batch variations, video frames โ are often gated or unavailable.
ComfyUI is the open-source answer: a node-based visual workflow editor for image and video generation that runs entirely on your hardware. 75K+ GitHub stars, an enormous community workflow library, and support for the best open models (FLUX.1, Stable Diffusion, and the open video models).
This is a deep, opinionated, ~5,000-word look at what ComfyUI is, how to install and run it, a beginner walkthrough from blank canvas to first image, prompt engineering, model management, a real character-sheet workflow you can copy, custom nodes, LoRA, upscaling, video, teams, troubleshooting, what it costs on your own GPU, how it compares to the field, and โ the part this blog lives for โ where your data goes when you use it.
1. The Local Image-Generation Dilemma
Let me be concrete about why people reach for local generation. A studio I advise produces character concept art. On Midjourney they hit three walls monthly: (a) a reference image they uploaded "for style" quietly became training fodder elsewhere; (b) they needed 400 variations of one pose for a storyboard and the waitlist + rate limits made it a two-day chore; (c) a client contract explicitly forbade sending briefs to a third-party AI, so the junior artist was hand-painting what a local pipeline could do in minutes.
None of those are "Midjourney is bad." They are "Midjourney is a service, and services have terms, limits, and a server in the middle." When your work is commercial, sensitive, or high-volume, those three facts stop being footnotes.
ComfyUI removes the server in the middle. You bring the model weights, the GPU, and the graph; it executes on your metal. That is the entire value proposition, and it is a large one for the right user.
2. What ComfyUI Actually Is
ComfyUI is a graphical workflow engine for generative models, not a model itself. You lay out nodes โ load checkpoint, encode prompt, run sampler, decode image, upscale, save โ and connect them into a graph. Hit queue, and it executes the pipeline on your GPU.
Key facts:
- Models are plug-ins. FLUX.1 (Black Forest Labs), SD1.5/SDXL, SD3, and others drop in as checkpoints. You bring the weights.
- ControlNet / LoRA / IP-Adapter supported as nodes โ precise pose, style, and character control.
- Batch processing via grid nodes or queued prompts.
- REST API for headless/server use โ script it, wire it into other tools.
- Custom nodes from a huge community registry extend everything.
License: GPL-3.0 (community). Free, local, offline-capable. The "product" is yours; you supply the model weights and the GPU.
What ComfyUI is not:
- Not a model. It ships no weights; you download FLUX/SDXL yourself.
- Not a chat box. It is a pipeline tool; pair it with Open WebUI if you want a conversational front door.
- Not beginner-proof. The node canvas is powerful but overwhelming on first launch.
3. Installation
Recommended: a portable or venv Python install, or the community Docker images.
# Clone and run (simplified)
git clone https://github.com/comfyanonymous/ComfyUI
cd ComfyUI
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt
python main.py --listen 0.0.0.0 --port 8188
Then open http://localhost:8188. Drop your model checkpoints into the models/ folder (FLUX.1, SDXL, etc.). Load a shared workflow JSON and you are generating.
For a one-command self-hosted stack, many pair it with Ollama + Open WebUI in a single Docker Compose (image gen wired into the chat UI).
GPU floor (realistic):
- SDXL / FLUX.1-dev (Q4): ~12โ24 GB VRAM.
- FLUX.1-full (non-quantized): 24 GB+ (RTX 4090 / A6000 territory).
- Video models (HunyuanVideo, CogVideoX, LTX-Video): 24 GB+ and slow; 48 GB+ for comfort.
The one mistake everyone makes: putting models in the wrong folder. ComfyUI expects checkpoints in models/checkpoints/, LoRAs in models/loras/, VAEs in models/vae/. A model in the wrong directory silently never appears in the loader. Ten minutes of "why is FLUX not showing up" is always a folder-path issue.
4. Beginner Walkthrough: Blank Canvas to First Image
The first launch is intimidating โ a blank canvas and a "Load" button. Here is the minimum path that gets a real image out, so the abstract "node editor" becomes concrete.
1. Load a workflow. Do not start from scratch. Click "Load" and pick a community FLUX workflow JSON (hundreds are shared on the ComfyUI Discord and Civitai). It auto-fetches missing nodes. You now have a working graph.
2. Select the checkpoint. In the Load Checkpoint node, pick your FLUX.1-dev Q4 from the dropdown. If it is empty, you put the model in the wrong folder (see ยง3).
3. Write a positive prompt. In the CLIP Text Encode (positive) node, type something concrete: "cinematic portrait of an old sailor, weathered face, rim light, 35mm, sharp focus."
4. Leave the negative as-is for the first run (FLUX barely uses negatives; SDXL wants them).
5. Set the KSampler. Steps 28, CFG 3.5, sampler euler, scheduler simple, seed -1 (random).
6. Queue. Click "Queue Prompt." Watch the graph light up node by node, then an image appears in the Save Image / Preview node.
That is the whole loop. The power is that this graph is now yours โ save it, reload it in six months, get the same result. That reproducibility is the thing cloud tools cannot give you without hoarding every seed by hand.
The second realization: every "button" in a cloud tool is just a preset graph like this one. Once you see that, you stop being limited by someone else's presets.
5. Prompt Engineering Inside ComfyUI
Because you control the graph, prompting here is richer than a single text box โ but the fundamentals still apply, and getting them wrong wastes generations.
Be concrete, not abstract. "A beautiful woman" is a slot machine. "A 30-year-old violinist mid-performance, sweat on brow, stage spotlight from upper left, shallow depth of field, 85mm lens" is a target. Local models reward specificity because they have no "make it pretty" safety net โ what you describe is what you get.
Use the two-text-encoder structure deliberately. The positive encoder drives what appears; for SDXL the negative encoder ("blurry, watermark, extra fingers, deformed hands") filters failure modes. FLUX responds weakly to negatives โ keep them minimal or omit. Knowing which model wants what saves you from "why is my FLUX ignoring my negative?"
Seed discipline. Fix the seed while you iterate on composition and prompt; randomize it only for final batches. A fixed seed turns prompting into a controllable experiment: change one word, see one effect. This is how you learn your model's vocabulary fast.
Chain prompts across nodes. Advanced graphs feed the output of one sampler into a second (e.g., generate โ face-detail โ upscale), each with its own prompt. You are not writing one prompt; you are writing a sequence โ base scene, then "refine the eyes," then "sharpen to 4K." That layered prompting is the local advantage cloud UIs hide from you.
6. Model Management: Where They Live and How to Organize
As soon as you generate seriously, you will accumulate dozens of checkpoints, LoRAs, VAEs, and upscalers. Chaos here costs hours. A few practices that pay off:
- Folder discipline. Checkpoints โ
models/checkpoints/, LoRAs โmodels/loras/, VAEs โmodels/vae/, upscalers โmodels/upscale_models/, ControlNet โmodels/controlnet/. The loader only looks in the right place; a misplaced file is an invisible file. - Name with provenance.
flux1-dev-Q4_K_S.ggufbeatsmodel2.gguf. When you have ten checkpoints, the name is the only thing telling them apart. - Verify checksums on big downloads. A corrupted 12 GB checkpoint fails cryptically mid-run. Hash-check before you queue.
- Where to get them. Hugging Face (FLUX.1, SDXL, video models) and Civitai (LoRAs, community checkpoints). Both are where the open ecosystem lives; both require you to read licenses โ many community LoRAs are non-commercial.
- Quantized vs full. FLUX.1-dev Q4 fits a 16 GB card; the full model wants 24 GB+. Quantization is the lever that makes local viable, at a small quality cost on the finest detail.
7. Hands-On: What Using It Feels Like
Real capabilities once you are rolling:
- Reproducible output: the graph is the recipe. Re-run months later, same result. This is impossible on Midjourney without saving every seed by hand.
- Compositional control: pose via ControlNet, style via LoRA, character consistency via IP-Adapter โ things cloud tools abstract away.
- Video: node chains that animate, interpolate, and render โ fully local.
- Scriptable: the REST API lets you generate thousands of variations from a CSV.
The honest delight: once a workflow is built, it is a product you own, not a prompt you re-type. A studio can encode its entire visual style into one graph and hand it to any artist.
8. A Real Workflow Walkthrough (Character Sheet)
Let me walk one concrete graph so the abstract "node editor" becomes concrete:
1. Load Checkpoint node โ FLUX.1-dev Q4.
2. CLIP Text Encode (positive) โ "cinematic portrait, rim light, 35mm, sharp focus."
3. CLIP Text Encode (negative) โ "blurry, watermark, extra fingers."
4. Load LoRA โ a face LoRA trained on your character (optional, for consistency).
5. ControlNet (openpose) โ a pose reference image, so the character always stands the same way.
6. KSampler โ steps 28, cfg 3.5, sampler euler, scheduler simple, seed fixed.
7. VAE Decode โ image tensor to pixels.
8. Save Image โ writes to output/.
Queue 50 prompts from a CSV (hat variations, expressions) โ 50 consistent character sheets in minutes. Swap the LoRA and the whole batch is a different character. This is the local advantage: the graph is a reusable asset, and the batch is free after the GPU is paid for.
Tuning the KSampler: steps 20โ30 is the sweet spot for FLUX; more rarely helps and costs time. CFG 3.0โ4.0 for FLUX (it is sensitive โ too high looks plastic). For SDXL, CFG 6โ8 and DPM++ 2M Karras at 25โ35 steps is the classic. Seed: fix it while iterating on composition, then randomize for final batches.
9. Custom Nodes: Extending the Engine
The base install is deliberately minimal. Almost everything useful is a custom node pack installed from the in-app Manager (or git clone into custom_nodes/). The packs that matter most:
- ComfyUI-Manager: the package installer. Install this first; everything else flows from it.
- ComfyUI_IPAdapter_plus: character/style consistency from reference images.
- ComfyUI_ControlNet_aux: pose, depth, scribble, lineart preprocessors.
- ComfyUI_VideoHelperSuite / KJNodes: video frame handling, interpolation, batching.
- Derfuu_ComfyUI_ModdedNodes: quality-of-life math and switches.
10. LoRA: Style and Character in a Small File
A LoRA (Low-Rank Adaptation) is a small weight file (tens to hundreds of MB) that nudges a base model toward a style, a character, or a concept. This is how you get "my character, every time" without retraining the whole model.
Two ways to get them:
- Download community LoRAs (Civitai, Hugging Face) and drop into
models/loras/. - Train your own on 10โ30 images of a subject. Tools like Kohya_ss or OneTrainer run locally; a character LoRA trains in an hour or two on a 24 GB GPU.
11. Upscaling and Post-Processing
Base generations are often 1024px. For print or large displays you upscale:
- Latent upscale node: doubles resolution inside the sampler (cheap, sometimes soft).
- Model-based upscale (ESRGAN/4x-UltraSharp): a separate model that sharpens real detail. Drop the upscaler into
models/upscale_models/. - Face detailer (via a detector + refiner): re-renders just the face at higher fidelity โ the difference between "AI face" and "real face" on close-ups.
12. Video: Animating the Pipeline
The open video models โ HunyuanVideo, CogVideoX, LTX-Video โ run as ComfyUI nodes. A video graph typically: loads a video model checkpoint (24 GB+ VRAM), encodes a first frame or a prompt, samples a sequence of latent frames, decodes them to an image batch, and wraps the batch into an MP4.
Realistic expectations: a 5-second 480p clip can take 10โ30 minutes on a 4090. This is not real-time. But it is local, unfiltered, and scriptable โ you can generate 200 variant clips from a CSV overnight, all on your hardware. For a studio producing social cuts or storyboards, that throughput at zero per-clip cost is the entire point.
The current ceiling: consistency across long clips is hard, and motion coherence is behind the best closed video models. But the gap is closing monthly, and "it runs on your GPU, privately" is already true today.
13. ComfyUI for Teams: Sharing Graphs, Not Just Images
The underrated production feature is that a workflow JSON is a shareable asset. A senior artist builds the "house style" graph once; the whole team loads it and only changes prompts. That is how a studio turns a tool into a pipeline.
Practices that make it work:
- Version the graphs. Commit workflow JSONs to git alongside the art repo. A broken graph is a
git revert, not a re-build from memory. - Standardize the model folder. If everyone's
models/differs, a shared graph breaks on load. A shared mount or a documented model inventory keeps graphs portable. - Pin ComfyUI and node versions across the team (see ยง9). A graph that works on one machine and fails on another is almost always a version drift.
- Expose via the REST API behind a queue. For high-volume studios, a small service accepts prompt CSVs, runs ComfyUI headless, and drops outputs to a shared folder โ no one touches the canvas.
14. Common Beginner Mistakes
The ones that cost the most time, in order:
1. Wrong model folder. 80% of "model not found" is a misplaced checkpoint. Know the models/ subfolders cold (ยง6).
2. CFG too high on FLUX. Looks plastic, over-saturated. Drop to 3.0โ4.0.
3. Hoping a 7B/16GB card runs FLUX-full. It won't, usefully. Q4 or a smaller base.
4. Randomizing the seed while learning. You cannot tell what changed. Fix it, iterate one variable at a time.
5. Unpinned ComfyUI after a working graph. Update breaks a node; the graph dies. Pin for production.
6. Stacking LoRAs at 1.0 each. They fight. Start at 0.6โ0.8 and tune.
7. Expecting cloud-level video speed. Local video is slow; plan overnight batches, not live preview.
None are model failures. All are configuration discipline โ the same pattern this whole blog repeats.
15. Performance & Cost
ComfyUI itself is light; the cost is the GPU and electricity.
| Setup | One-time | Electricity (per 1k images) | vs Midjourney |
|--------|-----------|----------------------------|---------------|
| Used RTX 4070 Ti (16GB) | ~$500 | ~$0.20 | No sub needed |
| RTX 4090 (24GB) | ~$1,600 | faster, ~$0.15 | No sub needed |
| Midjourney Pro | $0 | $60โ$120/month | baseline |
Breakeven vs a $60/mo Midjourney sub: roughly a year of heavy use on owned hardware. After that, the GPU is amortized and generation is essentially free. Local image gen is a sovereignty + cost decision โ you own the pipeline and the outputs outright.
Where local wins on cost: high-volume, repetitive generation. If you render 5,000 product variations a week, the Midjourney bill scales; your GPU stays flat. Where it loses: occasional single images โ a $30/mo sub is simpler and you skip the VRAM math.
16. Data Sovereignty: Where Your Data Goes
This is the section that justifies covering ComfyUI at all.
By default, ComfyUI is fully offline. Once model weights are local, prompting, reference images, and outputs never leave your machine. No account, no telemetry by default, no server in the middle. Your concept art, your product photos, your client briefs stay on your disk.
Caveats:
- Downloading weights needs internet once. Do it, then disconnect.
- Custom-node registries may phone home during install โ review what you pull.
- If you wire it to a cloud model (some nodes call hosted APIs), you have defeated the privacy point. Keep the backends local.
- The output files are yours. No watermark, no license clawback, no "we may use your images to improve." That clause, present in many cloud TOS, simply does not exist in a local pipeline.
17. How It Compares
| Tool | Angle | License | Local? |
|------|-------|---------|--------|
| ComfyUI | Node workflow, max control | GPL-3.0 | Yes |
| AUTOMATIC1111 | Classic SD WebUI, big plugin ecosystem | AGPL-3.0 | Yes |
| Forge | Faster, lower VRAM than A1111 | AGPL-3.0 | Yes |
| Midjourney | Hosted, curated, no control | Closed | No |
ComfyUI wins on control and reproducibility; A1111 wins on plug-and-play familiarity. Both are local; Midjourney is not. For a team that needs a reusable visual pipeline, ComfyUI's graph-as-recipe model is the differentiator.
18. Troubleshooting (the stuff nobody posts)
- "Model not in loader": wrong folder. Checkpoints โ
models/checkpoints/, LoRA โmodels/loras/, VAE โmodels/vae/. - CUDA out of memory: lower the resolution, use a quantized model (Q4), or move to a bigger GPU. FLUX-full at 24GB is the floor; 16GB needs Q4 or dev.
- Black image / NaN: bad VAE or a strength set too high. Swap the VAE, drop LoRA strength to 0.7.
- Missing custom node: open Manager โ Install Missing, or the graph came from a newer ComfyUI. Update or pin.
- First launch is blank: you loaded a workflow JSON but no checkpoint is selected in the Load Checkpoint node. Pick one.
- Slow first gen, fast after: the first run compiles/loads; subsequent runs are cached. Normal.
- Video run OOMs: video models are VRAM-hungry; lower resolution or frame count, or move to 48GB+.
19. FAQ
Is ComfyUI harder than Midjourney? Yes, to start. No, once you have one graph. The learning curve is front-loaded; the payoff is permanent.
Can I use it without a GPU? Barely. Tiny CPU models exist but are toys. A 16 GB GPU is the real floor.
Do I need to know Python? No. The UI is visual. Python only if you script the API.
Where do I get models? Hugging Face and Civitai for FLUX/SDXL/LoRA. Verify checksums on big downloads. Read the licenses โ many LoRAs are non-commercial.
Can it do video? Yes โ HunyuanVideo, CogVideoX, LTX-Video as nodes. Slow, but local.
Will my outputs have a watermark? No. You own them.
Can my team share one graph? Yes โ that is the production model. Version the JSON, standardize the model folder, pin versions (ยง13).
20. Who Should (and Shouldn't) Use It
Use ComfyUI if: you want reproducible, scriptable, unfiltered image/video pipelines; you own a capable GPU; you care that prompts and outputs never leave your machine; you generate commercially or at volume.
Skip it if: you want a one-line "make me an image" box (use a hosted tool or Forge); you lack a real GPU; you are allergic to node graphs; your volume is low and a $30/mo sub is simpler.
21. Wiring It Into the Rest of Your Stack
ComfyUI is not an island. The productive setups combine it:
- Open WebUI + ComfyUI: expose the ComfyUI REST API to Open WebUI so non-technical users get "generate an image" as a chat action, backed by your local GPU.
- n8n + ComfyUI: a workflow that watches a folder, generates variations, and posts them to a CMS โ fully automated, no cloud.
- Ollama + ComfyUI: use a local LLM to write the prompts, ComfyUI to render them. The whole creative loop stays on your hardware.
22. Conditioning Deep-Dive: How Prompts Actually Control Pixels
To use ComfyUI well, it helps to know what the prompt nodes are doing. The CLIP Text Encode node does not "describe the image to the model" in English โ it encodes your text into the latent space the sampler operates in. Two encoders, two jobs:
- Positive encoder: maps "cinematic portrait, rim light" into a direction in latent space the model pulls the generation toward.
- Negative encoder: maps "blurry, watermark, extra fingers" into a direction the sampler is pushed away from.
A second insight: the prompt is not the only conditioning. The Load Checkpoint node sets the model's base distribution, the LoRA node nudges it, the ControlNet node injects a structural constraint (pose/depth), and the IP-Adapter node injects a style/identity. The final image is the intersection of all of these conditionings. When output "ignores" your prompt, it is usually being over-ruled by a stronger conditioner โ a ControlNet pose the prompt cannot contradict, or a LoRA trained to always look a certain way. Debug conditioning left to right: checkpoint โ LoRA โ ControlNet โ prompt.
23. A Second Workflow: Product Mockup From a Brief
The character-sheet workflow (ยง8) is the classic. Here is a different one that shows the graph's commercial value: turning a text brief into a product mockup batch.
1. Load Checkpoint โ SDXL base (Q4).
2. CLIP positive โ "product photo, matte black wireless earbuds, studio lighting, floating on white, soft shadow, 4k product render."
3. CLIP negative โ "blurry, lowres, watermark, hands, text."
4. ControlNet (canny) โ a rough silhouette sketch you drew in two minutes; the model fills it with photoreal detail while keeping your composition.
5. KSampler โ DPM++ 2M Karras, 30 steps, cfg 7, fixed seed.
6. Background swap node โ replaces the grey with pure white.
7. Save Image โ output/earbuds_v1.png.
Queue 30 seeds โ 30 composition-locked variations. Hand the client 6, iterate on the one they like. The local advantage again: the canny sketch is your IP, the generations are yours, and the 30 variations cost you watts, not a per-image subscription fee. A studio can template this graph and reuse it for every SKU.
24. The REST API: Scripting Thousands of Images
The UI is the on-ramp; the REST API is the product. Once a graph works, you can drive it headless from any script โ and that is where local generation stops being a hobby and becomes infrastructure.
The shape. ComfyUI exposes a /prompt endpoint: POST a workflow (with the prompt text swapped for a CSV row), and it queues a generation. A small Python loop reads 500 product SKUs from a spreadsheet, fills the prompt template, posts each, and collects the outputs to a folder. Overnight, you have 500 on-brand mockups. The marginal cost is watts.
Why local wins here specifically. A cloud service meters per image โ 500 variations is 500 billable events, and a "generate me 500" job is exactly the kind of volume that makes a subscription sting. Your GPU does not care if it renders 5 or 5,000; the curve is flat. This is the same volume argument as the LLM case (ยง15): local gen is a sovereignty and cost win precisely when volume is high.
A real example. A furniture studio needs every product on a consistent white background with the same lighting. They build one graph (SDXL + canny control from a hand-drawn silhouette + a background-swap node), then feed it 300 SKUs. Each output is composition-locked to the brand template. A human art director reviews a contact sheet and flags 12 to regenerate with a tweak. What used to be a week of freelance is an overnight batch plus a morning of review.
Failure handling at volume. At 500 generations, some will fail โ OOM on one oversized input, a NaN from a bad seed, a node timeout. Script defensively: wrap each POST in try/except, log failures to a retry queue, and cap concurrency so you do not melt the GPU. The batch succeeds as "480 good + 20 to re-run," not "all or nothing." This is the operational maturity that separates a pipeline from a demo โ and it only matters once you stop clicking and start scripting, which is exactly the moment ComfyUI pays for itself.
The caveat. Scripting means you are now an operator, not a clicker โ you need the workflow JSON as a file, a tiny script, and the discipline to version both. But that is the whole point of ComfyUI: the graph is an asset you can automate, not a button you must press. For anyone generating at volume, the API is not an advanced feature; it is the reason to use ComfyUI over a click-only tool.
25. Verdict
ComfyUI is the local image-and-video toolkit with no ceiling and no leash. It is harder than a chat box, demands a GPU, and makes you manage your own models โ but in exchange you get the entire Midjourney-class pipeline, running offline, on hardware you own, with outputs you fully control.
The limitations are configuration discipline, not capability gaps. None of them compromise the core promise: your prompts and your images stay on your hardware. For the right user โ commercial, sensitive, or high-volume work โ that promise is worth the learning curve.
Related:
- Ollama: Run LLMs on Your Own Machine โ the text-model backend to pair with it
- Open WebUI: The Self-Hosted ChatGPT Frontend โ wire image gen into a chat UI
- DeepSeek V4: The MIT-Licensed Model That Runs on Consumer Hardware โ the LLM to write your prompts
Comments (0)
No comments yet. Be the first to comment!