Image Generation
Turn a sentence into a picture nobody has ever taken.
Image Generation · Text-to-Image
- What it is—what AI "drawing" actually does: is it patching together existing images?
- Two core things—what needs to be solved in the middle to turn a sentence into an image.
- What the engine is—what specifically draws the image.
- How to make it more controllable—a single sentence is too coarse; how to control it precisely.
- What it still can't do well—what longstanding difficulties current image generation has.
1What is image generationIntuition
This section answers: When AI "draws" a picture that nobody has photographed, is it pieced together from an image library?
Image generation can be understood as "creating image candidates based on conditions": It solves the problem of "having no ready-made image but wanting a new picture that matches a description", rather than finding the most similar image from an image library.
Inputcan be a piece of text, and may also include a reference image, sketch, or pose;Outputis one or more pixel images. For example, with "a corgi wearing a hat", the system first converts the input conditions into an internal representation, then samples a new representation based on the image patterns learned during training, and finally decodes it into a visible picture. Heresamplingis selecting one result from many possible images, so running the same sentence multiple times can produce different outputs.
Boundary:The output is usually not an image that exists exactly as is in the training set, but the model may still memorize and reproduce training fragments; therefore, "generating a new image" does not prove that it is absolutely original, factually correct, or can be used without restrictions.
2It Must Do Two Things at OnceEngineering
“One sentence → one image,” there are actually two hurdles to cross in between.
Two-stage generation describes the complete process from “understanding the request” to “forming the image.”, to solve the problem that text meaning and image pixels cannot correspond directly one-to-one. Itsinputis the prompt and optional control conditions,intermediate outputis a machine-usable conditioning representation,final outputis the image candidate.
| What to do | What It Solves | What It Relies On |
|---|---|---|
| ① Understand the text | Understand “a corgi wearing a hat, oil painting, seaside” in image-level terms. | text understanding + image-text alignment (see “Embedding,” “CLIP,” “Multimodal Models”) |
| ② Draw the image | Generate in pixel, latent variable, or visual token space, then decode into a coherent new image. | Diffusion, autoregressive, and other generative engines |
How it works:The system first encodes “corgi, hat, seaside, oil painting” into a conditioning representation, then has the generative engine form an image representation according to these conditions, then decodes it into an image, and finally verifies objects, relationships, and style item by item.Image-text alignmentis to make text and corresponding images close in internal representation;visual tokenare the small discrete units used when the model processes images; you can temporarily think of them as “image-version word blocks.”
Scroll horizontally to view the full diagram on small screens.
| Running example: fixed 4 verification items | Sample 1 | Sample 2 |
|---|---|---|
| Corgi subject | ✓ | ✓ |
| Hat worn on head | ✗ (floating) | ✓ |
| Seaside scene | ✓ | ✓ |
| Oil painting style | ✓ | △ (leans toward photo) |
Results and Boundaries:A candidate that overall is very similar to the prompt only indicates that the general direction may be aligned; it does not prove that every object, count, and spatial relationship is correct. This “two-stage” explanation is suitable for building task intuition, but different models may combine encoding, generation, and decoding into different structures; you cannot treat the boxes in the diagram as fixed components that all systems must use.
3Engine: Today It's Mainly DiffusionIntuition
What exactly is used to draw the picture?
Diffusion models refer to engines that generate data through repeated denoising, it addresses the question, “How do you get from a random starting point to an image that is both natural and matches the text requirements?” Here,noisecan be thought of as random snow with no recognizable objects;conditionis the text or control image used to constrain the direction of generation.
Inputis a blob of random noise, the current denoising step, and the text condition,The output of each stepis a slightly more structured intermediate image; only the final output is a complete picture. The system first predicts which changes in the current noise look more like the target picture, then removes some of the noise, and then feeds the result into the next step; after repeating this many times, the structure of “corgi, hat, seaside” gradually emerges.
Boundary:Increasing the number of denoising steps does not necessarily keep improving quality, and it cannot guarantee correct counts, text, or facts. Diffusion is only the current mainstream route, not equivalent to image generation itself;Autoregressiveinstead predicts the next visual unit in sequence, forming the image block by block like writing a sentence word by word; the two routes each involve trade-offs in speed and control.
4Complete Example: How to Make It More ObedientCase Walkthrough
Only giving one sentence makes the control too coarse — what if you want precise composition, poses, or local modifications?
What controllable generation does is add structural constraints to the text-to-image generation process, to solve the problem that “the text is correct, but positions, poses, or local details are still uncontrollable.” Itsinputis to add, in addition to the prompt, a pose skeleton, depth map, line art, or local mask,outputis an image that both satisfies the textual semantics and complies with these constraints as much as possible.
- Controllable Generation: in addition to text, addextra conditions—a line draft, a human pose, a depth map, to make the generation strictly follow it (e.g., ControlNet); or only redraw in the imagea specified region(local inpainting), keeping the rest unchanged (see “Controllable Generation”).
- Image Editing: not generation from scratch, butmodify an existing image according to instructions(changing the background, removing an object, expanding the image), the workhorse in practical scenarios (see “Image Editing”).
| Step | Input | Output to inspect at this step |
|---|---|---|
| 1 Determine the goal | “A person in a red jacket stands on the left side of a bridge, with a rainy city night as the background” | The subject, position, clothing, and scene are broken down into checkable conditions |
| 2 Constrain the composition | A pose skeleton of the person and a depth map of the bridge deck | The person's position and perspective follow the control maps |
| 3 Generate candidates | Fix 4 random seeds under the same conditions | Check each for prompt alignment, naturalness, and diversity among candidates |
| 4 Local revision | Mask only the erroneous hand region | The hand is improved, and the face, clothing, and bridge deck are not accidentally altered |
How it works:First break the vague wish into checkable conditions, then constrain the overall composition with structural maps, generate multiple candidates, and finally mask only the erroneous region for local repainting.random seedis a number that generates the random starting point; fixing it helps compare differences before and after changes, but this number itself does not indicate quality.
How to interpret the results:If only a few candidates fail, it indicates fluctuation caused by random sampling; if all four candidates fail in the same location, it is more likely that the conditions are unclear, the constraints conflict, or the model has not learned this combination well. You should re-examine the input instead of continuing to “draw cards.”
Boundary:More control conditions are not necessarily better. If a pose map requires the person to stand on the left while the text requires the person to be on the right, the system cannot fully obey both at the same time; local repainting may also incidentally change adjacent areas, so you must clarify priorities and recheck the unmasked parts.
5What It Still Doesn't Do WellEngineering
Capabilities change rapidly with models; the list below is not a permanent defect list, but high-risk dimensions that should be continuously evaluated during deployment.
Capability-boundary evaluation records model performance across different failure types, which addresses the problem of “demo images look great, but you don’t know where the system will fail in real tasks.” Itsinputis a set of test prompts covering text, counting, spatial relationships, consistency, and factuality, along with the specified number of samples for each prompt;outputare the success rates, failure rates, and representative examples for each error type.
- Text and layout in images: New models have significantly improved short text, but long text, multiple languages, precise layout, and editable fonts still need to be evaluated model by model.
- Composition and complex structure: Hands are just one example; more generally, counting, spatial relationships, occlusion, symmetry, and binding multiple object attributes can go wrong.
- Consistency: makingthe same characterlook consistent across multiple images is difficult.
- Factuality: what it generates is a “looks right” image, not a “really accurate” image—and large modelhallucinationis of the same source. Especially verify when requiring scientifically accurate diagrams (see “Hallucination”).
How it works:First, design a fixed set of prompts for each high-risk dimension, then have the same model repeatedly generate at a uniform resolution and sampling budget, then record error types according to clear annotation rules, and finally compare across dimensions and versions. Seeing one successful image only shows that that particular sample succeeded; only when a certain type of prompt still fails frequently across many generations does it indicate a reproducible capability weakness.
6How to Evaluate: Good Looks, Prompt Alignment, and Safety Are Not One ScoreEngineering
What does “this image is high quality” really mean?
Multi-dimensional evaluation compares different aspects of “quality” rather than seeking a single universal score, used to prevent a good-looking image from masking prompt-alignment errors, identity drift, or safety risks. Itsinputis a fixed prompt set, model version, random seed strategy, sampling budget, and human scoring rules;outputis not a single universal score, but per-dimension scores, failure types, and high-risk cases.
At minimum, evaluate these separately:perceptual qualityis whether the image looks natural,prompt alignmentis whether objects, relationships, and text meet the requirements,diversityis whether candidates are merely near-copies,identity consistencyis whether the same character maintains features across images,factuality and safetychecks whether the content is credible and compliant.human blind evaluationis having raters not know which model an image came from, reducing brand and order bias;safety red teamingis proactively designing high-risk requests to find system vulnerabilities.
How it works:First fix the prompts, resolution, and number of samples; then generate repeatedly for each prompt; then let automated metrics perform batch screening and have humans blind-evaluate each dimension; finally, separately aggregate failure types and safety issues.
How to interpret and applicable boundaries:A higher score on a dimension only means that dimension is better under this set of test conditions; it does not automatically imply the whole is better. Automated metrics struggle to cover composition preferences and high-risk semantics, and human scoring is also affected by culture, task, and order; safety failures cannot be averaged away by high scores on other dimensions.
7The New Problems It BringsSafety
Being able to conjure lifelike images from thin air also brings problems beyond pure technology.
Governance of generated images can be understood as risk control that runs throughout before generation, during generation, and after publication, used to reduce copyright infringement, impersonation of real people, fraud, and false dissemination. Its input includes user requests, the people and materials involved, authorization status, and usage scenarios; output includes decisions to allow, restrict, or reject, as well as labeling of generated content, provenance records, and human review results.
- Copyright disputes: The training data contains a large number of copyright-protected works, and the boundary of “learning someone else’s painting style and then generating” is still under debate.
- Deepfakes: can generate highly realistic images of people, used for spreading rumors, fraud, and infringement.
- Detection and watermarking: This has given rise to technologies for “determining whether an image is AI-generated”, and these technologies are in an arms race with generative capabilities (see “AI-generated Content Detection and Watermarking”).
How it works: Before generation, it checks identity, authorization, and purpose; during generation, it restricts clearly high-risk requests; after generation, it adds provenance information, retains audit records, and conducts human review of sensitive content.Watermarking is the addition of provenance clues to the file or image; the detector estimates whether an image is AI-generated based on statistical features; the two serve different purposes.
8Connecting the Entire Causal ChainSynthesis
From text semantics, condition injection, image sampling, to item-by-item acceptance, break “looks good” into checkable mechanisms.
- Image generation creates a brand-new image, not a retrieval-based collage.(§1)
- It must do two things simultaneously: understand the text (image-text alignment) + draw the picture (generation engine).(§2)
- Today's engine is mainly diffusion: denoising generation, with text injected as conditioning.(§3)
- Controllable generation and image editing provide finer-grained steering.(§4)
- Weaknesses: text in images, fingers, consistency, factuality (same root as hallucination).(§5)
- Evaluation must break apart aesthetics, prompt alignment, consistency, factuality, and safety; it cannot report only a single total score.(§6)
- It also brings problems that go beyond technology: copyright, deepfakes, detection and watermarking.(§7)
9Common MisconceptionsIntuition
| Misconception | More Accurate Understanding |
|---|---|
| AI drawing is collage from an image library | It usually samples image representations from a learned distribution, not simple retrieval and collage; but the model may memorize and reproduce training snippets. |
| Image generation = diffusion models | Image generation is the task; diffusion is today's mainstream engine (along with GANs, autoregressive models, etc.). |
| The images it generates are accurate | It is 'looks right'; it may not match the facts and is of the same origin as hallucination. |
| Writing a good prompt can make correct text appear in the image | Writing accurate text in images remains a long-standing challenge; it requires dedicated capabilities/tools. |
| Generation capability is only a technical issue | It also involves ethical and legal issues such as copyright, deepfakes, detection and watermarking. |
10Check whether you really understandSelf-test
- Is AI image generation just retrieving and collaging existing images? What does it actually do?
- What two things must be accomplished simultaneously to turn “one sentence into an image”? What does each rely on?
- What is the relationship between “image generation” and “diffusion models”?
- What methods are available if you want to precisely control composition/pose or only modify a local area?
- What stable weaknesses does image generation currently have? Why is the “factuality” problem said to have the same source as hallucination?
- What problems beyond technology has it brought?
Reference Answers
- It is not simple collage; it samples and decodes in pixel, latent variable, or visual token space, but still needs to guard against memorization and reproduction of training samples.
- Reading the text (text understanding plus image-text alignment) and creating the image (generation engine); these respectively rely on CLIP-style alignment and diffusion and other engines.
- Image generation is the task; diffusion is currently the most mainstream engine for implementing it. In addition, there are routes such as GANs and autoregressive models.
- Controllable generation (adding extra conditions such as line art/pose/depth, as well as local inpainting) and image editing (modifying an existing image according to instructions).
- Writing text in images, fingers and complex structures, character consistency across multiple images, factuality; factuality is because it generates “looks right” rather than “truly accurate”, sharing the same origin as large model hallucination.
- Training data copyright disputes, deepfakes, and the AI-generated content detection and watermarking that have arisen as a result.
11Concept Dependencies and Further LearningRoadmap
| Learning Level | Concepts Covered |
|---|---|
| Prerequisite | Diffusion Models, Embedding, CLIP, Multimodal Models |
| Core of This Page | Task vs Engine, Text Understanding + Alignment, Weaknesses (text rendering/consistency/factuality) |
| Immediate Extensions | Controllable Generation, Image Editing, Image Super-resolution, Generative Adversarial Network (GAN) |
| Further | Video Generation, AI-generated Content Detection and Watermarking, Hallucination, AI Governance |
- Rombach et al., Latent Diffusion Models: latent-space text-to-image generation and conditional cross-attention.
- Saharia et al., Imagen: text encoding, cascaded diffusion, and image-text alignment.
- Ramesh et al., Hierarchical Text-Conditional Image Generation with CLIP Latents: CLIP latents and diffusion decoding.