The generate() method in Hugging Face Transformers is the primary interface for text generation with large language models, but its behavior depends heavily on the decoding strategy and configuration parameters you choose. The default greedy search works for short, deterministic tasks but often repeats itself on longer outputs, while sampling and beam search offer alternatives for creative or input-grounded generation respectively.
Understanding the Three Core Decoding Strategies
Transformers provides three basic decoding methods that serve as starting points for most text generation tasks. Greedy search selects the single most probable token at each step and is the default strategy unless a model's GenerationConfig specifies otherwise. It works well for tasks like transcription or translation where creativity is not desired, but it tends to produce repetitive loops when generating longer sequences. Sampling, enabled with do_sample=True and num_beams=1, randomly selects tokens based on the full probability distribution, giving every non-zero probability token a chance to appear. This reduces repetition and creates more diverse, creative outputs suitable for story writing or chat applications. Beam search maintains multiple candidate sequences (beams) simultaneously and selects the one with the highest overall probability after several steps, allowing it to "look ahead" past locally optimal choices. It excels at input-grounded tasks such as image captioning or speech recognition where fidelity to the prompt matters more than novelty.
Configuring Generation with GenerationConfig
All generation settings are encapsulated in a GenerationConfig object that can be inspected via model.generation_config. You can override parameters directly in generate() calls or create reusable configurations for different tasks. Key parameters include max_new_tokens to control output length (highly recommended since the default is only 20 tokens), temperature to adjust unpredictability (higher values above 0.8 for creativity, lower below 0.4 for focused reasoning, requires do_sample=True), repetition_penalty to discourage loops (values above 1.0), and eos_token_id to define custom stop tokens. Configurations can be saved with save_pretrained() and loaded separately for distinct use cases like creative writing with sampling versus summarization with beam search, enabling a single model to serve multiple generation profiles.
Common Pitfalls That Degrade Output Quality
Several subtle issues frequently undermine generation results. The default output length of 20 tokens often truncates responses prematurely, so explicitly setting max_new_tokens is essential. Decoder-only models return the input prompt concatenated with generated tokens, meaning you must slice the output if only new text is needed. Padding side matters critically: LLMs are not trained to continue from padding tokens, so tokenizer.padding_side must be set to "left" when batching inputs of different lengths; right padding causes the model to generate from padding tokens, producing garbage like repeated numbers. Prompt format is equally important for chat models, which expect structured conversation templates with roles and content rather than raw strings; passing a plain prompt to a chat-tuned model yields suboptimal responses.
Extending Generation with Custom Methods
Transformers supports custom generation methods that can implement specialized behaviors like conditional reasoning, rollback on uncertainty, or custom KV cache handling. These are distributed as model repositories with a custom_generate tag containing a generate.py file, optional requirements.txt, and a README documenting input/output differences. When loading such a repository with trust_remote_code=True, its generate() method overrides the default. Crucially, the custom_generate parameter lets you apply any published custom method to any compatible model without installing extra packages, enabling community sharing of decoding innovations. The custom generate function receives the model instance and all generation arguments, allowing it to reuse Transformers' built-in input preparation (batching, attention masks, logits processors, stopping criteria) while replacing only the decoding loop.
Practical Selection Guide for Developers
For deterministic, factual tasks like code completion or data extraction, use greedy search with low max_new_tokens and no sampling. For creative writing, roleplay, or brainstorming, enable do_sample=True with temperature around 0.7-0.9 and consider top-k or top-p filtering to maintain coherence. For translation, summarization, or captioning where accuracy to the input is paramount, use beam search with num_beams=4-6 and early_stopping=True. Always set max_new_tokens explicitly, configure left padding for batching, and apply chat templates for instruction-tuned models. Start with the basic strategies before exploring custom methods, and test configurations on representative prompts since benchmark scores don't guarantee real-world reliability for your specific domain.