Large language models now lead on most language tasks, but the label hides three different designs. This paper compares decoder-only, encoder-only and encoder-decoder architectures through a literature review: what each is built for, how it performs across domains, and which limits (bias, hallucination, compute cost) come with it. Code models get their own section because they show the clearest payoff.
01One label, three designs
Almost every modern language model is built on the Transformer and its attention mechanism 1. What differs is which half of it a model keeps. That choice decides what the model is good at, long before any training data is chosen.
Decoder-only models predict the next token and so lead on generative work. Encoder-only models read a whole sequence at once and excel at understanding and extracting information. Encoder-decoder models read, then write, which suits translation and summarisation 2.
02How they compare
The comparison runs across coding, reasoning and understanding, text generation and question answering, plus domain-specific models 3. Coding is scored on HumanEval pass@1, among other benchmarks 4.
03Code models
Code LLMs turn a natural-language description into source code (NL2Code) and go on to completion, translation and repair. Pre-training objectives such as span corruption and fill-in-the-middle are what make that possible 4.
04Limits and what comes next
Five limits recur: bias inherited from training data, hallucination (confident text that is false), the compute and energy cost of scale, a lack of interpretability, and security and ethical risks 5. The open directions follow from them: more efficient models, factual grounding such as retrieval-augmented generation, fairness, interpretability, multimodal and multilingual models, and agents that use tools and environments.
References
- 1. A. Vaswani et al. Attention is all you need. Advances in Neural Information Processing Systems 30, 2017.
- 2. S. Minaee et al. Large language models: a survey. 2025.
- 3. M. Shao, A. Basit, R. Karri, M. Shafique. Survey of different large language model architectures: trends, benchmarks, and challenges. IEEE Access 12, 2024.
- 4. J. Jiang et al. A survey on large language models for code generation. 2024.
- 5. R. Patil, V. Gudivada. A review of current trends, techniques, and challenges in large language models (LLMs). Applied Sciences 14(5), 2024.