Large Language Models (LLMs) — Complete Guide

What are Large Language Models?

Large Language Models (LLMs) are AI systems trained on massive text datasets to understand and generate human language. They are the technology behind ChatGPT, Claude, Gemini, and other modern AI assistants.

How LLMs Work

Tokenization

LLMs don’t process raw text — they break it into tokens (words or sub-words):

# Conceptual tokenization
def simple_tokenizer(text):
    """Demonstrates how tokenizers split text"""
    # Simplified word-piece tokenization
    vocab = {
        'the': 0, 'a': 1, 'cat': 2, 'sat': 3, 'on': 4,
        'mat': 5, '##s': 6, '##ing': 7, 'un': 8, '##happy': 9,
        '[PAD]': 10, '[UNK]': 11, '[CLS]': 12, '[SEP]': 13,
    }

    tokens = text.lower().split()
    token_ids = [vocab.get(t, vocab['[UNK]']) for t in tokens]

    print(f"Text: '{text}'")
    print(f"Tokens: {tokens}")
    print(f"Token IDs: {token_ids}")
    return token_ids

simple_tokenizer("The cat sat on the mat")
simple_tokenizer("Unhappy cats")

The Transformer Architecture

LLMs use the Transformer architecture, which processes all tokens simultaneously:

import torch
import torch.nn as nn

class SimplifiedLLM(nn.Module):
    """Simplified LLM architecture for understanding"""

    def __init__(self, vocab_size=50000, d_model=512,
                 nhead=8, num_layers=6, max_seq_len=2048):
        super().__init__()

        # Token embedding
        self.token_embedding = nn.Embedding(vocab_size, d_model)
        # Positional encoding
        self.pos_embedding = nn.Embedding(max_seq_len, d_model)

        # Transformer layers
        encoder_layer = nn.TransformerEncoderLayer(
            d_model=d_model, nhead=nhead,
            dim_feedforward=d_model * 4,
            batch_first=True
        )
        self.transformer = nn.TransformerEncoder(
            encoder_layer, num_layers=num_layers
        )

        # Output projection to vocabulary
        self.output_head = nn.Linear(d_model, vocab_size)

    def forward(self, input_ids):
        seq_len = input_ids.size(1)
        positions = torch.arange(seq_len, device=input_ids.device).unsqueeze(0)

        # Embed tokens + positions
        x = self.token_embedding(input_ids) + self.pos_embedding(positions)

        # Process through transformer layers
        x = self.transformer(x)

        # Predict next token for each position
        logits = self.output_head(x)
        return logits

# Create a small LLM (real ones are 1000x larger)
model = SimplifiedLLM()
total_params = sum(p.numel() for p in model.parameters())
print(f"Model parameters: {total_params:,}")
print(f"Model size: ~{total_params * 4 / 1e6:.1f} MB (float32)")

Next Token Prediction

The fundamental task of LLMs is predicting the next token:

def next_token_prediction_concept():
    """How LLMs generate text one token at a time"""

    # Training: Given "The cat sat on the", predict "mat"
    # Input:  [The, cat, sat, on, the]
    # Target: [cat, sat, on, the, mat]  (shifted)

    # Generation: Autoregressive sampling
    prompt = "The future of AI is"

    # Step 1: Model sees "The future of AI is"
    # Step 2: Predicts probability distribution for next token
    #         {"bright": 0.3, "uncertain": 0.25, "exciting": 0.2, ...}
    # Step 3: Sample "bright" → sequence becomes "The future of AI is bright"
    # Step 4: Model sees "The future of AI is bright"
    # Step 5: Predicts next → "and promising"
    # Step 6: Repeat until stop token

    print("LLM Generation Process:")
    print("Step 1: Tokenize prompt")
    print("Step 2: Forward pass → next token probabilities")
    print("Step 3: Sample from distribution (with temperature)")
    print("Step 4: Append token to sequence")
    print("Step 5: Repeat from Step 2 until stop token")

next_token_prediction_concept()

Temperature and Sampling

import numpy as np

def softmax_with_temperature(logits, temperature=1.0):
    """How temperature affects token selection"""
    scaled_logits = np.array(logits) / temperature
    exp_logits = np.exp(scaled_logits - np.max(scaled_logits))
    return exp_logits / exp_logits.sum()

# Example: Model output logits for next token
logits = [2.0, 1.5, 0.8, 0.3, 0.1]
tokens = ["AI", "future", "technology", "world", "today"]

print("Temperature effects on text generation:")
for temp in [0.1, 0.7, 1.0, 1.5, 2.0]:
    probs = softmax_with_temperature(logits, temp)
    top_idx = np.argmax(probs)
    print(f"  T={temp:.1f}: {tokens[top_idx]} ({probs[top_idx]:.1%}) — "
          f"{'Focused' if temp < 0.8 else 'Balanced' if temp < 1.2 else 'Creative'}")

Major LLM Families

GPT Series (OpenAI)

# Using OpenAI's GPT API
from openai import OpenAI

client = OpenAI(api_key="your-api-key")

response = client.chat.completions.create(
    model="gpt-4",
    messages=[
        {"role": "system", "content": "You are a helpful coding assistant."},
        {"role": "user", "content": "Explain what a binary search tree is."}
    ],
    temperature=0.7,
    max_tokens=500
)

print(response.choices[0].message.content)

Claude (Anthropic)

# Using Anthropic's Claude API
import anthropic

client = anthropic.Anthropic(api_key="your-api-key")

message = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=500,
    messages=[
        {"role": "user", "content": "Explain recursion with a Python example."}
    ]
)

print(message.content[0].text)

Open-Source LLMs (Llama, Mistral)

# Using open-source LLMs with HuggingFace
from transformers import AutoTokenizer, AutoModelForCausalLM

def load_open_source_llm(model_name="meta-llama/Llama-2-7b-chat-hf"):
    """Load and use an open-source LLM"""
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    model = AutoModelForCausalLM.from_pretrained(
        model_name,
        torch_dtype="auto",
        device_map="auto"
    )

    # Format prompt for Llama
    prompt = """<s>[INST] What is the difference between a stack and a queue? [/INST]"""

    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    outputs = model.generate(**inputs, max_new_tokens=256)
    response = tokenizer.decode(outputs[0], skip_special_tokens=True)
    print(response)

# load_open_source_llm()  # Uncomment with proper model access

Comparing LLMs

Model	Creator	Parameters	Key Strength
GPT-4	OpenAI	~1.8T (est.)	Reasoning, code
Claude 3.5	Anthropic	Not disclosed	Safety, long context
Gemini 1.5	Google	Not disclosed	Multimodal, long context
Llama 3	Meta	8B-405B	Open source
Mistral Large	Mistral	Not disclosed	European, efficient
Qwen 2.5	Alibaba	0.5B-72B	Multilingual

Fine-Tuning LLMs

# Fine-tuning a small LLM with LoRA (Low-Rank Adaptation)
from peft import LoraConfig, get_peft_model, TaskType
from transformers import AutoModelForCausalLM

def setup_lora_finetuning():
    """Configure LoRA for efficient LLM fine-tuning"""

    # Load base model
    base_model = AutoModelForCausalLM.from_pretrained(
        "meta-llama/Llama-2-7b-hf",
        torch_dtype="auto"
    )

    # LoRA configuration — only trains a small number of parameters
    lora_config = LoraConfig(
        task_type=TaskType.CAUSAL_LM,
        r=16,                      # Rank of update matrices
        lora_alpha=32,             # Scaling factor
        lora_dropout=0.1,
        target_modules=["q_proj", "v_proj"],  # Which layers to adapt
    )

    # Apply LoRA
    model = get_peft_model(base_model, lora_config)
    trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
    total_params = sum(p.numel() for p in model.parameters())

    print(f"Trainable parameters: {trainable_params:,}")
    print(f"Total parameters: {total_params:,}")
    print(f"Trainable percentage: {100 * trainable_params / total_params:.2f}%")

# setup_lora_finetuning()

LLM Evaluation

# Key metrics for evaluating LLMs
evaluation_metrics = {
    "Perplexity": {
        "description": "How well the model predicts text (lower is better)",
        "range": "1.0 (perfect) → 100+ (poor)",
    },
    "BLEU Score": {
        "description": "N-gram overlap with reference text (for translation/summarization)",
        "range": "0-100",
    },
    "Human Evaluation": {
        "description": "Human judges rate output quality, helpfulness, and safety",
        "range": "1-5 rating scale",
    },
    "Benchmark Scores": {
        "description": "Performance on standardized tests (MMLU, HumanEval, etc.)",
        "range": "0-100%",
    },
}

for metric, info in evaluation_metrics.items():
    print(f"\n{metric}:")
    print(f"  {info['description']}")
    print(f"  Range: {info['range']}")

Frequently Asked Questions

What is the difference between GPT and Claude?

GPT (by OpenAI) excels at coding and complex reasoning tasks. Claude (by Anthropic) focuses on safety and handles very long contexts well. Both are excellent for general conversation and analysis.

Can I run LLMs locally?

Yes. Smaller models like Llama 3 (8B) and Mistral (7B) can run on consumer GPUs with 8GB+ VRAM. Tools like Ollama make local deployment simple.

How do LLMs handle different languages?

LLMs are trained on multilingual data and can understand many languages. Performance varies — they typically work best in English but are increasingly capable in other languages.

What is RAG (Retrieval-Augmented Generation)?

RAG enhances LLMs by retrieving relevant documents before generating a response. This grounds the model’s answers in real data, reducing hallucinations and enabling access to up-to-date information.

---

Want to learn more about LLMs? Join our Telegram Community for LLM tutorials, prompt engineering tips, and discussions about the latest AI models.