Interactive exploration of transformer model architecture and data flow
Used for sequence-to-sequence tasks (T5)
• Converts discrete tokens to dense vector representations
• Learned during training to capture semantic meaning
• Typically 512 or 768 dimensional vectors
• Foundation for all subsequent processing
Input tokens are converted to embeddings
Positional encoding is added to preserve sequence order
Multi-head attention computes relationships between tokens
Residual connection and layer normalization
Feed-forward network processes each position
Another residual connection and normalization
Output probabilities are generated
Used for understanding tasks (BERT)
Used for generation tasks (GPT)
Used for sequence-to-sequence tasks (T5)
Self-attention mechanism allows parallel processing unlike RNNs
Residual connections help with gradient flow in deep networks
Layer normalization stabilizes training dynamics
Positional encoding enables understanding of sequence order