vLLM with the Transformers Modelling Backend

vLLM with the Transformers Modelling Backend

• vLLM with the Transformers Modelling Backend
• Harry Mellor
• HuggingFace
• tutorial

NoteNotes

• Tools:
  • Transformers
  • torchtitan
  • Axolotl
  • TRL
  • Unsloth

1. Transformers - covers transformes serve, from_pretrained and generate_batch
2. vLLM - covers llm serve and the LLM class.
3. transformers backend how vLLm can run a transformers model implementation without reimplementing it from scratch.
   • c.f. Building a compatible model backend for inference
4. Bring your own transformers model to vLLM
   • When making a model compatible with the Transformers backend, watch out for:
     • Missing kwargs at any level** — The most common issue. If OlmoeModel.forward accepted **kwargs but OlmoeDecoderLayer.forward didn’t, attention_instances would be silently dropped.
     • Custom attention not using ALL_ATTENTION_FUNCTIONS — Models that compute attention inline can’t be dispatched to vLLM’s kernels. The model must use the standard dispatch pattern.
     • Incorrect TP plans — Misspecifying “colwise” vs “rowwise” will produce wrong results silently. Remember: projections that increase dimension (Q, K, V, gate, up) are typically “colwise”, and projections that decrease dimension (O, down) are “rowwise”.
     • Non-standard attention mask handling — Models that manipulate attention weights directly (e.g., adding positional bias to attention scores after softmax) may not be expressible through the standard attention interface.