[Roadmap] LMFlow Roadmap

[wheresmyhair]

This document includes the features in LMFlow's roadmap. We welcome any discuss or contribute to the specific features at related Issues/PRs. 🤗

Main Features

• Data
  • DPO dataset format [Feature] reward model inferencer and dpov2 aligner #867
  • Conversation template in DPO [Feature] Iterative DPO #883
  • jinja template
  • Tools in conversation dataset function-call-finetune #884 Modify the formatter of function and observation #892
  • Packing with block diagonal attention
• Model
  • Backend
    • 🏗️ Accelerate support
  • Tokenization
    • Tokenization update, using hf method
• Pipeline
  • Train/Finetune/Align
    • DPO (multi-gpu) [Feature] reward model inferencer and dpov2 aligner #867
    • Iterative DPO [Feature] reward model inferencer and dpov2 aligner #867 [Feature] Iterative DPO #883
    • PPO
    • LISA (multi-gpu, qwen2, chatglm) [Feature] LISA multi GPU support #899
    • Batch size and learning rate recommendation (arxiv)
    • No trainer version pipelines, allowing users to customize/modify based on their needs
    • Sparse training for moe models [New Feature] Is Mixtral supported? #879
  • Inference
    • vllm inference [Feature] vllm inferencer and memory safe vllm inferencer #860 [Feature] Add vllm inference example #863
    • Reward model scoring [Feature] reward model inferencer and dpov2 aligner #867
    • Multiple instances inference (vllm, rm, others) [Feature] Iterative DPO #883
    • Inference checkpointing and resume from checkpoints
    • Inference accelerate EAGLE
    • Inferencer for chat/instruction models, and chatbot.py upgrade Conversation_template #917

Usability

• Make some packages/functions (gradio, vllm, ray, etc.) optional, add conditional import. [usability] deps streamlining #905
• Inference method auto-downgrading (vllm>ds, etc.), and make vllm package optional
• Merging similar model methods into hf_model_mixin
• Set torch_dtype='bfloat16' when bf16 is specified, etc. (bf16 is in FinetunerArguments but torch_dtype is in ModelArguments, thus cannot handle in __post_init__(). )

Bug fixes

• model.generate() with dsz3 [BUG] The text cannot be generated successfully during the Raft step #861
• merge_lora lora with abs path merging
• load_dataset long data fix [Bug Fix] update load_dataset to support long data #878
• src/lmflow/utils/common.py create_copied_dataclass compatibility when python version >= 3.10 (kw_only issue) [BUG]TypeError: Field.__init__() missing 1 required positional argument: 'kw_only' #903 [usability] deps streamlining #905

Issues left over from history

• use_accelerator -> use_accelerate typo fix (with Accelerate support PR)
• model_args.use_lora leads to truncation of the sequence, mentioned in [Feature] reward model inferencer and dpov2 aligner #867
• Make ports, addresses, and all other settings in distributed training tidy and clear (with Accelerate support PR)

Documentation

• Approx GPU memory requirement w.r.t model size & pipeline
• Dev handbook, indicating styles, test list, etc.

[wheresmyhair]

Note on multiple instances inference:
In vllm inference, the number of attn heads should be devisible by vllm tensor parallel size. If we have a 14 heads LLM, then the options for tp is 1&2 (7 will cause another division issue, but I just forget what that issue is).
Say we have 8 gpus, then to utilize these devices, multiple instances vllm inference is necessary (tp=1 -> 8 instances, and tp=2 -> 4 instances)
Also, same for rm inference, and any other inference pipelines.

[wheresmyhair]

Now supports Iterative DPO #883