Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LayerNormalization broadcast (limited support for axis=2) #23297

Open
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

tianleiwu
Copy link
Contributor

@tianleiwu tianleiwu commented Jan 9, 2025

Description

Spec of LayerNormalization supports broadcasting (tensors Scale and B should be unidirectional broadcastable to tensor X).
https://onnx.ai/onnx/operators/onnx__LayerNormalization.html
However, current implementation only allow scale and bias size to be X.shape()[axis:].

Example of input tensors that normalized with axis=2:

X shape Scale shape B shape Before After
(B, S, D) (D) (D) Supported Supported
(B, S, D) (1, 1, D) (1, 1, D) Supported Supported
(B, S, D) (B, 1, D) (B, 1, D) Not Supported Supported
(B, S, D) (1, S, D) (1, S, D) Not Supported Supported
(B, S, D) (B, S, D) (B, S, D) Not Supported Supported

Here we add limited support: axis=2; scale/bias has same shape; scale/bias/X have same number of dimensions. It could support common use case in LLM and vision models.

Motivation and Context

Support Stable Diffusion 3.x and Flux model.

@tianleiwu tianleiwu marked this pull request as draft January 9, 2025 07:26
@tianleiwu tianleiwu marked this pull request as ready for review January 9, 2025 22:28
@tianleiwu tianleiwu requested a review from jiafatom January 9, 2025 22:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant