[BUG]

[Arkowne]

Hello, I just ran the program and entered my wandb API key, and the program stopped. From the error, the program seems to refuse to use 4D tensors, do you know where the problem is?

Password:
/Users/electrocard/Documents/GitHub/ark-project/ark1/lib/python3.9/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020
  warnings.warn(
2024-12-01 12:45:33.775 | INFO     | __main__:__init__:76 - Initialized streaming dataset
2024-12-01 12:45:33.794 | INFO     | lfm_torch.liquid_t_moe:__init__:55 - Initialized LiquidCell with input_size=768, hidden_size=768, dropout=0.1
2024-12-01 12:45:33.817 | INFO     | lfm_torch.liquid_t_moe:__init__:55 - Initialized LiquidCell with input_size=768, hidden_size=768, dropout=0.1
2024-12-01 12:45:33.841 | INFO     | lfm_torch.liquid_t_moe:__init__:55 - Initialized LiquidCell with input_size=768, hidden_size=768, dropout=0.1
2024-12-01 12:45:33.862 | INFO     | lfm_torch.liquid_t_moe:__init__:55 - Initialized LiquidCell with input_size=768, hidden_size=768, dropout=0.1
2024-12-01 12:45:33.888 | INFO     | lfm_torch.liquid_t_moe:__init__:55 - Initialized LiquidCell with input_size=768, hidden_size=768, dropout=0.1
2024-12-01 12:45:33.908 | INFO     | lfm_torch.liquid_t_moe:__init__:55 - Initialized LiquidCell with input_size=768, hidden_size=768, dropout=0.1
wandb: Using wandb-core as the SDK backend.  Please refer to https://wandb.me/wandb-core for more information.
wandb: Currently logged in as: electrocard (electrocard-hexhal). Use `wandb login --relogin` to force relogin
wandb: Tracking run with wandb version 0.18.7
wandb: Run data is saved locally in /Users/electrocard/Documents/GitHub/ark-project/wandb/run-20241201_124536-17noep84
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run glamorous-dew-2
wandb: ⭐️ View project at https://wandb.ai/electrocard-hexhal/liquid-transformer
wandb: 🚀 View run at https://wandb.ai/electrocard-hexhal/liquid-transformer/runs/17noep84
2024-12-01 12:45:37.316 | INFO     | __main__:__init__:146 - Trainer initialized successfully
2024-12-01 12:45:37.317 | INFO     | __main__:train:221 - Starting training
Training:   0%|                                                                                                                                                                            | 0/100000 [00:00<?, ?it/s]2024-12-01 12:45:40.400 | DEBUG    | lfm_torch.liquid_t_moe:forward:196 - Input shape to TransformerLayerWithLiquid: torch.Size([1, 16, 512, 768])
2024-12-01 12:45:40.402 | ERROR    | __main__:train_step:192 - Error in train_step: query should be unbatched 2D or batched 3D tensor but received 4-D query tensor
2024-12-01 12:45:40.406 | ERROR    | __main__:train:272 - Training error: query should be unbatched 2D or batched 3D tensor but received 4-D query tensor
2024-12-01 12:45:43.224 | INFO     | __main__:save_checkpoint:214 - Saved checkpoint at step 0 to checkpoints/checkpoint_step_0.pt
Training:   0%|                                                                                                                                                                            | 0/100000 [00:05<?, ?it/s]
2024-12-01 12:45:44.090 | INFO     | __main__:save_checkpoint:214 - Saved checkpoint at step 0 to checkpoints/checkpoint_step_0.pt
2024-12-01 12:45:44.094 | INFO     | __main__:train:279 - Training completed after 0 steps
2024-12-01 12:45:44.094 | ERROR    | __main__:main:314 - Training failed with error: query should be unbatched 2D or batched 3D tensor but received 4-D query tensor
wandb:                                                                                
wandb: 🚀 View run glamorous-dew-2 at: https://wandb.ai/electrocard-hexhal/liquid-transformer/runs/17noep84
wandb: ⭐️ View project at: https://wandb.ai/electrocard-hexhal/liquid-transformer
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Find logs at: ./wandb/run-20241201_124536-17noep84/logs
Traceback (most recent call last):
  File "/Users/electrocard/Documents/GitHub/ark-project/LFM-main/liquid_transformer_train.py", line 320, in <module>
    main()
  File "/Users/electrocard/Documents/GitHub/ark-project/LFM-main/liquid_transformer_train.py", line 311, in main
    trainer.train(train_dataset)
  File "/Users/electrocard/Documents/GitHub/ark-project/LFM-main/liquid_transformer_train.py", line 240, in train
    current_loss = self.train_step(batch)
  File "/Users/electrocard/Documents/GitHub/ark-project/LFM-main/liquid_transformer_train.py", line 174, in train_step
    outputs = self.model(embedded_input)
  File "/Users/electrocard/Documents/GitHub/ark-project/ark1/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/Users/electrocard/Documents/GitHub/ark-project/ark1/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/electrocard/Documents/GitHub/ark-project/LFM-main/lfm_torch/liquid_t_moe.py", line 267, in forward
    x = layer(x, self.hidden_state)
  File "/Users/electrocard/Documents/GitHub/ark-project/ark1/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/Users/electrocard/Documents/GitHub/ark-project/ark1/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/electrocard/Documents/GitHub/ark-project/LFM-main/lfm_torch/liquid_t_moe.py", line 201, in forward
    attention_output, _ = self.attention(x, x, x)
  File "/Users/electrocard/Documents/GitHub/ark-project/ark1/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/Users/electrocard/Documents/GitHub/ark-project/ark1/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/electrocard/Documents/GitHub/ark-project/ark1/lib/python3.9/site-packages/torch/nn/modules/activation.py", line 1368, in forward
    attn_output, attn_output_weights = F.multi_head_attention_forward(
  File "/Users/electrocard/Documents/GitHub/ark-project/ark1/lib/python3.9/site-packages/torch/nn/functional.py", line 6014, in multi_head_attention_forward
    is_batched = _mha_shape_check(
  File "/Users/electrocard/Documents/GitHub/ark-project/ark1/lib/python3.9/site-packages/torch/nn/functional.py", line 5825, in _mha_shape_check
    raise AssertionError(
AssertionError: query should be unbatched 2D or batched 3D tensor but received 4-D query tensor

[github-actions]

Hello there, thank you for opening an Issue ! 🙏🏻 The team was notified and they will get back to you asap.

[github-actions]

Stale issue message