PoliBa-DeepLearning/NanoSocrates
2
0
Fork 0
Code Issues 3 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Deleted MultiHeadAttention

Browse Source
This commit is contained in:
Christian Risi 2025-10-07 16:36:11 +02:00
parent 9b5bb6d5f8
commit f9545aca1d
1 changed files with 0 additions and 24 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

24
Project_Model/Libs/Transformer/Classes/MultiHeadAttention.py
View File

@ -1,24 +0,0 @@
# multi-head attention -> (then to) ff
# attention: qkv -> score = qk -> divide -> softamx
# multihead -> QKV diferent in each head ( built by : X*[WQ/QK/WV])
# z = soft(Q*K'/sqr(d))*V
# recombine Z: 1) concatenate. 2) [z01234] * W = Z
# we expect later to have padding token
########################
# WIP
########################
import torch.nn as nn
embed_dim = 256
num_heads = 8
multihead_attn = nn.MultiheadAttention(embed_dim, num_heads)
class MultiheadAttention:
def __init__(
self,
num_heads=8,
) -> None:
pass