原书出处
Build a Reasoning Model (From Scratch)
next_cache = (keys, values) keys = keys.repeat_interleave( #D self.group_size, dim=1 #D ) #D values = values.repeat_interleave( #D self.group_size, dim=1 #D ) #D attn_scores = queries @ keys.transpose(2, 3) attn_scores = attn_scores.masked_fill(mask, -torch.inf) attn_weights = torch.softmax( attn_scores / self.head_dim**0.5, dim=-1 ) context = (attn_weights @ values).transpose(1, 2) context = context.reshape(b, num_tokens, self.d_out) return self.out_proj(context), next_cache #A The shape is (b, num_tokens, num_heads * head_dim)
原书定位详情
来源:a19c7933-4eda-4176-aebd-30fe7e1ad292
正文位置:extracted/Build a Reasoning Model (From Scratch)/_epub-src/OEBPS/Text/appendix-c.html
EPUB SHA-256:52d4d07986020be70f5aac21762dfcf60fcba08db0ba29d4f63957caf12aa377