← 返回卡片

原书出处

Build a Reasoning Model (From Scratch)

MEAP Version 5(早期访问版) · 核验版本的摘录快照 · 导入于 2026-11-14T08:12:58.964Z

next_cache = (keys, values) keys = keys.repeat_interleave( #D self.group_size, dim=1 #D ) #D values = values.repeat_interleave( #D self.group_size, dim=1 #D ) #D attn_scores = queries @ keys.transpose(2, 3) attn_scores = attn_scores.masked_fill(mask, -torch.inf) attn_weights = torch.softmax( attn_scores / self.head_dim**0.5, dim=-1 ) context = (attn_weights @ values).transpose(1, 2) context = context.reshape(b, num_tokens, self.d_out) return self.out_proj(context), next_cache #A The shape is (b, num_tokens, num_heads * head_dim)

此预览只包含定位所需的摘录和上下文。

原书定位详情

来源:a19c7933-4eda-4176-aebd-30fe7e1ad292

正文位置:extracted/Build a Reasoning Model (From Scratch)/_epub-src/OEBPS/Text/appendix-c.html

EPUB SHA-256:52d4d07986020be70f5aac21762dfcf60fcba08db0ba29d4f63957caf12aa377