原书出处
Build a Reasoning Model (From Scratch)
queries = apply_rope(queries, cos, sin, offset=start_pos) keys_new = apply_rope(keys_new, cos, sin, offset=start_pos) if cache is not None: prev_k, prev_v = cache keys = torch.cat([prev_k, keys_new], dim=2) values = torch.cat([prev_v, values_new], dim=2) else: start_pos = 0 #C keys, values = keys_new, values_new next_cache = (keys, values) keys = keys.repeat_interleave( #D self.group_size, dim=1 #D ) #D values = values.repeat_interleave( #D self.group_size, dim=1 #D ) #D
原书定位详情
来源:a19c7933-4eda-4176-aebd-30fe7e1ad292
正文位置:extracted/Build a Reasoning Model (From Scratch)/_epub-src/OEBPS/Text/appendix-c.html
EPUB SHA-256:52d4d07986020be70f5aac21762dfcf60fcba08db0ba29d4f63957caf12aa377