← 返回刚才阅读的位置

知识卡片

普通读书笔记卡 · 1980.z.1.e.3

在线扩缩须保护Shuffle

理解等级:高级

动态增加或移除执行器时,下游仍可能读取上游生成的shuffle数据;若直接销毁上游和临时文件,就会破坏计算。外部shuffle服务或保留仍被引用的执行器可隔离此生命周期。另一方案是暂停、保存一致检查点并用新并行度恢复。应用并行度与集群可用容量是两个扩缩层次,必须同时满足。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • Heavyweight Framework Microservices · 查看原书依据

      The first strategy allows you to remove, add, or reassign application instances without stopping the application or affecting processing accuracy. It is available only in some heavyweight streaming frameworks, as it requires careful handling of both state and shuffled events. The addition and removal of instances requires redistributing any assigned stream partitions and reloading state from the last checkpoint. Figure 11-6 shows a regular shuffle, where each downstream reduce operation

    参考来源

    - 位置:《Building Event-Driven Microservices Leveraging Organizational Data at Scale (Adam Bellemare)》第11章《Heavyweight Framework Microservices》(来源项目:a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Building Event-Driven Microservices Leveraging Organizational Data at Scale (Adam Bellemare)/_epub-src/OEBPS/ch11.xhtml;规范化段落起句:The first strategy allows you to remove) - 结论依据:原文对该对象的定义、运行说明或对比支持以下归纳:动态增加或移除执行器时,下游仍可能读取上游生成的shuffle数据;若直接销毁上游和临时文件,就会破坏计算。本卡同时保留其适用条件与失败边界。 - 原始内容:The first strategy allows you to remove, add, or reassign application instances without stopping the application or affecting processing accuracy. It is available only in some heavyweight streaming frameworks, as it requires careful handling of both state and shuffled events. The addition and removal of instances requires redistributing any assigned stream partitions and reloading state from the last checkpoint. Figure 11-6 shows a regular shuffle, where each downstream reduce operation