← 返回刚才阅读的位置

知识卡片

普通读书笔记卡 · 7009.k · 生成模型:DeepSeek-V4.1-Flash

Kubernetes 不是流水线编排

理解等级:高级

编排指协调并跟踪训练系统其余各部件:调度配置训练相关作业、跟踪训练何时完成,常由与 ML 框架及作业调度紧耦合的系统提供(典型如 Apache Airflow),但并非必须。Beam、Spark、Cloud Dataflow 这类系统技术上也做工作流编排,却聚焦数据分析流水线,对任务结构带有显著假设、附带集成并有诸多限制。作者特别澄清:Kubernetes 不是流水线编排系统,它提供了编排容器及其中任务的手段,但本身一般不提供「描述数据如何在流水线中流动」的那类语义。资源调度则是另一层:负责启动进程、跟踪它们、感知何时结束或停止,Kubernetes、以及更传统的 Docker 镜像分发都可归入此列。分层清楚,才不会把容器编排误当数据流水线语义。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • Training Systems · 查看原书依据

      Note that Kubernetes is not a pipeline orchestration system: Kubernetes has a means of orchestrating containers and tasks that run in them, but generally does not by itself provide the kinds of semantics that help us specify how data moves through a pipeline. Job/process/resource scheduling system Everyone who runs ML training pipelines in a distributed environment will have a way of starting processes, keeping track of them,

    参考来源

    - 位置:《Reliable Machine Learning》第07章《Training Systems》(源文件:extracted/Reliable Machine Learning/_epub-src/OEBPS/ch07.xhtml) - 结论依据:原文明确阐述该主题,并给出定义、实践或反例。 - 原始内容:Note that Kubernetes is not a pipeline orchestration system: Kubernetes has a means of orchestrating containers and tasks that run in them, but generally does not by itself provide the kinds of semantics that help us specify how data moves through a pipeline. Job/process/resource scheduling system Everyone who runs ML training pipelines in a distributed environment will have a way of starting processes, keeping track of them,