知识卡片
管道粒度避免重复处理
为每个细小产品分别建立管道,可能重复读取同一源、造成运行间隔不一致并破坏可组合的时间状态。可按完整应用状态组织共享处理,再生成多个清晰产品;粒度应依据依赖和完整性,不能机械地一产品一管道。
这张卡片看不懂?
这张卡片已通过人工审查,不再接受失败反馈。
已有反馈 · 0 条
还没有人提交反馈。
在整书笔记中的位置
原书出处 · 1 处
- Data Product Management · 查看原书依据
To answer this question, consider the granularity of your data products and the corresponding data pipelines. Suppose you batch process the data from one application and build several coarse-grained data products (datasets) from it, each using its own data pipeline. It’s likely that multiple pipelines will be processing the same underlying source system data because input data for data products often overlaps. This means some
参考来源
- 位置:《Data Management at Scale》第04章《Data Product Management》(来源项目 a19c7933-4eda-4176-aebd-30fe7e1ad292,源文件:extracted/Data Management at Scale/_epub-src/OEBPS/ch04.html)
- 结论依据:原文对moving specific parts of the data的解释、约束及上下文支持本卡;内容为中文归纳,不把架构建议当作所有组织的必然规律。
- 原始内容:To answer this question, consider the granularity of your data products and the corresponding data pipelines. Suppose you batch process the data from one application and build several coarse-grained data products (datasets) from it, each using its own data pipeline. It’s likely that multiple pipelines will be processing the same underlying source system data because input data for data products often overlaps. This means some