知识卡片
Spark承担批量转换
Spark 是分布式计算引擎,可在外部存储上执行摄取、清洗、转换和 SQL 分析,也可用 Structured Streaming 处理流。它不自带湖仓持久存储;低延迟要求和表格式读写能力要按具体版本测试。
这张卡片看不懂?
这张卡片的反馈已满 10 条,暂时无法继续提交。
已有反馈 · 0 / 10 条
还没有人提交反馈。
原书出处
参考来源
- 位置:《Practical Lakehouse Architecture》第5章《Compute Engines for Lakehouse Architectures》(源文件:_epub-src/OEBPS/ch05.html)
- 结论依据:原文将 Spark 描述为依赖 HDFS 或对象存储的计算框架,列出 ETL、Spark SQL 和 Structured Streaming 用途。
- 原始内容:Spark is a distributed, in-memory processing framework. It has no storage component but rather uses distributed storage like HDFS or cloud object storage like S3, ADLS, or GCS to store data. Key features, benefits, and limitations of Spark are listed below: Spark offers high reliability and scalability while processing large volumes of data. Unlike its