例如把缓存(Memcached)和全文搜索(Elasticsearch、Solr)从主数据库剥离出来后,让缓存和索引与主数据库保持同步就成了应用代码的责任。对外的 API 隐藏这些细节,实际上你已经用通用组件搭出了一个新的专用数据系统,它可能需要提供某些保证(如写入时让缓存作废或更新,使客户端看到一致的结果)。
结论是:组合使用这些工具的开发者,同时也是数据系统设计者,要回答数据出错时如何保证正确完整、部分降级时如何保持性能、负载增加时如何扩容、什么样的 API 才好等问题。影响设计的还有团队技能、历史遗留、交付时限、风险容忍度和监管约束。全书聚焦其中三个普遍问题:可靠性、可伸缩性、可维护性。
[[全序广播的伸缩限制]]下,无因果关系的并发事件可任意排序,同一对象的更新路由到同一分区即可有序。但因果关系可能很隐蔽:一方分手后把前任移出好友,再向好友抱怨,期望前任看不到。若好友关系与消息存在不同地方,通知服务可能先处理消息再处理解除好友,把通知推给前任。这本质上是[[三种流连接]]的时序问题。应对思路有:逻辑时间戳;给决策事件赋 ID 供后续引用;冲突解决算法。目前没有简单通用的答案。
flowchart TD
A[组合多种存储处理工具] --> B[联合数据库: 统一读取]
A --> C[分拆数据库: 统一写入]
B --> D[统一查询接口跨底层引擎,如PostgreSQL外部数据包装器]
C --> E[基于CDC与事件日志,把索引维护拆成跨技术的可靠写入组件]
D -.关系型传统,实现复杂但优雅.-> F[解决跨系统只读查询]
E -.Unix传统,小工具+统一接口+高层组合.-> G[解决跨系统同步写入,更困难的工程问题]
传统 Web 应用假设客户端无状态,浏览器读一次数据后对变化一无所知,除非轮询。单页应用与移动应用有了本地状态,重新点燃了离线优先:尽量用本地数据库工作,联网时后台同步。此时设备状态可视为服务器状态的缓存。EventSource、WebSocket 让服务器主动推送变化,相当于把[[写路径与读路径]]中的写路径延伸到终端设备。离线重连不丢消息,则可沿用[[消费者偏移量]]的技术。
Building Event-Driven Microservices Leveraging Organizational Data at Scale (Adam Bellemare) · Integrating Event-Driven Architectures with Existing Systems
Building Event-Driven Microservices Leveraging Organizational Data at Scale (Adam Bellemare) · Integrating Event-Driven Architectures with Existing Systems
Building Event-Driven Microservices Leveraging Organizational Data at Scale (Adam Bellemare) · Integrating Event-Driven Architectures with Existing Systems
Building Event-Driven Microservices Leveraging Organizational Data at Scale (Adam Bellemare) · Integrating Event-Driven Architectures with Existing Systems
flowchart LR
A["客户端写入master"] --> B["master记录binlog"]
B --> C["slave的I/O线程订阅binlog<br/>写入本地relay log"]
C --> D["slave的SQL线程重放relay log<br/>完成主从数据同步"]
B -.Canal伪装成slave同样订阅binlog.-> E["Canal Server<br/>无需改动业务代码"]
E --> F["Canal Client消费变更事件"]
F --> G["同步缓存 / 数据异构 / 下发任务通知下游系统"]
当只需要获取数据库”从某时刻起发生了什么变化”(增量数据)而非每次全量扫描时,业界总结出四种技术方案,权衡各不相同。定时扫描整表:周期性把整张表读一遍再比对差异,实现最简单但性能低效、延迟高,数据量越大越不可行。写双份数据:在业务代码里,每次更新数据库的同时手动再发一份变更消息,问题是业务逻辑和数据同步逻辑耦合在一起,一致性难以保证(一旦忘记加、或两次写入不是原子的,两边就会不一致),且不利于系统演化。利用触发器机制:让数据库在表变化时自动触发一段逻辑发送变更,避免了业务代码改造,但触发器管理复杂、且会对数据库主库性能产生额外压力。解析事务/提交日志(如MySQL的binlog):不改业务代码、也不使用触发器,但仍需承担日志读取、保留和消费开销;它模拟主备复制协议,从数据库天然产生的复制日志中还原出所有变更记录,这是书中列举的高效方案(阿里Canal、LinkedIn Databus均采用此思路)。这四种方案的本质分歧在于”变更捕获的位置”——日志解析减少业务代码侵入,但仍需权衡数据库日志、消费端开销与实现复杂度;离数据源越远(定时扫描全表)实现越简单,但性能和实时性代价越大。可迁移启发:设计任何”感知数据变化”的系统时,可以用这个坐标(侵入性 vs 实现复杂度 vs 实时性)去比较候选方案,而不是想当然地选”最省事”的定时轮询或”看起来最直接”的业务层双写。
结构图:
flowchart TB
A["CDC四种方案<br/>(按捕获位置比较)"]
A --> B["定时扫描整表<br/>实现最简单/性能低效/延迟高"]
A --> C["业务层写双份数据<br/>无需额外组件/一致性难保证/耦合业务代码"]
A --> D["数据库触发器<br/>无需改代码/管理烦琐/影响数据库性能"]
A --> E["解析事务/提交日志(binlog)<br/>无需业务双写或触发器/需考虑日志和消费开销<br/>如Canal/Databus"]
Building Event-Driven Microservices Leveraging Organizational Data at Scale (Adam Bellemare) · Integrating Event-Driven Architectures with Existing Systems
展开或收起下级:1980.z.1耐久事件连接服务路径上下文Building Event-Driven Microservices Leveraging Organizational Data at Scale (Adam Bellemare) · Why Event-Driven Microservices 理解等级:中级
展开或收起下级:1980.z.1.a复制数据的隐藏耦合路径上下文Building Event-Driven Microservices Leveraging Organizational Data at Scale (Adam Bellemare) · Why Event-Driven Microservices 理解等级:中级
展开或收起下级:1980.z.1.a.1数据解放的迁移边界主题成员Building Event-Driven Microservices Leveraging Organizational Data at Scale (Adam Bellemare) · Integrating Event-Driven Architectures with Existing Systems 理解等级:中级
1980.z.1.a.1.a查询式捕获的取舍主题成员Building Event-Driven Microservices Leveraging Organizational Data at Scale (Adam Bellemare) · Integrating Event-Driven Architectures with Existing Systems 理解等级:高级
1980.z.1.a.1.b日志捕获的恢复边界主题成员Building Event-Driven Microservices Leveraging Organizational Data at Scale (Adam Bellemare) · Integrating Event-Driven Architectures with Existing Systems 理解等级:高级
展开或收起下级:1980.z.1.a.1.c原始CDC会泄漏模型主题成员Building Event-Driven Microservices Leveraging Organizational Data at Scale (Adam Bellemare) · Integrating Event-Driven Architectures with Existing Systems 理解等级:中级
1980.z.1.a.1.c.1事件化隔离内部关联主题成员Building Event-Driven Microservices Leveraging Organizational Data at Scale (Adam Bellemare) · Integrating Event-Driven Architectures with Existing Systems 理解等级:中级
展开或收起下级:1980.z.1.a.1.dOutbox事务避免漏事件路径上下文Building Event-Driven Microservices Leveraging Organizational Data at Scale (Adam Bellemare) · Integrating Event-Driven Architectures with Existing Systems 理解等级:高级
1980.z.1.a.1.d.2触发器捕获的维护成本主题成员Building Event-Driven Microservices Leveraging Organizational Data at Scale (Adam Bellemare) · Integrating Event-Driven Architectures with Existing Systems 理解等级:高级
1980.z.1.a.1.eSink与平台责任划分主题成员Building Event-Driven Microservices Leveraging Organizational Data at Scale (Adam Bellemare) · Integrating Event-Driven Architectures with Existing Systems 理解等级:中级
展开或收起下级:1980.z.1.d基础客户端的能力边界路径上下文Building Event-Driven Microservices Leveraging Organizational Data at Scale (Adam Bellemare) · Basic Producer and Consumer Microservices 理解等级:中级
1980.z.1.d.1Sidecar适配遗留数据主题成员Building Event-Driven Microservices Leveraging Organizational Data at Scale (Adam Bellemare) · Basic Producer and Consumer Microservices 理解等级:中级
展开或收起下级:3333模块化编程按功能隔离路径上下文Principles of Software Architecture Modernization · What’s Wrong with Monoliths? 理解等级:初级
展开或收起下级:3333.i服务数据仅经公开接口访问路径上下文Principles of Software Architecture Modernization · Principles of Proper Services 理解等级:中级
3333.i.3远程读取保边界增依赖主题成员Software Architecture The Hard Parts Modern Trade-Off Analyses for Distributed Architectures (Neal Ford, Mark Richards, Pramod Sadalage etc.) · 分布式数据访问 理解等级:高级
3333.i.4列复制换取本地读取主题成员Software Architecture The Hard Parts Modern Trade-Off Analyses for Distributed Architectures (Neal Ford, Mark Richards, Pramod Sadalage etc.) · 分布式数据访问 理解等级:高级
展开或收起下级:3333.i.5三种缓存共享方式主题成员Software Architecture The Hard Parts Modern Trade-Off Analyses for Distributed Architectures (Neal Ford, Mark Richards, Pramod Sadalage etc.) · 分布式数据访问 理解等级:中级
展开或收起下级:3333.i.5.a复制缓存保留单一写者主题成员Software Architecture The Hard Parts Modern Trade-Off Analyses for Distributed Architectures (Neal Ford, Mark Richards, Pramod Sadalage etc.) · 分布式数据访问 理解等级:高级
3333.i.5.a.1复制缓存有启动容量边界主题成员Software Architecture The Hard Parts Modern Trade-Off Analyses for Distributed Architectures (Neal Ford, Mark Richards, Pramod Sadalage etc.) · 分布式数据访问 理解等级:高级
3333.i.5.a.2专家资料缓存先算规模主题成员Software Architecture The Hard Parts Modern Trade-Off Analyses for Distributed Architectures (Neal Ford, Mark Richards, Pramod Sadalage etc.) · 分布式数据访问 理解等级:高级
3359数据库隔离包含资源层主题成员Principles of Software Architecture Modernization · Anti-Patterns: Lack of Isolation 理解等级:高级
3385数据库隔离要看实际读写主题成员Principles of Software Architecture Modernization · Assessments 理解等级:高级
3421数据拆分优先识别独立读写集合主题成员Principles of Software Architecture Modernization · Data Migrations 理解等级:高级
3424复制切换时明确唯一写入真源主题成员Principles of Software Architecture Modernization · Data Migrations 理解等级:高级
3425CDC从事务日志追踪增量变化主题成员Principles of Software Architecture Modernization · Data Migrations 理解等级:中级
3427数据迁移可先搬再改或同步重塑主题成员Principles of Software Architecture Modernization · Data Migrations 理解等级:高级
3428拆分单体数据先迁写者再迁读者主题成员Principles of Software Architecture Modernization · Data Migrations 理解等级:高级
展开或收起下级:4355逐层扩容会迁移瓶颈路径上下文Fundamentals of Software Architecture · Space-Based Architecture Style 理解等级:中级
展开或收起下级:4355.a内存网格移开同步数据库路径上下文Fundamentals of Software Architecture · Space-Based Architecture Style 理解等级:高级
展开或收起下级:4355.a.3数据泵异步传递缓存更新主题成员Fundamentals of Software Architecture · Space-Based Architecture Style 理解等级:中级
4355.a.3.a数据写入器按领域或单元拆分主题成员Fundamentals of Software Architecture · Space-Based Architecture Style 理解等级:高级
4355.a.3.b数据抽象层隔离缓存与库结构主题成员Fundamentals of Software Architecture · Space-Based Architecture Style 理解等级:高级
4355.a.3.c异步数据泵连接云端与本地库主题成员Fundamentals of Software Architecture · Space-Based Architecture Style 理解等级:中级
4355.a.3.d数据泵拥堵拉长最终一致窗口主题成员Fundamentals of Software Architecture · Space-Based Architecture Style 理解等级:高级
4355.a.4后备库拓扑取决于数据用途主题成员Fundamentals of Software Architecture · Space-Based Architecture Style 理解等级:高级
5105关联键扩展远端数据主题成员RESTful Web API Patterns and Practices Cookbook Connecting and Orchestrating Microservices and Distributed Data (Mike Amundsen) · 分布式数据 理解等级:中级
展开或收起下级:5366数据拆分的相反驱动路径上下文Software Architecture The Hard Parts Modern Trade-Off Analyses for Distributed Architectures (Neal Ford, Mark Richards, Pramod Sadalage etc.) · 拆分操作数据 理解等级:高级
展开或收起下级:5366.a数据域保留内部关系路径上下文Software Architecture The Hard Parts Modern Trade-Off Analyses for Distributed Architectures (Neal Ford, Mark Richards, Pramod Sadalage etc.) · 拆分操作数据 理解等级:高级
展开或收起下级:5366.a.3共享数据域扩大变更边界路径上下文Software Architecture The Hard Parts Modern Trade-Off Analyses for Distributed Architectures (Neal Ford, Mark Richards, Pramod Sadalage etc.) · 数据所有权与分布式事务 理解等级:高级
5366.a.3.a共享数据域以表为契约主题成员Software Architecture The Hard Parts Modern Trade-Off Analyses for Distributed Architectures (Neal Ford, Mark Richards, Pramod Sadalage etc.) · 分布式数据访问 理解等级:高级