知识卡片

普通读书笔记卡 · 7010.b · 生成模型:DeepSeek-V4.1-Flash

QPS 负载与三条扩容路径

理解等级:中级

服务环境的第一个判据是流量规模,按需请求时称每秒查询数(QPS):面向数百万日活的模型可能要扛每秒数万查询,手机上监听唤醒词的音频识别器可能只有几 QPS,而房地产估价模型也许根本不按需服务、只作为大型批处理流水线的一环运行。应对高负载有三条基本路径:一是把模型复制到多台机器并行,常借云平台做流量分发并随需求轻松扩容;二是上更强硬件(GPU 等专用加速器),但芯片太强往往瓶颈在输入输出而非计算,因此要把请求批处理以摊薄成本;三是调低模型自身的计算成本——少用特征、更少层或更少参数的深度模型、量化与稀疏化。此外模型级联也常能有效降本:便宜模型先对容易样本做初判,只有难样本才转交昂贵模型。

这张卡片看不懂?

已有反馈 · 0 条

    还没有人提交反馈。

    在整书笔记中的位置

    原书出处 · 1 处

    • Serving · 查看原书依据

      What Will Be the Load to Our Model? The first thing to understand about our serving environment is the level of traffic that our model will be asked to handle—often referred to as queries per second (QPS) when queries are made on demand. A model serving predictions to millions of daily users may be asked to handle tens of thousands of queries per second.

    参考来源

    - 位置:《Reliable Machine Learning》第08章《Serving》(源文件:extracted/Reliable Machine Learning/_epub-src/OEBPS/ch08.xhtml) - 结论依据:原文明确阐述该主题,并给出定义、实践或反例。 - 原始内容:What Will Be the Load to Our Model? The first thing to understand about our serving environment is the level of traffic that our model will be asked to handle—often referred to as queries per second (QPS) when queries are made on demand. A model serving predictions to millions of daily users may be asked to handle tens of thousands of queries per second.