知识卡片
AI负载不只看请求频率
相同请求次数可能包含不同长度输入和输出,消耗明显不同。噪声邻居策略需把请求规模、模型和频率纳入观察与约束,而不是只限定每秒调用次数;具体负载模型应由实际服务数据验证。
这张卡片看不懂?
这张卡片的反馈已满 10 条,暂时无法继续提交。
已有反馈 · 0 / 10 条
还没有人提交反馈。
在整书笔记中的位置
原书出处 · 1 处
- GenAI and Multi-Tenancy · 查看原书依据
neighbor footprint of your environment. With GenAI workloads, the complexity of requests and responses also have some correlation to the level of load that is being placed on the service. This requires us to add another layer to our noisy neighbor model, introducing constructs that can evaluate the number of tokens and complexity of individual requests, ensuring that tenants aren’t
参考来源
- 位置:《Building Multi-Tenant SaaS Architectures》第16章《GenAI and Multi-Tenancy》(来源项目 a19c7933-4eda-4176-aebd-30fe7e1ad292;源文件:extracted/Building Multi-Tenant SaaS Architectures/_epub-src/OEBPS/ch16.html)
- 结论依据:原文具体说明相同请求次数可能包含不同长度输入和输出,消耗明显不同;本卡据此归纳机制及适用边界。
- 原始内容:neighbor footprint of your environment. With GenAI workloads, the complexity of requests and responses also have some correlation to the level of load that is being placed on the service. This requires us to add another layer to our noisy neighbor model, introducing constructs that can evaluate the number of tokens and complexity of individual requests, ensuring that tenants aren’t