{"componentChunkName":"component---src-templates-blog-post-js","path":"/blog/2020-06-17-account-service-design/","result":{"data":{"markdownRemark":{"id":"ee4fdb78-bd89-5f04-b0e2-5103fb9d9f67","html":"<p><img src=\"/img/flavor_wheel.jpg\" alt=\"flavor wheel\"></p>\n<p>主要需求：对网站访问量进行技术（视频观看量、博客网站粉丝数、微服务调用量计数）。</p>\n<div id=\"markmap-2\" class=\"gatsby-markmap\" data-markmap=\"{&quot;t&quot;:&quot;root&quot;,&quot;d&quot;:0,&quot;v&quot;:&quot;&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:1,&quot;v&quot;:&quot;需求&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;需求沟通&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;list_item&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;需求模糊需要确认细节&quot;},{&quot;t&quot;:&quot;list_item&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;需求不明确下，不同技术背景的人会给出不同的方案&quot;}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;需求澄清&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;场景用例&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;list_item&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;用户群体是谁&quot;},{&quot;t&quot;:&quot;list_item&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;用户如何使用这个系统&quot;}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;量级规模（读/写）&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;list_item&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;每秒查询请求量&quot;},{&quot;t&quot;:&quot;list_item&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;每个请求查询多少数据&quot;},{&quot;t&quot;:&quot;list_item&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;每秒处理多少个视频观看记录&quot;},{&quot;t&quot;:&quot;list_item&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;流量模式？是否有流量高峰&quot;}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;性能&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;list_item&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;预期从写入到读取数据的延迟？&quot;},{&quot;t&quot;:&quot;list_item&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;预期 p99 读请求延迟是多少&quot;},{&quot;t&quot;:&quot;list_item&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;高可用性（一般隐含要求）&quot;}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;成本&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;list_item&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;开发成本有什么限制？&quot;},{&quot;t&quot;:&quot;list_item&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;运维成本有什么限制？&quot;},{&quot;t&quot;:&quot;list_item&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;开发成员组成结构&quot;}]}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;功能需求&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;非功能需求&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;list_item&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;高可用&quot;},{&quot;t&quot;:&quot;list_item&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;高性能&quot;},{&quot;t&quot;:&quot;list_item&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;可扩展&quot;},{&quot;t&quot;:&quot;list_item&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;可维护性&quot;},{&quot;t&quot;:&quot;list_item&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;成本&quot;}]}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:1,&quot;v&quot;:&quot;架构设计&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;简单架构图&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;存储设计&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;存什么&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;单个事件（每次点击）&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;聚合数据&quot;}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;数据库选型&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;结构化数据库&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:5,&quot;v&quot;:&quot;数据分区&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:5,&quot;v&quot;:&quot;主从复制&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;paragraph&quot;,&quot;d&quot;:6,&quot;v&quot;:&quot;读写分离&quot;}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:5,&quot;v&quot;:&quot;ShardingSphere&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;paragraph&quot;,&quot;d&quot;:6,&quot;v&quot;:&quot;本身需要高可用&quot;}]}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;NoSql 数据库&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:5,&quot;v&quot;:&quot;Cassandra&quot;}]}]}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;计数服务设计&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;数据聚合&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;消息队列&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;kafka&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:5,&quot;v&quot;:&quot;检查点（Checkpointing）&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:5,&quot;v&quot;:&quot;分区（Partitioning）&quot;}]}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;计数消费者（详细设计）&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;分区消费者（kafka client）&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;计算方法&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;内部队列&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;DB Writer&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:5,&quot;v&quot;:&quot;死信队列&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:5,&quot;v&quot;:&quot;数据填充（Data Enrichment）&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;paragraph&quot;,&quot;d&quot;:6,&quot;v&quot;:&quot;处理了每分钟的需求，如果还有更粗粒度的需求（小时、天、月）可以再次把消息发送到队列中进行计算。这是流处理的一种思想&quot;}]}]}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;数据接收路径（Data Ingestion path）&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;paragraph&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;代理用户请求，发送数据到 MQs&quot;}]}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;相关技术点&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;如果实现 API Gateway 的负载均衡访问&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;API Gateway 如何发现 Account Service 实例&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;集中式做法&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;微服务注册中心&quot;}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;MQ 的生产者和分区之间、MQ 的消费者和分区之间的服务发现和负载均衡&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;网关如何实现容错限流&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;如何通过网关实现实时的防爬虫&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;MQ Client 同步和异步生产问题&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;消息格式&quot;}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;查询服务设计&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;数据存储&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;Distributed Cache&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;paragraph&quot;,&quot;d&quot;:5,&quot;v&quot;:&quot;热点数据&quot;}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;Database&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;paragraph&quot;,&quot;d&quot;:5,&quot;v&quot;:&quot;热点数据&quot;}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;Object Store&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:5,&quot;v&quot;:&quot;历史数据归档&quot;}]}]}]}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:1,&quot;v&quot;:&quot;技术站选型&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;流程图&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;Spring Boot&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;负载均衡 - F5&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;Zull or spring cloud gateway&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;euraka and ribbon&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;Kafka&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;Redis&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;Cassandra or Hbase&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;Hadoop&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;OSS&quot;}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:1,&quot;v&quot;:&quot;更多的问题&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;如何定位系统瓶颈&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;性能和压力测试&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;bullet_list&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;list_item&quot;,&quot;d&quot;:5,&quot;v&quot;:&quot;技术服务&quot;},{&quot;t&quot;:&quot;list_item&quot;,&quot;d&quot;:5,&quot;v&quot;:&quot;消费者服务&quot;},{&quot;t&quot;:&quot;list_item&quot;,&quot;d&quot;:5,&quot;v&quot;:&quot;查询服务&quot;}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;定位性能和多线程问题 &amp;amp; 系统调优&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;对生产部署进行容量估算&quot;}]}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;如何监控系统健康状况&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;blockquote&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;paragraph&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;对系统做细粒度的监控和埋点&quot;}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;服务监控&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;调用量&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;调用延迟&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;调用错误数&quot;}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;硬件资源&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;CPU&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;内存&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;磁盘&quot;}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;Java 项目&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;垃圾回收活动&quot;}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;队列&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;消息堆积&quot;}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;监控手段&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;日志监控&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;Metrics 监控&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;调用链监控&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;健康检查&quot;}]}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;如何确保线上系统运行结果正确&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;完整的线下测试&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;确保线上服务正确行&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;开发一套线上模拟程序&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:4,&quot;v&quot;:&quot;创建线下校验系统（Lambda Architecture）&quot;}]}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;如何解决热分区的问题&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;视频 ID 添加时间戳&quot;}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;如何监控慢消费者&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;TODO&quot;}]}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:1,&quot;v&quot;:&quot;总结&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;功能需求（API）&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;需求用例&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;输入输出&quot;}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;非功能需求&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;高可用&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;高性能&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;可扩展&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;成本&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;其他&quot;}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;总体设计&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;写入路径&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;读取路径&quot;}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;详细设计&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;存储设计&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;写入处理设计&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;读取查询设计&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;分布式系统概念和技术&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;技术栈选型&quot;}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;评估&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;性能测试/瓶颈定位&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;监控&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;扩容升级&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;线上校验&quot;}]}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:1,&quot;v&quot;:&quot;扩展&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;监控系统&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;微服务调用量 or 错误数&quot;}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;欺诈检测系统&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;账户使用情况&quot;}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;限流系统&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;访问用户 IP&quot;}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;推荐系统&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;用户浏览和购买商品计数&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;推荐系统处理加工&quot;}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;今日热点&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:3,&quot;v&quot;:&quot;热点文章技术&quot;}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;等&quot;}]},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:1,&quot;v&quot;:&quot;参考&quot;,&quot;c&quot;:[{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;&lt;a href=\\&quot;https://www.youtube.com/watch?v=bUHFg8CzFws\\&quot; title=\\&quot;\\&quot; target=\\&quot;_blank\\&quot; rel=\\&quot;noopener noreferrer\\&quot;&gt;System Design Interview - Step by Step Guide&lt;/a&gt;&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;&lt;a href=\\&quot;https://github.com/donnemartin/system-design-primer\\&quot; title=\\&quot;\\&quot; target=\\&quot;_blank\\&quot; rel=\\&quot;noopener noreferrer\\&quot;&gt;System design primer&lt;/a&gt;&quot;},{&quot;t&quot;:&quot;heading&quot;,&quot;d&quot;:2,&quot;v&quot;:&quot;&lt;a href=\\&quot;https://github.com/Jaskey/ConsistentHash\\&quot; title=\\&quot;\\&quot; target=\\&quot;_blank\\&quot; rel=\\&quot;noopener noreferrer\\&quot;&gt;ConsistentHash&lt;/a&gt;&quot;}]}]}\"></div>\n<h1>需求</h1>\n<h2>需求沟通</h2>\n<ul>\n<li>需求模糊需要确认细节</li>\n<li>需求不明确下，不同技术背景的人会给出不同的方案</li>\n</ul>\n<h2>需求澄清</h2>\n<h3>场景用例</h3>\n<ul>\n<li>用户群体是谁</li>\n<li>用户如何使用这个系统</li>\n</ul>\n<h3>量级规模（读/写）</h3>\n<ul>\n<li>每秒查询请求量</li>\n<li>每个请求查询多少数据</li>\n<li>每秒处理多少个视频观看记录</li>\n<li>流量模式？是否有流量高峰</li>\n</ul>\n<h3>性能</h3>\n<ul>\n<li>预期从写入到读取数据的延迟？</li>\n<li>预期 p99 读请求延迟是多少</li>\n<li>高可用性（一般隐含要求）</li>\n</ul>\n<h3>成本</h3>\n<ul>\n<li>开发成本有什么限制？</li>\n<li>运维成本有什么限制？</li>\n<li>开发成员组成结构</li>\n</ul>\n<h2>功能需求</h2>\n<h2>非功能需求</h2>\n<ul>\n<li>高可用</li>\n<li>高性能</li>\n<li>可扩展</li>\n<li>可维护性</li>\n<li>成本</li>\n</ul>\n<h1>架构设计</h1>\n<h2>简单架构图</h2>\n<p>部署图</p>\n<h2>存储设计</h2>\n<h3>存什么</h3>\n<h4>单个事件（每次点击）</h4>\n<h4>聚合数据</h4>\n<h3>数据库选型</h3>\n<h4>结构化数据库</h4>\n<h5>数据分区</h5>\n<h5>主从复制</h5>\n<blockquote>\n<p>读写分离</p>\n</blockquote>\n<h5>ShardingSphere</h5>\n<blockquote>\n<p>本身需要高可用</p>\n</blockquote>\n<h4>NoSql 数据库</h4>\n<h5>Cassandra</h5>\n<p>仲裁读写\n支持最终一致性</p>\n<p>支持动态添加节点</p>\n<p>支持多数据中心</p>\n<h2>计数服务设计</h2>\n<h3>数据聚合</h3>\n<h3>消息队列</h3>\n<h4>kafka</h4>\n<h5>检查点（Checkpointing）</h5>\n<h5>分区（Partitioning）</h5>\n<h3>计数消费者（详细设计）</h3>\n<h4>分区消费者（kafka client）</h4>\n<p>kafka client 可自动提交偏移量</p>\n<h4>计算方法</h4>\n<h4>内部队列</h4>\n<h4>DB Writer</h4>\n<h5>死信队列</h5>\n<p>保证消息的高可用</p>\n<h5>数据填充（Data Enrichment）</h5>\n<p>添加 Cache 提高性能。</p>\n<blockquote>\n<p>处理了每分钟的需求，如果还有更粗粒度的需求（小时、天、月）可以再次把消息发送到队列中进行计算。这是流处理的一种思想</p>\n</blockquote>\n<h3>数据接收路径（Data Ingestion path）</h3>\n<blockquote>\n<p>代理用户请求，发送数据到 MQs</p>\n</blockquote>\n<h2>相关技术点</h2>\n<h3>如果实现 API Gateway 的负载均衡访问</h3>\n<p>Gateway 也是集群部署的，前置也需要负载均衡。</p>\n<p>可以考虑 F5 或者 Nginx SLB</p>\n<h3>API Gateway 如何发现 Account Service 实例</h3>\n<h4>集中式做法</h4>\n<p>Nginx + DNS</p>\n<h4>微服务注册中心</h4>\n<p>e.g. Eureka 注册中心 + Robbin 客户端</p>\n<h3>MQ 的生产者和分区之间、MQ 的消费者和分区之间的服务发现和负载均衡</h3>\n<p>相对复杂</p>\n<h3>网关如何实现容错限流</h3>\n<h3>如何通过网关实现实时的防爬虫</h3>\n<h3>MQ Client 同步和异步生产问题</h3>\n<h3>消息格式</h3>\n<p>protobuf/json</p>\n<h2>查询服务设计</h2>\n<h3>数据存储</h3>\n<h4>Distributed Cache</h4>\n<blockquote>\n<p>热点数据</p>\n</blockquote>\n<p>提高查询效率</p>\n<h4>Database</h4>\n<blockquote>\n<p>热点数据</p>\n</blockquote>\n<h4>Object Store</h4>\n<h5>历史数据归档</h5>\n<p>分钟小时数据数据量大，一直存储比较消耗数据库内存，数据库性能会逐步下降。对于已经聚合完成的一个月以前的分钟小时数据，把它们存储到 Object Store。大部分场景下用户只关注近期数据，偶尔查询历史详情数据。</p>\n<h1>技术站选型</h1>\n<h2>流程图</h2>\n<h2>Spring Boot</h2>\n<h2>负载均衡 - F5</h2>\n<h2>Zull or spring cloud gateway</h2>\n<h2>euraka and ribbon</h2>\n<h2>Kafka</h2>\n<h2>Redis</h2>\n<h2>Cassandra or Hbase</h2>\n<h2>Hadoop</h2>\n<p>线下计算</p>\n<h2>OSS</h2>\n<p>对象存储（长期存储）</p>\n<h1>更多的问题</h1>\n<h2>如何定位系统瓶颈</h2>\n<h3>性能和压力测试</h3>\n<p>核心服务：</p>\n<ul>\n<li>技术服务</li>\n<li>消费者服务</li>\n<li>查询服务</li>\n</ul>\n<h4>定位性能和多线程问题 &#x26; 系统调优</h4>\n<h4>对生产部署进行容量估算</h4>\n<p>获取基准性能数据</p>\n<h2>如何监控系统健康状况</h2>\n<blockquote>\n<p>对系统做细粒度的监控和埋点</p>\n</blockquote>\n<h3>服务监控</h3>\n<h4>调用量</h4>\n<h4>调用延迟</h4>\n<h4>调用错误数</h4>\n<h3>硬件资源</h3>\n<h4>CPU</h4>\n<h4>内存</h4>\n<h4>磁盘</h4>\n<h3>Java 项目</h3>\n<h4>垃圾回收活动</h4>\n<h3>队列</h3>\n<h4>消息堆积</h4>\n<p>消息堆积是系统需要扩容的一个重要信号</p>\n<h3>监控手段</h3>\n<h4>日志监控</h4>\n<h4>Metrics 监控</h4>\n<h4>调用链监控</h4>\n<h4>健康检查</h4>\n<h2>如何确保线上系统运行结果正确</h2>\n<h3>完整的线下测试</h3>\n<p>确保系统功能正确。但是线下测试不能完全覆盖线上的真是流量，一些场景对数据准确性要求比较高。</p>\n<h3>确保线上服务正确行</h3>\n<h4>开发一套线上模拟程序</h4>\n<p>数据是模拟的，还不能完全覆盖生产流量。</p>\n<h4>创建线下校验系统（Lambda Architecture）</h4>\n<p>引入一套线下批处理系统，比如 Hadoop/Map reduce。同时做相同计算，线下有一个校验系统对两套系统产生的结果进行校验。</p>\n<h2>如何解决热分区的问题</h2>\n<p>对于热点视频，进行特殊处理。对应的分区会产生频繁的读写。</p>\n<h3>视频 ID 添加时间戳</h3>\n<p>热点视频按时间再分区</p>\n<h2>如何监控慢消费者</h2>\n<h3>TODO</h3>\n<h1>总结</h1>\n<p>分布式系统设计流程主要分为五个步骤</p>\n<h2>功能需求（API）</h2>\n<h3>需求用例</h3>\n<h3>输入输出</h3>\n<h2>非功能需求</h2>\n<h3>高可用</h3>\n<h3>高性能</h3>\n<h3>可扩展</h3>\n<h3>成本</h3>\n<h3>其他</h3>\n<h2>总体设计</h2>\n<h3>写入路径</h3>\n<h3>读取路径</h3>\n<h2>详细设计</h2>\n<h3>存储设计</h3>\n<h3>写入处理设计</h3>\n<h3>读取查询设计</h3>\n<h3>分布式系统概念和技术</h3>\n<h3>技术栈选型</h3>\n<h2>评估</h2>\n<h3>性能测试/瓶颈定位</h3>\n<h3>监控</h3>\n<h3>扩容升级</h3>\n<h3>线上校验</h3>\n<h1>扩展</h1>\n<h2>监控系统</h2>\n<h3>微服务调用量 or 错误数</h3>\n<h2>欺诈检测系统</h2>\n<h3>账户使用情况</h3>\n<h2>限流系统</h2>\n<h3>访问用户 IP</h3>\n<h2>推荐系统</h2>\n<h3>用户浏览和购买商品计数</h3>\n<h3>推荐系统处理加工</h3>\n<h2>今日热点</h2>\n<h3>热点文章技术</h3>\n<h2>等</h2>\n<h1>参考</h1>\n<h2><a href=\"https://www.youtube.com/watch?v=bUHFg8CzFws\">System Design Interview - Step by Step Guide</a></h2>\n<h2><a href=\"https://github.com/donnemartin/system-design-primer\">System design primer</a></h2>\n<h2><a href=\"https://github.com/Jaskey/ConsistentHash\">ConsistentHash</a></h2>\n<p>Java 实现的一致性 hash</p>","frontmatter":{"date":"June 17, 2020","title":"计数服务设计","description":"分布式项目设计","tags":null}}},"pageContext":{"id":"ee4fdb78-bd89-5f04-b0e2-5103fb9d9f67"}}}