企业级海量代码的检索与管理技术
作者:
作者单位:

作者简介:

刘志伟(1984-),男,辽宁朝阳人,学士,主要研究领域为智能软件开发;李涛(1982-),男,硕士,CCF专业会员,主要研究领域为智能软件开发;邢永旭(1985-),男,硕士,主要研究领域为智能软件开发;张晓东(1990-),男,博士,CCF学生会员,主要研究领域为代码搜索,软件测试与验证;于澔(1985-),男,硕士,主要研究领域为智能软件开发.

通讯作者:

刘志伟,E-mail:liuzhiweihome@gmail.com

中图分类号:

基金项目:

国家重点研发计划(2018YFB1003900)


Retrieval and Management Technology for Industrial-scale Massive Code
Author:
Affiliation:

Fund Project:

National Key Research and Development Program of China (2018YFB1003900)

  • 摘要
  • |
  • 图/表
  • |
  • 访问统计
  • |
  • 参考文献
  • |
  • 相似文献
  • |
  • 引证文献
  • |
  • 资源附件
  • |
  • 文章评论
    摘要:

    在大型IT企业中,尤其像Google或者百度,代码搜索已是软件开发过程中不可或缺且频繁的活动,其通过借鉴或复用已有代码,加速开发过程的速度.多年以来,已有大量的研究人员关注代码搜索,且设计出很多优秀的工具.但是已有的研究和工具主要是在小规模或者编程语言单一的代码数据集上,没有从企业实际搜索需求出发,且对用户的查询输入也有所限制,尚缺少一套针对企业级海量代码的检索与管理技术方案.提出了一套企业级海量数据代码搜索引擎的方案和系统实现,面向开发过程中用户最直接的需求,通过离线分析与在线分析,完成对海量代码库的索引构建与检索.其中,离线分析负责代码相关数据的获取与分析、构建索引集群.在线过程负责变换用户的query、对搜索的结果进行高级排序、生成摘要.本系统部署在百度代码库上,为数十TB级的Git代码库构建了索引,平均一次检索时间在1s之内.在百度推出应用以来,访问量逐步增加,现每周平均用户有数千人,每周查询平均有数万次,广受百度工程师好评.

    Abstract:

    In large IT companies, especially like Google or Baidu, code search is an indispensable and frequent activity in the software development process, which speeds up the development process by learning or reusing existing code. Over the years, a large number of researchers have focused on code search and designed many excellent tools. However, the existing research and tools are mainly on a small-scale or single programming language code data set, not from the actual requirement of industries, and the user's query input is also limited; there is still a lack of a set of industrial-scale massive code retrieval and management technology solutions. This study proposes a code search engine solution and system implementation based on industrial-scale massive data, oriented to the most direct needs of users in the development process, through offline analysis and online analysis, complete the index construction and retrieval of massive code base. Among them, offline analysis is responsible for the acquisition and analysis of code-related data and building an index cluster. The online process is responsible for transforming the user's query, sorting the results of the search, and generating a summary. The system is deployed on the Baidu code base, and the index is built for dozens of TB-level Git code bases. The average retrieval time is within 1s. Since the launch of Baidu's application, the number of visits has gradually increased. There are thousands of users per week and tens of thousands of times searching. The system is widely praised by Baidu engineers.

    参考文献
    相似文献
    引证文献
引用本文

刘志伟,邢永旭,于澔,李涛,张晓东.企业级海量代码的检索与管理技术.软件学报,2019,30(5):1498-1509

复制
分享
文章指标
  • 点击次数:
  • 下载次数:
  • HTML阅读次数:
  • 引用次数:
历史
  • 收稿日期:2018-08-31
  • 最后修改日期:2018-10-31
  • 录用日期:
  • 在线发布日期: 2019-05-08
  • 出版日期:
您是第位访问者
版权所有:中国科学院软件研究所 京ICP备05046678号-3
地址:北京市海淀区中关村南四街4号,邮政编码:100190
电话:010-62562563 传真:010-62562533 Email:jos@iscas.ac.cn
技术支持:北京勤云科技发展有限公司

京公网安备 11040202500063号