Bing Liu (刘冰) Head of Research @ Scale AI GenAI, LLM, Agents San Francisco Bay Area PhD, Carnegie Mellon University Email: hadoopit [at] gmail dot com [Publications] [Work Experiences] [Education] [CV] My Github @hadoopit

I currently lead research at Scale AI, where we focus on LLM evaluation, data, post-training, and agentic systems. We build leading benchmarks, evaluation frameworks, and data infrastructure for GenAI, and collaborate with leading AI labs to advance frontier models through high-quality data and rigorous evaluation.

Previously, I contributed to the development of Llama 3 and the initial release of the Meta AI Assistant at Meta GenAI. Before that, I led the NLU team at Meta Reality Labs, where we built the digital assistant for AR/VR. Earlier in my career, I worked at Google Research and Capio.ai (acquired by Twilio).

I did my Ph.D. in Electrical and Computer Engineering from Carnegie Mellon University (CMU), where my research centered on deep and reinforcement learning for spoken dialog systems.

I have served as an area chair and senior committee member for major NLP and machine learning conferences, including ACL, EMNLP, and ICLR. I am also an adjunct professor in the Computer Science department at UC Santa Cruz.

• 2025/12: New benchmark release on MCP Atlas - a large-scale, real-server benchmark for agentic tool use! [Paper]
• 2025/12: New benchmark release on Audio MultiChallenge - a multi-turn eval for spoken dialog systems! [Paper]
• 2025/11: New paper release on ResearchRubrics - a new eval for evaluating deep research agent! [Paper]
• 2025/11: New benchmark release - Professional Reasoning Benchmark (PRBench), a benchmark for real-world professional problems like Finance and Law. [Paper]
• 2025/10: New benchmark release - Remote Labor Index, the 1st benchmark that test how well AI agents can complete real, paid freelance work. [Paper]
• 2025/10: New benchmark release - VisualToolBench, for multimodal reasoning with tool use! [Paper]
• 2025/10: 2 new papers on RL with rubrics: Chasing the Tail [paper] & Online Rubrics Elicitation [paper]
• 2025/09: New eval platform release - Showdown, a platform for evaluating frontier models in the wild with granular insights! [Leaderboard] [Technical Report]
• 2025/09: New benchmark release - SWE-Bench Pro, for evaluating LLM coding agents on real, enterprise-grade software engineering tasks! [Paper]
• 2025/09: New benchmark release - MCP Atlas, for evaluating LLM function calling and tool use in the real world! (benchmark data and paper release coming soon!)
• 2025/09: New paper release on Rubrics as Rewards - doing RL beyond verifiable domains!
• 2025/09: New benchmark release - TutorBench, for evaluating LLM's tutoring capabilities! [Paper]
• 2025/07: New benchmark release - MultiNRC, for multilingual reasoning! [Paper]
• 2025/03: New benchmark release - MultiChallenge, for multi-turn instruction following! [Paper]
• 2025/02: We released Humanity’s Last Exam, a new AI benchmark that was designed to test the limits of AI knowledge at the frontiers of human expertise! [Paper]
• 2024/11: After 6 years at Meta, I joined Scale AI to lead the applied research on LLM post-training, data, and eval!
• 2024/07: We released Llama 3.1, the latest Llama model with multilingual capability, 128K context window, and much stronger tool use and reasoning capabilities!
• 2024/03: Appointed as an Adjunct Professor at the Computer Science and Engineering department of University of California, Santa Cruz.
• 2024/01: Serving as the Publication Chair for ACL 2024.
• 2023/05: Gave an invited talk at University of California, Santa Cruz, Seminar in NLP, on "ChatGPT and Large Language Models".
• 2023/01: Serving as the Area Chair for the Large Language Models track at ACL 2023, and the Publication Chair for the 5th Workshop on NLP for Conversational AI.
• 2022/07: Serving on the Senior Program Committee for AAAI 2023 and ICASSP 2023.
• 2022/05: Serving as the General Chair to organize the 4th Workshop on NLP for Conversational AI at ACL 2022.

• MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers
  Chaithanya Bandi, Ben Hertzberg, Geobio Boo, Tejas Polakam, Jeff Da, Sami Hassaan, Manasi Sharma, Andrew Park, Ernesto Hernandez, Dan Rambado, Ivan Salazar, Rafael Cruz, Chetan Rane, Ben Levin, Brad Kenstler, Bing Liu
  In arXiv, 2025
[bib] [paper] [data] [code]


• Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interaction
  Advait Gosai, Tyler Vuong, Utkarsh Tyagi, Steven Li, Wenjia You, Miheer Bavare, Arda Uçar, Zhongwang Fang, Brian Jang, Bing Liu, Yunzhong He
  In arXiv, 2025

  @article{gosai2025audio,
    title={Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interaction},
    author={Gosai, Advait and Vuong, Tyler and Tyagi, Utkarsh and Li, Steven and You, Wenjia and Bavare, Miheer and U{\c{c}}ar, Arda and Fang, Zhongwang and Jang, Brian and Liu, Bing and others},
    journal={arXiv preprint arXiv:2512.14865},
    year={2025}
  }
  

[bib] [paper] [data]


• ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents
  Manasi Sharma, Chen Bo Calvin Zhang, Chaithanya Bandi, Clinton Wang, Ankit Aich, Huy Nghiem, Tahseen Rabbani, Ye Htet, Brian Jang, Sumana Basu, Aishwarya Balwani, Denis Peskoff, Marcos Ayestaran, Sean M. Hendryx, Brad Kenstler, Bing Liu
  In arXiv, 2025

  @article{sharma2025researchrubrics,
    title={ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents},
    author={Sharma, Manasi and Zhang, Chen Bo Calvin and Bandi, Chaithanya and Wang, Clinton and Aich, Ankit and Nghiem, Huy and Rabbani, Tahseen and Htet, Ye and Jang, Brian and Basu, Sumana and others},
    journal={arXiv preprint arXiv:2511.07685},
    year={2025}
  }
  

[bib] [paper]


• PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning
  Afra Feyza Akyürek, Advait Gosai, Chen Bo Calvin Zhang, Vipul Gupta, Jaehwan Jeong, Anisha Gunjal, Tahseen Rabbani, Maria Mazzone, David Randolph, Mohammad Mahmoudi Meymand, Gurshaan Chattha, Paula Rodriguez, Diego Mares, Pavit Singh, Michael Liu, Subodh Chawla, Pete Cline, Lucy Ogaz, Ernesto Hernandez, Zihao Wang, Pavi Bhatter, Marcos Ayestaran, Bing Liu, Yunzhong He
  In arXiv, 2025

  @article{akyurek2025prbench,
    title={PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning},
    author={Aky{\"u}rek, Afra Feyza and Gosai, Advait and Zhang, Chen Bo Calvin and Gupta, Vipul and Jeong, Jaehwan and Gunjal, Anisha and Rabbani, Tahseen and Mazzone, Maria and Randolph, David and Meymand, Mohammad Mahmoudi and others},
    journal={arXiv preprint arXiv:2511.11562},
    year={2025}
  }
  

[bib] [paper] [data] [code]


• Remote Labor Index: Measuring AI Automation of Remote Work
  Mazeika, Mantas, et al. jointly by Scale AI and Center for AI Safety
  In arXiv, 2025

  @article{mazeika2025remote,
    title={Remote Labor Index: Measuring AI Automation of Remote Work},
    author={Mazeika, Mantas and Gatti, Alice and Menghini, Cristina and Sehwag, Udari Madhushani and Singhal, Shivam and Orlovskiy, Yury and Basart, Steven and Sharma, Manasi and Peskoff, Denis and Lau, Elaine and others},
    journal={arXiv preprint arXiv:2510.26787},
    year={2025}
  }
  

[bib] [paper]


• SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
  Xiang Deng, Jeff Da, Edwin Pan, Yannis Y. He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa R Kundurthy, Sean M. Hendryx, Zifan Wang, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu, Brad Kenstler
  In arXiv, 2025

  @article{deng2025swe,
    title={SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?},
    author={Deng, Xiang and Da, Jeff and Pan, Edwin and He, Yannis Yiming and Ide, Charles and Garg, Kanak and Lauffer, Niklas and Park, Andrew and Pasari, Nitin and Rane, Chetan and others},
    journal={arXiv preprint arXiv:2509.16941},
    year={2025}
  }
  

[bib] [paper] [data] [code]


• Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoning
  Xingang Guo, Utkarsh Tyagi, Advait Gosai, Paula Vergara, Ernesto Gabriel Hernández Montoya, Chen Bo Calvin Zhang, Bin Hu, Yunzhong He, Bing Liu, Rakshith Sharma Srinivasa
  In arXiv, 2025

  @article{guo2025beyond,
    title={Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoning},
    author={Guo, Xingang and Tyagi, Utkarsh and Gosai, Advait and Vergara, Paula and Montoya, Ernesto Gabriel Hern{\'a}ndez and Zhang, Chen Bo Calvin and Hu, Bin and He, Yunzhong and Liu, Bing and Srinivasa, Rakshith Sharma},
    journal={arXiv preprint arXiv:2510.12712},
    year={2025}
  }
  

[bib] [paper] [data]


• Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training
  Junkai Zhang, Zihao Wang, Lin Gui, Swarnashree Mysore Sathyendra, Jaehwan Jeong, Victor Veitch, Wei Wang, Yunzhong He, Bing Liu, Lifeng Jin
  In arXiv, 2025

  @article{zhang2025chasing,
    title={Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training},
    author={Zhang, Junkai and Wang, Zihao and Gui, Lin and Sathyendra, Swarnashree Mysore and Jeong, Jaehwan and Veitch, Victor and Wang, Wei and He, Yunzhong and Liu, Bing and Jin, Lifeng},
    journal={arXiv preprint arXiv:2509.21500},
    year={2025}
  }
  

[bib] [paper]


• Online Rubrics Elicitation from Pairwise Comparisons
  MohammadHossein Rezaei, Robert Vacareanu, Zihao Wang, Clinton Wang, Bing Liu, Yunzhong He, Afra Feyza Akyürek
  In arXiv, 2025

  @article{rezaei2025online,
    title={Online Rubrics Elicitation from Pairwise Comparisons},
    author={Rezaei, MohammadHossein and Vacareanu, Robert and Wang, Zihao and Wang, Clinton and He, Yunzhong and Aky{\"u}rek, Afra Feyza},
    journal={arXiv preprint arXiv:2510.07284},
    year={2025}
  }
  

[bib] [paper]


• Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains
  Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu, and Sean Hendryx
  In arXiv, 2025

  @article{gunjal2025rubrics,
    title={Rubrics as rewards: Reinforcement learning beyond verifiable domains},
    author={Gunjal, Anisha and Wang, Anthony and Lau, Elaine and Nath, Vaskar and Liu, Bing and Hendryx, Sean},
    journal={arXiv preprint arXiv:2507.17746},
    year={2025}
  }
  

[bib] [paper]


• TutorBench: A Benchmark To Assess Tutoring Capabilities Of Large Language Models
  Rakshith S Srinivasa, Zora Che, Chen Bo Calvin Zhang, Diego Mares, Ernesto Hernandez, Jayeon Park, Dean Lee, Guillermo Mangialardi, Charmaine Ng, Ed-Yeremai Hernandez Cardona, Anisha Gunjal, Yunzhong He, Bing Liu, Chen Xing
  In arXiv, 2025

  @article{srinivasa2025tutorbench,
    title={TutorBench: A Benchmark To Assess Tutoring Capabilities Of Large Language Models},
    author={Srinivasa, Rakshith S and Che, Zora and Zhang, Chen Bo Calvin and Mares, Diego and Hernandez, Ernesto and Park, Jayeon and Lee, Dean and Mangialardi, Guillermo and Ng, Charmaine and Cardona, Ed-Yeremai Hernandez and others},
    journal={arXiv preprint arXiv:2510.02663},
    year={2025}
  }
  

[bib] [paper] [data]


• MultiNRC: A Challenging and Native Multilingual Reasoning Evaluation Benchmark for LLMs
  Alexander R. Fabbri, Diego Mares, Jorge Flores, Meher Mankikar, Ernesto Hernandez, Dean Lee, Bing Liu, Chen Xing
  In arXiv, 2025

  @article{fabbri2025multinrc,
    title={MultiNRC: A Challenging and Native Multilingual Reasoning Evaluation Benchmark for LLMs},
    author={Fabbri, Alexander R and Mares, Diego and Flores, Jorge and Mankikar, Meher and Hernandez, Ernesto and Lee, Dean and Liu, Bing and Xing, Chen},
    journal={arXiv preprint arXiv:2507.17476},
    year={2025}
  }
  

[bib] [paper] [data]


• Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Models
  Lucas Bandarkar, Benjamin Muller, Pritish Yuvraj, Rui Hou, Nayan Singhal, Hongjiang Lv, Bing Liu
  In ICLR, 2024

  @article{Bandarkar2024LayerSF,
    title={Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Models},
    author={Lucas Bandarkar and Benjamin Muller and Pritish Yuvraj and Rui Hou and Nayan Singhal and Hongjiang Lv and Bing Liu},
    journal={International Conference on Learning Representations},
    year={2024}
  }
  

[bib] [paper]


• The Llama 3 Herd of Models
  Llama Team, AI @ Meta
  In arXiv, 2024

  @article{dubey2024llama,
    title={The llama 3 herd of models},
    author={Dubey, Abhimanyu and Jauhri, Abhinav and Pandey, Abhinav and Kadian, Abhishek and Al-Dahle, Ahmad and Letman, Aiesha and Mathur, Akhil and Schelten, Alan and Yang, Amy and Fan, Angela and others},
    journal={arXiv preprint arXiv:2407.21783},
    year={2024}
  }
  

[bib] [paper]


• Introducing Semantics into Speech Encoders
  Derek Xu, Shuyan Dong, Changhan Wang, Suyoun Kim, Zhaojiang Lin, Bing Liu, Akshat Shrivastava, Shang-Wen Li, Liang-Hsuan Tseng, Guan-Ting Lin, Alexei Baevski, Hung-yi Lee, Yizhou Sun, Wei Wang
  In ACL 2023

  @inproceedings{xu-etal-2023-introducing,
      title = "Introducing Semantics into Speech Encoders",
      author = "Xu, Derek  and
        Dong, Shuyan  and
        Wang, Changhan  and
        Kim, Suyoun  and
        Lin, Zhaojiang  and
        Liu, Bing  and
        Shrivastava, Akshat  and
        Li, Shang-Wen  and
        Tseng, Liang-Hsuan  and
        Lin, Guan-Ting  and
        Baevski, Alexei  and
        Lee, Hung-yi  and
        Sun, Yizhou  and
        Wang, Wei",
      booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
      year = "2023",
  }
  

[bib] [paper]


• KETOD: Knowledge-Enriched Task-Oriented Dialogue
  Zhiyu Chen, Bing Liu, Seungwhan Moon, Chinnadhurai Sankar, Paul Crook, William Yang Wang
  In NAACL 2022

  @inproceedings{chen-etal-2022-ketod,
    title = "{KETOD}: Knowledge-Enriched Task-Oriented Dialogue",
    author = "Chen, Zhiyu and Liu, Bing and Moon, Seungwhan and Sankar, Chinnadhurai and Crook, Paul and Wang, William Yang",
    booktitle = "Findings of the Association for Computational Linguistics: NAACL 2022",
    year = "2022",
  }
  

[bib] [paper] [data]

• Scale AI
Head of Research / Director (2024 - current)

• Led multimillion-dollar data roadmap across text, multimodal, reasoning, and agentic capabilities.
• Drove eval strategies for frontier models. Launched new benchmarks adopted by top AI labs.
• Built the AI research team with 30+ scientists and engineers, partnering with leading AI labs.
• Advised executive leadership on GenAI research direction and product roadmap.


• Meta (formerly Facebook)
Tech Lead and Manager (2018 - 2024)

• 2023 - 2024 @ GenAI: Led a team of research scientists and engineers on multilingual post-training and eval for Llama3. Drove product data flywheel and post-training for Meta AI i18n. Grew MAU from <100M to 700M.
• 2018 - 2022 @ Meta Reality Labs: Led a ML team to build NLU and conversational AI for Meta's first voice AI Assistant. Launched to serve millions of users on Portal, Oculus, and Ray-Ban Stories Smart Glasses.


• Google Research
Software Engineer (internship) (2017)

• Conducted foundational research in deep RL for spoken dialog systems.


• Capio.ai (Acq. by Twilio)
Software Engineer (part time) (2014 - 2016)

• Built speech infrastructure and APIs. Developed core NLU systems for Capio’s Voice AI products.


• Oracle, Singapore
Member of Technical Staff (2011 - 2013)

• Delivered the 1st Oracle Big Data Appliance solution in APAC to a major telecom client.

• University of California, Santa Cruz
Adjunct Professor (2024 - present)

• Affiliated with the Computer Science and Engineering department. Gave seminars on LLMs and NLP. Guided graduate research in LLM, and multimodal AI.

• Publication Chair: ACL 2024
• Area Chair: Large Language Models track @ ACL 2023, Senior PC: ACL, EMNLP, AAAI, ICASSP
• General Chair: NLP for Conversational AI Workshop @ ACL 2022
• Journal Reviewer: IEEE TASLP, IEEE SPL, Neural Networks, Info Systems
• Program Committee: ICLR, ACL, EMNLP, NAACL, ICASSP, Interspeech, AAAI, etc

• Carnegie Mellon University
• Ph.D. in Electrical and Computer Engineering
• Research Area: Machine Learning, Deep Learning, Natural Language Understanding, Spoken Dialog Systems.
• Thesis Committee: Ian Lane, John Paul Shen, Alexander Rudnicky, Dilek Hakkani-Tür, Antoine Raux

• Nanyang Technological University, Singapore
• B.Eng in Electrical and Electronic Engineering
• Minor Degree: Business
• First Class Honors (GPA 4.9/5.0, top 2%); President Research Scholar; Dean’s List

• KTH Royal Institute of Technology, Sweden
• Exchange student in school of Information and Communication Technology

Last updated: Oct, 2025