Bing Liu (刘冰) Head of Research @ Scale AI GenAI, LLM, RL, Agents San Francisco Bay Area PhD, Carnegie Mellon University Email: hadoopit [at] gmail dot com [Publications] [Work Experiences] [Education] [CV] @vbingliu My Github @hadoopit

I lead research at Scale AI, where I focus on evaluation infrastructure, post-training data, and agentic RL for frontier models. My work includes Humanity's Last Exam, SWE-Bench Pro, and MCP Atlas, benchmarks used across leading AI labs to assess reasoning, coding, and agentic tool-use capabilities. I work closely with frontier model developers to build scalable, high-fidelity evaluation systems and training data pipelines that advance long-horizon reasoning and agentic AI.

Previously, I led multilingual post-training for Llama 3 at Meta, helping scale Meta AI to over 700 million monthly active users across 40+ languages. Before that, I worked on conversational AI and AR/VR assistants at Meta Reality Labs and Google Research. I received my Ph.D. from Carnegie Mellon University and currently serve as Adjunct Professor at UC Santa Cruz, where I advise on large language models, evaluation, and agentic AI systems.

• 2026/05: Gave an invited talk and penal at the Stanford Graduate School of Business on "Evaluating AI: From Language Models to Agentic Systems and Beyond".
• 2026/05: We have 4 papers accepted to ICML 2026!
• 2026/04: Gave an invited talk and penal at the ICLR 2026 Workshop: Agents in the Wild on "From LLMs to Agents: The Evaluation Challenge".
• 2026/04: Gave an invited talk at the ICLR 2026 Workshop: AI with Recursive Self-Improvement on "Evaluation as the Engine for Self-Improving AI".
• 2026/04: We have 3 papers accepted to ACL 2026 the Main Conference!
• 2026/03: New benchmark release on SWE Atlas - a suite of SWE agent evals that is beyond patch generation.
• 2026/01: We have 5 papers accepted to ICLR 2026. See you in Rio!
• 2025/12: New benchmark release on MCP Atlas - a large-scale, real-server benchmark for agentic tool use. [Paper]
• 2025/12: New benchmark release on Audio MultiChallenge - a multi-turn eval for spoken dialog systems. [Paper]
• 2025/11: New paper release on ResearchRubrics - a new eval for evaluating deep research agent. [Paper]
• 2025/11: New benchmark release - Professional Reasoning Benchmark (PRBench), a benchmark for real-world professional problems like Finance and Law. [Paper]
• 2025/10: New benchmark release - Remote Labor Index, the 1st benchmark that test how well AI agents can complete real, paid freelance work. [Paper]
• 2025/10: New benchmark release - VisualToolBench, for multimodal reasoning with tool use. [Paper]
• 2025/10: 2 new papers on RL with rubrics: Chasing the Tail [paper] & Online Rubrics Elicitation [paper]
• 2025/09: New eval platform release - Showdown, a platform for evaluating frontier models in the wild with granular insights! [Leaderboard] [Technical Report]
• 2025/09: New benchmark release - SWE-Bench Pro, for evaluating LLM coding agents on real, enterprise-grade software engineering tasks. [Paper]
• 2025/09: New benchmark release - MCP Atlas, for evaluating LLM function calling and tool use in the real world! (benchmark data and paper release coming soon!)
• 2025/09: New paper release on Rubrics as Rewards - doing RL beyond verifiable domains.
• 2025/09: New benchmark release - TutorBench, for evaluating LLM's tutoring capabilities. [Paper]
• 2025/07: New benchmark release - MultiNRC, for multilingual reasoning! [Paper]
• 2025/03: New benchmark release - MultiChallenge, for multi-turn instruction following. [Paper]
• 2025/02: We released Humanity’s Last Exam, a new AI benchmark that was designed to test the limits of AI knowledge at the frontiers of human expertise. [Paper]
• 2024/11: After 6 years at Meta, I joined Scale AI to lead the applied research on LLM post-training, data, and eval!
• 2024/07: We released Llama 3.1, the latest Llama model with multilingual capability, 128K context window, and much stronger tool use and reasoning capabilities.
• 2024/03: Appointed as an Adjunct Professor at the Computer Science and Engineering department of University of California, Santa Cruz.
• 2024/01: Serving as the Publication Chair for ACL 2024.

• SciPredict: Can LLMs Predict the Outcomes of Scientific Experiments in Natural Sciences?
  Udari Madhushani Sehwag, Elaine Lau, Haniyeh Ehsani Oskouie, Shayan Shabihi, Erich Liang, Andrea Toledo, Guillermo Mangialardi, Sergio Fonrouge, Ed-Yeremai Hernandez Cardona, Paula Vergara, Utkarsh Tyagi, Chen Bo Calvin Zhang, Pavi Bhatter, Nicholas Johnson, Furong Huang, Ernesto Gabriel Hernandez Montoya, Bing Liu
  In ICML, 2026

  @article{sehwag2026scipredict,
    title={SciPredict: Can LLMs Predict the Outcomes of Scientific Experiments in Natural Sciences?},
    author={Sehwag, Udari Madhushani and Lau, Elaine and Oskouie, Haniyeh Ehsani and Shabihi, Shayan and Liang, Erich and Toledo, Andrea and Mangialardi, Guillermo and Fonrouge, Sergio and Cardona, Ed-Yeremai Hernandez and Vergara, Paula and others},
    journal={arXiv preprint arXiv:2604.10718},
    year={2026}
  }
  

[bib] [paper]


• Online Rubrics Elicitation from Pairwise Comparisons
  MohammadHossein Rezaei, Robert Vacareanu, Zihao Wang, Clinton Wang, Bing Liu, Yunzhong He, Afra Feyza Akyürek
  In ICML, 2026

  @article{rezaei2025online,
    title={Online Rubrics Elicitation from Pairwise Comparisons},
    author={Rezaei, MohammadHossein and Vacareanu, Robert and Wang, Zihao and Wang, Clinton and He, Yunzhong and Aky{\"u}rek, Afra Feyza},
    journal={arXiv preprint arXiv:2510.07284},
    year={2025}
  }
  

[bib] [paper]


• SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
  Xiang Deng, Jeff Da, Edwin Pan, Yannis Y. He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa R Kundurthy, Sean M. Hendryx, Zifan Wang, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu, Brad Kenstler
  In ICML, 2026

  @article{deng2025swe,
    title={SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?},
    author={Deng, Xiang and Da, Jeff and Pan, Edwin and He, Yannis Yiming and Ide, Charles and Garg, Kanak and Lauffer, Niklas and Park, Andrew and Pasari, Nitin and Rane, Chetan and others},
    journal={arXiv preprint arXiv:2509.16941},
    year={2025}
  }
  

[bib] [paper] [data] [code]


• Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains
  Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu, and Sean Hendryx
  In ICLR, 2026

  @article{gunjal2025rubrics,
    title={Rubrics as rewards: Reinforcement learning beyond verifiable domains},
    author={Gunjal, Anisha and Wang, Anthony and Lau, Elaine and Nath, Vaskar and Liu, Bing and Hendryx, Sean},
    journal={arXiv preprint arXiv:2507.17746},
    year={2025}
  }
  

[bib] [paper]


• Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training
  Junkai Zhang, Zihao Wang, Lin Gui, Swarnashree Mysore Sathyendra, Jaehwan Jeong, Victor Veitch, Wei Wang, Yunzhong He, Bing Liu, Lifeng Jin
  In ICLR, 2026

  @article{zhang2025chasing,
    title={Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training},
    author={Zhang, Junkai and Wang, Zihao and Gui, Lin and Sathyendra, Swarnashree Mysore and Jeong, Jaehwan and Veitch, Victor and Wang, Wei and He, Yunzhong and Liu, Bing and Jin, Lifeng},
    journal={arXiv preprint arXiv:2509.21500},
    year={2025}
  }
  

[bib] [paper]


• ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents
  Manasi Sharma, Chen Bo Calvin Zhang, Chaithanya Bandi, Clinton Wang, Ankit Aich, Huy Nghiem, Tahseen Rabbani, Ye Htet, Brian Jang, Sumana Basu, Aishwarya Balwani, Denis Peskoff, Marcos Ayestaran, Sean M. Hendryx, Brad Kenstler, Bing Liu
  In ICLR, 2026

  @article{sharma2025researchrubrics,
    title={ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents},
    author={Sharma, Manasi and Zhang, Chen Bo Calvin and Bandi, Chaithanya and Wang, Clinton and Aich, Ankit and Nghiem, Huy and Rabbani, Tahseen and Htet, Ye and Jang, Brian and Basu, Sumana and others},
    journal={arXiv preprint arXiv:2511.07685},
    year={2025}
  }
  

[bib] [paper]


• MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in Language Models, More than Outcomes
  Yu Ying Chiu, Michael S. Lee, Rachel Calcott, Brandon Handoko, Paul de Font-Reaulx, Paula Rodriguez, Chen Bo Calvin Zhang, Ziwen Han, Udari Madhushani Sehwag, Yash Maurya, Christina Q Knight, Harry R. Lloyd, Florence Bacus, Mantas Mazeika, Bing Liu, Yejin Choi, Mitchell L Gordon, Sydney Levine
  In ICLR, 2026

  @article{chiu2025morebench,
    title={MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in Language Models, More than Outcomes},
    author={Chiu, Yu Ying and Lee, Michael S and Calcott, Rachel and Handoko, Brandon and de Font-Reaulx, Paul and Rodriguez, Paula and Zhang, Chen Bo Calvin and Han, Ziwen and Sehwag, Udari Madhushani and Maurya, Yash and others},
    journal={arXiv preprint arXiv:2510.16380},
    year={2025}
  }
  

[bib] [paper] [data] [code]


• Agentic Rubrics as Contextual Verifiers for SWE Agents
  Mohit Raghavendra, Anisha Gunjal, Bing Liu, Yunzhong He
  In ACL, 2026

  @article{raghavendra2026agentic,
    title={Agentic Rubrics as Contextual Verifiers for SWE Agents},
    author={Raghavendra, Mohit and Gunjal, Anisha and Liu, Bing and He, Yunzhong},
    journal={arXiv preprint arXiv:2601.04171},
    year={2026}
  }
  

[bib] [paper]


• Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interaction
  Advait Gosai, Tyler Vuong, Utkarsh Tyagi, Steven Li, Wenjia You, Miheer Bavare, Arda Uçar, Zhongwang Fang, Brian Jang, Bing Liu, Yunzhong He
  In ACL, 2026

  @article{gosai2025audio,
    title={Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interaction},
    author={Gosai, Advait and Vuong, Tyler and Tyagi, Utkarsh and Li, Steven and You, Wenjia and Bavare, Miheer and U{\c{c}}ar, Arda and Fang, Zhongwang and Jang, Brian and Liu, Bing and others},
    journal={arXiv preprint arXiv:2512.14865},
    year={2025}
  }
  

[bib] [paper] [data]


• PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning
  Afra Feyza Akyürek, Advait Gosai, Chen Bo Calvin Zhang, Vipul Gupta, Jaehwan Jeong, Anisha Gunjal, Tahseen Rabbani, Maria Mazzone, David Randolph, Mohammad Mahmoudi Meymand, Gurshaan Chattha, Paula Rodriguez, Diego Mares, Pavit Singh, Michael Liu, Subodh Chawla, Pete Cline, Lucy Ogaz, Ernesto Hernandez, Zihao Wang, Pavi Bhatter, Marcos Ayestaran, Bing Liu, Yunzhong He
  In ACL, 2026

  @article{akyurek2025prbench,
    title={PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning},
    author={Aky{\"u}rek, Afra Feyza and Gosai, Advait and Zhang, Chen Bo Calvin and Gupta, Vipul and Jeong, Jaehwan and Gunjal, Anisha and Rabbani, Tahseen and Mazzone, Maria and Randolph, David and Meymand, Mohammad Mahmoudi and others},
    journal={arXiv preprint arXiv:2511.11562},
    year={2025}
  }
  

[bib] [paper] [data] [code]


• MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers
  Chaithanya Bandi, Ben Hertzberg, Geobio Boo, Tejas Polakam, Jeff Da, Sami Hassaan, Manasi Sharma, Andrew Park, Ernesto Hernandez, Dan Rambado, Ivan Salazar, Rafael Cruz, Chetan Rane, Ben Levin, Brad Kenstler, Bing Liu
  In arXiv, 2025
[bib] [paper] [data] [code]


• Remote Labor Index: Measuring AI Automation of Remote Work
  Mazeika, Mantas, et al. jointly by Scale AI and Center for AI Safety
  In arXiv, 2025

  @article{mazeika2025remote,
    title={Remote Labor Index: Measuring AI Automation of Remote Work},
    author={Mazeika, Mantas and Gatti, Alice and Menghini, Cristina and Sehwag, Udari Madhushani and Singhal, Shivam and Orlovskiy, Yury and Basart, Steven and Sharma, Manasi and Peskoff, Denis and Lau, Elaine and others},
    journal={arXiv preprint arXiv:2510.26787},
    year={2025}
  }
  

[bib] [paper]


• Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoning
  Xingang Guo, Utkarsh Tyagi, Advait Gosai, Paula Vergara, Ernesto Gabriel Hernández Montoya, Chen Bo Calvin Zhang, Bin Hu, Yunzhong He, Bing Liu, Rakshith Sharma Srinivasa
  In arXiv, 2025

  @article{guo2025beyond,
    title={Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoning},
    author={Guo, Xingang and Tyagi, Utkarsh and Gosai, Advait and Vergara, Paula and Montoya, Ernesto Gabriel Hern{\'a}ndez and Zhang, Chen Bo Calvin and Hu, Bin and He, Yunzhong and Liu, Bing and Srinivasa, Rakshith Sharma},
    journal={arXiv preprint arXiv:2510.12712},
    year={2025}
  }
  

[bib] [paper] [data]


• TutorBench: A Benchmark To Assess Tutoring Capabilities Of Large Language Models
  Rakshith S Srinivasa, Zora Che, Chen Bo Calvin Zhang, Diego Mares, Ernesto Hernandez, Jayeon Park, Dean Lee, Guillermo Mangialardi, Charmaine Ng, Ed-Yeremai Hernandez Cardona, Anisha Gunjal, Yunzhong He, Bing Liu, Chen Xing
  In arXiv, 2025

  @article{srinivasa2025tutorbench,
    title={TutorBench: A Benchmark To Assess Tutoring Capabilities Of Large Language Models},
    author={Srinivasa, Rakshith S and Che, Zora and Zhang, Chen Bo Calvin and Mares, Diego and Hernandez, Ernesto and Park, Jayeon and Lee, Dean and Mangialardi, Guillermo and Ng, Charmaine and Cardona, Ed-Yeremai Hernandez and others},
    journal={arXiv preprint arXiv:2510.02663},
    year={2025}
  }
  

[bib] [paper] [data]


• MultiNRC: A Challenging and Native Multilingual Reasoning Evaluation Benchmark for LLMs
  Alexander R. Fabbri, Diego Mares, Jorge Flores, Meher Mankikar, Ernesto Hernandez, Dean Lee, Bing Liu, Chen Xing
  In arXiv, 2025

  @article{fabbri2025multinrc,
    title={MultiNRC: A Challenging and Native Multilingual Reasoning Evaluation Benchmark for LLMs},
    author={Fabbri, Alexander R and Mares, Diego and Flores, Jorge and Mankikar, Meher and Hernandez, Ernesto and Lee, Dean and Liu, Bing and Xing, Chen},
    journal={arXiv preprint arXiv:2507.17476},
    year={2025}
  }
  

[bib] [paper] [data]


• Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Models
  Lucas Bandarkar, Benjamin Muller, Pritish Yuvraj, Rui Hou, Nayan Singhal, Hongjiang Lv, Bing Liu
  In ICLR, 2024

  @article{Bandarkar2024LayerSF,
    title={Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Models},
    author={Lucas Bandarkar and Benjamin Muller and Pritish Yuvraj and Rui Hou and Nayan Singhal and Hongjiang Lv and Bing Liu},
    journal={International Conference on Learning Representations},
    year={2024}
  }
  

[bib] [paper]


• The Llama 3 Herd of Models
  Llama Team, AI @ Meta
  In arXiv, 2024

  @article{dubey2024llama,
    title={The llama 3 herd of models},
    author={Dubey, Abhimanyu and Jauhri, Abhinav and Pandey, Abhinav and Kadian, Abhishek and Al-Dahle, Ahmad and Letman, Aiesha and Mathur, Akhil and Schelten, Alan and Yang, Amy and Fan, Angela and others},
    journal={arXiv preprint arXiv:2407.21783},
    year={2024}
  }
  

[bib] [paper]


• Introducing Semantics into Speech Encoders
  Derek Xu, Shuyan Dong, Changhan Wang, Suyoun Kim, Zhaojiang Lin, Bing Liu, Akshat Shrivastava, Shang-Wen Li, Liang-Hsuan Tseng, Guan-Ting Lin, Alexei Baevski, Hung-yi Lee, Yizhou Sun, Wei Wang
  In ACL 2023

  @inproceedings{xu-etal-2023-introducing,
      title = "Introducing Semantics into Speech Encoders",
      author = "Xu, Derek  and
        Dong, Shuyan  and
        Wang, Changhan  and
        Kim, Suyoun  and
        Lin, Zhaojiang  and
        Liu, Bing  and
        Shrivastava, Akshat  and
        Li, Shang-Wen  and
        Tseng, Liang-Hsuan  and
        Lin, Guan-Ting  and
        Baevski, Alexei  and
        Lee, Hung-yi  and
        Sun, Yizhou  and
        Wang, Wei",
      booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
      year = "2023",
  }
  

[bib] [paper]


• KETOD: Knowledge-Enriched Task-Oriented Dialogue
  Zhiyu Chen, Bing Liu, Seungwhan Moon, Chinnadhurai Sankar, Paul Crook, William Yang Wang
  In NAACL 2022

  @inproceedings{chen-etal-2022-ketod,
    title = "{KETOD}: Knowledge-Enriched Task-Oriented Dialogue",
    author = "Chen, Zhiyu and Liu, Bing and Moon, Seungwhan and Sankar, Chinnadhurai and Crook, Paul and Wang, William Yang",
    booktitle = "Findings of the Association for Computational Linguistics: NAACL 2022",
    year = "2022",
  }
  

[bib] [paper] [data]

• Scale AI
Head of Research / Senior Director (2024 - current)

• Defined and executed eval strategy for frontier models; launched benchmarks adopted across top AI labs, including Humanity's Last Exam, SWE-Bench Pro, MCP-Atlas, and Remote Labor Index.
• Led post-training and data roadmap across text, multimodal, reasoning, and agentic capabilities, partnering directly with OpenAI, Anthropic, Google DeepMind, and Amazon AGI.
• Scaled SEAL Labs to 40+ research scientists and engineers.
• Advised CEO and executive leadership on GenAI research direction and product roadmap.


• Meta (formerly Facebook)
Tech Lead and Manager (2018 - 2024)

• 2023 - 2024 @ GenAI: Led a team of research scientists and engineers on multilingual post-training and eval for Llama3. Drove product data flywheel and post-training for Meta AI i18n. Grew MAU from <100M to 700M across 40+ languages.
• 2018 - 2022 @ Meta Reality Labs: Led a ML team to build NLU and conversational AI for Meta's first voice AI Assistant. Launched to serve millions of users on Portal, Oculus, and Ray-Ban Stories Smart Glasses.


• Google Research
Software Engineer (internship) (2017)

• Conducted foundational research in deep RL for spoken dialog systems.


• Capio.ai (Acq. by Twilio)
Software Engineer (part time) (2014 - 2016)

• Built speech infrastructure and APIs. Developed core NLU systems for Capio’s Voice AI products.


• Oracle, Singapore
Member of Technical Staff (2011 - 2013)

• Delivered the 1st Oracle Big Data Appliance solution in APAC to a major telecom client.

• University of California, Santa Cruz
Adjunct Professor (2024 - present)

• Affiliated with the Computer Science and Engineering department. Gave seminars on LLMs and NLP. Guided graduate research in LLM, and multimodal AI.

• Publication Chair: ACL 2024
• Area Chair: Large Language Models track @ ACL 2023, Senior PC: ACL, EMNLP, AAAI, ICASSP
• General Chair: NLP for Conversational AI Workshop @ ACL 2022
• Journal Reviewer: IEEE TASLP, IEEE SPL, Neural Networks, Info Systems
• Program Committee: NeurIPS, ICLR, ACL, EMNLP, NAACL, ICASSP, Interspeech, AAAI, etc

• Carnegie Mellon University
• Ph.D. in Electrical and Computer Engineering
• Research Area: Machine Learning, Deep Learning, Natural Language Understanding, Spoken Dialog Systems.
• Thesis Committee: Ian Lane, John Paul Shen, Alexander Rudnicky, Dilek Hakkani-Tür, Antoine Raux

• Nanyang Technological University, Singapore
• B.Eng in Electrical and Electronic Engineering
• Minor Degree: Business
• First Class Honors (GPA 4.9/5.0, top 2%); President Research Scholar; Dean’s List

• KTH Royal Institute of Technology, Sweden
• Exchange student in school of Information and Communication Technology

Last updated: April, 2026