Bing Liu (刘冰) Director of Applied Research @ Scale AI GenAI, LLM, Agents San Francisco Bay Area PhD, Carnegie Mellon University Email: hadoopit [at] gmail dot com [Publications] [Work Experiences] [Education] [CV] My Github @hadoopit

I currently lead the research team at Scale AI, where we focus on LLM post-training, reasoning, agent, and eval. At Scale, we are building the data engine for generative AI and collaborating with leading AI labs to enhance frontier models through high-quality data and evals.

My previous experience includes building Llama3 and the Meta AI Assistant at Meta GenAI. Before that, I led the NLU team at Meta Reality Labs, developing the digital assistant for AR/VR. Earlier in my career, I worked at Google Research and Capio.ai (which was acquired by Twilio).

I hold a Ph.D. in Electrical and Computer Engineering from Carnegie Mellon University (CMU), where my research centered on deep and reinforcement learning for spoken dialog systems.

I have served as an area chair and senior committee member for major NLP and ML conferences such as ACL, EMNLP, and ICLR. Additionally, I am an adjunct professor in the Computer Science department at UC Santa Cruz.

• 2025/02: We released Humanity’s Last Exam, a new AI benchmark that was designed to test the limits of AI knowledge at the frontiers of human expertise.
• 2024/11: After 6 years at Meta, I joined Scale AI to lead the applied research on LLM post-training, data, and eval!
• 2024/07: We released Llama 3.1, the latest Llama model with multilingual capability, 128K context window, and much stronger tool use and reasoning capabilities!
• 2024/03: Appointed as an Adjunct Professor at the Computer Science and Engineering department of University of California, Santa Cruz.
• 2024/01: Serving as the Publication Chair for ACL 2024.
• 2023/05: Gave an invited talk at University of California, Santa Cruz, Seminar in NLP, on "ChatGPT and Large Language Models".
• 2023/01: Serving as the Area Chair for the Large Language Models track at ACL 2023, and the Publication Chair for the 5th Workshop on NLP for Conversational AI.
• 2022/07: Serving on the Senior Program Committee for AAAI 2023 and ICASSP 2023.
• 2022/05: Serving as the General Chair to organize the 4th Workshop on NLP for Conversational AI at ACL 2022.

• Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Models
  Lucas Bandarkar, Benjamin Muller, Pritish Yuvraj, Rui Hou, Nayan Singhal, Hongjiang Lv, Bing Liu
  In ICLR, 2024

  @article{Bandarkar2024LayerSF,
    title={Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Models},
    author={Lucas Bandarkar and Benjamin Muller and Pritish Yuvraj and Rui Hou and Nayan Singhal and Hongjiang Lv and Bing Liu},
    journal={International Conference on Learning Representations},
    year={2024}
  }
  

[bib] [paper]


• The Llama 3 Herd of Models
  Llama Team, AI @ Meta
  In arXiv, 2024

  @article{dubey2024llama,
    title={The llama 3 herd of models},
    author={Dubey, Abhimanyu and Jauhri, Abhinav and Pandey, Abhinav and Kadian, Abhishek and Al-Dahle, Ahmad and Letman, Aiesha and Mathur, Akhil and Schelten, Alan and Yang, Amy and Fan, Angela and others},
    journal={arXiv preprint arXiv:2407.21783},
    year={2024}
  }
  

[bib] [paper]


• Introducing Semantics into Speech Encoders
  Derek Xu, Shuyan Dong, Changhan Wang, Suyoun Kim, Zhaojiang Lin, Bing Liu, Akshat Shrivastava, Shang-Wen Li, Liang-Hsuan Tseng, Guan-Ting Lin, Alexei Baevski, Hung-yi Lee, Yizhou Sun, Wei Wang
  In ACL 2023

  @inproceedings{xu-etal-2023-introducing,
      title = "Introducing Semantics into Speech Encoders",
      author = "Xu, Derek  and
        Dong, Shuyan  and
        Wang, Changhan  and
        Kim, Suyoun  and
        Lin, Zhaojiang  and
        Liu, Bing  and
        Shrivastava, Akshat  and
        Li, Shang-Wen  and
        Tseng, Liang-Hsuan  and
        Lin, Guan-Ting  and
        Baevski, Alexei  and
        Lee, Hung-yi  and
        Sun, Yizhou  and
        Wang, Wei",
      booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
      year = "2023",
  }
  

[bib] [paper]


• KETOD: Knowledge-Enriched Task-Oriented Dialogue
  Zhiyu Chen, Bing Liu, Seungwhan Moon, Chinnadhurai Sankar, Paul Crook, William Yang Wang
  In NAACL 2022

  @inproceedings{chen-etal-2022-ketod,
    title = "{KETOD}: Knowledge-Enriched Task-Oriented Dialogue",
    author = "Chen, Zhiyu and Liu, Bing and Moon, Seungwhan and Sankar, Chinnadhurai and Crook, Paul and Wang, William Yang",
    booktitle = "Findings of the Association for Computational Linguistics: NAACL 2022",
    year = "2022",
  }
  

[bib] [paper] [data]


• Zero-Shot Dialogue State Tracking via Cross-Task Transfer
  Zhaojiang Lin, Bing Liu, Andrea Madotto, Seungwhan Moon, Zhenpeng Zhou, Paul Crook, Zhiguang Wang, Zhou Yu, Eunjoon Cho, Rajen Subba, Pascale Fung
  In EMNLP 2021

  @inproceedings{lin-etal-2021-zero-shot-dst,
    title={Zero-Shot Dialogue State Tracking via Cross-Task Transfer},
    author={Lin, Zhaojiang and Liu, Bing and Madotto, Andrea and Moon, Seungwhan and Zhou, Zhenpeng and Crook, Paul and Wang, Zhiguang and Yu, Zhou and Cho, Eunjoon and Fung, Pascale and Subba, Rajen},
    journal={Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
    year={2021}
  }
  

[bib] [paper] [code]


• Continual Learning in Task-Oriented Dialogue Systems
  Andrea Madotto, Zhaojiang Lin, Zhenpeng Zhou, Seungwhan Moon, Paul Crook, Bing Liu, Zhou Yu, Eunjoon Cho, Zhiguang Wang
  In EMNLP 2021

  @inproceedings{madotto2020continual,
    title={Continual Learning in Task-Oriented Dialogue Systems},
    author={Madotto, Andrea and Lin, Zhaojiang and Zhou, Zhenpeng and Moon, Seungwhan and Crook, Paul and Liu, Bing and Yu, Zhou and Cho, Eunjoon and Wang, Zhiguang},
    journal={Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
    year={2021}
  }
  

[bib] [paper]


• NUANCED: Natural Utterance Annotation for Nuanced Conversation with Estimated Distributions
  Zhiyu Chen, Honglei Liu, Hu Xu, Seungwhan Moon, Hao Zhou, Bing Liu
  In EMNLP 2021

  @inproceedings{chen2020nuanced,
    title={NUANCED: Natural Utterance Annotation for Nuanced Conversation with Estimated Distributions},
    author={Chen, Zhiyu and Liu, Honglei and Xu, Hu and Moon, Seungwhan and Zhou, Hao and Liu, Bing},
    journal={Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
    year={2021}
  }
  

[bib] [paper] [data]


• Leveraging Slot Descriptions for Zero-Shot Cross-Domain Dialogue State Tracking
  Zhaojiang Lin, Bing Liu, Seungwhan Moon, Paul Crook, Zhenpeng Zhou, Zhiguang Wang, Zhou Yu, Andrea Madotto, Eunjoon Cho, Rajen Subba
  In NAACL 2021

  @inproceedings{lin-etal-2021-leveraging,
    title={Leveraging Slot Descriptions for Zero-Shot Cross-Domain Dialogue State Tracking},
    author={Lin, Zhaojiang and Liu, Bing and Moon, Seungwhan and Crook, Paul and Zhou, Zhenpeng and Wang, Zhiguang and Yu, Zhou and Madotto, Andrea and Cho, Eunjoon and Subba, Rajen},
    journal={Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
    year={2021}
  }
  

[bib] [paper] [code]


• Adding Chit-Chats to Enhance Task-Oriented Dialogues
  Kai Sun, Seungwhan Moon, Paul Crook, Stephen Roller, Becka Silvert, Bing Liu, Zhiguang Wang, Honglei Liu, Eunjoon Cho, and Claire Cardie
  In NAACL 2021

  @inproceedings{sun-etal-2021-adding,
    title={Adding Chit-Chat to Enhance Task-Oriented Dialogues},
    author={Sun, Kai and Moon, Seungwhan and Crook, Paul and Roller, Stephen and Silvert, Becka and Liu, Bing and Wang, Zhiguang and Liu, Honglei and Cho, Eunjoon and Cardie, Claire},
    booktitle={Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
    year={2021}
  }
  

[bib] [paper] [code]


• Analyzing the Forgetting Problem in the Pretrain-Finetuning of Dialogue Response Models
  Tianxing He, Jun Liu, Kyunghyun Cho, Myle Ott, Bing Liu, James Glass, Fuchun Peng
  In EACL 2021

  @inproceedings{he-etal-2021-analyzing,
    title={Analyzing the Forgetting Problem in Pretrain-Finetuning of Open-domain Dialogue Response Models},
    author={He, Tianxing and Liu, Jun and Cho, Kyunghyun and Ott, Myle and Liu, Bing and Glass, James and Peng, Fuchun},
    booktitle={Proceedings of EACL 2021},
    year={2021}
  }
  

[bib] [paper]


• User Memory Reasoning for Conversational Recommendation
  Hu Xu, Seungwhan Moon, Honglei Liu, Bing Liu, Pararth Shah, Bing Liu, Philip S Yu
  In COLING 2020

  @inproceedings{xu-etal-2020-user,,
    title={User Memory Reasoning for Conversational Recommendation},
    author={Xu, Hu and Moon, Seungwhan and Liu, Honglei and Liu, Bing and Shah, Pararth and Yu, Philip S},
    booktitle={Proceedings of the 28th International Conference on Computational Linguistics},
    year={2020}
  }
  

[bib] [paper]


• Reinforcement Learning of Multi-Domain Dialog Policies Via Action Embeddings
  Jorge A. Mendez, Alborz Geramifard, Mohammad Ghavamzadeh, Bing Liu
  In NeurIPS 2019 Workshop on Conversational AI.

  @inproceedings{mendez2019reinforcement,
    title={Reinforcement Learning of Multi-Domain Dialog Policies Via Action Embeddings},
    author={Mendez, Jorge A and Geramifard, Alborz and Ghavamzadeh, Mohammad and Liu, Bing},
    booktitle={3rd Workshop on Conversational AI, NeurIPS},
    year={2019}
  }
  

[bib] [paper]


• Learning Task-Oriented Dialog with Neural Network Methods
  Bing Liu
  PhD Thesis

  @phdthesis{liu2018learning,
    title={Learning Task-Oriented Dialog with Neural Network Methods},
    author={Liu, Bing},
    year={2018},
    school={Carnegie Mellon University}
  }
  

[bib] [thesis]


• Adversarial Learning of Task-Oriented Neural Dialog Models
  Bing Liu, Ian Lane.
  In SIGDIAL 2018.

  @inproceedings{liu2018adversarial,
    author    = {Bing Liu and Ian Lane},
    title     = {Adversarial Learning of Task-Oriented Neural Dialog Models},
    booktitle = {SIGDIAL},
    year      = {2018}
  }
  

[bib] [paper]


• Dialogue Learning with Human Teaching and Feedback in End-To-End Trainable Task-Oriented Dialogue Systems
  Bing Liu, Gokhan Tur, Dilek Hakkani-Tur, Pararth Shah, and Larry Heck
  In NAACL 2018.

  @inproceedings{liu2018dialogue,
    author={Liu, Bing and Tur, Gokhan and Hakkani-Tur, Dilek and Shah, Pararth and Heck, Larry}, 
    title     = {Dialogue Learning with Human Teaching and Feedback in End-To-End Trainable Task-Oriented Dialogue Systems},
    booktitle = {NAACL},
    year      = {2018}
  }
  

[bib] [paper] [poster]


• Bootstrapping a Neural Conversational Agent with Dialogue Self-Play, Crowdsourcing and On-Line Reinforcement Learning
  Pararth Shah, Dilek Hakkani-Tur, Bing Liu, and Gokhan Tur
  In NAACL 2018.

  @inproceedings{shah2018bootstrapping,
    author={Shah, Pararth and Hakkani-Tur, Dilek and Liu, Bing and Tur, Gokhan}, 
    title     = {Bootstrapping a Neural Conversational Agent with Dialogue Self-Play, Crowdsourcing and On-Line Reinforcement Learning},
    booktitle = {NAACL},
    year      = {2018}
  }
  

[bib] [paper]


• End-to-End Learning of Task-Oriented Dialogs
  Bing Liu, Ian Lane.
  In NAACL 2018 SRW (thesis proposal paper).

  @inproceedings{liu2018e2e,
    author={Liu, Bing and Lane, Ian}, 
    title     = {End-to-End Learning of Task-Oriented Dialogs},
    booktitle = {NAACL Student Research Workshop},
    year      = {2018}
  }
  

[bib] [paper] [slides] [video]


• Customized Nonlinear Bandits for Online Response Selection in Neural Conversation Models
  Bing Liu, Tong Yu, Ian Lane, and Ole Mengshoel.
  In AAAI 2018.

  @inproceedings{liu2018customized,
    author    = {Liu, Bing and Yu, Tong and Lane, Ian and Mengshoel, Ole},
    title     = {Customized Nonlinear Bandits for Online Response Selection in Neural Conversation Models},
    booktitle = {AAAI},
    year      = {2018}
  }
  

[bib] [paper]


• Iterative Policy Learning in End-to-End Trainable Task-Oriented Neural Dialog Models
  Bing Liu, Ian Lane.
  In ASRU 2017.

  @inproceedings{liu2017iterative,
    author    = {Bing Liu and Ian Lane},
    title     = {Iterative Policy Learning in End-to-End Trainable Task-Oriented Neural Dialog Models},
    booktitle = {Proceedings of 2017 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU)},
    year      = {2017}
  }
  

[bib] [paper]


• End-to-End Optimization of Task-Oriented Dialogue Model with Deep Reinforcement Learning
  Bing Liu, Gokhan Tur, Dilek Hakkani-Tur, Pararth Shah, and Larry Heck
  In NIPS 2017 Workshop on Conversational AI.

  @inproceedings{liu2017e2e,
    title={End-to-End Optimization of Task-Oriented Dialogue Model with Deep Reinforcement Learning},
    author={Liu, Bing and Tur, Gokhan and Hakkani-Tur, Dilek and Shah, Pararth and Heck, Larry}, 
    booktitle={NIPS Workshop on Conversational AI},
    year={2017}
  }
  

[bib] [paper]


• Multi-Domain Adversarial Learning for Slot Filling in Spoken Language Understanding
  Bing Liu, Ian Lane.
  In NIPS 2017 Workshop on Conversational AI.

  @inproceedings{liu2017multi,
    title={Multi-Domain Adversarial Learning for Slot Filling in Spoken Language Understanding},
    author={Liu, Bing and Lane, Ian},
    booktitle={NIPS Workshop on Conversational AI},
    year={2017}
  }
  

[bib] [paper] [slides]


• An End-to-End Trainable Neural Network Model with Belief Tracking for Task-Oriented Dialog
  Bing Liu, Ian Lane.
  In Interspeech 2017.

  @inproceedings{Liu2017,
    author={Bing Liu and Ian Lane},
    title={An End-to-End Trainable Neural Network Model with Belief Tracking for Task-Oriented Dialog},
    year={2017},
    booktitle={Interspeech 2017}
  }
  

[bib] [paper] [slides]


• Dialog Context Language Modeling with Recurrent Neural Networks
  Bing Liu, Ian Lane.
  In ICASSP 2017.

  @inproceedings{liu-lane:2017:ICASSP,
    title={Dialog Context Language Modeling with Recurrent Neural Networks},
    author={Liu, Bing  and  Lane, Ian},
    booktitle={Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference on},
    year={2017},
    organization={IEEE}
  }
  

[bib] [paper]


• Joint Online Spoken Language Understanding and Language Modeling With Recurrent Neural Networks
  Bing Liu, Ian Lane.
  In SIGDIAL 2016.

  @InProceedings{liu-lane:2016:SIGDIAL,
    author    = {Liu, Bing  and  Lane, Ian},
    title     = {Joint Online Spoken Language Understanding and Language Modeling With Recurrent Neural Networks},
    booktitle = {Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue},
    year      = {2016}
  }
  

[bib] [paper] [slides] [video] [code]


• Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling
  Bing Liu, Ian Lane.
  In Interspeech 2016.

  @inproceedings{Liu+2016,
    author={Bing Liu and Ian Lane},
    title={Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling},
    year=2016,
    booktitle={Interspeech 2016}
  }
  

[bib] [paper] [code]


• Recurrent Neural Network Structured Output Prediction for Spoken Language Understanding
  Bing Liu, Ian Lane.
  In NIPS Workshop 2015.

  @inproceedings{liu2015recurrent,
    title={Recurrent Neural Network Structured Output Prediction for Spoken Language Understanding},
    author={Liu, Bing and Lane, Ian},
    booktitle={Proc. NIPS Workshop on Machine Learning for Spoken Language Understanding and Interactions},
    year={2015}
  }
  

[bib] [paper]

• Scale AI
Director of Research (2024 - current)

• Built and scaled the applied research team with 40+ research scientists and engineers responsible for GenAI data and eval, partnering with top-tier AI labs.
• Led multimillion-dollar data roadmap across text, multimodal, reasoning, and agentic capabilities. Drove eval strategies for frontier models. Launched new leaderboards adopted by top AI labs.
• Advised executive leadership on GenAI research direction and product roadmap.


• Meta (formerly Facebook)
Tech Lead and Manager (2018 - 2024)

• 2023 - 2024 @ GenAI: Led a team of research scientists and engineers on multilingual post-training and eval for Llama3. Drove product data flywheel and post-training for Meta AI i18n. Grew MAU from <100M to 700M.
• 2018 - 2022 @ Meta Reality Labs: Led a ML team to build NLU and conversational AI for Meta's first voice AI Assistant. Launched to serve millions of users on Portal, Oculus, and Ray-Ban Stories Smart Glasses.


• Google Research
Software Engineer (internship) (2017)

• Conducted foundational research in deep RL for spoken dialog systems.


• Capio.ai (Acq. by Twilio)
Software Engineer (part time) (2014 - 2016)

• Built speech infrastructure and APIs. Developed core NLU systems for Capio’s Voice AI products.


• Oracle, Singapore
Member of Technical Staff (2011 - 2013)

• Delivered the 1st Oracle Big Data Appliance solution in APAC to a major telecom client.

• University of California, Santa Cruz
Adjunct Professor (2024 - present)

• Affiliated with the Computer Science and Engineering department. Gave seminars on LLMs and NLP. Guided graduate research in LLM, and multimodal AI.

• Publication Chair: ACL 2024
• Area Chair: Large Language Models track @ ACL 2023, Senior PC: ACL, EMNLP, AAAI, ICASSP
• General Chair: NLP for Conversational AI Workshop @ ACL 2022
• Journal Reviewer: IEEE TASLP, IEEE SPL, Neural Networks, Info Systems
• Program Committee: ACL, EMNLP, NAACL, ICASSP, Interspeech, AAAI, etc

• Carnegie Mellon University
• Ph.D. in Electrical and Computer Engineering
• Research Area: Machine Learning, Deep Learning, Natural Language Understanding, Spoken Dialog Systems.
• Thesis Committee: Ian Lane, John Paul Shen, Alexander Rudnicky, Dilek Hakkani-Tür, Antoine Raux

• Nanyang Technological University, Singapore
• B.Eng in Electrical and Electronic Engineering
• Minor Degree: Business
• First Class Honors (GPA 4.9/5.0, top 2%); President Research Scholar; Dean’s List

• KTH Royal Institute of Technology, Sweden
• Exchange student in school of Information and Communication Technology

Last updated: May, 2025