YI REN
≡
profile photo

Yi (Joshua) Ren

Hey, I am a Ph.D. (2020-now) student who is working on machine learning under the supervision of Prof. Danica J. Sutherland at the University of British Columbia (UBC). I am trying to figure out how to train a model that generalizes well systematically. This idea originates from my master's study at the University of Edinburgh (2018-2019) with Prof. Simon Kirby and Prof. Shay Cohen. We find that not only humans, but the neural network might also prefer the highly compositional mapping when trained under different tasks. Such a preference might make the learned representations become more compositional if we train the model iteratively. To further explore this interesting phenomenon, I visited Prof. Aaron Courville's group at Mila, during which we find that "soft-discretization" might play an important role. In the remainder of my Ph.D. study, I hope we can build a theoretical model to explain the aforementioned preference during neural network training, and also design more efficient algorithms for real applications. Recently, I surprisingly found that the Bayesian iterated learning framework (a hypothetic framework depicting human cultural evolution used in cognitive science) has the potential to approximate the behavior of the evolution of large language models (LLMs). Hope this theoretical framework can bring valuable insights to the LLM era!

Email  /  Google Scholar  /  GitHub  /  Twitter  /  CV

UBC Machine Learning  /  MILD  /  AML-TN  / 

News

  • 01/2024, one paper accepted by ICLR-2024, depicting sample difficulty in NTK space.

  • 09/2023, one paper accepted by NeurIPS-2023, iterated learning is helpful in representation learning.

  • 09/2023, start an internship at Borealis AI, working on time series prediction project.

  • 02/2023, one paper accepted by ICLR-2023, leave enough energy for feature adaptation.

  • 12/2022, one paper presented in the 1st workshop on interpolation and beyond at NeurIPS-2022.

  • 08/2022, visiting Professor Aaron Courville's group at Mila for 4 months, really enjoy living in Montreal.

  • 02/2022, two papers are accepted by ICLR-2022, code and camera-ready will be released soon.

  • 10/2021, finally, after one year's waiting, I arrived in Vancouver to start my 2nd-year Ph.D. study.

    • Highlight Topics

    I believe "clustering" my work by topics can provide a good overview of my research interest. Here are some.

  • Neural Iterated Learning

  • Emergent Communication

  • Representation Learning

    • Talks

  • 03/2024 Happy to give a talk at Chalmers University of Technology about the application and understanding of neural iterated learning. (Slides)

    • Publications

    Preprints:

    1. Language Model Evolution: An Iterated Learning Perspective
      Yi Ren, Shangmin Guo, Linlu Qiu, Bailin Wang, Danica J. Sutherland
      Preprint 2024 | pdf | code

    2. Economics arena for large language models
      Shangmin Guo, Haoran Bu, Haochuan Wang, Yi Ren, Dianbo Sui, Yuming Shang, Siting Lu
      Preprint 2024 | pdf

    3. AdaFlood: Adaptive Flood Regularization
      Wonho Bae, Yi Ren, Mohamad Osama Ahmed, Frederick Tung, Danica J Sutherland, Gabriel L Oliveira
      Preprint 2023 | pdf

    Journal and Low-Acceptance-Rate Conference Papers:

    1. lpNTK: Better Generalisation with Less Data via Sample Interaction During Learning
      Shangmin Guo, Yi Ren, Stefano V. Albrecht, Kenny Smith
      ICLR 2024 | pdf

    2. Improving Compositional Generalization using Iterated Learning and Simplicial Embeddings
      Yi Ren, Samuel Lavoie, Mikhail Galkin, Danica J. Sutherland, Aaron Courville
      NeurIPS 2023 | pdf | code

    3. How to prepare your task head for finetuning
      Yi Ren, Shangmin Guo, Wonho Bae, Danica J. Sutherland
      ICLR 2023 | pdf | code

    4. Better Supervisory Signals by Observing Learning Paths
      Yi Ren, Shangmin Guo, Danica J. Sutherland
      ICLR 2022 | pdf | code

    5. Expressivity of Emergent Language is a Trade-off between Contextual Complexity and Unpredictability
      Shangmin Guo, Yi Ren, Kory Mathewson, Simon Kirby, Stefano V. Albrecht, Kenny Smith
      ICLR 2022 | pdf | code | workshop-version

    6. Compositional languages emerge in a neural iterated learning model
      Yi Ren, Shangmin Guo, Matthieu Labeau, Shay B. Cohen, Simon Kirby
      ICLR 2020 | pdf | code | workshop-version

    7. The Emergence of Compositional Languages for Numeric Concepts Through Iterated Learning in Neural Agents
      Shangmin Guo, Yi Ren, Serhii Havrylov, Stella Frank, Ivan Titov, Kenny Smith
      EmeCom@NeurIPS 2019 | pdf

    Welcome to my home page *^_^*
    No. Visitor Since Jan 2022. Powered by w3.css