Wai Keen Vong (@wkvong) / X

Wai Keen Vong

52 posts

Wai Keen Vong

@wkvong

building neural interfaces @Meta @RealityLabs, prev: cognitive science @nyuniversity @Rutgers_Newark @UniOfAdelaide

New York City

Joined April 2009

Pinned
Wai Keen Vong
@wkvong
Feb 1, 2024
1/ Today in Science, we train a neural net from scratch through the eyes and ears of one child. The model learns to map words to visual referents, showing how grounded language learning from just one child's perspective is possible with today's AI tools. science.org/doi/10.1126/sc…
1M
Wai Keen Vong
@wkvong
Feb 1, 2024
Replying to @wkvong
6/ Results: Even with limited data, we found that the model can acquire word-referent mappings from merely tens to hundreds of examples, generalize zero-shot to new visual datasets, and achieve multi-modal alignment. Again, genuine language learning is possible from a child's
25K
Wai Keen Vong
@wkvong
Feb 1, 2024
Replying to @wkvong
5/ We ran exactly this experiment. We trained a neural net (which we call CVCL, related to CLIP by its use of a contrastive objective) on headcam video, which captured slices of what a child saw and heard from 6 to 25 months. It’s an unprecedented look at one child’s experience,
41K
Wai Keen Vong
@wkvong
Feb 1, 2024
Replying to @wkvong
10/ Finally, our findings address a classic long-standing debate in philosophy and cognitive science: What ingredients do children need to learn words? Given their everyday experience, do they (or any learner) need language-specific inductive biases or innate knowledge to get
20K
Wai Keen Vong
@wkvong
Feb 1, 2024
Replying to @wkvong
4/ To test this, what better than to train a neural network, not on enormous amounts of data from the web, but only on the input that a single child receives? What would it learn then, if anything?
GIF
20K
Wai Keen Vong
@wkvong
Feb 1, 2024
Replying to @wkvong
8/ Limitations: A typical 2-year-old's vocabulary and word learning skills are still out of reach for the current version of CVCL. What else is missing? Note that the modeling, and the data, are inherently limited compared to children’s actual experiences and capabilities. - CVCL
21K
Wai Keen Vong
@wkvong
Feb 1, 2024
Replying to @wkvong
2/ First, we are incredibly grateful for the authors of the SAYCam paper and dataset (@mcxfrank, @andyperfors, and others not on twitter), which made our work possible and we were not involved in collecting. SAYCam provides an unprecedented look at a child's egocentric,
24K
Wai Keen Vong
@wkvong
Feb 1, 2024
Replying to @wkvong
3/ Motivation: There is an enormous gap in how AI systems like GPT-4 and children learn language. The best AI systems now train on text with a word count in the trillions; it would take a child about 100K years to match that amount of data. Due to this data gap, many researchers
19K
Wai Keen Vong
@wkvong
Feb 1, 2024
Replying to @wkvong
7/ CVCL can also learn words given naturalistic experience and generalize to very different images in a more “laboratory” style evaluation. Here, note how everyday examples of butterflies in books can help support performance in vocabulary tests.
14K
Wai Keen Vong
@wkvong
Feb 1, 2024
Replying to @wkvong
9/ In ongoing work, we are excited to add components (where possible) and see how much closer models can get to child-like language learning. Our goal is to find the minimal set of ingredients to get there. Nevertheless, our work provides a new conceptual blueprint for how
13K
Wai Keen Vong
@wkvong
Feb 1, 2024
Replying to @wkvong
11/ A special thanks to my fantastic collaborators: Wentao Wang (@wentaow10), Emin Orhan and Brenden Lake (@lakebrenden), as well as all of the reviewers and commenters who provided feedback on earlier versions of this work. We're very excited to finally share this research!
11K
Wai Keen Vong
@wkvong
Nov 6, 2018
Pretty stoked to catch Cixin Liu (author of The Three-Body Problem) interviewed by @JiayangFan at the China Institute this evening! Snagged a copy of his latest book too
Wai Keen Vong
@wkvong
Feb 2, 2024
Replying to @ChrisKortge
We do use a pretrained vision model, but the pretraining is done using only the video frames from the baby, not using outside sources of data. We also train a blank slate vision model in conjunction with language and find that it performs comparably!
824
Wai Keen Vong
@wkvong
Feb 3, 2024
Replying to @gchrupala @antalvdb and @asifa_majid
Yes, currently we are piggybacking on a strong speech recognition model (human research assistants who helped transcribe the dataset!). But this is an important limitation we spell out in the paper, and we hope to address this in follow-up work to more closely mimic the learning
75