In 1969, while auditioning for a role, Loretta Mae Long was approached by representatives from a room over. “We hear you’re a teacher,” they said, and she corrected them. “I am an actress who substitute teaches on the side.”
“Perfect!”
In her next audition, Long got a whole test audience of children to jump up and perform “I’m a little teapot” with her. From then on, she was Susan Robinson, who makes her home on Sesame Street. In the series test pilot, she sings the very first iteration of the famous “one of these things is not like the others.”
If you look at the comments of this YouTube video, you’ll see that some people are incredulous that the first instance of “one of these things” is so simple. In later episodes the objects on display fall into a category into which one does not fit, but this time, the three are exactly the same. Each of them is the number two. At the end, Loretta even pulls out a fourth two, which she places over the other twos and the errant W. Then at the end she comforts the watchers, assuming they didn’t get it.
This makes sense if you realize that children don’t just have to learn about letters and numbers, they have to learn about the game itself. They have to learn what a game is. So much knowledge we take for granted is required to perform the simplest tasks.
At the beginning, a neural network is like one of these children who can’t solve the simplest puzzle. Before it’s trained, a network’s encoded knowledge is completely randomized, the machine learning equivalent of an empty brain. To succeed at an English task, it doesn’t need to learn just the task, it needs to learn English. Therefore, a model learning about, say, what emotion a sentence like “I hated this movie!” represents, requires hundreds of thousands or millions of datapoints.
But what if it didn’t start from nothing? What if, like Sesame Street did for millions of children in our nation, we could give neural networks a foundation of general knowledge before they even begin a specific task?
In 2018, ELMo came onto the AI scene and did just that. It wouldn’t change the face of the field, though, until it got help from an unlikely source.
I take requests. If you have a fictional AI and wonder how it could work, or any other topic you’d like to see me cover, mention it in the comments or on my Facebook page.
Samuel Munk 