Week 8: Monday—What is AI? A Short History.

Data Literacy

What Comes to Mind?

• When you hear “Artificial Intelligence,” what do you picture?
• Where did it actually come from?

Open with discussion. Let students throw out answers: ChatGPT, robots, self-driving cars, something scarier from a movie. Point being: AI feels like it appeared overnight around 2022, but the idea is over 70 years old. We’re not at the beginning of this story, we’re in a new chapter of it. Today’s slides trace that arc and then explain what’s actually happening under the hood, without the math.

Alan Turing (1950)

• “Can machines think?”
• The Turing Test

• Turing was a mathematician, codebreaker, and early computer scientist. In 1950 he asked a simple, radical question: can machines think?
• He proposed the Turing Test as a practical stand-in for that question: if you can’t tell whether you’re talking to a human or a machine, does the difference matter? We are still arguing about this today.
• Turing describes the new form of the problem in terms of a three-person party game called the “imitation game”, in which an interrogator asks questions of a man and a woman in another room in order to determine the correct sex of the two players. Turing’s new question is: “Are there imaginable digital computers which would do well in the imitation game?”
• This question, Turing believed, was one that could actually be answered. In the remainder of the paper, he argued against the major objections to the proposition that “machines can think”.

The Term Gets Coined

• 1956, Dartmouth Summer Research Project
• “Artificial Intelligence” is named

A small group of researchers at Dartmouth coined the term “Artificial Intelligence” in 1956. They believed a machine as smart as a human was maybe 20 years away. They were off by a lot. That gap between promise and delivery repeats throughout this history, and it’s worth watching for today, too.

Decades of Slow Progress

• Rule-based systems, “if this, then that”
• “AI winters”
• The bottleneck wasn’t imagination, it was computation

Through the 1960s-80s, the dominant approach was rule-based, “expert systems” that encoded human knowledge as explicit if-then rules. Progress was real but slow. Funding would surge, then dry up when results didn’t match the hype, periods researchers now call “AI winters.” Computers simply didn’t have the memory or speed for the ideas researchers already had on paper.

AI Is a Big Umbrella

• AI: the broad goal
• Machine Learning (ML): today’s dominant approach

AI is the broad goal of building machines that do things which seem to require intelligence. Machine learning is the dominant approach to that goal today: instead of programming explicit rules, you show a system huge amounts of data and let it find the patterns itself. Almost everything called “AI” right now is ML underneath.

Machine Learning (ML)

• Supervised learning
  • Regression models
  • Convolution Neural Networks (CNNs)
  • Recurrent Neural Networks (RNNs)
• Unsupervised Learning
  • Clustering (K-means)
  • Dimensionality Reduction (PCA)

• Supervised learning is the most common form of ML today. You give the system labeled examples and it learns to predict the label for new examples. CNNs are used for images, RNNs for sequences like text or music.
  • Regression models are the simplest form of supervised learning, where you predict a continuous value based on input features. For example, predicting house prices based on square footage, number of bedrooms, etc.
  • A CNN is a type of neural network that is particularly good at processing data that has a grid-like topology, such as images. It uses convolutional layers to automatically and adaptively learn spatial hierarchies of features from input images.
    • We give the CNN a bunch of labeled images, and it learns to recognize patterns that correspond to those labels. For example, we might give it thousands of images of cats and dogs, and it will learn to distinguish between them.
  • An RNN is a type of neural network that is designed to handle sequential data, such as time series or natural language. It has connections that form directed cycles, allowing it to maintain a ‘memory’ of previous inputs in the sequence. This makes RNNs suitable for tasks like language modeling, where the context of previous words is important for predicting the next word.
• Unsupervised learning, on the other hand, involves training a model on data without explicit labels. The model tries to find hidden patterns or structures in the data.
  • Clustering algorithms like K-means group similar data points together,
  • dimensionality reduction techniques like PCA reduce the number of features in the data while preserving as much information as possible.

What Modern AI Is Actually Doing

• Predicting what comes next
• Next word, next frame, next note

A huge share of the AI you interact with is doing one thing: predicting what comes next, the next word in a sentence, the next frame in a video, the next note in a song. That’s the trick. The interesting part isn’t the idea itself, it’s what it took to make that prediction fast and good enough to feel intelligent.

The Old Way: RNNs

• Read one word at a time, in order
• Word 500 has to wait for everything before it

For years, the standard tool for “predict the next word” was the Recurrent Neural Network, or RNN. RNNs read text one word at a time, in order, carrying a memory forward as they go, like reading a sentence with a finger tracing under each word. That’s a strictly step-by-step process: word 2 has to wait for word 1, word 500 has to wait for everything before it. RNNs also tend to “forget” things from early in a long passage by the time they reach the end. This kept them slow to train and limited in size.

The Shift: Transformers (2017)

• “Attention Is All You Need”
• Look at all the words at once

A 2017 paper called “Attention Is All You Need” introduced the transformer architecture and changed the trajectory of the field. The core idea, without the math: instead of reading one word at a time in order, look at all the words at once and weigh how much each one matters to the others. We don’t need the mechanics today, what matters is what this unlocked.

Why It Was a Breakthrough

• No more waiting word by word
• Spread the work across thousands of processors
• More data and compute kept making it better

Because transformers don’t have to process words one-by-one in sequence, that work can be spread across thousands of processors running at the same time. That parallelism is the real story, it’s a computational and engineering breakthrough as much as an algorithmic one. Suddenly, throwing more data and more computing power at the problem kept making it better, which is why we now talk about “scaling.” This is the direct line from Turing’s 1950 question to the tools you use today.

So, “Is It AI?”

• Yes, technically
• But not “thinking” the way the word suggests

Yes, technically, in the sense the field has used since 1956. But it’s not “thinking” the way the term suggests to most people. It’s an extremely well-trained, extremely fast pattern-predictor, built on decades of ideas that finally had enough data and computing power behind them. Keep that in mind for everything else we discuss this semester.

How Does AI Work?

1. Tokenization
Text breaks down into standard sub-word units.

2. Embeddings
Tokens map to high-dimensional vectors.

3. Attention
Vectors are compared pair-by-pair.

“Truthfully, the bewilderment I experienced was surprising.”

Becomes:

["Truth", "fully", ",", "the", "be", "wilder", "ment", "I", ...]

Each token becomes a vector of numbers:

• OpenAI’s models use ~1,536 dimensions per token.
• A 1,000-token prompt creates 1,000 parallel vectors.

[
  [12, 1231, 234, ... 1536 numbers for Token 1],
  [89, 4321, 567, ... 1536 numbers for Token 2],
  ...
]

• Comparing two vectors is just math, a dot product
• Attention compares every token’s vector to every other token’s vector
• 1000 tokens means roughly 1,000 x 1,000 comparisons
• Double the prompt, and the work roughly quadruples
• This is why context windows have limits

We’re doing math on vectors here, not on words. Tokenization breaks text into pieces, embeddings turn each piece into a list of numbers, and attention is the step that decides how much each token’s vector should influence the others. Comparing two vectors is cheap: a dot product. But attention compares every token to every other token, so the number of comparisons grows with the square of the number of tokens. That quadratic cost, not some limit on vocabulary or meaning, is why models have a maximum context window: past a certain length, the math becomes too expensive to run. This is the flip side of the transformer breakthrough. Parallelism made the per-step math fast, but the amount of math still grows quadratically with how much text you feed it.

• In fact, we create three vectors for each token: a query vector, a key vector, and a value vector. The query vector is compared to the key vectors of all other tokens to determine how much attention to pay to each one. The value vectors are then combined according to those attention weights to produce the output for that token. This process is repeated for each token in the input sequence.
• This is a lot of calculation.