Matthew Rascoff: Hello, everyone. My name is Matthew Rascoff. I'm Vice Provost for Digital Education at Stanford University, and I'm so glad to be joining you and your colleagues at Tec de Monterrey today. If you're watching this recording, it means something went wrong with our technology setup, so I'm sorry about that, but I'm glad we have this backup plan in place, and I look forward to engaging you today and in the future as well. I've also been promised a chance to visit the campus, which I've heard many wonderful things about over the years, and I hope we can find a way to do that sometime — maybe after this pandemic is over, God willing, one day.
So today I want to talk about high-touch learning at scale, and the way I want to do that is through an analogy from architecture and urban design. One of the favorite classes that I took at Columbia University, where I was an undergrad, was an architectural history class with Professor Barry Bergdoll, and he turned me on to the idea of urban design as something that was about more than just cities — it was really about how we choose to live together, and the fundamental questions about communities and learning as well. And that's kind of the starting point for my talk today, which was inspired by my own education in art history and architectural history, but brings it together with work that I've done over the past two decades in digital learning and online learning, and the way that technology can support learning and our educational values.
So let me start with a bit of a history lesson. This is a picture of Denver, Colorado, in the US in 1925. It was a thriving Western city at the time, with buildings, as you can see, of many different sizes and shapes. Pay special attention to the clock tower in the center of this picture here — that's going to take on some importance. That's the Daniels and Fisher Tower, which was part of a department store of the same name, built in 1910 to a height of 325 feet. It was the tallest building in that era between the Mississippi River and California, and it was modeled after the Campanile in Piazza San Marco in Venice.
Now, fast forward a few decades later. In the subsequent years after the 1920s, downtown Denver went into a steep decline that was driven by new technologies. In this case it was the automobile and trucks that led to the suburbanization of industry and the consolidation of the small workshops that were depicted in that previous picture into larger factories by the 1970s. So 50 years after that first picture was taken, the Denver Urban Renewal Authority took control of many of those small lots that fell into disrepair. They demolished many of the old buildings, and they repackaged the land into larger parcels and then sold them to office developers. Many of those original buildings were just one- to three-story warehouses and light industrial, but as you can see, the new buildings that were constructed were these really gigantic office towers.
In the subsequent years the redevelopment effort basically failed. The Urban Renewal Authority cleared far more land than was needed for these developments, and many of those offices had already moved to the suburbs. Meanwhile, these towers in downtown Denver created these huge parts of the city that were basically empty at night, and as you can see, in their wake were these gigantic parking lots that were kind of wastelands in terms of what we expect from a city, the life of a city.
In his 1971 speech "The Room, the Street, and Human Agreement," the architect Lou Kahn offered an alternative vision for urban planning to what you saw happen in Denver, and of course in many other cities in the US and around the world in the 20th century. Kahn was an architect and an educator. He was a professor of architecture and taught at the University of Pennsylvania for many years. He was not a professional city planner, and I think that's important context. In his 1971 speech, his vision — which he later published — was a statement of his philosophy about architecture, and it was rooted in individuals, or more precisely, as you can see here, in dialogue among individuals. That's what's happening there — there's two people, it's a little bit hard to see in this picture, two people talking. And he designed for the human scale, not from the city scale down, but from that conversation, those two people engaging one another, out and up from there.
Kahn said in his speech: "The room is the beginning of architecture. It is the place of mind. You're in the room, with its dimensions, its structure, its light, responds to its character, its spiritual aura, recognizing that whatever the human proposes and makes can become a life." The next level up in Kahn's vision, from the room, was the street. The street, as he described it in his speech, was a room for the whole community. He said: "The street is a room of agreement. The street is dedicated by each house owner to the city in exchange for common services. Dead-end streets in cities still retain this room character, but through streets, since the advent of the automobile" — he was saying in 1971 — "have entirely lost their room quality. I believe that city planning can start with the realization of this loss by directing the driver, and to reinstate the street, where people live, learn, shop, and work out, as the room out of commonality."
So from the room to the street, and then the next level up in his vision of designing from the small to the medium to the large is the city. Kahn said in his 1971 speech: "From a simple settlement, the city becomes the place of assembled institutions. The settlement was the first institution. The talents found their places. The carpenter directed building, the thoughtful person became the teacher, the strong one the leader. The desire to learn made the first schoolroom. It was of human agreement. The institution became the modus operandi, and the agreement has the immediacy of rapport, the inspiring force which recognized its commonality and that it must be part of the human way of life supported by all people."
So what Kahn is doing here, in his alternative vision to that urban renewal model, is building up from the smallest unit of space, the room, to the largest. And with each step comes greater organization, greater sophistication, and greater specialization. Yet there is a fundamental unity in his design, which preserves the integrity of the room even as it scales up. It scales in a cellular, modular, almost fractal format.
So I want to share a little bit about, you know, what that might mean at the city scale. This is an image also made by Louis Kahn — not drawn from that speech, not published; it actually was produced earlier in his career, and it's more of a theoretical statement. It's an urban plan for the city of Philadelphia, and it's kind of an alternative vision for how transportation might work in a city. So in his plan for Philadelphia, which was, you know, connected to the philosophical ideas of the 1971 speech, he proposed a new traffic pattern, and that was really the key idea, as represented in this image.
I want to quote just from the catalog of the Museum of Modern Art, which owns this drawing, just to describe what you're seeing here and to kind of give it a little bit of a description, because it's a technical drawing that's a little bit hard to understand, and there's no text. The catalog entry says: "To untangle traffic congestion and to mitigate the proliferation of parking lots that plagued postwar cities, Kahn reordered the streets according to a functional hierarchy. The drawing's notational system corresponds to different tempos of traffic, such as the stop-and-go movement of trucks" — those are the dotted lines — "the fast flow of vehicles around the periphery" — those are the arrows at the edge — "and the stasis of cars in parking garages" — those are the spirals, also at the edge of the image here. "To explain his movement, he invoked a historical analogy: the girdle of expressways and parking towers circling the city center metaphorically recalled the walls and towers that protected the medieval cities of Europe."
So this was how a city could play out. This is how movement could be organized in a city that was designed in that kind of bottom-up fashion, starting from the room to the street and then to the city level. This is what it looked like in the plan for a real city. Of course, it was never enacted in Philadelphia, but this was his vision for how a city could work. Lou Kahn was mostly ignored in his era from the perspective of urban design, and what he created for Philadelphia was dismissed, actually, by the city government there. But he wrote in 1971 that we can begin by planting trees on all existing residential streets, by redefining the order of movement, which would give these streets back to more intimate use, which would stimulate the feelings of well-being and unique street expression. The street is a community room.
And I want to posit that his ideas have become really important, actually, to architecture and to urbanism, because 50 years after his death, a park that he designed in New York — the Four Freedoms Park — was actually built, even though he, of course, was long gone; he died in the 1970s. And cities around the world are redesigning streets to look much more like his vision of a shared room. You can see a picture, you know, at the top of a street dedicated basically to the movement of cars, and then look at the picture at the bottom of the same street, a street in Paris which was redesigned and reopened according very much to the philosophy that Kahn described, of planting trees and reclaiming the street as a community room.
So why am I sharing this extended analogy from architecture and urban planning? Because I want to argue today that Kahn's ideas are important for how we think about spaces of learning — both digital and physical spaces of learning — and the learning communities that those spaces can either support or, like urban renewal, they can undermine. So let's bring this back to the design of learning spaces. I believe there's really an important lesson for us in the way we think about learning spaces and the institutions and organizations that provide them. In the 20th century, they were responding to the advent of the automobile. That was the technological force that brought about the urban renewal movement in Colorado and Denver, and the response from Lou Kahn. Today, of course, the technologies are different, but I would argue that some of the design considerations are similar in the space of learning.
Take the example of Coursera, just to get us started, which is a sort of digital skyscraper, and you can see that kind of almost represented in this chart here. Coursera is a giant platform that in 2022 passed 100 million learners, perhaps second only to YouTube in the size of its user base. But each of those learners effectively travels in a private vehicle with little communication or interaction with others. They work in basically a cubicle of learning, a self-paced isolation from their fellow learners — hardly a community room for learning. If it were a physical classroom, Coursera might be represented as a gigantic amphitheater. They've essentially recreated the technologies of the private automobile and the skyscraper, and all of the isolation that's associated with them.
Meanwhile, the traditional providers of higher education in the United States, at least — and that's where these data are drawn from — are in steep decline. The growth of giant online learning platforms, with the potential for winner-take-all economies of scale, puts smaller-scale institutions at risk. The process accelerated during the pandemic, but it was already well underway, as you can see from these data, before it. This is the percentage change in enrollments in different sectors of higher education in the United States. So to me, I think the question for us is: will smaller schools be swallowed up? Will they be demolished and turned into the institutional equivalent of parking lots? Will the smaller-scale model of learning — that kind of one- to two-story building that you saw represented in the first image of Denver, the 1925 image — will that model become obsolete? And will the walkable, mixed-use educational neighborhoods give way to digital skyscrapers and internet highways?
I want to offer today an alternative and more optimistic vision for the future, in which diversity and pluralism and design at different scales is able to survive — we're able to retain the community benefits of those smaller spaces for learning even as we gain some of the cost economies of scale by serving larger numbers of students, of learners. And it's based on a model from a project at Stanford that was developed during the pandemic called Code in Place. That model is designed to be bigger than the face-to-face residential experience of Stanford, but much higher touch than a MOOC. I'm going to let the faculty leader of Code in Place, Professor Chris Piech of the Stanford CS Department, describe it to you in his own words, and then I want to talk a little bit about what I think it means for how we think about design and how we support students and learners.
Chris Piech: Hi, I'm Chris. I'm an associate professor of computer science, and I have the wonderful privilege of being part of this great team working on education in the context of a MOOC. So sometimes when people hear AI and education, they might think we're going to have fewer teachers, or machines that'll be teaching us, and there's two responses to that. First, we just learned that the feeling, and mattering so much, just can't replace the sort of mentorship and change in identity you get from interacting with another person. Teaching and learning is one of the most social things we do; it's a defining characteristic of humans. Even if AI was fantastic, it just doesn't really hold a candle to the sorts of impact a real human being could have.
But on the other hand, we've learned something else during this pandemic ourselves. Schoolhouse.world — we talked to a lot of people, where they get volunteer teachers to come join in the process, and we all collectively learned how astounding it is, the magnitude of people who want to teach. So to that extent, our work has really been to augment this social learning experience. How could AI make students better at learning and teachers better at communicating knowledge? And then also, how could they augment a collaborative experience between students and teachers?
At the start of Code in Place, my fellow teachers and I were about to start a new term. A few of us came together and thought, how could we make this transition an opportunity to give back? We weren't doctors, we couldn't help in the hospitals, but we were teachers. Maybe we could bring that as some good for society. Because we've spent decades on our teaching, and because we already started the process of developing useful AI tools for students, we were ready to offer a pretty wonderful, high-quality experience. So we ran a course — we called it Code in Place — but it really taught the first two of our Introduction to Programming. And we were able to use tools, AI techniques that can help teachers get feedback on their own teaching process, and we could do it in the context of the human, community-centered education that we believe in.
The interesting thing about this research is, often we would have just done this — we think about the research, and then maybe in 10 years we would think about how we could deploy it. But because of the pandemic, we deployed immediately. We did two things. We gave feedback to thousands of students, on an exam where they would have otherwise not been given feedback. We gave feedback to all these students — 16,000 pieces of feedback. 90% of them got feedback from AI, 10% got feedback from humans, and they actually rated the AI feedback as more helpful. On the other hand, we had 2,000 teachers who we had to train. For these 2,000 teachers, we first gave them the standard teacher training that you'd expect, but we also developed a system where they look at their process of teaching students, take their transcripts, find the moments where they revoiced students, find the moments where they had missed what a student had said, and use that to give them automatic feedback on their teaching. And this led to things like them asking more questions, revoicing what the students had more, and the students who were the beneficiaries of this came to class more and were more likely to find class engaging. So we have two great stories of deployment actually helping students get the feedback they need, and helping teachers improve their own practice.
So what does a grant allow us to do? We could have imagined a world in which the pandemic hit and each of us were siloed in our own different parts of the university — I was sitting there thinking about artificial intelligence, my friends thinking about their role in education. We were scrambling. We would not have been able to form the team that we needed. We happened to have started a few months before the pandemic, so we were already in active collaboration. It was just the right combination of people in the right room at the right moment. Our dream for 10 years now is a symbiosis of course content and also mentor support that augments learning. When you're learning online, there's likely hundreds or even thousands of people going through the exact same experience somewhere around the world. We leverage this great potential for collaboration to make the future of learning less lonely, more collaborative, and also to give many more people the chance to play that role of peer. That's going to take a lot of AI, though — you have to make sure things are safe, you have to put the right pair of people together. When people are learning and teaching, we have to help them become better learners and teachers. But I think we have the ingredients. I think that in 10 years, we could move toward that education being augmented.
Matthew Rascoff: So I just want to describe what's happening in the Code in Place course, in case it wasn't clear entirely from the video. This is a model of scalable online learning that reached thousands of students, learners around the world. But each of those learners was placed in a section of 10 students that was led by a volunteer section leader — and that's represented on the left of this image here. Then each of those section leaders was themselves placed in a cohort of 20, led by a teaching leader who was responsible for training the trainer. And we were able to reach, in this kind of modular, fractal format, 12,000 students, taught by thousands of co-instructors, each of whom was given support — both human and AI support. So the AI is helping to improve both the performance of the learner and of the instructor at the same time in this course.
And I just want to talk a little bit about the impact that this had on various stakeholders who participated in this, because I think, as Chris says in this video, there's a real model for impact here. 99.6% of the section leaders completed their responsibilities, even though they weren't being paid in this pilot. 56% of the students completed the course — much higher than what you would see in a typical MOOC, that might have five or six percent of the students completing from beginning to end. 30% of students said they'd like to lead a section, so wanted to step up into that responsibility. Stanford measures its teaching performance on a five-point scale; this course was given a 4.95 on that scale. And net promoter scores of 90 for the students and 70 for the section leaders — net promoter, for those who don't know, is a scale that goes from negative 100 to 100, so those are astronomically high scores in terms of the satisfaction of both the students and of the teaching leaders.
I want to just talk also about the educational impact that section leadership has on the experiences of our students. Most of those section leaders were either Stanford students or alumni. This is a quote from not Code in Place, but a similar project that we've done using the same teaching and learning model, but oriented towards high school students. And you can see, this is a CS master's student at Stanford who led a section in this course, and talks about, in an essay that he wrote for the student newspaper, the impact that it had on him — and in fact on the trajectory of his career. He said it is going to lead him, this kind of teaching, to a different career than he would have had, and to founding a nonprofit dedicated to this kind of educational innovation.
There's also a really powerful research connection that Chris, you know, mentioned in that video, because inside this course the AI was providing feedback to the instructors on the quality of their facilitation of these sessions. The AI was virtually listening in to all of the sessions, developing a transcript, and then using an analysis of the transcript to give feedback to the instructors to help them improve their active learning techniques in the class, and to allow them to use these more sophisticated strategies beyond just repeating what a student said. We found 24% improvement in instructors' uptake of student contributions using this AI, with a randomized control trial. So that to me is the powerful connection between educational impact projects of this type and the research opportunities that flow from them — by building into these projects this kind of research lab behind the scenes of an educational offering, like the one that Code in Place did.
And what you can see here is a kind of virtuous cycle of improvement that starts with an insight from cognitive science about how active learning can improve performance for students, then creating these kind of teaching and learning test beds in large courses like Code in Place, supported with data-driven tools to help the instructors improve their performance — and, you know, the AI feedback for the instructors, or the feedback for the students, are both examples of that. And then that can feed into the data science research to improve those tools over time. And what you saw in the previous slide was a paper that came out of the Code in Place project by faculty and students at Stanford who were doing this kind of data science research — educational data science research — that of course can then feed back into the cognitive science insights, in which we can refine what we understand from small psychology experiments using the big data techniques that the data scientists are able to pioneer.
To me, this model has huge implications and huge potential for how we think about high-scale but also high-touch learning. You know, in this model, what we have are really new combinations of humans and AI that I think hold the key to scaling — but scaling with quality, and scaling with a human touch, and scaling without demolishing that small-scale building of the room that is the basis for the dialogue, that is the basis for the human connection. What would it take to actually realize this vision beyond a small pilot course that Stanford offered during the pandemic, but to really think about what the implications could be worldwide, for other institutions, other faculty members to adopt? To me, I think we need to plan for community engagement for learners, and training and feedback for the instructors, and what we need are technologies that support building out that community. You know, Chris says in the video, supporting the pairs of learners, helping them find one another, and supporting them in the process of peer review and coding and processing and learning together, engaging with one another — but also giving that, you know, machine-style feedback to the instructor to help them improve their active learning facilitation in a small section that would otherwise not be feasible to provide to thousands of instructors.
So, you know, Code in Place was a kind of R&D project that Stanford launched during the pandemic, but there are actually real technologies that are now available in the marketplace that are realizing some of these ideas. And I just want to point to just one of these as a model of what I think is possible, because I think it demonstrates that this is not just happening in research labs — it's actually something that is available and doable using the machine learning techniques that we have right now. So this is a product that I love called TeachFX. It's both an app that you can use on your phone and a web technology that integrates into Zoom. And the idea of TeachFX is to give feedback to instructors using AI that mines the transcript of a session in a class. It can be either a face-to-face class using the iPhone app or an online class using the Zoom integration, and it gives feedback to the instructor that allows them to improve their teaching and learning practice and to use more evidence-based techniques in the class. So it's very much like that machine learning paper that you saw about Code in Place, but this is a real product, and it's available in the marketplace, and it's something that schools in the United States are already using — K–12 as well as higher education.
And I just want to show you a little bit about what it's like to use this app, just a kind of small demonstration that I'll talk you through of how this works in real life, with real students, with a real class. So I'm just going to start this video here. So this is me doing just a demo lesson — the names of the students are not real — and what you can see with those bars is how much participation there's been from teacher and student, how much silence there is in the class, and how much group discussion there is. Now we're going to go into a demo class that's going to mine what happened in this Zoom recording of a class. You can see the words that were spoken by the teachers and the students, how much talk there was by the teacher — me, let's say — and the students, individually and in small group. The green represents silence, the blue is students, and the red is the teacher.
Then you have a transcript of the entire lesson, and you can go through the lesson and you can see, in that chart right below the transcript, who was talking and how much and when. You can see it in terms of minutes on the right, as well as percentage terms. And I can go back, okay, and say, at that time — fast forward, go backwards — what was I saying at that time, what led me to speak so much? Now I've got some examples of short student responses from the lesson, so, like, what they actually said in response to the questions. I'm a teacher going back to see what happened in class today and, did I use these evidence-based techniques in my class? And I'm getting feedback from this algorithm that basically mines what happened in the class, for who spoke how much and what they said, that allows me to improve my performance the next time I do the class.
I have a roster of all the students in the class, and how much they spoke, what time they joined, whether, you know, their degree of participation, how many times they talked and how long they talked, what percentage of the class that represented. And then there's longest stretches of teacher talk — this is me doing too much exposition, really. You can see, okay, what was I doing at that point, why was I speaking so much in the class? Maybe I could let the students participate more fully. Now we've got some different models of student participation. Ping-pong means going back and forth between the teacher and the students, rather than allowing them to go back and forth amongst themselves. We can see, you know, the students getting an immediate response. Think time is a really important concept, and it's the idea of allowing students a moment to think. Before you call on a student, give them a few seconds of think time, just to process the question that you just asked. And after they've responded, give another few seconds of think time — sometimes called wait time — after they've said what they said, to allow their classmates to think about what they said.
All of these ideas are represented really powerfully in the TeachFX interface, and I think this is just such a good example of how the AI can give superpowers to educators. It can help that teacher do her job so much more effectively than she could on her own. And that shows, I mean, the potential for, you know, allowing students to participate more fully in a class like this. So teachers — here's some data from TeachFX — teachers using this app raise student talk time by 273% in the nine months subsequent to their adoption of this software. And that, to me, really shows the potential for taking what we know from the teaching and learning research — from, you know, evidence based on the scholarship of Rachel Lotan, a scholar at the Stanford Graduate School of Education — turning it into, you know, algorithmic tools that help teachers realize those ideas in classrooms at scale, and then measuring the impact on teaching and learning, and allowing us to see that these kinds of apps can help us take an idea that previously could only be represented really in professional development. We could tell a teacher, this is a good idea, try to do this in your class, but it was not feasible to actually give this kind of feedback, to have an assistant listening in on a free class and giving that feedback live. Now we actually can do that, and we can do that in a face-to-face class and in an online class.
And I see, you know, with data like this, the huge potential of AI augmentation for teachers, that allows us to improve learning outcomes for students and preserve that high touch. This is not an AI that's teaching the students, it's not taking the teacher out of the equation — the AI here is really a human-in-the-loop model, which means a combination of a human plus an AI working in tandem to try to improve the overall experience with the learner.
So I'm going to bring things to a close by returning to my architectural analogy. In Louis Kahn's life, he was really only able to complete a handful of projects, and most of them were for individual buildings. But the New Urbanism is a movement that revived many of Louis Kahn's ideas 50 years later, and now, you know, in the 21st century, some of them are coming to life with real projects. And they're happening at the level of cities and streets, not just the individual rooms and the individual buildings, which is what he was effectively able to manage in his own lifetime. The images here are from a new development in Costa Rica called Las Catalinas. It's still under construction, but it has several neighborhoods that have already been built. It was designed by an architect, Douglas Duany, who is a founder of the New Urbanism movement that's taken many of Louis Kahn's ideas into the 21st century. There are no cars in this development, the streets are all walkable and bikeable, they're safe for children to navigate themselves. It's mixed use, and the residences are mostly multifamily, though there's diversity and some of them are single-family homes as well.
You can see in this picture on the right, the buildings have a warmth to them, and there's a sense of community and of a village in the design. This looks like it might be a medieval Italian hill town, but it's actually been built just in the past few years. There's no reason, in my view, why we cannot design towns to be livable, walkable, and enjoyable the way this is designed to be. And similarly, in my view, there's no reason why we cannot design digital spaces and communities that we ourselves would want to live in, to learn in, and to allow our children to explore in. Thank you so much for listening today. I look forward to engaging with you, and thank you for the invitation to participate in this summit. Take care.