Matthew Rascoff: Greetings, hello. Can you hear me okay?
Bryan Alexander: You're just fine, good.
Should I ask you for Matt, or Matthew? Please, forgive me. Good to see you. I love the visual, having your state behind you.
Matthew Rascoff: I'm here covering the breadth of North Carolina, Bryan, as you said. I work for the UNC system office, which includes 17 campuses across the state of North Carolina. And I don't think people outside our state always know quite how diverse a system we have, but it's nine hours across from the eastern tip to the western tip of North Carolina, from Elizabeth City State University on the coast near Virginia to Western Carolina University on the Tennessee border in the western part of the state. And I think the map kind of represents the breadth and the diversity of our state in a very nice way.
Bryan Alexander: Well, that's quite extraordinary, going from the Atlantic coast all the way over to Appalachia.
Matthew Rascoff: Exactly. And I think it's, to me, metaphorical for the diversity of our students across the system. Now, unlike, say, the University of California system, which is just the research universities of California, the UNC system includes five HBCUs, includes mid-tier regional institutions which are, you know, bachelor's- and master's-level and all of that. And we have two flagships at NC State and UNC Chapel Hill. Not many know that NC State is part of the UNC system, in fact. And I think that's part of what makes us distinctive as a university system, that we've got 225,000 students from all walks of life, mostly from North Carolina but also from around the country and around the world, both online and on-campus now. And I think that's what makes the job challenging and also what makes it interesting.
Bryan Alexander: Is there any other state public university system that's managed to have both research-one universities, more traditional public universities, and historically black colleges in one initiative?
Matthew Rascoff: It's interesting. There's been a lot of movement in the design of systems recently. So I think I heard you mention somebody's here from Tennessee. Tennessee just shut down one of their system offices and is devolving more control down to campuses. So Tennessee Tech used to be part of a system, and now it's out on their own. SUNY has moved in the other direction: SUNY has now combined all the community colleges of New York with four-year institutions, and it's now got 60-something institutions in the SUNY system. So there's some active thinking and movement going on, I think, part of it driven by the challenges of technology and thinking about platforms and network effects and where the scale economy is, where the benefits of being part of a system are versus devolving authority down to the front lines and giving control to campus leaders and faculty. That's where the energy comes from.
Bryan Alexander: Of course. Do you think, if we can generalize, do you think nationwide the tendency among higher education is toward greater centralization or greater decentralization?
Matthew Rascoff: In the scale economy, there's no question the technology drives centralization. I mean, look what happens with platforms in technology: they tend to create winner-take-all dynamics, they tend to reward centralization. You know, in social networks, in e-commerce, we've gone from more diversity to more centralization pretty much in every area of the web. And I think that's certainly true of online learning, and I think online learning to some extent drives a centralizing movement. But on the flip side, what I see as really exciting about what's happening in teaching and learning is all the experimentation that's happening in the classroom. And that's not driven by, you know, a system administrator saying, thou shalt innovate in the classroom, and here are the experiments you'll try and you will report back. You know, we can support it and create the policy infrastructure for those experiments to happen, but there's really exciting and interesting stuff going on in the front line. So I think the challenge for all administrators at all levels is, you know, how do you strike the balance between the benefits of scale, reducing costs, improving quality of technology, and the grassroots energy of experimentation in the classroom that comes only from direct contact with students and the use cases that students present.
Bryan Alexander: Well, in a minute we're going to start talking about one of your projects, but I'm really struck by your observation about this combination, on the one hand centralization, on the other hand diversity, pedagogical diversity, teaching stimulation. Is one of the functions of your office to be able to connect experiments between campuses across the system?
Matthew Rascoff: I think at our best we do that. And I think we need, as a system of higher education, better mechanisms for supporting that experimentation and then scaling up effective practices. One of the greatest frustrations that I've observed among fellow leaders of digital learning and online learning is that even when you see an experiment succeed, when you see them developing results and even publishing the results, sometimes those innovations don't diffuse effectively, even maybe the majority of times they don't diffuse. So I'll give you an example of a course redesign section in economics on one of our campuses that has a wonderful evaluation proving that they've actually closed the achievement gaps between Hispanics and whites, and the students, they cut in half achievement gaps between blacks and whites. It's an active learning intervention that's clearly getting great results for the students, improved the outcomes for all the students, and yet it hasn't scaled from one section of intro econ to the three others it's being taught in on that very campus in the same department. And I think the challenge there is, what is the incentive system by which we can take a model that we know works to scale? And we sort of take for granted, like, oh, if you've got something that's effective, of course it's going to be widely adopted. And I think the lessons of history and the diffusion of innovation from other sectors are that not all innovations scale. You actually have to think about creating the right infrastructure and the right kind of highways for the knowledge to flow and to be adopted widely. And so I do think a lot about that, and I think about creating programs that don't just support new ideas but also support the scale-up of ideas. Course redesign is hardly new. The National Center for Academic Transformation and Carol Twigg have been supporting course redesign models for 20-something years now. We've got great results. I think the challenge is not an experimental challenge, it's much more, if you think in kind of pharmaceutical terms, we're past the phase one and phase two and even phase three trials, and now we're trying to get the doctors to prescribe.
Bryan Alexander: Quite true. That's a nice analogy, especially if you think about this as a healthy move. Can you mention the one program that you do, or one strategy, to make those kinds of highways function, to extend the pharmaceutical metaphor?
Matthew Rascoff: Sure. So the main program that my office does with faculty, and I should add the caveat that at the system office we rarely work directly with faculty, most of what we do here is policy assistance, so just working with faculty at all is to some extent counterculture for any system office, we run every summer a course development incubator called the Instructional Innovation Incubator. And it's a week-long fellowship for faculty to build their first online or blended course, and they do it in a highly supported environment where we bring the instructional design and learning management system support to them, and they architect their course on-site. The premise is usually you bring it to them.
Bryan Alexander: So you have the workshop on their campus?
Matthew Rascoff: No, so we bring faculty from across the system together at one of our campuses, and they travel and spend a week building their course. And we bring instructional designers from across the system to that institution, and we all basically do a kind of week-long design charrette together, faculty from across the university system, across all disciplines, but what they have in common is that they're all novices to digital learning, online learning. And the basic premise is that most professional development fails because you may go off to a workshop or a training program and learn some new abstract ideas, and then you go home and you have to figure out how to apply them to your discipline. And our premise is that the application to your discipline is the most difficult part of changing your practice, and we want faculty to be supported in that process. So that's what we have them do. We do a kind of new-ideas seminar discussion every morning and then apply those ideas to your course in the afternoon.
Bryan Alexander: So you're building the course as you're learning these new concepts of formative assessment or how to use video, that sort of thing?
Matthew Rascoff: Applying and learning new concepts, and applying and learning new concepts. And it's an iterative kind of building-and-learning process that all happens together, and you come out of this week-long fellowship, which faculty compete to be part of, with the basic bones of your course already built, which you typically don't have. Typically you go to professional development and you're like, what do I have to do now, how do I apply this? And you've just got a big binder that says lots of new ideas, and it's pretty intimidating to try to figure out how to apply it. So that's, I'd say, our core mechanism for creating a cadre of faculty innovators who could then go back to their campuses and engage more fruitfully, I think, with the instructional designers on their campuses and other resources for support, because they're more educated consumers of those supports. And then they're also advocates, I think, in department meetings when there's debate about, you know, is online good, is digital learning good, is this helping or hurting. This is, you know, hundreds of faculty members who are going out into the field and saying, I've had this positive experience and I know it can be good because I built this myself and I've seen it in action. I'd say that's our core kind of scale-up mechanism. Last year a quarter of UNC system faculty taught an online course, and I think our infrastructure hasn't nearly kept up with the scale of demand for professional development and preparing faculty to do this work when, in fact, they've mostly never taken an online course, let alone designed one. And that's a structural challenge facing all of American higher education.
Bryan Alexander: I agree, agree. We're in a transition moment. Just to remind everybody, if you have any questions about his work with the UNC system, remember to use the raised hand or the ask button, a tweet, or the chat box. Matthew, this one we've been talking about off and on over the year. The forum, we've been discussing in different ways the inter-institutional collaboration work. You had people doing this in different sectors, different proportions of the country, and I'm going to follow up with this over the next year. So it's really exciting to hear about this one week, I'm sure that you did, but also more to the point, and you just brought this up, this website that you launched, I want to say in January, around the New Year, was it?
Matthew Rascoff: Yes, yeah.
Bryan Alexander: And let's just make sure everyone has a link to this. This is UNC Learn Trials, unclearntrials.com, and I just put it out on Twitter so people can see it. So tell us a bit about the genesis of this site, but I don't know one like it at all anywhere in education. You all came up with it.
Matthew Rascoff: So the premise behind the site, I think, was encapsulated very nicely in a blog post that I actually just read recently by Jonathan Rees from Colorado State. It's a post called "There Be Dragons," I'll post it to the Twitter, but he basically argues in this post, Bryan, that the learning management system has become an impediment rather than an enabler of digital learning innovation. And in fact the enterprise model of technology more broadly, of a single system that's adopted by a campus and is to some extent imposed on the educators on that campus, is the wrong way to think about how to drive innovation at scale in higher education. And that's very much my premise, too. I think we have moved to an app-style economy in our consumer lives, you know, pull out your phone and count how many apps you have on your phone, but we're still very much in like the one big Microsoft Office-style system in our teaching and learning lives. And the learning management system is to some extent like this Swiss Army knife of mediocrity, where it does a lot of things but none of them particularly well. And what I observed was that our most innovative faculty used it the least. And I thought, you know, the ones who are most creative, most technologically adept, most experimental were the ones who hated the LMS the most and were most inclined to just ignore it and sidestep it. And to me that was kind of a window into the future, and it indicated that we needed better mechanisms for empowering faculty to use the ed tech tools that I see rising around us, help them figure out how to adopt those tools, but to do so in a way that follows the law, you know, like FERPA, and complies with university policy, gives some control to IT administrators because there's a reason they need that control, and doesn't add too much overhead and too much friction to management. So what we tried to do was create a kind of app store, modeled on what you might see in Android or iOS, that allows faculty more control over picking the tools that they use. And our premise was that those tools might be different in different disciplines with different student populations, allow them to decide which ones are most effective for their students and use the crowd to review which ones were getting results in terms of learning outcomes and which ones weren't, rather than one big RFP committee deciding which is going to be the blessed system for our campus. Instead, have a plurality of tools, a diversity of tools, that we can let faculty adopt with relatively low cost, relatively low risk, low friction, and then the review comes on the back end where they try it out, see what gets results, publish those results to, you know, our 20,000 faculty members across the system, and let the quality rise to the top through the impact that it has on teaching and learning from the front lines.
Bryan Alexander: Let me walk back through this, because there are a lot of great ideas, I just want to pull them out, not least because of your accurate and damning statement about the most innovative faculty using the LMS the least. What led you to the app store model, of all possible paradigms?
Matthew Rascoff: I think it was to some extent having worked in K-12 ed tech before. I spent three years working for a company called Wireless Generation, which was a medium-sized ed tech company that often ran up against the challenges of integrating with student information systems in fifteen thousand-odd districts across the country. You know, K-12 ed tech is arguably more fragmented than higher ed tech that way, and the integration costs became a big impediment and a big explanation for why K-12 ed tech had historically been dominated just by a handful of publishers, because they were the ones who could, you know, wait out the 18-month to 24-month sales cycle of the RFP committee, and they were the ones who had the capacity to do these very expensive integrations with very painful student information systems, which were designed in a very proprietary business model, locking down the data. And so my vision was more of an open ecosystem where ultimately the data was shared across our institutions, was more accessible, where companies didn't compete on the basis of, you know, their ability to wait out patiently our slowness, but in fact competed on the features, the quality of those platforms. And I think, you know, if you look around in tech, what are those very competitive marketplaces? The ones that are really consumer-centric, where you can find anything you need pretty much, I think, are in mobile apps. You know, compared to kind of enterprise software, those are the ones that are most focused on the needs of the users, where the buyer's and the users' interests are really most closely aligned. And so that was one input. The other input was that I saw a lot of value created in the pre-agreed terms of service that any app developer who wants to play in the Apple App Store has to agree to with Apple. If you want to sell an app on the App Store, you have to agree to give up thirty percent of your revenue to Apple, you have to agree to their terms, and while that may be frustrating for an app developer, it actually creates a lot of value, because it means that there's no one-off negotiation, there's no red-lining of agreements going back and forth between app developers and Apple, and simultaneously between Apple and its end users. It's just a standard agreement between the app developer and the platform provider, Apple, and another standard agreement between the platform provider and the end user. And we did that in our project. We came up with a standard set of terms and conditions and basically said, take it or leave it, to vendors: we're not going to cheat on them. And that has the potential to massively reduce the amount of time that it takes to do a deal with us, because we now have these pre-agreed terms with all the participants in our learning technology commons, and any of our faculty across the system can adopt one of these tools with pre-agreed pricing, pre-agreed terms, no legal department involvement. And that's good for both the vendor and good for us, and that's, I think, a big source of the value that app stores create that we wanted to recreate.
Bryan Alexander: That's fantastic, and it makes me wonder, why isn't this widespread, how come people aren't doing it? Before I ask that question, we have a better question from a fine person whose name I'm going to mangle, unfortunately, which I believe is Esther. Esther asks, Matthew, how do you curate the tools that all these faculty use?
Matthew Rascoff: We had a committee that helps us do this, and we applied a very light filter of curation on the applicants, because we really wanted the curation to happen on the back end. So we let in almost any vendor who is willing to accept our standard terms and conditions, publish their pricing, and tell us a little bit about what they do. But there was no quality filter, there was no curation on quality, because we want results to speak, not the committee to determine for the faculty. I think the use cases now are just too diverse for any central decision-making model to tell faculty on the front lines in every discipline under the sun, and we have nearly 400 online programs in every discipline, let alone all the learning that's happening on campus. There is no committee that could ever master all the different use cases and make central decisions even on a single tool. I think it's a lost cause. So what we said is, let's get out of the business of curation, let the crowd decide what works, and we'll basically just keep out the bad actors as best we can, and in terms of evaluating quality, you've got to figure it out for yourself. And that's pretty much what the Apple App Store is like. Almost any app provider can get their product listed on the App Store as long as, you know, it follows a basic set of guidelines that are laid down by Apple, not being abusive and not being, you know, pornographic. Those are the kinds of criteria that Apple uses. And how many stars you get on your app, that's determined by the users. And so we took a similarly laissez-faire approach to the evaluation of quality and impact.
Bryan Alexander: That's a great answer to a really good question. Let me press on a couple of points there. First, how is that committee composed, of faculty and educational technologists from across the system?
Matthew Rascoff: We had a few of them, and we also had several of our IT procurement and finance procurement folks on that committee, because they helped us craft that system-wide set of terms and conditions that we really needed to get traction with all of our buyers of technology across the system. And that was a bureaucratic challenge, I would say, in getting buy-in from, you know, 17 campuses which previously had negotiated one-off with vendors, and getting them to say, okay, here's the core concerns that we have in common, and let's lay down one set of those concerns and publish them, and if any vendor agrees to them, then we agree that these are good enough for our campus too. So it was really the buyers in finance and IT departments and the people who enable deals to go through, they were really the crucial participants in that committee. Like I said, we weren't evaluating the quality of ed tech on this committee, so we didn't really need that many faculty members, because it wasn't about the user interface, it wasn't about the learning outcome, that all comes later, that comes as these apps come in contact with students and faculty, that's when the quality metrics come in. And what we did do, though, was work with our partner on this platform called Learn Trials to design a rubric, so that we know we're asking faculty the right questions about these apps, and the questions aren't just Yelp-style questions like, did you like the restaurant? The questions are designed for educational apps, like, tell us how long you've used this, tell us what the impact has been on the educational outcomes. And we put a lot of attention into that rubric because we wanted to make sure we were gathering meaningful data and sharing meaningful data across our campuses that would inform decision-making, improve the quality of decision-making, and potentially, to your question earlier, Bryan, scale up those apps that were getting results, that did have efficacy data behind them. That was really, I think that's part of the power of a platform like this, to show where the impact is and to scale up that impact more quickly. You know, if you've got a hot app in an app store, it can really take off if you get featured and you're getting results. And we wanted this to be a mechanism for that kind of scale.
Bryan Alexander: I have a couple more questions, but once again the audience has better ones. So let me pause. Someone asks, what does support look like in this model, if a faculty member wants to try something new but isn't an expert? Who provides guidance?
Matthew Rascoff: So, you know, that's part of what's scary about a project like this, I think, for IT departments and to some extent for faculty, because there is no official support as you would get for the learning management system, and you're kind of, you know, using this stuff at your own risk a little bit. And of course many vendors provide support for end users, smart ones do, especially with faculty who are experimenting with the tool and have a voice to 20,000 peers across the system, and use that voice to express positive or negative sentiment. But there isn't support, generally, in the sense that you would typically get from the IT department with a tool that's formally adopted by the IT department. So we recognize that limits the scale potential of these tools to faculty who are more comfortable with technology. But I think that's okay. I think of this as like a second tier of technology. There's the, you know, first tier of fully supported apps like the student information system where there's 24/7 support, and maybe the LMS has nine-to-five support, and then there's maybe another tier of apps that are, you know, productivity tools for your desktop, and you can generally call somebody but maybe you'll get a student support person, and if something's really wrong then you'll get to your tier two. And what we created was, I think, like another, lower level of apps that are less supported, self-supported you could say, where vendors are generally the first line. And, you know, it's not for the most high-risk, most critical technologies on the campus. This is designed to support instructional innovation, where the risks are there but they're usually limited to one class or another, and a bad outcome looks something like maybe you wasted a few minutes trying something, or learning how to use some tool that you didn't necessarily want to use. But I think it's unreasonable to ask IT departments to support the 4,000 tech tools that are out there in higher ed now, it has listed in our index. And so, you know, the alternatives are unpalatable: either they say no to everything, that's not acceptable, or they only support two or three things and they forbid you from using everything else. And that's what, you know, Jonathan Rees was complaining about, that the LMS is the impediment to innovation. So what we were trying to do was navigate in between those kind of bad extremes.
Bryan Alexander: So we had John on the forum, and he was as passionate about this. We have another question that's come in, and this is from Nepal. He says, is it safe to say that the apps that you speak of and offer on Learn Trials at UNC are only iOS-based and are not for Android users?
Matthew Rascoff: So none of these are platform-specific. Or, let me put that another way: these are apps, I use "apps" in a generic way to say mobile apps or web-based apps, and they run on a variety of different platforms. You can discover what platforms they run on in the index pages, you know, that list information about these tools. The URL is unclearntrials.com, that is the correct URL for this project. unclearntrials.com is where faculty from our system can log in using a single sign-on system, so no matter where they come to our system, they can use their campus credentials, and where vendors come to apply to be part of our app store, to publish information about their tools and to accept our terms and conditions. We actually have the vendor competition open again, we've reopened it, and we're doing it on a kind of semester basis, where we keep it open for a few months, have the committee meet, include a new cohort, and then close it and reopen it every semester. That's our vision for inclusion.
Bryan Alexander: This is really interesting. So this is basically OS-agnostic?
Matthew Rascoff: Exactly. I use "apps" in the most generic possible way to mean unbundled teaching and learning technologies. They're generally micro-solutions, not necessarily but generally, you know, tools like a MobLab, let's say, which is a social science simulation tool. It's designed for economics, sociology courses where you want to bring in a simulation to make some theory more realistic, and it's both an authoring tool for creating simulations and courses like that, and also has some pre-existing simulations that, you know, faculty in business schools can adopt and use to teach these courses. Another one I thought was very interesting was called Gradescope. It's a productivity tool to make it easier for faculty to quickly grade assignments, either that are entered online or scanned in, and to develop rubrics and to grade them more quickly. There was a third one that I thought was very impressive called WriteLab, which is a combination of AI-driven and human-driven tools for helping students improve their writing, so, you know, they'll upload the writing assignment and get high-quality feedback on their writing using WriteLab. Those are some of the apps that I think are doing, you know, one thing really well. They're often, you know, apps from a company that only has one product, unlike the big publishers, that are really focused on making that product right. They're generally startups, and they're offering some technology that's just not available in the big bundles of learning management.
Bryan Alexander: Oh, this is great. I'm killing my poor little MacBook Air by popping open tab after tab for you as you're showing some really good stuff, which is a nice second-order impact of your app store. We have another question, but this time it's from Mark Ulit, who wants to come on stage. So, Lydia, why don't you pull me off and bring Mark up so he can talk?
Mark Ulit: Hi there, Matthew, can you hear me?
Matthew Rascoff: I can, okay, great.
Mark Ulit: Hi everybody. I just had a great question. Really love, love what you're doing here, love the approach and sort of broad democratizing aspect going on here. But I wonder as well, like, this is taking place in sort of a landscape where universities are focusing more and more on the student experience, and some of these micro-solutions can create, so first of all, it can create a few different things, really ragged a student experience, generally, especially if you're looking at a broad state system, but also potentially a really disruptive system where the students are from class to class to class to class having to go into completely different systems, have different apps. I actually sort of experienced or ran into this in my own personal situation, but it was with sort of high school students who were expected to use six or seven different websites, LMSes, tools, and it was really disruptive to them in their experience. So I wonder if you could speak a little bit to, you know, how you see this playing out and how this relates to that student experience.
Matthew Rascoff: Yeah, I think it's a good question. I think there's a debate going on in the community about the merits of consistent UI and keeping students kind of together on the same page, as it were, and the merits of democratizing diversity and, like you said, you know, enabling more freedom, more experimentation. And I think what we're trying to do is strike the balance between those. Learning management systems are, I think, you know, very much on the former extreme, and, you know, maybe the mobile app world is going too far in terms of teaching and learning. But where I see our students in their technology lives is in a very pluralistic place, you know, where they generally engage in maybe four or five different social networks for different kinds of interactions. You know, they're on Instagram and Snapchat and Facebook and Twitter, and they've learned what you use those different social networks for, and they're all different kinds of interactions with different networks. And the kinds of things that I say on LinkedIn, I know, are very different from the kinds of things I say on Facebook. And sure, in some, yeah, do I wish it was easier to post to all the social networks at once? Like, yeah, sometimes there's something I want to put out to every network, but usually I'm thinking about different tools and different networks for different audiences. And I've come to, I think, recognize that that's where the technology is, that's where most of my friends and most of our students are as well. So I don't think it's unreasonable to say that there should be three or four apps per class, maybe. I think six or seven sounds crazy, but then of course you put together a bunch of classes at the same time for a full-time student, and that adds up. I think it's a reasonable question to ask, like, at what point are there diminishing returns, and where is the value of, say, you know, bundling these together, aggregating them, coming up with a standard UI? And I think that's probably coming to ed tech. You know, I think you see big publishers buying up ed tech startups, and to some extent that's what they do, they buy them up, they put them in a standard UI, they do a single sign-on, they make a more consistent experience. You know, this is basically Pearson's strategy, they don't build a lot themselves, they buy up a lot of companies. I think they do it partially because of the reason you said, because there's value created in the aggregation. So I don't think it's, you know, all of one or all of the other. I think there's probably a balance that can be struck, and it's going to be about what students need, like you said, and coming up with a model that supports them but also gives freedom to faculty to use the best tools that are out there. That's the perspective that everyone—
Mark Ulit: That's good, that's really great, thank you. I mean, I would imagine as well that it's one of these, you know, these are in a sense uncharted waters that we're heading into, and it may be that, you know, these things swing back and forth at different school systems, as you pointed out earlier, different school systems are driving toward one side versus the other as they sort of navigate this. So it may be, you know, the next 20, 30 years are sort of going back and forth between emphasizing the experience versus sort of really trying to create that innovative situation.
Matthew Rascoff: I see this, you know, in our campuses. Each campus has adopted its own learning management system, and as a result we have four different LMSes in our 17 campuses. We have Moodle, Blackboard, Sakai, and Canvas. When a student cross-registers from one campus to another, as they often do in online courses, we're presenting them typically with a new learning management system, new login and password, and nobody in my three years here has ever complained about that. They complain about other frictions, like the different academic calendars on our campuses and the fact that they have to give up spring break in order to cross-register, they don't love that, but it's never been an issue in my three years here that they have to figure out Canvas when they're coming from a Blackboard school. And as a result, I haven't invested any effort in trying to standardize on the learning management system, because I think they're all kind of, you know, boring and similar, that there isn't value created in the standardization. And if nobody is really complaining about it, why go through the pain of uprooting one and changing them out and retraining faculty? I mean, I think that's just like a little micro-expression of the diversity as it was expressed in an earlier generation of ed tech, and I think there's probably a similar story now to be told in the app economy that we're living in.
Mark Ulit: Wonderful, thank you very much.
Bryan Alexander: Thank you, Mark, for a good question that really strikes at the harder things here. Remember, folks, if you'd like to ask more questions, as you can see, those two orange dots do the trick, the Twitter is available for you, and you can chat with your compadres in the chat box. Matthew, I usually break around this point to do a mingle session, but I think the tempo's too high, so let's keep going. Okay, you mentioned one thing that I thought was really powerful, I wanted to just press you a little bit. You mentioned that you got the procurement officers across 17 campuses to agree to the app store. How'd you do it? What's your secret?
Matthew Rascoff: The way you get anybody to agree to anything in higher education: you form a committee, and you talk a lot, and you go through the process, more seriously, of engaging the stakeholders and understanding their concerns and hearing them out and coming up with a compromise. And, you know, I do have the benefit of being at the system office in a fairly centrally governed university system where we have a fair amount of formal power, but I didn't use any of that formal power in this process. All I did was use my convening power and help, you know, my colleagues across the system see the benefit of doing this. You know, they don't love the red-lining process with vendors, it's not like, you know, they get a lot of joy out of changing the jurisdiction of agreements from the state of Delaware to the state of North Carolina as required by North Carolina law. But, you know, the track-changes features, it's nobody's friend, per se.
Bryan Alexander: No.
Matthew Rascoff: But that's, they spend their lives doing this and rewriting agreements, and, you know, I don't think it's particularly joyful for anybody. So if there's a way to make that process more efficient through these pre-agreed terms, why not try something like that? Within many of their own campuses, and the larger ones, you know, like at UNC Chapel Hill, they've already done this within their campus, working across the schools of a single campus, they've come up with a standard set of terms that the IT department negotiated. So I thought, you know, if we could do that within a campus across schools, why couldn't we do that across multiple campuses in the university system? We're all subject to the same laws of, you know, the government of North Carolina, that's what really drives the requirements in these documents, it's North Carolina law, we all have that in common. It turns out the differences are much smaller than the commonalities. That's what we discovered in getting this committee together.
Bryan Alexander: Did you have to bring in counsel at some point?
Matthew Rascoff: I did, yes. Our legal counsel's office was very closely involved with the project, at the system level, and they published, you know, these standard terms to the general counsels of each of our individual institutions and said, these are blessed.
Bryan Alexander: Very good. This is powerful stuff. I mean, what you describe is the nitty-gritty of making collaboration work.
Matthew Rascoff: Exactly, it's unbelievably unsexy, but I think arguably this is the part of the project that has demonstrably created the most value most quickly. You know, the outcomes data and the sharing and the feedback, all that's going to take a long time to realize, where we, you know, as we grow faculty usage and get them on the platform and using it to drive decisions, that's going to take years. But, you know, having standard terms has already saved time in transactions that I've seen go through with vendors, and making them more efficient, reducing the friction in negotiation.
Bryan Alexander: That's really good to see. I appreciate having an immediate response and also your long-term plan for this. You mentioned Learn Trials, what are they like to work with?
Matthew Rascoff: They've been a fantastic partner on this project. Their mission is to help educators use data more effectively to drive tech decision-making, and they are an ed tech management platform that white-labels projects like this one on behalf of university systems and also K-12 school districts and charter management organizations. They work with, you know, organizations like ours to create platforms like the one we've built here, and we basically just license their platform. They made a few small changes to our specs, but all that's been adopted into the Learn Trials platform, and they've been a wonderful partner in helping to, you know, learn, for example, from what the K-12 system has done with their K-12 partners. In ed tech management, in many ways K-12 is ahead of higher ed in the ed tech world, there are a lot more tools out there, teachers find it much easier, I think, to adopt them, there are a lot of free tools, a lot of premium tools. I think it's partly just because, you know, there hasn't been as much kind of central resources available, so teachers have been more on their own, and they've had to be kind of more experimental and more entrepreneurial in adopting technology, because, you know, the district IT department is four people trying to manage the hardware primarily. They don't have the kind of big organizations that have the ability to kind of impose solutions. So it's always been more classroom-centric as a result, there's more teacher-driven ed tech, and as a result, you know, the ed tech economy is more diverse, not a lot of companies necessarily with thriving business models, I would say, but there are a lot of products out there. And so I think there's a lot that higher ed can learn from K-12 ed tech. Learn Trials to some extent helped us do that by taking this platform that was originally designed for K-12 and bringing it to higher ed.
Bryan Alexander: Right. Do you find that when you were working with them, that they connect you with any similar projects going on in higher ed?
Matthew Rascoff: I'm aware of, I think, projects, I would say whose theory of action is like ours. So I would point to the work of the Jefferson Education Accelerator at UVA, which is organizing a big symposium next year focused on ed tech efficacy. I think there's a movement around the efficacy of ed tech that's converging and recognizing that, you know, a fair amount of money is now being spent in this area without necessarily good evidence and good infrastructure for collecting and sharing evidence about where the results are happening. And the vision that Bart Epstein and the Jefferson Education Accelerator have is, you know, to create a competitive market for ed tech that's driven by educational outcomes, not by the talent of the marketing team and the sales force of a company. And I think that's definitely a vision that I share. So I see our project as philosophically aligned with what that project is doing. There's another one that I think is relevant, maybe, to your audience. It's called Courseware in Context. This is a Gates Foundation-funded effort run by Tyton Partners in Boston, and the idea there is to create an evaluative rubric for courseware, including adaptive courseware, that gives selection committees more tools for deciding what works and what doesn't, other than just, you know, this is the textbook we've always used, and it comes from McGraw-Hill, so we're going to keep using it. It's basically to create a quality rubric that gives open courseware and adaptive courseware a fair shot at competing for adoptions by committees, and it's open source, it's coursewareincontext.org. They're taking a different approach, which is really more of an expert-driven approach rather than a crowd-driven approach, and they've really focused on creating a quality rubric, as we did as well. And it's for content solutions, so it's not ed tech broadly, it's just courseware, it's educational content. But I see them as very much part of the same movement of trying to let efficacy speak, let the evidence speak, and share results more freely in mechanisms that are other than just, you know, the published paper, which may come out five years after the product has been built and isn't current enough to drive decision-making by practitioners like me.
Bryan Alexander: Well, these are two very, very different projects. I like the way that I can see the spiritual affiliation, but also very different paradigms, you have crowdsourcing going on for yours, and for Courseware in Context, expert-rated.
Matthew Rascoff: Exactly. We would love for other universities to be part of the learning technology commons. That's one thing I would add: we want nothing more than to partner on this project, and I know Learn Trials would like that very much.
Bryan Alexander: Do they want other universities to take an approach like this?
Matthew Rascoff: I've been approached by probably half a dozen other institutions and systems that want to either join in or build something similar. I think there are a lot of network effects in going in together on something like this and sharing our reviews with another university system, and vice versa. There isn't anything proprietary, you know, in these reviews, it's about, you know, I used this tool and I got these results with this kind of student. There's no reason why we couldn't share that with other states and other universities as well, and we'd be very excited to do that. So we were approached, for example, by a university actually in the UK that was interested in, you know, a similar model and sharing it across a consortium of institutions, and, you know, I'd be thrilled to do that, and I jumped at the chance to share the project with them, as I would any other institution.
Bryan Alexander: Sure. Speaking of jumping, just to imagine, we're almost at time, so if you have a question or comment that you would like to share, this is a great time to click one of the orange buttons, or on Twitter. Just to blog a little bit, thinking back on the time that you spent developing this project, and looking ahead to what you're going to do with it, can you share any lessons that you learned about inter-institutional collaboration, Matthew?
Matthew Rascoff: I would say I wish I had paid more attention to the rollout of the project and spent more time, I think, with the front-line IT departments and faculty development centers, all the different kind of brokers of adoption at the campus level, to explain the project, you know, after it was built, but maybe while it was underway as well. I think, you know, I would love to see more usage on this platform, more faculty coming to it and using it. And I think, you know, that means getting, you know, our campus CIOs on board, and I've asked them to link to it and to talk about it and to use it, but I think really firing them up and making them excited about the project is the thing that will really determine whether this is successful. You know, this can't ultimately be driven by a system office, I think, and if it isn't adopted wholeheartedly by our campuses, I don't think a project like this is sustainable. And I think I wish I had spent more time focusing on that kind of implementation, not just getting this platform built and making it work. And, you know, it's kind of a two-sided market: there's the vendor side, and then there's the user side, and there's a chicken-and-egg problem with every two-sided market. You know, if there are no vendors in it, then no faculty are going to come to review, but if we've got a lot of vendors and no faculty use it, then the vendors aren't going to stick around and we're not going to get new ones. So you really need to focus on both sides kind of at the same time, that's the chicken-and-egg dynamic, everything needs to arise at the same time in order to create a healthy two-sided ecosystem. And that's challenging, and I think I'd probably put a little bit more emphasis on the vendors, not quite enough on our end users, in retrospect.
Bryan Alexander: Have you taken this to the EDUCAUSE community at all? No? I'm thinking, if you wanted to get CIOs fired up, that's—
Matthew Rascoff: Sure, in general, that's a great idea. I haven't. I didn't go to the last EDUCAUSE, but I went to the one before that, I think it was in San Antonio. And I think that, you know, doing something for EDUCAUSE Review or maybe the ELI group, I think, makes a lot of sense.
Bryan Alexander: Yeah, to talk about this project. Thank you, that's a very good idea. I think this approach would make a lot of sense to those leaders. Friends, if you don't have any more questions to ask, I have one more myself, but if you do, this is your last chance. It's like an auction, right? Seeing none, going, going. Let me ask, looking ahead: you mentioned, for the future of this project, that you'd like to accumulate a lot more data about the usage and be able to start drawing lessons from that. What else is ahead in the future for this site?
Matthew Rascoff: I think the next technology step that maybe we will take, or maybe somebody else will do, is figuring out a better integration mechanism that allows apps to tie into the student data that's held in student information systems. Most of the data in the U.S. is locked down, either in PeopleSoft or Banner or other student information systems. The business model of, you know, Oracle and Ellucian is to keep that data as close as possible, not to, you know, publish it securely to app developers and enable a competitive ecosystem of teaching and learning tools on top of it. They're not going to do it, I don't think, unless they're pushed, and I think it's our obligation as a community to, for example, do projects like Brigham Young's API project. I think they're really the leader nationally in thinking about how do you unlock the student data so that it can be accessed both by students themselves and faculty for educational uses, but also, with the permission of students, for educational technologies that come from the marketplace. And I see that really as a model for unbundling, unlocking data, making it more widely available, reducing the friction in integration. So, you know, what we've built is kind of a parallel system, we haven't yet solved that data integration problem with this project technically. You know, we did a very light integration for single sign-on, but no student data is flowing into these apps through this project. Ultimately, though, I think that's where, that's a big technology impediment, and I think it's going to be educational institutions like ours that need to push the vendors, or build APIs on their own, as we have done here. We have now a system-wide student API that we can give to a vendor who's serving students across our system and allow them to tap into our database. It took an enormous amount of technology to do that, and I think it's unreasonable to ask, you know, smaller institutions, for example, to figure out how to do that on their own. I think we need our vendors to move to a more API-driven, more open technology infrastructure.
Bryan Alexander: Well, that's a wonderful, wonderful note to end this on, in part because you're grappling with some of the grand information architecture questions for higher education technology today, but also because this really nicely anchors a fascinating project in the larger landscape. I've got to say, Matthew, thank you so much for taking the time to share this with us.
Matthew Rascoff: Thank you, this was a pleasure, and I really enjoyed the questions from the audience and the participation from the audience. And Bryan, I have to say, you have a voice for radio or podcasting, or maybe for shindigging. You're a masterful convener of this. It was really a pleasure to be with you and to reconnect with you.
Bryan Alexander: Well, thank you so much. We may have some podcast work coming up, introducing, okay, good, great, but we'll follow up and be in touch. And if you have a major milestone or a major event, please reach out to me so I can share the book. Wonderful. Good weekend, everyone. This was a treat, a major pleasure. Now, Olivia will be inside so that she can show us what happens next. But again, just want to say how spot-on Matthew was to thank you guys, because of the terrific questions, that's what makes this format work.
Now, next week we have the last forum session of 2016, and it's a party, we're going to have a holiday session. So you can show up, and on camera you can drink champagne if you're allowed to, you can have funny clothes or funny hats, and we can basically hang out, relax, and have our own Future Trends Forum holiday party. So that's for next Wednesday, December 21st, so please come. I promise no PowerPoint presentation, no macroeconomics, and we will mention as much technology as we feel happy with, and you can get as goofy as we want and let our hair down. So in the meantime, if that's not enough for you, we're also finishing up our reading. So I have finished the last blog post and the last chunk of "We Make the Road by Walking," and we've seen extraordinary discussion happening. We've seen blog posts all over the place, we're more than 20 now, from multiple, multiple people, we've seen art, we've seen data projects, we've seen resources grow, and this is really a striking and exhilarating event. In fact, this week I heard of the first meetup in Mexico City. So next week we'll have a final discussion of this, so please join in. And if you have ideas about what book you should read next, please contact me, because that's the next step for us. Now, if you are new to our forum, or if you've been here before, let me remind you of some of the context. If you'd like to learn more about us and what we do, go to the Shindig.com site, which can tell you all about this great technology, or go to FTTE to learn more about the Future Trends Forum. I know you all have to go back to your lives, but I'll stay here for a few minutes if you want to chat. Thank you all so much for a really rich, illuminating conversation. Look forward to seeing you next week for our party. Bye-bye.