No AI needed. Just an old fashioned depth first search through a graph of skills and dependencies.
I made it to help me practice music, but I have been branching out and using it to study math for a few weeks. I find myself saying "just one question more" and then spending another half hour in it.
Still needs more material to be useful to other people but it's a solid experience. I learned and memorized how to play most of the notes in the saxophone with good intonation in about a week, as a complete beginner.
Trane looks good. I've been trying to design a similar system, but for mathematics: the difference there is that there can be multiple routes towards a particular understanding, with different routes better for different people.
Prerequisites for calculus: an understanding of the standard reals OR an understanding of the hyperreals. Yes, real and hyperreal analysis are radically different in many ways, but they're provably equivalent, and they get you to the same answers. Some people find one more intuitive than the other, and some problems are easier in one than the other.
Imo, the biggest issue with mathematics education is that it tries to push you down one, standardised way of thinking about things: if you don't get it, that cripples your ability to think and reason for yourself. The main thing I do in one-to-one mathematics tutoring is finding how the student thinks, and nurturing that (instead of stamping it out and replacing it with a "more formal" approach).
I'd like to see something branchy, but I haven't found a good solution. Do you have any ideas for tackling this?
I started working on this a few weeks ago: https://github.com/trane-project/trane-math It currently needs a README, but you could take a look at the courses on how I am building the flashcards. It's easy to reference external resources, so that's what I have been doing, rather than trying to create exercises of my own.
I am starting with a very basic olympiad-style book and a book based on Euclid's Elements, because I don't have the understanding required to clearly work out the dependencies of more advanced stuff. And I also would like to start at the beginning to make sure I don't miss anything.
The ideal end state is to have courses that cover all the undergrad and grad math curriculum. I am also curious on whether this could be used by researchers to keep up to date with the latest research on their fields. But all of that is a long way out.
As for your question, there are a couple of ways that Trane could handle multiple paths through similar material.
1. Just have separate curriculums. You could copy the courses, but the second copy has different dependencies, courses/lessons IDs. For example, one could have a series of courses teaching the undergrad MIT math curriculum and another the Harvard curriculum. They might share a lot of the material, but the order will be different.
2. Trane does not lock you into a specific order. There are filters that let you specify which parts of the graph you want to study. You are free to get questions from specific courses and lessons. You can also use the metadata in the courses to say things like "give me questions from all lessons teaching linear algebra" or "give me questions from all courses on real analysis but not from the lessons on set theory". The dependencies between the lessons that match that metadata are still respected. There are a few more options, but you get my point. The dependencies are not set in stone, and there's freedom to jump around and study specific topics.
I actually use option 2 most days. If I want to practice guitar, I just set a filter to give me exercises from the guitar. Similar thing when I want to practice the saxophone.
Mathematics doesn't really have dependencies. Everything can be thought of in terms of everything else: you need an in, which is usually arithmetic or geometry (but, for physicists, is often physics – and for programmers, is often concepts related to their favourite programming languages), and then everything can be defined in terms of something else.
However, many people find it easier if you have concrete anchors, rather than making everything an abstract ivory tower. A certain kind of musician might get Fourier series quite quickly by analogy to the behaviour of sound, and then pick up Taylor series by analogy to that; somebody like me might find it easier to translate the most general Fourier series to a Maclaurin series, then compare coefficients. All these approaches would give you an understanding of Fourier, Maclaurin and Taylor series', which then lets you learn things that build upon them.
The "separate courses" approach is doable (and the best I've come up with so far), but it doesn't scale: courses that cover the same things without providing further insight don't get merged together. It can't handle the combinations that, intuitively, I feel like there should be a way to handle. Human teachers can cope with it just fine!
One nitpick on trane-math: running `cargo test` will wipe out the .trane directory, if it exists. That seems not very good; perhaps there should be a way to point trane at a temporary directory, so you can test it better?
Thanks for pointing that out. I'll fix it, but for now just have the .trane directory in the folder right above.
I guess you are right. The textbooks definitely have dependencies, but that seems to be the very problem that you have with them. Unfortunately, I think other than different curriculums for each way to approach the subject, this is beyond the current scope of Trane. I made this thinking of music at first, which has a more rigid way of approaching the subject that tends to work for a wide set of students. There's no person on earth that will be comfortable studying highly chromatic music without mastering harmonically easier material first.
Yeah, trane's design works really well for practical stuff like learning an instrument. It's probably good for language learning, too: perhaps a series of books or texts, each of which builds on the language skills learned from the previous…
It's a good idea to limit the scope of your projects. Jack of all trades is a master of none, and the fact trane exists (and is GPL'd) makes it just that bit easier for me to solve the other problem.
This looks great, thanks for sharing. I do think it would be more ergonomic to have this served via a website given that it feels CRUD-like with some browsing of exercises, relationships between them, etc.
The CLI approach assumes people are comfortable using the terminal, which unfortunately limits the number of potential users
Definitely. I have not worked in frontend/ux stuff for over 10 years, and I am working on this by myself, so I have been focusing on the CLI to quickly iterate and prove that it works reasonably well. Part of the reason of trying it out on math was to a find a more technical group of users.
The ideal state is a graphic interface where the exercises are presented on the screen to reduce the friction. Gotta start somewhere.
Sure. I would love some help. I am visiting my family all March (I am typing this from the airport), so I don't know how much time I'll have to work on trane for the next month. But maybe that gives you some time to use it, and read the code and docs.
I am not going to put my personal email here, and I don't see an option to send private messages on hn, although it's probably out there somewhere. Probably in the commits, lol. Worst case, feel free to open an issue on the trane repo.
I figured writing the UI in rust is probably not a good idea. The ecosystem for UI is very immature and the language itself is probably overkill. But doing it in typescript/html/css requires you to understand the internal data structures (all the JSON files you see in the courses are just serialized rust objects).
I found that repo and it seems like a promising approach to autogenerate the types and make the interaction easier.
I'm more than happy to do a CLI app, but I think this needs a way to run it on a phone (noticed I don't necessarily mean "app" ;-). I can see me doing this on my phone while waiting for the barista to make my coffee.
I'm going to do something similar with Anki. I'll try to compare the two...
Main difference with anki is that anki explicitly discourages trying to make dependencies between flashcards or decks. I tried to work out something like that into Anki, but as I saw it, dependencies were core to what I had in mind. There's some similar functionality in supermemo, but it's based on a tree, not a graph, so it was inherently limited.
No experience in mobile at all so that'll have to wait, but a GUI/website/app that works on most platforms is something I would like to work out eventually.
I know it's possible. But the last time I did any frontend work, jQuery was the state of the art. So I decided that for now my efforts are better spent nailing the core experience and creating more courses.
This is cool! I've had similar ideas, but when it comes to creating the curriculum it was always the time consuming part. If the graph was in something like a wiki, where people could contribute, add linkages, add new lessons, etc., and a system to automatically prune the graph to each student, there could be a globally sourced and maintained "school" for all human knowledge.
The courses are markdown and JSON files in GitHub so close enough. It does restrict a bit because it assumes some technical proficiency, but there's probably some middle ground (open a GitHub issue to suggest changes and have someone else write the PR, for example).
I really like simple, reliable, tools like this - just watched the demo video. I hadn't heard of mastery learning before, but now I'm thinking about trying to map some Excel lessons towards this and see if I can implement it at work.
I've found that our youngest employees are coming out of school with little-to-no knowledge of how to work spreadsheet software - even those who are otherwise pretty technically proficient.
While programmers tend to eschew tools like Excel, in consulting (or at least the consulting we do) it's critical to being able to understand how a process works before designing an automated solution. Excited to potentially have a way to share that understanding.
Thanks for the feedback. An excel course should be doable. Currently the most general way to write material is to use this: https://trane-project.github.io/generated_courses/knowledge_...
Basically a folder of lessons where the flashcards are pairs of markdown files.
There's also another tool to write simple flashcards and lessons in a json file, run a command, and build all those directories and markdown files for you, but I have not written the documentation yet. Here's an example for a course based on a reworking of Euclid's Elements: https://github.com/trane-project/trane-math/blob/master/cour...
But plain markdown files for now should get you going.
This sounds very similar to the system behind http://mathacademy.com . You follow a graph of skills and dependencies and because you always have the necessary prerequisites it is always doable.
Their team of mathematicians have created a curriculum from k-12 to a bachelors in mathematics.
What in tarnation. For 100 bucks/month you can get 1-1 mastery learning sessions from math PhDs/postdocs that need money (like me! I am the postdoc!).
If you want to learn some math and are willing to fork over $50 a month, my advice is to find a competent tutor online*, tell them what your goals are, and work with them to produce an evolving, customized mini-course.
Have them pick material suitable for your goals, work through it in-session together so they can spot gaps in your knowledge/technique and make progressive problem sets to cover those gaps, and so on.
*A local tutor would be better, but I presume they will be prohibitively expensive in the average HN user's location.
The problem is that finding a postdoc that is also a good _teacher_ is a monumental task of its own.
Compared to an online platform that's being used by a ton of people, has lots of reviews and recommendations and that is, presumably, actively optimized through a feedback loop, it's an easy choice between the two. Also, a platform removes the friction of cancelling the engagement if the need to do so arises for whatever reason.
I mean, if someone I know points at a postdoc and says "this guy is excellent", then their recommendation will prevail. But chances of that happening are next to zero.
I have no experience with those platforms, so I don't know how deep the instruction is. After taking, say, a Calculus course there, would you be comfortable doing Spivak's exercises? If yes, that seems great.
No, you can't do anything in 1h/week, 3 hours should be the minimum, 4 is ideal, depending on the level of the class (I would ask for more than 100 for 4 hours a week if I am to teach advanced harmonic analysis, for example).
Yeah, this is cool and similar to the end state I have in mind. The main difference is that I am not trying to create the curriculum/exercises. I am just creating flaschards that say "Solve exercise x.y.z in that textbook". Plus the whole free-software angle and being able to share the material as text files.
The math thing is just a side project at the moment to try to see how it works for other fields. I am primarily using this for music. I was hoping something like this existed already, but all the solutions are either very specialized like this one or do not support dependencies as a core feature, like Anki. Had Anki supported something like this, I wouldn't have needed to make my own thing.
It would be nice to simply include support for dependency tracking in existing flashcard software. Unfortunately, the most commonly used flashcard software, viz. Anki, has not been adding such features. I can only assume that the project is very lightly maintained and/or working on paying down technical debt - new innovative features don't seem to be a priority.
Agreed, I've also been waiting to see a system with spaced repetition (to help memorize and retain) + dependency graph (to choose what new topics to present). Not sure what the AI value add would be?
I have two children and I was glad that teachers are willing to use new tools (I never asked if they were forced to use them, but I assume not because each teacher seems to use different websites). I'm sure some kids still get bored, but they can let the ones who enjoy math practice at their own pace and provide special guidance to them while still spending the majority of their time helping those who need it.
I think ultimately it's a very different approach to their current one, so it might be too difficult to do in the existing codebase. Maybe not, I have not looked at the code. But the public docs and learning philosophy around anki all seemed to discourage it, so I decided to make my own thing.
And yeah, I don't think Anki gets a lot of support. I am an open-source maintainer myself, so I know that story.
This looks great. My son uses “beast academy” from Art of Problem Solving (https://beastacademy.com/), which is fantastic. But mathacademy looks like it might be a good competitor at the post year 6 level.
My observation is that the paper books are unnecessary because the online experience on beastacademy is so comprehensive - and your answers are auto checked. YMMV.
Sorry, but how is that exactly a solution to the problem in the article? From what I can see, yours is just a (rather user unfriendly) guideline to self-study.
That's 80% of how to use it. Most of the rest is how to select individual parts of the graph to study. So other than setting it up and writing material for it (which I don't envision most people needing to do), I don't see how that's less user friendly than a website. But I understand how using a CLI is inherently limiting. Not enough time as a solo project so I am not focusing on that for now.
As for how it's a solution, see my answer to the other person that asked the same question.
The effect is about how individually-tutored students consistently out-perform classroom students. How does a study guide, such as your program, help with that?
The problem is bridging the gap between the ideal solution (everyone gets an individual tutor) that scales for everyone.
It's not meant to replace teachers, but it reduces the amount of individual teaching needed. In my own case, once I am done with this and can afford the time, I'll get a private music teacher and go through any tough spots I encountered since our last meeting.
Sorry, but you came in way too pretentious. A system like yours may support the student a bit, but it's in no way closing the gap with an individual tutor. Don't say it's a solution. It's an interactive homework list.
I never said I had solved it, only that I was working on solving it. Nowhere in the documentation or in my comments here I intended to give the impression this is anywhere close to done or intended for the general public at this time. There's an entire half of the project intended for instructors that I have barely mentioned publically or had had time to properly sketch out.
Mathacademy is pretty much the same thing from what I can gather and has already been applied very successfully in a school district. Same thing with the DARPA tutor also mentioned here.
All of the principles these tools are based on are already proven by mountains of research. And yet no free software, general solution exists that combines all of them. I know because I resisted working on this for months while looking for it.
And sure, it's an interactive homework list. I say so myself in the readme. One that automatically reviews skills at variable intervals depending on the mastery of the material, interleaves questions from separate topics for increased retention, gradually introduces new topics only after dependent knowledge is sufficiently mastered, balances the difficulties of the questions, and allows students to study specific portions of the graph.
So if we are giving each other advice on how we come across online, I find your comments lacking any criticism I can use to improve the project, and very dismissive of the hundreds of hours I have spent researching and working on the problem with the intention of giving all of my work away for free. If that ever changes and I charge a cent for an incomplete solution, your criticism will be more understandable.
Mastery education is great, and a form of it was shown to be successful in pilot schools throughout the US as early as 1919. The program was called “The Winnetka Plan,” and it was a result of progressive ideas coming from John Dewey.
My father attended a high school where the program was used, and he said it was great. He was able to get help from the teacher when he needed it, and he was able to finish his high school requirements a full one year early. This is because mastery is individualized and is based on reaching learning goals not “time in a seat” or “time in grade.”
There are many articles about the plan on the Internet. Here is a link to one of them:
The question is: why don’t all school systems use this approach? Heck, why don’t more teachers take this approach in their classrooms?
If the goal is mastery of certain basic concepts, then why allow some students (sometimes many students) to fail?
Unfortunately I don’t know the answer. It could be the fact that students can’t sit still and work independently when they are in a classroom filled with other kids. It could be that schools and teachers are there just to control students rather than teach them. Your guess is as good as mine.
It's just a LOT of work to tailor the lesson plan for 30 students, rather than one. A really really good tutor will customize the day's lesson every day for the child.
This is famously envisioned as one of the areas that AI might help with: bringing 1:1 tutoring to the poorest of the world's children. It's the key plot hook of Neal Stephenson's book "The Diamond Age: Or, A Young Lady's Illustrated Primer", wherein said Primer is an AI-powered tablet computer designed to interactively tutor young orphans and other street children to become highly educated.
> It's just a LOT of work to tailor the lesson plan for 30 students, rather than one.
Why bother? If they get it they can advance to the next lesson/unit/certification. If they don’t get out keep working individually or in a group with others of the same level, with the help of a teacher or a more advanced student. That’s how the monitorial[1] system worked. If you group by ability without regard to age the “need” to differentiate vanishes in a puff of smoke.
Ah interesting... that's a very good point! Then why aren't these programs rolled out all over the US and stuff in public schools? Is it just an example of government incompetence? Are there any younger public administrators seeking to fix the administration or whatever? Sorry I don't know much about this yet but it sounds very promising.
While I loved the idea of the Primer, I think one of the themes of the book was that it actually couldn’t work fully with just AI. All the children who end up “exceptional” (eg not just part of the mouse army) actually had a human tutor behind the scenes.
Agreed. I guess that's one gap between the rich and the poorest that will always be hard to bridge.
If we could adapt some concepts to helping kids thru a government program, it could still be pretty cool to think about allocating even a small % of what we spend on military procurement to maybe go into giving AI tablets to every child that taught the most basic stuff like healthy diet, keeping a personal budget, how to learn about elections and vote in them, etc., and give the kids access to creator tools too in a way where it's in the Start Menu or whatever but not forced upon them. That way maybe most kids just mostly play games, and do a few mandatory lessons about diet & how to vote, but the kids who are self-motivated and driven to explore can open up stuff like Unity, Roblox, Blender, Visual Studio Code, and more. And these kids who live w/ five siblings in crammed in a Projects apartment the size of a large closet, can still express their genius thru the tool of computing.
I hope it could maybe be a pretty inspiring huge improvement in the lives of a lot of kids who deserve better.
I think the hardest part about doing this in real life is that in the book the Primer's hardware is always speedy & cutting-edge but in real life a tablet computer that's more than 2-3 years old feels obsolete, and I don't want kids that have the government-issued tablets from when they were little to have to carry that as a mark of shame.
For a very literal real-world example, there was the One Laptop Per Child project. There are a lot of critiques of it but let's focus on the core of what they did which was give free laptops to impoverished children in the Global South. The laptops were quite well-designed (I did an EE tear-down of their first version laptop as a university EE project, and came away impressed). Parts of the schematic and architecture are almost on the level of something from Woz. But it's hard to imagine how children given the equivalent of a netbook will be able to keep up with other kids as the world moves on to VR/AR etc and these children are left behind, with no way to run Roblox or anything the cool kids are doing.
I'd definitely suggest though that you read about the project, which is still ongoing in some form a number of years later. It's really interesting and the closest thing there is right now to the Primer. Maybe you could lend your ideas to them or help with open source if educational technology is one of your passions, as it is mine and as it seems to be yours.
We don’t need hardware to make this possible. We need very well designed content - lessons and software - to drive equity in education. The web can do the rest. Every day more and more of the world gets connected to the web - the screen they use to access doesn’t need to be fancy. It’s just a door to the content.
I wonder how aging populations and large-scale language models are going to change the difficulty of this problem. It seems to me that we're just about to unlock 1-to-1 learning, without actually needing 1 teacher per child.
It isn't hard to imagine a language model (or something similar, I'm not in AI myself so I might not have the best idea of the specifics) giving personalized instruction to a student based on a knowledge bank and generative prompts + reinforcement from what the child understands.
Add to that aging populations leading to less children and higher worker-child ratios, + automation in other industries, and its plausible to imagine that there could be a large movement towards training new teachers, thereby reducing class sizes. This could increase feedback between the two, and probably reduce behavioural problems as teachers could address them more easily.
I guess one problem of this approach is that this 'atomizes' the student, where they only engage with their device with the language model, removing some of the collaborative aspects of modern instructional methods. One easy fix to this is to increase the amount of recess or structured play, thereby increasing socialization. This seems to work in Finland [1]. If advances in AI can make learning more efficient, we can increase the amount of time for unstructured play.
It's a nice dream. I would just caution that we will unlock people claiming that this works and foisting a half-baked version of it on large populations to save money, look modern, and make profits for tech companies... long before we unlock the actually good version.
Volunteer trials, pilot programs, slow rollouts... there are plenty of ways to test a beta version without making the entire population guinea pigs for someone's shower thought. But such careful methods mean the conman might not get his big score.
Always watch out for the false sense of urgency. Remember when we had to invade Iraq immediately because weapons inspections weren't working fast enough?
It seems to me the larger issue is most people don’t want to learn, or don’t have the environment conducive to learning. It’s already possible to learn in a self-directed way without a teacher. I didn’t even go to high school, all my higher education happened on my computer using the Internet, and still does. The tools for learning many things without tutors are available.
Imagine if we had a perfect tutor using a LLM. From my experience most people won’t take advantage of it anyway.
Look at television as a contrived example. Before it was widely available one could have imagined that everyone would be able to learn on video, once the cost fell to where everyone had a TV. But what happened is everyone used the TV to entertain themselves instead.
We need to study what amounts of 1:1 tutoring have positive outcomes. If society can’t afford 1:1 tutoring for everyone, all the time, can we afford 20% of the time?
HN tends to filter for people who are more academically inclined; people for who mastering a topic is a goal, and believe that education systems should support that.
Taking a wider view, this is not really something that society wants in a widespread system; mandatory education systems do two things: They provide a minimum skill level; and they filter people into groups based on whether or not they can pass some bar. Better outcomes for more spending only really becomes a topic where the minimum bar is not being met.
Education systems/techniques set up for effectiveness first do exist, but typically in spaces where there is a need for an individual to obtain a specific skill (military education, employer training in high demand and skill areas, private tutoring).
> Better outcomes for more spending only really becomes a topic where the minimum bar is not being met.
Usually we just get calls for more spending, without the better outcomes. It's all about throwing good money after bad, into a failing system that's already among the most expensive in the world.
My understanding is that the knee of the curve is ~10-12 students. Much cheaper than 1:1 or 1:2, but still more expensive than the 30+ classrooms of my youth.
You can teach large classes efficiently, but it requires a very direct, rote, scripted approach that's exactly what most teachers don't want to use - because fancy education schools have told them that it "demeans" the profession, and they've drunk the Kool-Aid. A key component is to come up with memorable text that the students must be able to repeat word for word when prompted - both as a group and individually. (Problem solving similarly starts with trivial cases that the students must answer on the spot; difficulty is very gradually increased, and there's a lot of retracing and review of earlier steps.)
This is as close as you can get to "mastery learning" in a large classroom; keep all students on lockstep and highly engaged, trying to ensure that no one falls through the cracks.
The college students who are using Anki to cram incredibly complex material in the form of text would surely disagree with your assessment that memorizing text is "worthless".
Are tests which gauge a college students ability to regurgitate text verbatim useful? Are these college students getting what we think is worthwhile learning out of these cramming sessions?
"As close as you can get" may be true, but it still doesn't sound very close. In fact the memorization part sounds profoundly miserable. "A lot of retracing" of stuff that, frankly, I probably understood the first couple times also sounds miserable, and inefficient. So, uh, the kool aid is pretty tasty I guess.
It's not "miserable" if the lesson scripts are designed for it. In fact, the challenge of gradually memorizing "the lesson" until you know it word for word can be playful and engaging.
I’d think you can get away with 5:1, a single good teacher can give a ton of individual attention to 5 kids studying completely different things on their own.
Has anyone used any of the "intelligent tutoring systems" mentioned in this Wikipedia article?[0]
Lots of comments here are saying "we'll have an intelligent tutoring system one day, based on LLMs" -- but there are many such systems that already claim to be getting good results now.[1] I'd be interested to hear from anyone who's used one.
Going through the list, it seems like all of them only work on very specialized subjects and/or are not published. I have read about the DARPA tutor, but I don't think it's been made publically available. So I doubt anyone here has used it, unless they were in that specific branch of the military at the time it was used.
I don't see how LLMs deliver on this. I think they could speed up how the curriculum is generated, and I have played around with ChatGPT to do just that, but the questions only need to be generated once. It's the guiding of students through the exercises that needs to be personalized, not the exercises themselves.
As for their effectiveness, I am the author of Trane (see my other comment). It's worked pretty well for me, but I am splitting my time between my job, working on the software, creating materials, and actual practice, so I can't really say what the upper limits are until I can focus on just the last part.
I am a customer of mathacademy.com, an intelligent tutoring system with a focus on math. Right now they are putting the finishing touches on their courses like abstract algebra and discrete mathematics. They already have linear algebra and calculus.
It absolutely works as advertised and I highly recommend it.
I took a look at the site you mentioned, but I don't see how it's useful for learning. It uses AI to determine what the learner's current level of knowledge is, but the way the learner learns seems completely typical. Then I may as well read classic textbooks. It doesn't seem to me to be that meaningful to determine the learner's current level of knowledge.
The benefit from 1-on-1 tutoring comes not just from finding what the student knows. There are any number of standardized tests that can determine that. Khan Academy does this for free and also adds lessons for mastery based learning.
The benefit comes from understanding how the student misunderstood a concept being taught to them and then fixing that misconception.
I think this claim is too strong. I’ve given and received more 1:1 instruction than most, plus my share of group instruction.
The correcting misunderstanding is a big deal. Don’t get me wrong. The big downside is it’s very dependent on teacher ability.
I’d say the bigger factor, and luckily one that scales well, is teaching the right thing to the student at that moment. You either have unmet dependencies of knowledge, needless repetition, or the right thing. You could also call this fast, slow, just right.
There’s a huge variance of what people already know, even if they’ve been through the same classes. This means if you put a group of people together, the way to teach the most to a group is to teach at a crawl. You’re not teaching to the bottom of the class, you’re teaching to the bottom of the class at any given moment.
1:1 you can just fly in comparison because you can scale up and down the time per topic easily 10x for new material. If you include there may be review, it’s totally reasonable to think one person may need 1 minute (check) or 100 minutes (learn).
The thing about this is that the math is really against you at any number greater than 1. Even 2 is a step change (also for social reasons). So trying to get a smaller class isn’t nearly as effective as springing for 1:1. I mean, you could even say it’s 2 SD better.
> I’d say the bigger factor, and luckily one that scales well, is teaching the right thing to the student at that moment.
I agree that this is a bigger factor for most in the current environment. I'd argue that this is already a solved problem though with Khan Academy for free. The benefit from 1-on-1 tutoring beyond that has little to do with understanding what the student knows and teaching what's next because that problem is already solved to a degree that is almost indistinguishable from what a skilled tutor can provide. The benefit comes mainly from the tutor being able to figure out why the student answered a question wrong and directly addressing it, speeding up deep understanding.
I disagree that teaching what’s next for the student is a solved problem. Usually the resolution is too coarse and the speed is barely variable at all.
Everyone here is asking the right questions. How fine grained are the assessments? What is the tempo? Etc. However, I don’t see any objections that haven’t already been addressed.
This graph traversal based learning is obviously an idea whose time has come and we will probably see many similar companies spring up in the coming years.
Our startup started working on solving this problem with extremely customized continuous AI Tutoring last year. In his research paper, Bloom says that private tutoring is the most effective learning method, but similar results can be obtained through a combination of scientific learning methods. However, in his time, these methods were too computationally expensive for teachers to implement.
Now, we have the technology (LLMs) for everyone to have an accelerated contextual learning experience. We started by tracking mastery learning with graphs, but the effort required was too much for users. Now, we're augmenting the previous system with ML to automatically track and explain new concepts. In the far future, I'm hoping we'll be able to accelerate learning rates by 150%, so that motivated highschool students can graduate with a bachelor's degree.
This research will be ignored because if they implemented this individualized learning program with AI, there'd still be a bunch of kids who would do poorly and the smart kids would do great and then people would complain even though all kids would do better.
This is kind of like how they got rid of AP classes in California because the racial outcomes were unequal[1].
I had a conversation about 10 years ago with a high-ranking DoEd official in Obama’s administration who was gushing over 2 sigma as a strategic framing for public education. Now this person is a very influential funder in edtech and ed reform. I think their heart is in the right place but it’s kinda crazy how no one above the school principal level is actively building strategy or executing tactics to address the basics of a good learning disposition: sleep, nutrition, safety, etc. By the time the robo-tutor is perfected, normalized trauma is going to make it useless for most school kids.
“… it’s kinda crazy how no one above the school principal level is actively building strategy or executing tactics to address the basics of a good learning disposition: sleep, nutrition, safety, etc”
By virtue of having been students, many people assume they understand how to be an effective educator - but many problems in schools are not, strictly speaking, education problems; they are social problems they come to school.
theres a lot of hope in the educational AI space that we can do personalized teaching and essentially raise everybody to the 90+ percentile implied by the 2 sigma problem (see https://lspace.swyx.io/p/chatgpt-gpt4-hype-and-building-llm 20 min mark - disclaimer its my podcast with openai's new representative)
i think this is a "last mile" type situation where we've seen great progress with ChatGPT on the first 90%, but the last mile will take the remaining 90% to do. not impossible, just won't come anywhere as soon or as easily as people want. still worth pursuing ofc, but not as a quick hustle but more if you are just intensely insightful and passionate about education and figure out how to apply AI to personalize it properly
Since ChatGPT often writes untrue statements, it seems like quite a poor teacher. I guess it could be used as an assist for an instructor (send me a question, chat GPT will answer it, I’ll check the output) but that seems not so revolutionary.
This is a topic that interest me. I see myself as a life long learner and I want to strive towards mastery. In some topics I’ll maybe get a tutor, it’s expensive but worth it.
I’ll share my workflow for studying and I am keen to hear about yours.
I start with exercises almost immediately when studying. Practice over theory. I have noticed that an unsolved problem, or a solved problem that I didn’t understand really bugs me. I use that irritation to fuel me when I go over the theory of a subject.
When doing exercises I count wrong answers from negligence as worse than just plain wrong. That’s because I believe it could stem from a lack of understanding disguised as negligence.
If I am struggling with a concept I’ll first try to copy the chapter of the book by hand. To internalize the author’s voice and framing. If that doesn’t work, I’ll search for another explanation and do the same thing.
I prefer books over video lectures. Video is to intense and I can’t as easily set the pace. Some video content out there is amazing, specially with math and visualization of math. That’s really helpful for me.
I don’t stress since I am going to this until I die. I play the long game.
The problem is that we are not giving the best education we can to everyone, as it is too costly to have one-to-one tutoring for all. So is there a method that works as well as one-to-one tutoring but affordable by our society?
I believe it’s that Bloom’s study implies that 1-on-1 pacing and teaching produces the best learning results, but that Bloom thought it was infeasible to scale to everyone: it’s “too costly for most societies to bear on a large scale”
Beyond that are the equity effects arising where availability of such tutoring primarily to wealthy families means that existing differentials in opportunity and advantage are further multiplied.
There are a few potential options that appear in or occur as one reads the literature (the linked Wikipedia article lists several references, beyond Bloom's own work). That includes inflection points in the effect, that is, some advantage occurs with lower degrees of specificity, particularly with instruction at a 1:10 -- 1:15 instructor/student ratio or so. Another option would be to randomly apportion tutors to the general population, or to assign tutors either to particularly talented students (regardless of family wealth/income), or to specific underprivileged populations (to reduce systemic and historic educational gulfs), or simply at random (which avoids numerous social, political, personal, and/or academic faddish biases).
montessori seems to be the solution here. it is essentially individual tutoring at scale because the teacher/tutor doesn't need to spend 100% of their time with one child. a lot of time the children are practicing on their own, and also they are learning from older students (the tutor doesn't always need to be the most qualified person around, they just need to be able to teach the material that the student is learning). they also don't move on to the next segment until they master the previous one.
but you just need to look at the montessori methodology to see how it works. it may not be as effective as pure one-on-one tutoring but it is certainly way more effective than regular classroom teaching.
keep in mind though that the name montessori is not protected and that there are many schools out there that are montessori in name only without actually having trained teachers.
the lack of good studies of montessori is frustrating to me. the montessori method is as scientific as education can get, free of any ideology or philosophy, unlike eg. waldorf, and therefore should be easy to apply on a wider scale. the teacher training is not expensive (about a year) and could easily be integrated into any general teacher education. the teaching material is more elaborate than just books, especially for the younger age groups, but the high cost now is mainly due to a lack of scale.
it is a core driver of compounding inequity, assuming the rich can afford personal tuition and the poor cannot, which then gives rich kids access to better opportunities to continue being rich. it gives the lie to “equality of opportunity not equality of outcome” because any game of consequence involves outcome being exclusive opportunity.
This is one of the most important problems to solve IMO, especially for people who have left the traditional academic system. I feel that having the pressure to perform under examinations, as horrible as it was, at least got some level competency over the line but not having that anymore and learning outside of work has been a tricky affair.
I suspect that the solution to this problem might be whatever is common in lots of different learning styles.
While I am a language model trained by OpenAI, I am not a substitute for human teachers and I do not have the ability to solve Bloom's two sigma problem on my own. However, I believe that AI and technology can play a significant role in addressing this problem and improving education outcomes.
In the future, AI and personalized learning systems could help to provide individualized feedback and support to students, similar to the benefits of one-on-one tutoring. AI can also be used to provide real-time feedback to teachers, helping them to adjust their teaching methods to better meet the needs of their students.
However, it's important to note that while technology can be a valuable tool in education, it cannot replace the human element of teaching entirely. Teachers play a crucial role in creating a supportive learning environment, building relationships with students, and providing guidance and mentorship that goes beyond what a machine can offer.
So while AI and technology can certainly help to address Bloom's two sigma problem, they should be seen as complementary tools to support teachers and enhance the learning experience, rather than a replacement for human teachers.
Wow, I think it's truly amazing that it seems like ChatGPT is the only one in the room mentioning, that «creating a supportive learning environment, building relationships with students» is key to success. I haven't read Bloom's paper in detail, but to me it seems very obvious, that the relationship aspect of 1:1 tutoring is a huge factor in the equation, which (as the AI rightly states), cannot be replaced by better machines.
ChatGPT is already amazing. I’ve let a smart kid loose on it and he instantly asked these questions:
1. How do you work?
2. What should I learn to understand what you wrote (I am 12)?
3. Write a script for a YouTube video walkthrough of xyz game (can’t remember).
"If you throw 30x more resources on a problem, you are usually able to achieve outstanding results even with mediocre quality inputs" - i don't know why it looks like a surprise to anyone.
i wonder how much all the redundant and slightly different resources on the internet today close that gap... this paper is from 1984.
also would be curious how good today's llms are at getting someone to grok a topic they're having trouble grasping... (this can be actually a hard problem as it can require understanding the misunderstanding in order to guide away from it effectively.)
If I ask ChatGPT how transformers work the results are sufficient but brief. It’s difficult to probe ChatGPT, and it doesn’t have an intuition for your weak points like a tutor does. If I ask ChatGPT about who influenced John Maynard Keynes however it will hallucinate a whole corpus of economists and papers that never existed but whose existence is proposed so confidently it is incredulous.
Mainly we should appreciate the dearth of capabilities that arise merely as emergent properties of ever larger language models. Some kind of architecture will be needed for each use case, which means we once again find ourselves facing difficult engineering problems.
The history of proposed technological replacements for human teachers is filled with highly enthusiastic and widely touted failures. It's not simply a matter of presenting information --- a book, or streamed lecture, or audio or video recording can do this --- or even interactive engagement ("educational software"), but a highly-attentive in-the-moment back-and-forth and with the student, sensing where they do and don't follow or understand a lesson, and most critically allowing them to discover for themselves a principle, concept, or fact.
It's not that these cannot happen alone, via self-directed (autodidactic) study, or with the assistance of technologically-mediated aides. But time and again the effectiveness of such methods falls far short of direct human instruction.
LLM AI chatbots can respond to written or verbal cues, to an extent, but as yet lack an underlying pedagogical framework and understanding of the context of a communication.
From what I've seen so far of such models, "lacking an underlying framework", the bones and skeleton of a conversation, if you will, which can be subsequently fleshed out, seems to be a general failing of the models, so far.
I'm not saying that this cannot emerge, and parts of this may in fact be a relatively simple additional step or advance. Given other advances such as sentiment and mood assessment and analysis they may even be largely in-place. What's left however is the motivation of such efforts, and alignment with the educational mission, again a place where previous technological education programmes have foundered, as most have been coopted by, if not outright conceived as, propaganda and cultural programming tools. (A part of traditional education as well, but far less effectively and uniformly delivered when mediated at 1:15 -- 1:30 or so ratios amongst millions of instructors[1]). It will all but certainly be difficult for corporate-based LLM models to resist such temptations, see the earlier case of Backrub and perverse incentives.[2]
How many teachers in the real world are actually engaged in "a highly-attentive in-the-moment back-and-forth and with the student, sensing where they do and don't follow or understand a lesson"? A lot of the time, the pretense of "allowing them to discover for themselves" simply turns into not teaching at all. This whole notion that students will simply educate themselves on their own may be a comforting idea, but plenty of evidence shows that it's just not true.
The question is how effective human teachers are relative to alternate technological methods, which I've already addressed:
But time and again the effectiveness of such methods falls far short of direct human instruction.
How human teachers perform relative to some abstract ideal isn't particularly interesting if that ideal isn't viably attainable. All pedagogical methods are relative.
Teaching efficacy does of course vary, though most studies show that other factors, including both environmental (e.g., student's home life, parents, neighbourhood, income/wealth) and institutional (school or district as a whole) matter far more than any individual teacher.
This is a key point in Cathy O'Neill's book Weapons of Math Destruction, mentioned in her Ted Talk here (at about 3 minutes): <https://yewtu.be/watch?v=_2u_eHHzRto>. Of NYC public school teachers tested, 665 had two scores, for which there was virtually no correlation.
Keep in mind that "institutional factors" frequently involves "eliminating educationally-disadvantaged students", by many methods. One such discussed in the past month at the New York Times, "How Educators Secretly Remove Students With Disabilities From School" (10 Feb 2023) <https://www.nytimes.com/2023/02/09/us/students-disabilities-...>
There's a long history of ed-tech failures. You might find Failure to Disrupt: Why Technology Alone Can’t Transform Education by Justin Reich to be a useful introduction:
The "institutional factor" described in the NYT article is teachers "informally removing" students that they view as disruptive from the classroom. If anything, this looks like a case where human teachers are being ineffective (if perhaps sometimes through no fault of their own) wrt. the goal of educating these students, whereas better ed tech might be especially helpful.
I feel we're reading different stories. The article clearly describes systemic policy and decisionmaking:
In kindergarten he became eligible for special education for what school officials described at the time as a “communication disorder,” but they opted instead to place him in a regular classroom and have him pulled out for instruction in a smaller group.
And later in the story, on the same case:
A few weeks later, the school team emailed Ms. LaVigne to set up another meeting, offering to add one class to Dakotah’s schedule in December.
Note the language "school officials" rather than, say, "a teacher".
Similarly in the second case described:
Records show that the Sacramento City Unified School District has a history of disciplining students with disabilities, particularly those who are Black, at a higher rate than most other public schools in the state.
Rather than get hung up on the specifics of one specific story that happened to come to mind, you might want to consider the general dynamic and other modes of student segmentation as well: segregated schools, charter schools (with selective admissions criteria), voucher programmes, and the like.
Again: all change the overall nature of the school far above and beyond the specific teaching abilities of any given instructor. A school or district whose policies and practices permit exclusion of challenging students isn't improving teacher's pedagogical skills, but rather selecting whom specifically they're permitting to be educated there.
Different educational modalities and/or intensive education might be another approach. That isn't what the story is describing or what you seem to be proposing.
Education is going to be radically changed when AI can manage to helpfully teach students with a Microsoft Tay situation happening.
Very quickly after that, I think a sort of global shift will begin, as schools all around the world begin to see the power of legitimate curriculums, instead of lobotomized "think how I think" bs. I'm thinking of Florida right now, but I know the same shit goes on in India, China, countries in Africa, and probably other places too.
On a longer scale, as models become more and more advanced, maybe there will be teachers that are constantly absorbing new information and are able to keep students up-to-date with best practices, new research, etc. That would not bode well for the leagues of McKinsey-ites and that whole ilk.
I am particularly hopeful that AI will be able to radically advance education.
That is, if this hypothetical technology is democratized and kept open, rather than horded and sequestered to the ruling capital class until there exists a financial, educational, biological and altogether insurmountable moat between the ultra-wealthy and the rest of us.
No AI needed. Just an old fashioned depth first search through a graph of skills and dependencies.
I made it to help me practice music, but I have been branching out and using it to study math for a few weeks. I find myself saying "just one question more" and then spending another half hour in it.
Still needs more material to be useful to other people but it's a solid experience. I learned and memorized how to play most of the notes in the saxophone with good intonation in about a week, as a complete beginner.