Replying to @meatballtimes
Unfortunately many teachers (and thus students) have no clue on what effective studying looks like (see last pic)
Borderline criminal when you actually think about it. Instead there's hours every day wasted on fluff techniques that barely move the needle but feel effective.
Where else can you find a legitimate 2-3x increase in A-grades from 1 oft-disregarded method?
shaeda.blog/s/research
Did you know Bloom's very famous "two-sigma problem" is widely misinterpreted - and often intentionally by EdTech apps.
1. Bloom performed zero experiments himself. He simply summarized two (small) PhD dissertations by his own students (Anania and Burke). These authors are rarely even mentioned or known. One dissertation was never published at all; the other has been cited only 77 times. Neither student ever followed up with replications or further tutoring research.
2. The iconic +2 sigma bell-curve graph everyone quotes is not real data, it's a hand-drawn illustration Bloom created to “show what it might look like.”
3. Bloom’s data distorts even his own cited source (Herb Walberg’s meta-analysis from one month earlier). Walberg reported ordinary tutoring at just 0.40 sigma. Bloom relabelled it narrowly as “peer tutoring,” and then inserted the 2.00 value to put tutoring at the top.
4. In his student's studies, topics were both deliberately narrow and also brand-new to the students involved (probability for 4th/5th graders, cartography for 8th graders). Of course novices on unfamiliar material produce outsized short-term gains, but do they generalise with wider topics and/or long term?
5. Tests were custom, researcher-created, and tightly matched to the exact 3-week lessons. They were not standardised or broad (SAT, state tests, etc.). Effects on such narrow tests are routinely 2–3× larger than on real standardised measures. This is 99% of EdTech summed up: apps say users are learning great to keep them paying and sharing with others. Duolingo will race to tell users that they are approaching fluency.
6. The entire “intervention” lasted only 3 weeks (!) (11 × 40-minute lessons) with immediate post-testing with no long-term follow-up
7. The “tutoring” condition was not pure one-on-one tutoring. It was a mix of one-on-one plus strict mastery thresholds. Roughly half the 2-sigma gain (~1.1 sigma) came from the mastery-aspect and informative feedback alone
8. Modern meta-analyses of actual randomised studies find average effects of only 0.33–0.37 sigma across 65–96 RCTs. Zero large-scale studies have ever replicated 2 sigma.
9. +2 sigma is crazy in real terms outside of very specific contexts (new material, cocktail approach, very short term): it would turn a median student into the 98th percentile or deliver 5+ years of school learning in three weeks.
10. He conflates formative assessment (short-term feedback quizzes) with summative claims about broad, long-term achievement. Once again, EdTech.
11. +2 sigma gains do exist, but only at the system level over several years via excellent whole-class instruction (e.g., England’s top Progress 8 schools on broad subjects across 600,000+ students). Not from short-term individual tutoring interventions.
12. Post-COVID governments poured huge amounts of money into programs they thought were explicitly justified by Bloom. Results: 0.06 sigma in most districts, at best 0.16–0.37 sigma in the best structured programs
educationnext.org/two-sigma-…
substack.nomoremarking.com/p…
i'm now approaching double-digit figures for people getting back in touch with me who have realised this
Replying to @yargward @canonical_pers
don't get enough reviews in general, especially on just facts based things, they were meant to release something for that but then just didn't, there used to be a guy who was pretty vocal about MA issues, @shaedapk and i used to fight him over it lol but he was right about a lot of the issues
just posted a quick article on the topic of self-made flashcards versus pre-made, consider reading if interested
shaeda.blog/p/should-you-cre…
Important thread for education-based folk 📝 💻
We (very) often hear in education circles that pen-and-paper note-taking is superior to tech-based. I’ve long been skeptical of these studies/articles (for unrelated reasons) but it’s good to see a breakdown of some actual statistical problems
(Not that either method is particularly effective for retention, although perhaps there’s statistical issues in that branch a well…)
Just 2 days ago
(@TolentinoTeach see above)
Replying to @TolentinoTeach
This is very commonly spoke about but there's a lot of nuance and context to it which makes it often misinterpreted, in my opinion
For example, off the back of this oft-heard finding I've seen many suggest abandoning, for instance, Anki, and instead opt for manual cards with a heavily-simplified interval. This is hugely flawed and detrimental for students long-term.
One study I would like to see though is simply adjusting for writing/typing speed. It's perfectly plausible that the primary driver of this (context-specific) finding is that students write slower and thus spend longer on the material. Well, what about if they typed slower?
Further, all the papers I've seen on this so far seem to be on a fixed (and small) amount of material. But learning outside the lab is not fixed, nor small. If a student has, say, 20% better recall on written versus typed content, but the typed student has covered 3x more material, the typed student would actually be better off, but the finding from research would be the opposite.
Did you know Bloom's very famous "two-sigma problem" is widely misinterpreted - and often intentionally by EdTech apps.
1. Bloom performed zero experiments himself. He simply summarized two (small) PhD dissertations by his own students (Anania and Burke). These authors are rarely even mentioned or known. One dissertation was never published at all; the other has been cited only 77 times. Neither student ever followed up with replications or further tutoring research.
2. The iconic +2 sigma bell-curve graph everyone quotes is not real data, it's a hand-drawn illustration Bloom created to “show what it might look like.”
3. Bloom’s data distorts even his own cited source (Herb Walberg’s meta-analysis from one month earlier). Walberg reported ordinary tutoring at just 0.40 sigma. Bloom relabelled it narrowly as “peer tutoring,” and then inserted the 2.00 value to put tutoring at the top.
4. In his student's studies, topics were both deliberately narrow and also brand-new to the students involved (probability for 4th/5th graders, cartography for 8th graders). Of course novices on unfamiliar material produce outsized short-term gains, but do they generalise with wider topics and/or long term?
5. Tests were custom, researcher-created, and tightly matched to the exact 3-week lessons. They were not standardised or broad (SAT, state tests, etc.). Effects on such narrow tests are routinely 2–3× larger than on real standardised measures. This is 99% of EdTech summed up: apps say users are learning great to keep them paying and sharing with others. Duolingo will race to tell users that they are approaching fluency.
6. The entire “intervention” lasted only 3 weeks (!) (11 × 40-minute lessons) with immediate post-testing with no long-term follow-up
7. The “tutoring” condition was not pure one-on-one tutoring. It was a mix of one-on-one plus strict mastery thresholds. Roughly half the 2-sigma gain (~1.1 sigma) came from the mastery-aspect and informative feedback alone
8. Modern meta-analyses of actual randomised studies find average effects of only 0.33–0.37 sigma across 65–96 RCTs. Zero large-scale studies have ever replicated 2 sigma.
9. +2 sigma is crazy in real terms outside of very specific contexts (new material, cocktail approach, very short term): it would turn a median student into the 98th percentile or deliver 5+ years of school learning in three weeks.
10. He conflates formative assessment (short-term feedback quizzes) with summative claims about broad, long-term achievement. Once again, EdTech.
11. +2 sigma gains do exist, but only at the system level over several years via excellent whole-class instruction (e.g., England’s top Progress 8 schools on broad subjects across 600,000+ students). Not from short-term individual tutoring interventions.
12. Post-COVID governments poured huge amounts of money into programs they thought were explicitly justified by Bloom. Results: 0.06 sigma in most districts, at best 0.16–0.37 sigma in the best structured programs
educationnext.org/two-sigma-…
substack.nomoremarking.com/p…
Replying to @kutluokan
Will exclude secondary or tertiary issues:
The simple answer is it's just not what it says on the (beautifully-marketed) tin - in it's current form. A couple small and neccesary changes and it would be better.
Last year I had a couple very passionate users state that I/we (who cancelled) must've just been using it wrong somehow, but A) were not able to say or even guess how, just "wrong", and B) these individuals we're not willing to take a simple reset-diagnostic (or even just a third-party test) under strict conditions (no guessing, referring to notes etc).
If MA is what it says on the tin, 100% of users should be able to take this reset diagnostic under strict conditions and be placed back exactly where they currently are in the system, maybe +/- ~2% due to their SRS config (another core issue and basic premise of SRS - prioritise old, not new)
In the absence of taking such a step (reset diag, or third-party), it is no different to a user passionately insisting they're fluent in a language because Duolingo told them while refusing to have a conversation. That's not to say they can't speak a word of course, just not at all fluent. Business 101.
Myself, I was at ~50% MF2 but realised for a very long time that I just didn't have any actual knowledge or understanding of the material. I was answering questions correctly for months and asking myself "Damn, if someone asked me to explain this I'd be in trouble". This is not ideal when you're paying $50p/m *to* have an understanding. ChatGPT is at olympiad-level, what good is advancing without understanding in this day and age? This realisation of minimal understanding is also common from others online and requires one to develop some sense of metacognitive awareness.
Anyway, I took the reset-diag and was regressed back from 50% MF2 to something like 80% MF1 + 30% MF2. I've forgot the exact numbers.
"The system didn't work for you/those" seems a very strange response given the context.
--
PS: Should also add that I'm very critical of 99% of EdTech in general (incl. Alpha). Criticisms above can be applied to probably the top 50 apps (excl. Anki) on the App Store right now.
6am where I am and haven't slept. Will reply in morning or DM on more if interested. End of the day, each to their own 🤝
c retweeted
Fuck this disgusting ‘sport’ and fuck anyone who watches and supports it.
bbc.co.uk/news/articles/cd70…
Replying to @meatballtimes
Unfortunately many teachers (and thus students) have no clue on what effective studying looks like (see last pic)
Borderline criminal when you actually think about it. Instead there's hours every day wasted on fluff techniques that barely move the needle but feel effective.
Where else can you find a legitimate 2-3x increase in A-grades from 1 oft-disregarded method?
shaeda.blog/s/research
Replying to @flowidealism
Outside of obvious concerns, research consistently finds that passive study (reading and rereading) is generally much less effective than active study (recall)
Worse still, students *feel* that passive study is more effective, so they carry on using them
xitter.cf/shaedapk/status/203062…
Replying to @JonHaidt
Most (all?) ed-based apps are built and optimised for business, not learning
Apps that are geared towards effective studying (i.e., Anki and/or flashcard principles) seem to quite consistently result in better performance - why wouldn't they?
This is almost certainly muddying the waters significantly.
shaeda.blog/s/research
The most effective apps are not the most complex or most gamified, they're actually the simplest.
Anki is the best but comes with a huge amount of friction in setting up. Luckily, there is shaeda.io soon.
Replying to @JonHaidt
Most (all?) ed-based apps are built and optimised for business, not learning
Apps that are geared towards effective studying (i.e., Anki and/or flashcard principles) seem to quite consistently result in better performance - why wouldn't they?
This is almost certainly muddying the waters significantly.
shaeda.blog/s/research
Replying to @MIT_CSAIL @CuriousMindsHub
Flashcard the things you need to know and review consistently - that's probably 90% of it.
Ditch any heavily-gamified apps and start actually learning.
xitter.cf/shaedapk/status/203062…
Replying to @JonHaidt
Most (all?) ed-based apps are built and optimised for business, not learning
Apps that are geared towards effective studying (i.e., Anki and/or flashcard principles) seem to quite consistently result in better performance - why wouldn't they?
This is almost certainly muddying the waters significantly.
shaeda.blog/s/research
Replying to @avikrishna @justalexoki
Have they published data on its estimation accuracy for non-trivial meals?
Fundamental problems such as lack of scale (is it 100g or 300g?) and hidden ingredients (butters, oils, sauces, food items underneath the top layer etc) are not AI problems they're camera problems. How can an LLM solve that?
Genuine question.
New blog post
MrBeast, Bald and Bankrupt and Education
shadeapinkramblings.substack…



































