Comment by ryuhhnn

Comment by ryuhhnn 17 hours ago

14 replies

View on Hacker News

Some very important context that the researchers don't mention: during the same period that they are claiming test scores improved because of phone bans, Florida changed the way they administer standardised tests. Starting in 2024, they switched from doing one end-of-year assessment and started administering more frequent tests throughout the year in order to better gauge a student's progress and provide a tighter feedback loop. (source: https://www.educationadvanced.com/blog/florida-standardized-...)

It's much more likely that simply changing the way they administer these tests had a more significant impact on test scores than phone bans.

rahimnathwani 12 hours ago

This criticism would be valid if the researchers had studied just one group of schools, and their methodology was just comparing before and after. But that's not all they did. They had two groups of schools, with low/high cell phone use before the ban. Their hypothesis was that the schools with high cell phone use would see a larger change in test scores (as they would have the largest drop in mobile phone usage).

  We then turn to our causal analysis comparing schools with different degrees of apparent pre-ban student cellphone use, after vs. before Florida’s cellphone ban. We show that the ban increased disciplinary incidents and suspensions significantly in the first year, immediately after the district started referring students for disciplinary action for cellphone use infractions. In particular, our difference-in-differences estimates suggest that the ban increased suspension rates by 12 percent (relative to the comparison group mean) and in-school suspension rates by roughly 20 percent in the first year.
There may be other reasons to criticize the paper, of course.
Reply View 0 replies
ecb_penguin 9 hours ago

This was controlled for in the study.

I swear sometimes people only exist to look for flaws in studies they didn't read.

Reply View 1 reply
  • [removed] 8 hours ago
    [deleted]
    Reply View 0 replies
jobs_throwaway 17 hours ago

> It's much more likely that simply changing the way they administer these tests had a more significant impact on test scores than phone bans.

Why do you think that's more likely?

Reply View 5 replies
  • ryuhhnn 17 hours ago

    Put yourself in the student's shoes: instead of being required to rote memorise every detail and hold that in your head until the end of the year, you are now only required to be assessed at the time that you are learning the material. Do you think you'd fare better on that type of test, or a test done months after you actually studied the material?

    One of the first things they teach you in educational research is that standardised test scores are significantly impacted based on how the tests are administered and what the test is actually assessing.

    Reply View 4 replies
    • kelnos 5 hours ago

      That also depends on how the tests work. If each test covers both the new material since the previous test, as well as older material, then that would require students retain all the material, not just the recent stuff.

      Or maybe the last test of the semester covers the entire semester's material, while the earlier tests only cover new material since the previous test.

      We can't say for sure without this information.

      Reply View 0 replies
    • MangoToupe 9 hours ago

      I still don't see where you're pulling the "more likely" from.

      Reply View 0 replies
    • johnnyanmac 14 hours ago

      A good student would do well regardless, a bad student would do bad regardless. Cell phones might help a bad student do a little less bad, but only a little.

      For the middle, it really depends on the material covered. if it's cumulative, then results might not change as much. if it's "learn and forget", then it might be testing the wrong incentives.

      Reply View 1 reply
      • ryuhhnn 13 hours ago

        > if it's "learn and forget", then it might be testing the wrong incentives

        The thing I find interesting is that when most people talk about standardised tests, they are talking about assessments that benchmark how much trivial knowledge about a given subject one has, and this has been the standard for most of the history of the American education system. I would argue that this is a flawed way to measure a student's literacy–in any subject for that matter.

        I would actually frame "learn and forget" as "learn and adapt" because I would much rather a student forget a piece of trivial knowledge, but still have the ability to figure it out on their own with the right resources than a student who can tell you the colour Benjamin Franklin wore on his 15th birthday, but couldn't explain the effects of imperialism on societies.

        For much of history, we have incentivised rote memorisation of trivial knowledge and accidentally de-valued critical thinking and problem solving skills. Do you remember the backlash that schools got from _parents_ when schools started implementing Common Core in an attempt to get students to think more abstractly? While I scoff at them, I genuinely don't blame parents for coming to the conclusion that we should just do math "the way we used to do it", but I can't help but point out that this is leading to the exact decline in general literacy that we have seen in public schools over the years. Now when you start comparing the educational attainment of students in public schools vs. private schools this becomes a who other conversation that cell phones can't even begin to explain.

        Reply View 0 replies
tootie 9 hours ago

This is also very closely following the pandemic. I'd imagine that massively pollutes their data. I didn't see a comparison to comparable districts that didn't implement a ban.

Just from anecdata of my own kids, enforcement is nearly impossible. Phones are banned citywide as of this year but it sounds like they are still being used pretty openly.

Reply View 1 reply
beastman82 14 hours ago

In other words, correlation does not imply causation

Reply View 2 replies
  • ecb_penguin 9 hours ago

    And of course, sometimes a correlation does in fact imply a causation!

    Reply View 1 reply
    • kelnos 5 hours ago

      No, correlation does not imply causation. However, sometimes you find both correlation and causation, depending on the evidence you have.

      Reply View 0 replies