False Positives

Although many do not follow forum guidelines about evidential support I have sometimes see the following statement "Statistically Significant".

This claim has also accompanied some of the outlandish AI generated stuff more recently.

I would draw members attention to the 2011 experiment by Simmons et al in which they statistically demonstrated that audiences who listened to a Beatles song literally became younger after the event.

Obviously impossible, but that's statistics for you if mishandled.

https://www.researchgate.net/publication/51725721_False-Positive_Psychology

Abstract

In this article, we accomplish two things. First, we show that despite empirical psychologists’ nominal endorsement of a low rate

of false-positive findings (≤ .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive

rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence

that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy

it is to accumulate (and report) statistically significant evidence for a false hypothesis. Second, we suggest a simple, low-cost,

and straightforwardly effective disclosure-based solution to this problem. The solution involves six concrete requirements for

authors and four guidelines for reviewers, all of which impose a minimal burden on the publication process.

I wonder if we shouln't have a sticky outlining the dangers of false positives in statistics ?