HeadlinesBriefing favicon HeadlinesBriefing.com

Detecting Scraper Bots Using Scroll Behavior

Hacker News •
×

Ever since I first read "Burstiness and Memory in Complex Systems" by Kwang-Il Goh, I have been obsessed with the two formulas showcased in the paper. Burstiness (B) and its just‑as‑important counterpart Memory (M) let us understand the dynamics of event‑based systems, revealing which patterns are human‑like and which aren’t.

Earlier work used B and M to differentiate between human and bot sessions based on request timings. However, advanced bots like Claude Bot can mimic a real browser, so the approach misidentified them as humans. I noticed that scrolling is a distinctly human behaviour, often bursty rather than linear. This led me to test whether B and M applied to scroll event inter‑times could predict bot vs. human.

Using the dataset from Ethan Wang et al., I calculated B and M for each page’s scroll events. Humans showed a higher Burstiness coefficient and almost no Memory, while the only agent that sometimes resembled humans was Chat GPT Agent. A Light GBM model trained on B and M alone achieved an accuracy of 73.4%, primarily confusing Humans with Chat GPT Agent—likely due to a small dataset (~150 data points per agent).

Scroll behaviour proves to be a promising data point for distinguishing bots from humans. Combining it with other actions such as mouse movement or typing could yield a robust model for protecting websites from scraper bots.