Reddit CEO Steve Huffman is standing by Reddit’s decision to block companies from scraping the site without an AI agreement.

Last week, 404 Media noticed that search engines that weren’t Google were no longer listing recent Reddit posts in results. This was because Reddit updated its Robots Exclusion Protocol (txt file) to block bots from scraping the site. The file reads: “Reddit believes in an open Internet, but not the misuse of public content.” Since the news broke, OpenAI announced SearchGPT, which can show recent Reddit results.

The change came a year after Reddit began its efforts to stop free scraping, which Huffman initially framed as an attempt to stop AI companies from making money off of Reddit content for free. This endeavor also led Reddit to begin charging for API access (the high pricing led to many third-party Reddit apps closing).

In an interview with The Verge today, Huffman stood by the changes that led to Google temporarily being the only search engine able to show recent discussions from Reddit. Reddit and Google signed an AI training deal in February said to be worth $60 million a year. It’s unclear how much Reddit’s OpenAI deal is worth.

Huffman said:

Without these agreements, we don’t have any say or knowledge of how our data is displayed and what it’s used for, which has put us in a position now of blocking folks who haven’t been willing to come to terms with how we’d like our data to be used or not used.

“[It’s been] a real pain in the ass to block these companies,” Huffman told The Verge.

  • _haha_oh_wow_@sh.itjust.worksEnglish
    4·
    3 months ago

    It seemed to happen to some people but I wouldn’t be surprised it it was just some sort of coincidental database fuck-up

    • finley@lemm.eeEnglish
      31·
      3 months ago

      i suspect that was more likely incomplete deletions than reddit restoring content. those scripts were pretty janky. i had to run mine several times to get everything, as it didn’t work fully the first couple of times. same with the overwrites. took a few times for those to get everything, especially on older accounts with lots of posts and comments.

      • _haha_oh_wow_@sh.itjust.worksEnglish
        3·
        3 months ago

        I’d read some claims that posts appeared to be deleted but then later came back. Could’ve even been some sort of caching shenanigans with their local browser though I guess.

        • finley@lemm.eeEnglish
          21·
          3 months ago

          oh, right. i suppose that’s possible. i’ve seen similar browser cache fuckery on other sites before.

          • finley@lemm.eeEnglish
            11·
            3 months ago

            as i said in a previous comment:

            so, unless reddit did a massive restore of everyone’s comments/posts except for my 4 accounts, then i don’t believe they did it at all except for a select number of top contributors who deleted their content.

            but there’s no evidence they’re keeping everyone’s deleted-but-restored comments from public view or whatever it is you’re suggesting. or even anything past whaat this one person found. in fact, there isn’t even any evidence that what happened to this user was intentional and not a bug or some other fluke.

            sure, reddit would have a vested interest in doing this, and what you’ve presented is suspicious, but it’s hardly conclusive of anything. all it does is raise more questions. but it doesn’t provide answers.