Close Menu
Chicago News Journal
    Facebook X (Twitter) Instagram
    • Contact us
    • About us
    • Amazon Disclaimer
    • DMCA / Copyrights Disclaimer
    • Privacy Policy
    • Terms and Conditions
    Facebook X (Twitter) Instagram YouTube TikTok
    Chicago News JournalChicago News Journal
    • Home
    • US News
    • Politics
    • Business
    • Science
    • Technology
    • LifeStyle
    • Music
    • Television
    • Film
    • Books
    • Contact
      • About us
      • Amazon Disclaimer
      • DMCA / Copyrights Disclaimer
      • Privacy Policy
      • Terms and Conditions
    Chicago News Journal
    Home»Technology

    Reddit to Update Web Standard to Block Automated Data Scraping From Its Website

    By July 1, 2024 Technology
    Facebook Twitter Pinterest LinkedIn Tumblr Email Reddit Telegram
    Reddit to Update Web Standard to Block Automated Data Scraping From Its Website

    Social media platform Reddit said on Tuesday it will update a Web standard used by the platform to block automated data scraping from its website, following reports that AI startups were bypassing the rule to gather content for their systems.

    The move comes at a time when artificial intelligence firms have been accused of plagiarizing content from publishers to create AI-generated summaries without giving credit or asking for permission.

    Reddit said that it would update the Robots Exclusion Protocol, or “robots.txt,” a widely accepted standard meant to determine which parts of a site are allowed to be crawled.

    The company also said it will maintain rate-limiting, a technique used to control the number of requests from one particular entity, and will block unknown bots and crawlers from data scraping – collecting and saving raw information – on its website.

    More recently, robots.txt has become a key tool that publishers employ to prevent tech companies from using their content free-of-charge to train AI algorithms and create summaries in response to some search queries.

    Last week, a letter to publishers by the content licensing startup TollBit said that several AI firms were circumventing the web standard to scrape publisher sites.

    This follows a Wired investigation which found that AI search startup Perplexity likely bypassed efforts to block its Web crawler via robots.txt.

    Earlier in June, business media publisher Forbes accused Perplexity of plagiarizing its investigative stories for use in generative AI systems without giving credit.

    Reddit said on Tuesday that researchers and organizations such as the Internet Archive will continue to have access to its content for non-commercial use.

    © Thomson Reuters 2024


    Affiliate links may be automatically generated – see our ethics statement for details.

    Read the original article here

    Share. Facebook Twitter Pinterest LinkedIn Tumblr Email Reddit Telegram

    You might also be interested in...

    Only the Seminole tribe can offer online gambling in Florida—and they're furious at DraftKings

    November 20, 2026

    Apple may be developing its own AI agent that runs your Mac | TechCrunch

    November 20, 2026

    Microsoft CEO Satya Nadella explains why he asked 60K employees to quit | TechCrunch

    November 19, 2026

    How to tell a cyberattack from a plain old outage

    November 18, 2026

    Voyager 1 at one light-day: Historian revisits Golden Record's forgotten details

    November 18, 2026

    Nvidia promises that revenue will keep soaring — for now | TechCrunch

    November 18, 2026
    Popular Posts

    Bruno Mars Mourns His Fathers Death: The Man That Taught Me Everything I Know

    RHONJ Fans Less Than Impressed With Season 15 Sneak Peek

    Director Steve Miner Has Shot New Friday the 13th Movie

    JPMorganChase reports cyberattack on third-party service providers

    Fantasy and Science Fiction Books Where the Rules Dont Apply – NewInBooks

    Book Riots Deals of the Day for November 19, 2026

    Categories
    • Books (2,459)
    • Business (3,450)
    • Events (34)
    • Film (1,064)
    • LifeStyle (2,924)
    • Music (2,798)
    • Politics (2,356)
    • Science (1,954)
    • Technology (1,819)
    • Television (3,630)
    • US News (3,295)
    Archives
    Useful Links
    • Contact us
    • About us
    • Amazon Disclaimer
    • DMCA / Copyrights Disclaimer
    • Privacy Policy
    • Terms and Conditions
    Popular Posts
    • MAHA vs. Trump over Supreme Court glyphosate case, farm billMAHA vs. Trump over Supreme Court glyphosate case, farm bill
    • Billy Reid’s Best Fall Wardrobe InvestmentsBilly Reid’s Best Fall Wardrobe Investments
    • Bon Iver to Perform at Kamala Harris Rally in WisconsinBon Iver to Perform at Kamala Harris Rally in Wisconsin
    • Sparks, Second Chances & Something More: New Romance PicksSparks, Second Chances & Something More: New Romance Picks
    • Good American Family Season 2 casts Isabela Merced and Patrick Schwarzenegger as Lorena and John BobbittGood American Family Season 2 casts Isabela Merced and Patrick Schwarzenegger as Lorena and John Bobbitt
    Archives
    Categories
    • Books
    • Business
    • Events
    • Film
    • LifeStyle
    • Music
    • Politics
    • Science
    • Technology
    • Television
    • US News
    Facebook X (Twitter) Instagram YouTube TikTok
    © 2026 Chicago News Journal. All rights reserved. All articles, images, product names, logos, and brands are property of their respective owners. All company, product and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement unless specified. By using this site, you agree to the Terms of Use and Privacy Policy.

    Type above and press Enter to search. Press Esc to cancel.