Close Menu

    Subscribe to Updates

    Get the latest creative news from FooBar about art, design and business.

    What's Hot

    Kleida Martiro is leading the AI scale conversation at TC All Stage

    July 1, 2025

    Nothing releases its first over-the-ear headphones, the $299 Headphone (1)

    July 1, 2025

    The electric Hummer is almost outselling the F-150 Lightning

    July 1, 2025
    Facebook X (Twitter) Instagram
    • Home
    • Technology
    • Gaming
    • Phones
    • Buy Now
    Facebook X (Twitter) Instagram Pinterest Vimeo
    My BlogMy Blog
    • Home
    • Features
      • Example Post
      • Typography
      • Contact
      • View All On Demos
    • Technology

      Is the Hyperloop Doomed? What Elon Musk’s Latest Setback Really Means

      March 10, 2022

      The Best Early Black Friday Deals on Gaming Laptops and Accessories

      March 10, 2022

      Apple Watch’s ECG Can Help Diagnose Heart Problem: Research

      January 19, 2021

      Simple Tips and Tricks to Take Care of Your Expensive DSLR Camera

      January 16, 2021

      Tech Study Reveals Effects of Mobile Technology on Professionals

      January 15, 2021
    • Typography
    • Phones
      1. Technology
      2. Gaming
      3. Gadgets
      4. View All

      Is the Hyperloop Doomed? What Elon Musk’s Latest Setback Really Means

      March 10, 2022

      The Best Early Black Friday Deals on Gaming Laptops and Accessories

      March 10, 2022

      Apple Watch’s ECG Can Help Diagnose Heart Problem: Research

      January 19, 2021

      Simple Tips and Tricks to Take Care of Your Expensive DSLR Camera

      January 16, 2021

      Game Development This Week: Save On Essential Tools and More

      November 19, 2022

      Riot Games Acquires a Wargaming Studio to Help With Live Game Development

      March 10, 2022

      Keep Talking and Nobody Explodes: A Boomer Gaming in VR

      March 12, 2021

      Hologate Announces New Plans for First Large Format World VR Arcade

      January 16, 2021
      8.9

      DJI Avata Review: Immersive FPV Flying For Drone Enthusiasts

      January 15, 2021
      8.9

      Bose QuietComfort Earbuds II: Noise-Cancellation Kings Reviewed

      January 15, 2021

      Thousands Of PC Games Discounted In New Black Friday Sale

      January 15, 2021

      Could Solar-Powered Headphones Be The Next Must-Have?

      January 15, 2021

      Will Using a VPN on Phone Helps Protect You from Ransomware?

      January 14, 2021

      Popular New Xbox Game Pass Game Being Review Bombed With “0s”

      January 14, 2021

      Google Says Surveillance Vendor Targeted Samsung Phones

      January 14, 2021

      Why Are iPhones More Expensive Than Android Phones?

      January 14, 2021
    • Buy Now
    Subscribe
    My BlogMy Blog
    Home»Uncategorized»Cloudflare launches a marketplace that lets websites charge AI bots for scraping
    Uncategorized

    Cloudflare launches a marketplace that lets websites charge AI bots for scraping

    Y U RajuBy Y U RajuJuly 1, 2025No Comments5 Mins Read
    Share Facebook Twitter Pinterest LinkedIn Tumblr Reddit Telegram Email
    Share
    Facebook Twitter LinkedIn Pinterest Email


    Cloudflare, a cloud infrastructure provider that serves 20% of the web, announced Tuesday the launch of a new marketplace that reimagines the relationship between website owners and AI companies — ideally giving publishers greater control over their content.

    For the last year, Cloudflare has launched tools for publishers to address the rampant rise of AI crawlers, including a one-click solution to block all AI bots, as well as a dashboard to view how AI crawlers are visiting their site. In a 2024 interview, Cloudflare CEO Matthew Prince told TechCrunch these products were laying a foundation for a new type of marketplace in which publishers could distribute their content to AI companies and be compensated for it.

    Now, Cloudflare is bringing that marketplace to life.

    It’s called Pay per Crawl, and Cloudflare is launching the “experiment” in private beta on Tuesday. Website owners in the experiment can choose to let AI crawlers, on an individual basis, scrape their site at a set rate — a micropayment for every single “crawl.” Alternatively, website owners can choose to let AI crawlers scrape their site for free, or block them altogether. Cloudflare claims its tools will let website owners see whether crawlers are scraping their site for AI training data, to appear in AI search responses, or for other purposes.

    Here’s what website owners see in Pay per Crawl (Credit: Cloudflare)

    At scale, Cloudflare’s marketplace is a big idea that could offer publishers a potential business model for the AI era — and it also places Cloudflare at the center of it all. The launch of the marketplace comes at a time when news publishers are facing existential questions about how to reach readers, as Google Search traffic fades away and AI chatbots rise in popularity.

    There’s not a clear answer for how news publishers will survive in the AI era. Some, such as the New York Times, have filed lawsuits against tech companies for training their AI models on news articles without permission. Meanwhile, other publishers have struck multi-year deals to license their content for AI model training and to have their content appear in AI chatbot responses.

    Even so, only large publishers have struck AI licensing deals, and it’s still unclear whether they provide meaningful sources of revenue. Cloudflare aims to create a more durable system where publishers can set prices on their own terms.

    The company also announced Tuesday that new websites set up with Cloudflare will now, by default, block all AI crawlers. Site owners will have to grant certain AI crawlers permission to access their site — a change Cloudflare says will give every new domain “the default of control.”

    Several large publishers, including Conde Nast, TIME, The Associated Press, The Atlantic, ADWEEK, and Fortune, have signed on with Cloudflare to block AI crawlers by default in support of the company’s broader goal of a “permission-based approach to crawling.”

    The business model that many of these publishers relied on for decades is slowly becoming unreliable. Historically, online publishers have allowed Google to scrape their sites in return for referrals in Google Search, which translated to traffic to their sites, and ultimately, ad revenue.

    However, new data from Cloudflare suggests that publishers may be getting a worse deal in the AI era than in the Google Search era. While some websites cite ChatGPT as a major traffic source, that doesn’t appear to be the case broadly.

    This June, Cloudflare says it found that Google’s crawler scraped its websites 14 times for every referral it gave them. Meanwhile, OpenAI’s crawler scraped websites 17,000 times for every one referral, while Anthropic scraped websites 73,000 times for every referral.

    Meanwhile, OpenAI and Google are building AI agents that are designed to visit websites on behalf of users, collect information, and deliver it back to users directly. A future in which these tools are mainstream has huge implications for publishers that rely on readers visiting their sites.

    Cloudflare notes that the “true potential” of Pay per Crawl may emerge in an “agentic” future.

    “What if an agentic paywall could operate at the network edge, entirely programmatically? Imagine asking your favorite deep research program to help you synthesize the latest cancer research or a legal brief, or just help you find the best restaurant in Soho — and then giving that agent a budget to spend to acquire the best and most relevant content,” Cloudflare said in a blog post.

    To participate in Cloudflare’s experimental marketplace, AI companies and publishers must both be set up with Cloudflare accounts. In their accounts, both parties can set rates at which they’d like to buy and sell a “crawl” of the publisher’s content. Cloudflare acts as the intermediary in these transactions, charging the AI company and distributing the earnings to the publisher.

    Cloudflare spokesperson Ripley Park tells TechCrunch there are no stablecoins or cryptocurrency involved in Pay per Crawl at this time, even though many have suggested digital currency would be perfect for something like this.

    Cloudflare’s marketplace feels like a bold vision for the future that requires a lot of publishers and AI companies to get on board. Still, there’s no guarantee publishers will get a good deal, and convincing AI firms to participate could be tough, given they’re currently scraping content for free.

    Nevertheless, Cloudflare seems like one of the few companies in a position to make a marketplace like this happen.



    Source link

    Share. Facebook Twitter Pinterest LinkedIn Tumblr Email
    Previous ArticleEverything you need to know about the AI chatbot
    Next Article xAI raises $10B in debt and equity
    Y U Raju

    Related Posts

    Uncategorized

    Kleida Martiro is leading the AI scale conversation at TC All Stage

    July 1, 2025
    Uncategorized

    Nothing releases its first over-the-ear headphones, the $299 Headphone (1)

    July 1, 2025
    Uncategorized

    The electric Hummer is almost outselling the F-150 Lightning

    July 1, 2025
    Add A Comment
    Leave A Reply Cancel Reply

    Demo
    Top Posts

    2025 will be a ‘pivotal year’ for Meta’s augmented and virtual reality, says CTO

    June 6, 202544 Views

    Still no AI-powered, ‘more personalized’ Siri from Apple at WWDC 25

    June 9, 202543 Views

    XRobotics’ countertop robots are cooking up 25,000 pizzas a month

    June 9, 202542 Views
    Stay In Touch
    • Facebook
    • YouTube
    • TikTok
    • WhatsApp
    • Twitter
    • Instagram
    Latest Reviews
    85
    Featured

    Pico 4 Review: Should You Actually Buy One Instead Of Quest 2?

    thf0oJanuary 15, 2021
    8.1
    Uncategorized

    A Review of the Venus Optics Argus 18mm f/0.95 MFT APO Lens

    thf0oJanuary 15, 2021
    8.9
    Editor's Picks

    DJI Avata Review: Immersive FPV Flying For Drone Enthusiasts

    thf0oJanuary 15, 2021

    Subscribe to Updates

    Get the latest tech news from FooBar about tech, design and biz.

    Demo
    Most Popular

    2025 will be a ‘pivotal year’ for Meta’s augmented and virtual reality, says CTO

    June 6, 202544 Views

    Still no AI-powered, ‘more personalized’ Siri from Apple at WWDC 25

    June 9, 202543 Views

    XRobotics’ countertop robots are cooking up 25,000 pizzas a month

    June 9, 202542 Views
    Our Picks

    Kleida Martiro is leading the AI scale conversation at TC All Stage

    July 1, 2025

    Nothing releases its first over-the-ear headphones, the $299 Headphone (1)

    July 1, 2025

    The electric Hummer is almost outselling the F-150 Lightning

    July 1, 2025

    Subscribe to Updates

    Get the latest creative news from FooBar about art, design and business.

    Facebook X (Twitter) Instagram Pinterest
    • Home
    • Technology
    • Gaming
    • Phones
    • Buy Now
    © 2025 ThemeSphere. Designed by ThemeSphere.

    Type above and press Enter to search. Press Esc to cancel.