What problems arise when AI agents increase web traffic?

AI agents send requests much faster and more repeatedly than humans, causing server load to surge and caches to collapse. Also, because they try to extract only data without looking at the screen, if structured responses are not provided, they are likely to collect incorrect information or misuse APIs.

What is the key to designing for both human and machine users?

The key is content negotiation that provides JSON, JSON-LD, and OpenAPI specifications for the same resource while keeping HTML for humans. Adding explicit access policies, idempotency, and rate limiting as safety mechanisms allows you to accommodate both types of users reliably.

How can agents built with vibe coding use the web politely?

At the agent generation stage, embed rules in prompts and templates: include contact information and purpose in the User-Agent, check robots.txt first, and honor Retry-After in 429 responses. For write operations, use idempotency keys to prevent incidents caused by duplicate requests.

Why are bot detection and rate limiting important?

If malicious bots and legitimate AI agents are not distinguished, the service may be paralyzed or the ecosystem may shrink due to excessive blocking. Applying multi-layered bot detection and staged rate limiting protects the server while allowing access for good agents.

Why are structured data such as JSON-LD and OpenAPI necessary?

AI agents accurately extract information by directly reading structured data instead of parsing HTML. JSON-LD helps machines understand page content, and OpenAPI enables agents to explore and call APIs without separate documentation, greatly improving automation efficiency.

AI Agents Are Taking Over the Internet: What Developers Must Redesign

md-logSeptember 17, 20269 min read

AI agents are emerging as primary web users, and interface design for humans alone has reached its limits. Developers must redesign with machine-readable APIs, structured data, and bot policies as fundamental premises.

As AI agents emerge as primary users of the web, interface design for humans alone has reached its limits. Developers must now treat machine-readable APIs, structured data, and bot policies as fundamental premises. This article analyzes the impact of increasing AI agent traffic on websites and APIs, and presents design principles and a practical checklist that consider both human and machine users.

AI agents are becoming the new primary actors in web traffic

The traditional web was designed for human users who read screens and click through browsers. However, AI agents that search for information on behalf of users, process reservations, generate code, and chain calls to multiple services are rapidly increasing. These agents prefer API responses and structured data over directly parsing HTML, and sometimes send large volumes of requests in ways humans would not. As a result, traditional websites face sudden traffic pattern shifts, server load spikes, and incorrect data collection. For example, an agent scraping search results may ignore pagination and send thousands of requests at once, causing caches to collapse and origin database load to surge.

Also, agents do not respond to visual layout or marketing copy like humans. No matter how impressive banners and interactive widgets are on screen, if the machine-readable structured information is not provided, agents cannot properly use the service. In September 2026, F5 unveiled Workforce AI Security, which emphasized enterprise-level control over AI and AI agent behavior used by employees. This signals that agent traffic is moving beyond mere technical curiosity and becoming a target of security and governance. Developers now face the challenge of providing both a "human-friendly web" and a "machine-readable web" within a single service, rather than separating them.

Design principles for interfaces that consider both human and machine users

The most important principle is to make progressive enhancement and content negotiation the foundation. Keep HTML for human users, but provide JSON, JSON-LD, and OpenAPI definitions for the same resource so agents can obtain exactly the data they need without unnecessary HTML parsing. For example, on a product information page, alongside human-readable details, embed Schema.org's Product schema as JSON-LD, allowing search agents to immediately extract price, inventory, and review ratings. If you operate an API, publish an OpenAPI 3.x specification at the root path and design each endpoint to return machine-understandable error messages and version information.

The second principle is to include explicit intent and safety mechanisms in the design. Humans change behavior based on visual cues in the UI, but agents may repeat default behavior or keep retrying bad input without explicit rules. Therefore, rate limits, retry policies, and including a Retry-After header in 429 responses are important. Also, use /robots.txt, /ai.txt, and meta tags to specify which paths allow agent access and what data may be collected. This extends past conventions for search engine crawlers into the AI agent era.

The third principle is to prioritize statelessness and idempotency in API design. If an agent sends the same request multiple times or retries after network errors, repeating the same result can cause serious issues such as data duplication or payment errors. Require an idempotency key for POST requests, keep GET read-only without side effects, and clearly separate mutations with PUT, PATCH, and DELETE. This ensures predictable behavior not only for human-click web applications but also for APIs called by agents.

Practical patterns to make vibe-coded agents use the web politely

Many developers now build agents quickly by conversing with AI in a "vibe coding" style. The problem is that generated code often does not know web etiquette. For example, an AI-written scraper may not include a User-Agent, may ignore pagination intervals, or may not check robots.txt. To correct this, "web citizen rules" must be embedded in prompts and code templates from the start.

Specifically, the following patterns can be applied. First, include a contactable identifier and purpose in the User-Agent, such as "MyResearchBot/1.0 (+https://example.com/bot)". Second, respect request intervals, and when a 429 response is received, read the Retry-After header and apply backoff. Third, check robots.txt and the site's meta robot tags first and do not access disallowed paths. Fourth, when fetching large data, use pagination or cursor-based APIs and limit the number of parallel requests. Providing these patterns as default context for vibe coding significantly changes the quality of generated agents.

Also, when agents perform write operations such as reservations, purchases, or message sending on behalf of humans, it is advisable to include idempotency keys and confirmation steps. For example, to prevent an airline ticket reservation agent from sending the same request twice and causing double payment, the server should validate a unique Idempotency-Key generated by the client and ignore duplicate requests. This applies both to those building agents and to services receiving them.

The importance of bot detection, rate limiting, and structured data

As machine users increase, distinguishing malicious bots from legitimate AI agents becomes more difficult. Traditional CAPTCHA not only degrades human user experience but is also increasingly likely to be bypassed by AI with improved image recognition. Therefore, multi-layered bot detection is needed, combining IP reputation, TLS fingerprints, request pattern analysis, header consistency, and behavior-based anomaly detection. Blocking all traffic can also block legitimate agents and shrink the service ecosystem, so it is better to apply rate limiting, challenges, and blocks in stages according to risk level.

Structured data benefits both humans and machines. JSON-LD helps search results and AI assistants understand page content accurately, and OpenAPI specifications allow agents to explore and call APIs on their own without separate documentation. As of late 2026, several companies are releasing API gateways and security solutions designed for AI agent access, and features like F5's Workforce AI Security that centrally control employee AI usage and agent behavior are attracting attention. Developers should leverage these tools, but remember that the core is clearly defining the contract between data and APIs.

Additionally, rate limiting goes beyond simple server protection and serves as a means of fair resource distribution. If a particular agent monopolizes the server with excessive requests, human users may slow down or the service may be interrupted. Using token bucket or sliding window algorithms to set limits per user, API key, or IP, and returning 429 with Retry-After when limits are exceeded allows agents to self-regulate their pace. Offering agent-specific pricing tiers or priority can also be a strategy.

An agent-friendly web service checklist you can apply right now

Checking the following items in order will help you move one step closer to a service that accommodates both humans and AI agents.

Provide /robots.txt and /ai.txt, and confirm that agent access policies and contact information are clearly stated.
Insert Schema.org-based JSON-LD (Product, Article, LocalBusiness, etc.) into key pages.
If you have an API, publish an OpenAPI 3.x specification at the root path and include descriptions, error codes, and examples for all endpoints.
GET is idempotent and free of side effects, and write requests require an Idempotency-Key.
Include a Retry-After header in 429 responses and document retry policies.
Monitor logs to ensure bot detection rules do not excessively block legitimate AI agents.
Implement logic to negotiate HTML and JSON responses based on User-Agent and Accept headers.
Establish caching and backpressure strategies in case AI agents send bulk requests.

Rather than applying this checklist perfectly all at once, it is realistic to prioritize based on the nature of your service and introduce items step by step. For example, an e-commerce site might start with product JSON-LD and inventory APIs, while a content site might first apply article structured data and crawler policies.

Conclusion: Designing in the agent era is about clarifying contracts

As AI agents become major users of the internet, the center of gravity in web development is shifting from "screens for humans" to "contracts for machines." Developers must treat APIs, structured data, and bot policies not as add-ons but as fundamental design elements. At the same time, it is necessary to embed rules from the generation stage so that agents created through vibe coding use the web politely.

Amid these changes, human-in-the-loop workflows—where people review and approve AI-generated requests, changes, and analysis results—are becoming increasingly important. md-log can be used as a layer to conveniently review AI-generated work on web and mobile, and to accumulate collaboration history as immutable versions every time you save. Ultimately, designing a machine-readable web while building a structure where humans can control and be accountable for the results is the core challenge developers must now redesign.

References

#AI agents #web development #API design #JSON-LD #OpenAPI #bot detection

Frequently asked questions

What problems arise when AI agents increase web traffic?: AI agents send requests much faster and more repeatedly than humans, causing server load to surge and caches to collapse. Also, because they try to extract only data without looking at the screen, if structured responses are not provided, they are likely to collect incorrect information or misuse APIs.
What is the key to designing for both human and machine users?: The key is content negotiation that provides JSON, JSON-LD, and OpenAPI specifications for the same resource while keeping HTML for humans. Adding explicit access policies, idempotency, and rate limiting as safety mechanisms allows you to accommodate both types of users reliably.
How can agents built with vibe coding use the web politely?: At the agent generation stage, embed rules in prompts and templates: include contact information and purpose in the User-Agent, check robots.txt first, and honor Retry-After in 429 responses. For write operations, use idempotency keys to prevent incidents caused by duplicate requests.
Why are bot detection and rate limiting important?: If malicious bots and legitimate AI agents are not distinguished, the service may be paralyzed or the ecosystem may shrink due to excessive blocking. Applying multi-layered bot detection and staged rate limiting protects the server while allowing access for good agents.
Why are structured data such as JSON-LD and OpenAPI necessary?: AI agents accurately extract information by directly reading structured data instead of parsing HTML. JSON-LD helps machines understand page content, and OpenAPI enables agents to explore and call APIs without separate documentation, greatly improving automation efficiency.

← All posts