Building the Agent-First Web: How We Optimized a Game Database for Humans and LLMs

In web development, we spend almost 100% of our energy designing for human eyes—vibrant colors, smooth transitions, and client-side interactivity. But in 2026, a massive portion of web traffic isn't human. It's AI agents, LLM crawlers, and search engines trying to ingest, synthesize, and answer questions using our data.

Today, we set out to build a Dual-Delivery Strategy Hub for the game Godforge. Our goal was simple yet ambitious: scrape 204 character guides, build a blazing-fast human experience featuring client-side RAG search, and optimize the site to be 100% agent-first so that LLMs can ingest it with zero friction.

Here is the story of what we built, what we audited, and how we optimized it.

The Blueprint: Dual-Delivery Architecture

A dual-delivery site serves two audiences:

1. Humans: Beautiful HTML layouts, interactive search, and readable copy.
2. AI Agents: Raw, semantic, structured markdown and JSON served via clean APIs and aggregate files.

                  ┌──────────────────────────────┐
                  │ Scraped Data: 204 Characters │
                  └──────────────┬───────────────┘
                                 │
                                 ▼
                  ┌──────────────────────────────┐
                  │   Astro Content Collection   │
                  └──────────────┬───────────────┘
                                 │
         ┌───────────────────────┼───────────────────────┐
         ▼                       ▼                       ▼
┌─────────────────┐    ┌───────────────────┐    ┌─────────────────┐
│  Human View:    │    │    Agent View:    │    │ Crawler Assets: │
│ HTML & RAG Search│    │  /api/:id.md APIs │    │ llms.txt & full │
└─────────────────┘    └───────────────────┘    └─────────────────┘

Step 1: Harvesting the Database (Concurrent Scraping)

We kicked off by writing a Python scraper using ThreadPoolExecutor to concurrently fetch and parse all 204 character strategy profiles from the game's database.

• Performance: Scraped and generated all files in under 10 seconds.
• Data Integrity: Extracted level 60 character stats (HP, ATK, DEF), scaling multipliers, and ability descriptions.
• Astro Schemas: Converted the raw data into type-safe Markdown files under src/content/characters/, strictly adhering to our Astro content configuration schemas.

Step 2: The Human Experience (Instant RAG Search)

For humans, we designed a client-side search directory on the homepage. Instead of a standard fuzzy-text search, we built a zero-dependency Client-Side RAG (Retrieval-Augmented Generation) Search Assistant:

• Build-time Indexing: Astro compiles the key text of all 204 guides into a compressed JSON index payload loaded on the client side.
• Local Tokenization & Scoring: When a user types a query (e.g., "Which hero has Banish?" or "Who applies Bleeds?"), the script tokenizes the query, ranks the guides using term frequency, and dynamically reorders the grid.
• Extractive Synthesis: The search assistant extracts the exact sentences containing the matching context, presenting a dynamic "AI Answer Card" citing the source guide and linking directly to the character.

Step 3: Auditing with Google Lighthouse

To verify our performance, accessibility, and SEO foundations, we ran remote Lighthouse audits on the homepage and character detail pages.

The results were outstanding:

• Performance: 100% (First Contentful Paint of 0.8s - 1.4s, 0ms Total Blocking Time, 0 Cumulative Layout Shift).
• Accessibility: 100% (Strict semantic landmark elements, contrast checks, and keyboard focus compliance).
• Best Practices: 100% (Secure HTTPS, clean console logs, and modern APIs).
• The Catch: SEO and Agentic Browsing scores were slightly depressed (91% and 67%). Lighthouse revealed that crawlers attempting to hit robots.txt and llms.txt were running into invalid HTML fallback pages because the files did not exist.

Step 4: The AI Agent-First Optimization

To fix the crawler issues and establish a best-in-class AI-friendly environment, we implemented four advanced discoverability enhancements:

1. Permission-Permissive robots.txt + XML Sitemap

We configured @astrojs/sitemap in the Astro engine to generate an XML sitemap on build. We then generated a standard robots.txt linking directly to it:

User-agent: *
Allow: /

Sitemap: https://godforgehub-astro.pages.dev/sitemap-index.xml

2. The Link Catalog (llms.txt)

We generated public/llms.txt listing all 204 guides with their names and descriptions, providing a clean semantic sitemap for AI crawlers like GPTBot and ClaudeBot.

3. The Aggregate Database (llms-full.txt)

Instead of forcing a scraper to make 204 separate HTTP requests to read your guides, we built a Python generator that concatenates all character guides into a single markdown file: /llms-full.txt. An AI can now ingest your entire strategy hub in one single request.

4. Structured Catalog Endpoint (api/catalog.json)

We compiled a machine-readable JSON array containing all character IDs, descriptions, and direct API endpoints.

5. Semantic Link Alternates (rel="alternate")

We injected alternate link tags into the HTML template heads of the directory and detail pages:

<link rel="alternate" type="text/markdown" href="/api/[id].md" />

This acts as a signpost at the edge. When an LLM bot hits a human-facing HTML URL, it reads this link tag and immediately redirects its request to the clean, header-less markdown endpoint (/api/[id].md), saving tokens and parsing overhead.

Measuring the Results

This dual-delivery site is now running as a live experiment. By checking Cloudflare's Edge Traffic Analytics (filtering by AI crawler user-agents like GPTBot and ClaudeBot hitting the /llms.txt and /api/* endpoints), we will measure how much more efficiently and frequently AI systems crawl this developer-friendly architecture compared to standard human-first sites.

The era of the agent-first web is here!