Use the Accept Header to Serve Markdown Instead of HTML to LLMs

Summary

September 27, 2025 , Nicholas Khami Use the Accept Header to serve Markdown instead of HTML to LLMs Agents don’t need to see websites with markup and styling; anything other than plain Markdown is just wasted money spent on context tokens. I decided to make my Astro sites more accessible to LLMs by having them return Markdown versions of pages when the Accept header has text/plain or text/markdown preceding text/html. This was very heavily inspired by this post on X from bunjavascript. Hopefully this helps SEO too, since agents are a big chunk of my traffic. The Bun team reported a 10x token drop for Markdown and frontier labs pay per token, so cheaper pages should get scraped more, be more likely to end up in training data, and give me a little extra lift from assistants and search. Note: You can check out the feature live by running curl -H "Accept: text/markdown" https://www.skeptrune.com or curl -H "Accept: text/plain" https://www.skeptrune.com in your terminal. Static Site Generators are already halfway there Static site generators like Astro and Gatsby already generate a big folder of HTML files, typically in a dist or public folder through an npm run build command. The only thing missing is a way to convert those HTML files to markdown. It turns out there’s a great CLI tool for this called html-to-markdown that can be installed with npm install -D @wcj/html-to-markdown-cli and run during a build step using npx. Here’s a quick Bash script an LLM wrote to convert all HTML files in dist/html to Markdown files in dist/markdown, preserving the directory structure: # convert-to-markdown.sh mkdir -p dist/markdown find dist/html -type f -name "*.html" | while read -r file; do relative_path="${file#dist/html/}" dest_path="dist/markdown/${relative_path%.html}.md" mkdir -p "$(dirname "$dest_path")" npx @wcj/html-to-markdown-cli "$file" --stdout > "$dest_path" done Once you have the conversion script in place, the next step is to make it run as a post-build action. Here’...