PerPageBot

You're seeing this page because PerPageBot/0.1 (+https://perpage.dev/bot) appeared in your server logs. PerPageBot is the fetcher behind PerPage, an API that converts public web pages into clean markdown and structured data for AI applications.

How it behaves

• On-demand, not bulk. PerPageBot fetches a page only when an API user requests that specific URL. It does not crawl the web on its own or build an index.
• Respects robots.txt. Your robots.txt is checked before every fetch (cached up to 1 hour). Rules for PerPageBot and * are honored.
• Public pages only. It never logs in, bypasses paywalls, or accesses anything behind authentication.
• Caching reduces load. Fetched pages are cached briefly, so repeated requests for the same URL don't re-fetch from your server.

Blocking PerPageBot

Add this to your robots.txt:

User-agent: PerPageBot
Disallow: /

It takes effect on the next fetch (within an hour at most, due to robots.txt caching).

Contact

Questions, problems, or removal requests: info@infospectrum.ai