Web Bot Auth

Summary

Web Bot Auth is an authentication method that leverages cryptographic signatures in HTTP messages to verify that a request comes from an automated bot. Web Bot Auth is used as a verification method for verified bots and signed agents. It relies on two active IETF drafts: a directory draft ↗ allowing the crawler to share their public keys, and a protocol draft ↗ defining how these keys should be used to attach crawler's identity to HTTP requests. This documentation goes over specific integration within Cloudflare. 1. Generate a valid signing key You need to generate a signing key which will be used to authenticate your bot's requests. Generate a unique Ed25519 ↗ private key to sign your requests. This example uses the OpenSSL ↗ genpkey command: openssl genpkey -algorithm ed25519 -out private-key.pem Extract your public key. openssl pkey -in private-key.pem -pubout -out public-key.pem Convert the public key to JSON Web Key (JWK) using a tool of your choice. This example uses jwker ↗ command line application. go install github.com/jphastings/jwker/cmd/jwker@latestjwker public-key.pem public-key.jwk By following these steps, you have generated a private key and a public key, then converted the public key to a JWK. You need to host a key directory which creates a way for your bot to authenticate its requests to Cloudflare. This directory should follow the definition from the active IETF draft draft-meunier-http-message-signatures-directory-01 ↗. Host a key directory at /.well-known/http-message-signatures-directory (note that this is a requirement). This key directory should serve a JSON Web Key Set (JWKS) including the public key derived from your signing key. Serve the web page over HTTPS (not HTTP). Calculate the base64 URL-encoded JWK thumbprint ↗ associated with your Ed25519 public key. Sign your HTTP response using the HTTP message signature specification by attaching one signature per key in your key directory. This ensures no one else can mirror your directory an...