Building an AI That Watches Rugby

Summary

Building an AI That Watches Rugby Heads up! This article was written over [[elapsed]]. While it may still be helpful, please verify any information, as my perspectives and practices may have evolved. There’s a gap in rugby data. We’ve got the big moments covered — the tries, conversions, and cards. Structured event feeds do a good job of telling you what happened. But they’re not so good at telling you why. At Gainline, we build our entire app around context. We want to give rugby fans a second-screen experience that feels alive — a commentary that goes deeper than the scoreline. We already pull in weather data, team stats, and player profiles. We enrich it with AI-generated summaries. But we’re limited by the data we get. We don’t know why the ref blew the whistle. We can’t tell if a prop is quietly dominating the scrum. We miss what the ref said to the captain. And that’s a problem — because these moments matter when you’re trying to tell the full story of a match. So we asked ourselves a simple question: What if we could watch the game and generate the data ourselves? That led me down a really fun rabbit hole. In this post, I’ll show you how I built a prototype system that watches a rugby game using AI. We’ll look at how we extracted the score and game clock from the broadcaster’s UI, how we used Whisper to transcribe referee and commentary audio, and what we learned about running these kinds of experiments cheaply and effectively. It’s scrappy — but it works. Context is Everything Gainline is our rugby app. It’s a clean, well-designed experience that gives fans what they need. We pull together data from a range of providers — live scores, player stats, team histories — and try to tell a richer story about what’s happening on the pitch. Most of it works well. If you want to know who scored the last try, who the fly-half is, or who’s made the most carries, we’ve got you covered. But rugby is messy. A lot happens between structured events. Penalties go unexplained....