Top Secret: Automatically filter sensitive information

Summary

We’ve written about how to prevent logging sensitive information when making network requests, but that approach only works if you’re dealing with parameters. What happens when you’re dealing with free text? Filtering the entire string may not be an option if an external API needs to process the value. Think chatbots or LLMs. You could use a regex to filter sensitive information (such as credit card numbers or emails), but that won’t capture everything, since not all sensitive information can be captured with a regex. Fortunately, named-entity recognition (NER) can be used to identify and classify real-world objects, such as a person, or location. Tools like MITIE Ruby make interfacing with NER models trivial. By using a combination of regex patterns and NER entities, Top Secret effectively filters sensitive information from free text—here are some real-world examples. If you want to see Top Secret in action, you might enjoy this live stream. Otherwise, see the examples below. It’s not uncommon to send user data to chatbots. Since the data might be free-form, we should be diligent about filtering it using the approach mentioned above. However, it’s likely we’ll want to “restore” the filtered values when returning a response from the chatbot. Top Secret returns a mapping that would allow for this. You’d likely want to provide instructions in the request. instructions = <<~TEXT I'm going to send filtered information to you in the form of free text. If you need to refer to the filtered information in a response, just reference it by the filter. TEXT The exchange might look something like this. Caller sends filtered text result = TopSecret::Text.filter("Ralph lives in Boston.") # Send this to the API result.output # => [PERSON_1] lives in [LOCATION_1]. # Save the mapping to "restore" response mapping = result.mapping # => { PERSON_1: "Ralph", LOCATION_1: "Boston" } API responds with filter "Hi [PERSON_1]! How is the weather in [LOCATION_1] today?" Caller can “restore” f...