tl;dr; How can prevent personal data from getting into logs and enable this for development teams without any work beyond npm update?
Spending the last 3 weeks side-of-desk looking into redaction, specifically around PCI/NPI (Payment Card Industry / Non Public Information) data (bank account, personal user data, etc) in logs. You break various laws when this happens, and it’s not a fun thing to clean it up.
This in turn taught me that logs aren’t the only sinks, e.g. exits of data out of your app that could have PCI/NPI data in them. Streaming data (e.g. Kinesis, Kafka, etc). HOW you get that data is also interesting. Sometimes you parse stuff through JSON.parse from a downstream, but get things you didn’t mean to. Sometimes a REST call comes in, and things you didn’t intend to keep around were from the headers. Some of this data can go into the sinks that you didn’t intentionally mean to log/stream.
I looked at how most are solving it. The dynamic language, or “lol, wat are types” crews are basically attacking with various Regular Expressions, and string parsing. Any string you log, they either abstract things atop OOP loggers, or write single points of exit sinks like in Pino/Tap. At these points, they’ll combine multiple RegExp’s together into a mass string replacer. I’ve even see a strong arm approach of JSON.stringify’ing an Object, then redacting the big string that comes out.
Some will mark that they redacted something. Some will mark what they redacted. Some will just say “[REDACTED]” and you don’t know if it was a bank account number (BAN), an email, or a number you wrote that was a false positive.
The types crew will use… types. They’ll either create Branded or Variant/Sum/Union types to designate something as a secret, and ensure logging, streaming, and metrics functions cannot take those types. Some will use type proofs in functions or methods to ensure even nested type records do not contain secrets; ensuring “if it compiles, it doesn’t leak PCI data”.
Some, particularly in the TypeScript, Python (gradually typed community) will use both. This is because the type system can, and will often, lie because developers consuming the library can hack their way around it using the 20 billion escape hatches & levers these type systems give you.
The amount of Regular Expression and string matchers is vast. I remember back in the video sharing with chat days, there were many string matchers to filter out bad words from chat, and many also included XSS helpers. This ensures that people who find they can’t type “badword” will type “b a d w o r d” and find that using spaces like that defeats the string matcher for bad words, so they can type other bad words, or even bad code snippets using spaces. The devs will counter with more aggressive string matchers and regular expressions, and they seem to err on the side of “false positives are fine because the cost of failure is quite bad”. While the chat will filter out random words you don’t think are bad, it gives business piece of mind from legal ramifications of NOT doing it. The User Experience costs here are interesting, both to users and to devs.
I’m finding my in own work, for our existing logger, I’m having to use that dual approach of “use types to prevent this from happening”, but also using a combination of regular expressions when the narrowness of type system breaks down. I’m also thinking of starting from scratch on a secondary implementation once the first is done that type narrow specifically to ensure we don’t even need the runtime checks (well, it’s TypeScript, almost anything can be defeated with an as unknown so I’ll have to run various tests). But it’s clear the hybrid approach causes the API to suffer because you’re constraining what the dev can log, but also have to do a ton of type work to compensate for when they (unknowingly) lie to you.
I really like the clean, typed approach as it lessens the attack surface, and I share a lot of this with the streaming and metrics use cases as well as those are less open to the developer so easier to constrain what gets in. It also helps when we attack the problem at the source, and immediately decode into secrets vs. “I don’t know man… they could be anywhere, let’s just JSON.stringify all the things and run a ball of RegExp on it”. Having schema libraries like Zod decode immediately to know secrets enables such a better security posture.
Before: “Where are your secrets?” “Uh, in this codebase… I think from a downstream REST call. You’ll have to go track down their Swagger, it may/may not point them out”
After: “Find all instances of the Secret class.”
I’ve found that some secrets aren’t even secrets. Devs will put credit card number in a secret container, and when you look inside, it’s actually a null. Having a schema validate it’s even an actual secret prevents a lot of this from even being a problem.
There’s also some tricky situations that reminds me of the 1989 Batman movie Joker plot. You’ll have a user’s last name and the last four digits of their credit card. Both aren’t necessarily PCI on their own… but together? Yes. Same with the chemicals Joker was pushing in the movie; isolated they were innocuous, but together, poison. There are various Intersection and Union types you can create to prevent these combinations so you can relax when it compiles.
It’s been interesting to tackle this problem from a Brownfield approach. “How do we prevent PCI data leakage?”
I think I’ve settled on improving the logger we have better runtime redaction, with some compile time preventative measures to help move in the right direction over time. A separate, ideal logger for where we’d like to go. This includes helpers to verify we even have secrets in the first place, and if we do, we use a Parse, Don’t Validate + Anti-Corruption Layer in the same place via a Zod transform.
Brownfield: npm/pip update, you’ll be safer.
Brownfield + migrate: move to this logger, you can offload the problem to the compiler, but log sinks/targets/streaming outputs remain the same.
Greenfield: same
Auditing: With schema parsing, we can confidently know what secrets we have, where, and what combinations we’ve forbidden.
The escape hatches have been interesting too. Getting at your source logger, whether that’s AWS Power Tools, or Pino, or Winston or whatever is often needed to configure it, but if it’s done in more than 1 place, you should get nervous. So we have to provide the ability to do that, but also mark that we’ve done that. I can solve this with a Linter.
For the secrets, you need to provide access to them to do business logic, but sometimes you may have to do that in multiple places unlike log/metric/streaming configurations. I … don’t know how to solve this beyond having an LLM track the Branded type that comes out and doing it’s own audit.
Hard problems, lots of ways to approach it. While I’m having fun, I keep reminding myself why I’m doing this:
- ensure we’re following the law
- we won’t have any surprise security work to do
- make this easy on developers now in Brownfield, and a “pit of success” for those in Greenfield
Still more to learn, but what a rabbit hole, man.
Leave a Reply