The Minimum Viable Pain Point, or MVPP

AI Disclosure

Claude was used as a thinking partner to help organize and flesh out ideas, generate an outline, generate images, fill in code examples and references, and do a final editing pass. All other content hand-crafted via ViM

What’s that one process?

You know the one…​
                SUPER high stakes,
                             all eyes on you.

ENORMOUS blast radius

You know…​
    the Resume Generating Event [1] if and when things go wrong?

For me, it was managing the re-key of our HashiCorp Vault instance.

A nervous person facing a giant glowing vault door covered in warning labels

The Scary Security Process

A later-stage fintech startup I worked for had chosen to use HashiCorp Vault to do secrets management, encryption, and just in time account provisioning. This is a perfectly reasonable solution as it provides a few different security needs in an audit-able, secure manner.

One of the ways that Vault protects the data in its care is to use a strong encryption key to protect the data that it writes to disk. This key is then protected in a few different ways:

  • Shamir’s Secret Sharing Scheme — SSSS

  • Key Management Service from your cloud provider

  • Hardware keys

This company had chosen to use SSSS and set up a quorum of what we called "Shardbearers" [2]

The downside is that nobody ever rotated these encryption keys after creating them. One of the shardbearers was the CTO — hire number three and, if I remember right, one of the technical founders.

Just after I was hired, this CTO decided his time at the company was coming to a close and started off-boarding. One of the tasks that was identified was that we needed to re-key this Vault shared key.

That’s where the anxiety came in. Vault’s rekey process is multi-step and uses some long and complex CLI commands.

$ vault operator rekey \
    -init \
    -key-shares=8 \
    -key-threshold=5 \
    -pgp-keys="keybase:steven,keybase:addison,keybase:blair,keybase:connie,keybase:sf_continuity,keybase:nyc_continuity,keybase:slc_continuity,keybase:goozbach"

$ vault operator rekey -status

$ vault operator rekey \
    -nonce="d3adb33f-0000-1111-2222-333344445555" \
    <unseal_key>

That last command gets run once per shardbearer, in turn, until quorum (five of the eight shares, in this example) is reached. Get the nonce wrong, or hand someone the wrong key, and you get to start over.

Fearing this process isn’t irrational — the tooling punishes mistakes. Restoring from the backup is a pain, and if the shardbearers which can unlock the key are no longer with the company in sufficient numbers then you’re locked out of your encrypted data.

Scroll badge representing the manual runbook

Facing the Fear

With no other option than to just "Get it done" I set out to do the first re-key that had happened in years. I read the docs, wrote a wiki page with EXACT commands to run, did test re-key events on non-production environments. I even exercised what would have happened if I mis-keyed the system and had to recover from a backup.

At this point we only had one extra shardbearer than what we needed to achieve quorum and when the CTO finally left we’d be down to exactly quorum.

The day came and we successfully re-keyed Vault and reset our quorum. However with the departure of the CTO and a fresh batch of new employees we noticed that there were other people who it made sense to be added to the ranks of our shardbearers.

This meant that we needed to re-key again soon. This process was still scary, but now we had a handle on it.

One of the things we wanted to exercise is the wiki documentation we had written. The best way to stress test documentation is to have another person run through it without any help from the author.

So for this second re-key event we had a different person run the whole ceremony. They did, and we found assumptions we’d made about environments and the commands that needed to run.

Around this time as well we hired a new head of security, and we were trying to reach some new compliance requirements, SOC 2 chief among them.

The new head of security set up some policies and asked us how often we re-keyed Vault. We responded "well it still scares us". In other words, it was still painful. The response: "Let’s do it more often then!"

This is where I discovered the first inklings of what I later came to call the Minimum Viable Pain Point.

Productionalizing the Process

At this point we had some robust, updatable, and easy to follow documentation for our process. We started doing monthly re-key events, even if it wasn’t needed.

A few observations started to appear:

  • Doing this on a VM in the cloud was a bit messy and had some limitations

  • Doing this on one employee’s laptop increased the Lottery Factor [3]

  • This didn’t work well for business continuity, if we lost quorum from some event we were hosed!

  • Distributing the generated key shards was a PAIN (copy and paste to an email, make sure that the PGP key matched the user, fix that when you mess it up, ohh this shardbearer doesn’t have their private key anymore)

We would never have uncovered these issues if we only re-keyed whenever we got up enough gumption to do it.

We also found ancillary improvements we could do:

  • Use USB PGP smart cards to create all-in-one business continuity shards.

  • Oh if we have USB PGP smart cards, then the machine which runs the ceremony becomes a HIGH VALUE TARGET and needs to be secured in a more robust way.

  • Storage of these HIGH VALUE TARGETS becomes an issue. We need a way to ensure they haven’t been tampered with, and do a regular audit of them.

I cribbed some examples from the DNSSEC root key signing ceremony to improve our security around the above. Things like hard-sided cases, tamper evident bags, and a safe to store it all in.

I used the EFF’s diceware passwords to ensure that we had random passwords for the PGP smart cards.

I used the US:GSA Form: SF702 "Security Container Check Sheet" to audit the contents of the safe.

This may have been overkill, but making this a formal process that we ran regularly with security audits, and strict processes made this process become second nature and ultimately better trusted and flexible.

Ribbon badge with a keyhole representing the security ceremony

Repetition Pain Points

Doubling down on the re-key operations we found that using copy and paste for the long commands was getting to be a bit of a pain.

Also we found that the shard distribution process was error prone and painful. Once the final shardbearer submits their key and quorum is reached, Vault hands back the new, PGP-encrypted shares in one shot:

$ vault operator rekey -nonce="d3adb33f-0000-1111-2222-333344445555" -format=json <unseal_key>
{
  "started": true,
  "nonce": "d3adb33f-0000-1111-2222-333344445555",
  "t": 5,
  "n": 8,
  "progress": 5,
  "required": 5,
  "pgp_fingerprints": [
    "aa11bb22cc33dd44ee55ff6600112233445566aa",
    "bb22cc33dd44ee55ff6600112233445566aabb11",
    "cc33dd44ee55ff6600112233445566aabb11cc22",
    "dd44ee55ff6600112233445566aabb11cc22dd33",
    "ee55ff6600112233445566aabb11cc22dd33ee44",
    "ff6600112233445566aabb11cc22dd33ee44ff55",
    "6600112233445566aabb11cc22dd33ee44ff5566",
    "00112233445566aabb11cc22dd33ee44ff556677"
  ],
  "backup": true,
  "verification_required": false,
  "complete": true,
  "keys": [],
  "keys_base64": [
    "wcBMA9...steven==",
    "wcBMA9...addison==",
    "wcBMA9...blair==",
    "wcBMA9...connie==",
    "wcBMA9...sf_continuity==",
    "wcBMA9...nyc_continuity==",
    "wcBMA9...slc_continuity==",
    "wcBMA9...goozbach=="
  ]
}

Note keys comes back empty and keys_base64 is populated instead — because we passed -pgp-keys, Vault encrypts each new share to its matching Keybase/PGP identity rather than handing back raw key material. Only the shardbearer holding the matching private key can decrypt their own entry.

You have to copy and paste the keys_base64 entries, figure out which PGP key it ties to, and email that to the proper person. More than once I ended up being a line off and sending the wrong key to the wrong person.

Wrench badge representing the Vault Helper tool

This led me to create Vault Helper as a wrapper around the entire re-key event. It handles the re-key command generation and execution, stores the output json artifacts, splits the per-person key shards, and saves the session’s metadata.

The config file for a re-key event ends up reading like a roster instead of a pile of fingerprints:

Vault Helper Config File
VAULT_HELPER_MODE=pgp
VAULT_HELPER_KEYSHARES=8
VAULT_HELPER_KEYTHRESH=5
VAULT_HELPER_PGPKEYS=keybase:steven,keybase:addison,keybase:blair,keybase:connie,keybase:sf_continuity,keybase:nyc_continuity,keybase:slc_continuity,keybase:goozbach
VAULT_HELPER_ROOTTOKPGPKEY=keybase:goozbach
VAULT_HELPER_VERBOSE=''
ENV_NAME=production
# Do not remove the below line
export ENV_NAME VAULT_HELPER_KEYSHARES VAULT_HELPER_KEYTHRESH VAULT_HELPER_PGPKEYS VAULT_HELPER_ROOTTOKPGPKEY VAULT_HELPER_MODE

And running the re-key itself is three commands instead of the hand-assembled CLI invocation from earlier, repeated once per shardbearer until quorum:

$ vault_helper rekey
$ vault_helper continue
$ vault_helper export

export signs a tarball of every artifact from the session — the PGP-encrypted shards, the JSON output, the nonce — with your own PGP key, so there’s a single auditable record of exactly what happened.

Full documentation, including the init, cancel, and status sub-commands, lives in the project’s README on GitLab.

Anatomy of The Minimum Viable Pain Point

This entire exercise is a perfect example of what I’ve come to call the "Minimum Viable Pain Point".

The minimum viable pain point is the smallest dose of pain that will actually justify fixing something properly. And while you can usually control how severe or risky something is, frequency is the one lever you can pull with the least resistance.

When do you know you’re looking at a minimum viable pain point?

  • If the people or processes responsible for generating the pain aren’t directly experiencing the pain.

  • If a process is only rarely done, or is disproportionally dangerous to its frequency.

  • If a process is only ever done by a single person.

  • If the situation which triggers a process doesn’t happen on a regular basis

Frequency, and danger level are the two main components of the Minimum Viable Pain Point with a lesser component based on who is feeling the pain.

Really there’s only so much one can do to change the danger level, so the lever that can easily get pulled to generate an MVPP is often the frequency lever. Occasionally the person feeling the pain being someone other than the person who feels it most often can move from "This hurts but is normal." to "This is TOO painful".

Management Buy-in

One note on using this heuristic: I’ve found that it works better if you get management buy-in for forcing systems, processes, or tooling into an MVPP state.

But keep in mind that you do want to ensure that the payoff is worth the work.

If you misdiagnose an MVPP you’ll burn political capital that may make it harder to get buy-in to fix a valid problem in the future.

Stronger by Accident

The same person, now relaxed and giving a thumbs up, with the vault door open to a calm meadow

The Vault re-key itself never got any less mathematically dangerous. The quorum math and the lock-yourself-out risk were exactly the same on the last re-key as they were on the first. What we did make safer was everything around it: Vault Helper replaced hand-copied PGP shards and manually-typed commands with a config file and three sub-commands, which cut out the copy-paste mistakes that used to send the wrong key to the wrong person.

Forcing the frequency up, and rotating who ran the ceremony each month, meant no one person held all the institutional knowledge, and running it often enough surfaced edge cases and bad assumptions that a single careful read-through never would have.

The most illuminating part of this process was that forcing the process didn’t just automate something scary, it revealed problems nobody was aware of: we had no real way to audit the physical shards, and no answer for business continuity if the entire team disappeared. We only found out because we were forced to run the process often enough to hit them, and fixed both along the way. The SF702 audits and the three-city continuity keys exist because this process got forced into the open, not because anyone planned for them up front. Running the process regularly finds the novel ways it can fail.

This is the real case for using a Minimum Viable Pain Point: it’s not just a productivity trick, it’s a discovery mechanism. You don’t know what’s wrong with a painful, rare process until you run it.

Next time you’ve got a process, tool, or app that scares everyone into avoiding it, don’t ask how to make it hurt less. Ask how to make the pain drive improvement. The pain will show you what needs fixing, usually in ways you could never have designed on purpose.

1. RGE — Resume-Generating Event. IT slang for a mistake severe enough to cost you your job.
2. h/t to Brandon Sanderson — StormLight Archive series, great read
3. I like this better than Bus factor, less gruesome