Continuous key rotation with Chef

Lets see if I can get this down on paper in a meaningful way.. Players: a) some server (has to be Chef Server) - aka: Key Master. b) the rest of the infrastructure Tools needed: a) chef vault b) admin key for the Key Master c) sublime text The flow: Key master converges a recipe that does a global search for all of the nodes. For each node it generates a new key pair. It rotates the key and places the new key into a vault with search criteria of only itself and the node. Each node on converge accesses the vault and retrieves a new key. Marks the vault as converged or deletes the vault after consumption. Faults: What happens if the node doesn't converge for a long time? How does key rotation actually work? Can a node even converge if the key has been rotated? >> probably this is the way << Perhaps the node has to generate a key and set the search criteria to itself and Key Master. Key Master consumes the key and runs ctl command. Do Nodes continue to fail converges until Key Master updated the key? How does key rotation actually work? Result: Ever converge the node rotates its own key. Same model can probably be done for SSH keys. Final thought(s): What does it actually buy? I don't know, but many customers ask about it. Should it be done? Should each node have a unique, individual vault? Most likely, if you really think about it, there isn't a reason. Node's should be grouped and each group should run off a same vault. Having 1 vault per node with identical info is meaningless. Especially, if there is an admin who has access to all of the nodes anyway.