Mishti: Create 90 day transition for hashes

[Jkd-eth]

User Story:

As the Passport team
We want to use the Mishti network
So that we can replace our dependency on our stored secret key for hashing

Acceptance Criteria

GIVEN a Passport user
WHEN they claim a stamp in Passport App
THEN the hash is generated using Mishti Human Keys

GIVEN a Passport users claiming stamps
WHEN some of the stamps have the new Mishti hashes and some of the user stamps have the old Passport hashes
THEN we need to establish a way of deduplicating the hashes until we can fully transition to the new Mishti hashes without requiring users to reclaim stamps they previously have obtained

Following up the story #3095 and story #3096 this issue will replace all of our existing credential hashes with new hashes using Mishti. This work is to implement the transition which needs to be a 90 day period once the new hahes have been implemented.

(See @nutrina notes below)

Product & Design Links:

Tech Details:

Open Questions:

Notes/Assumptions:

[nutrina]

Idea + example (dates are not fixed, they have been chosen just for the example):

• on 01.01.2025 we start issuing 2 versions for each stamps:
  • version with current hash
  • version with new hash
• we continue to score the stamps with the old hash
• on 01.04.2025 we can assume all stamps that exist and are not expired exist in both versions
  • hence we can start scoring the stamps with the new hash version

Tech details:

• we shall track multiple versions of a stamps in the ceramic cache table (depending on the hash version)
• the stamp version is relevant also in the HashScorerLink model
  • we will probably need to keep duplicate records in HashScorerLink to allow deduplication using the old version of the stamps and the new version of the stamps once we make the switch

[nutrina]

More tech details

Note

• given that we plan to track multiple hashing keys, we should also use the same (new) keys to for signing the stamps
• this means, that having the same version of a stamps, they will differ not inly in the hash field but also in the issuer

Infra

• need to be able to support multiple signing/hashing keys
• ideally we configure also the date ranges for when certain signing/hashing keys shall be used and the iam service uses the proper key automatically depending on the current time
• shall we look into AWS HSM already? Or do this as a next step?

Identity

• a package being created now

IAM

• update the verify API to return 2 versions of the stamps:
  • one signed with the current key
  • one signed with the future key
  • the identity package will need to be updated

Embed

• similar changes like in the IAM (the embed service depends on the same identity packages as the IAM)
• update the embed services in scorer as well

Scorer

• the user will basically have multiple versions of passports, one for each version of a key. We will need to actively track and work with:
  • the passport for the current version of the key
  • the one for the future version
• need to keep track of the current active version of the keys in at least the following models
  • CeramicCache
  • HashScorerLink
  • Score
  • Passport
  • Stamp
• all deduplication operations need to be mirrored on both version of the stamps
  • perform the scoring on current version of Passport and the future version of Passport
  • need to ensure that the scoring is identical:
    • we should raise an alarm if results are not identical for what reason ever
    • make sure to use DB transaction to ensure atomicity of these operations
• the transition of the keys (retiring the new key, promoting a future key to active and introducing a new future key) should be handled automatically:
  • we should have the same date & key version information configured in the scorer as we do in the IAM / embed
  • ideally we would enforce constraints regarding scoring at the DB level, for example:
    • reject storing a score computed for an retired key
    • reject storing a stamp for a retired key
    • we could enforce these constraints if we track the key versions and timeframes for each key in the DB
• banning by hash will be affected, we need to track the ban for all the hashes of a specific account

Passport App

• need to forward the stamps to the scorer backend
• when retrieving from the BE: return only the version of the stamp with the latest hash (the hash which expires last)
• update the ceramic schema (we will have a new field hashes)

Onchain stamps

Drop the current schemas in favour of 1 score.
Create 2 tickets for this:

• implementation
• upgrading on every chain

Impact

Infra

With these changes we expect:

• IAM:
  • slightly increased time for issuing stamps (verifications however will only be performed once)
• Embed:
  • more increased time for issuing stamps, given that this will also call the scoring service in the scorer
• Scorer:
  • probably double the time for scoring a passport
  • slightly increased times for reading / writing stamps
  • DB - double the storage requirement for stamps & scores given that we'll have to store & track everything twice

Public API

@lebraat we will need to come up with a strategy regarding what version of the stamps we return to users:
- do we return stamps created with retired keys? I would suggest we do not, but instead we could allow an optional argument ( return_retired which should default to false) which can allow a partner to pull retired versions of a stamp. I guess this will be particularly relevant for historical scores / stamps

Ceramic

• we'll store all versions of the stamps in ceramic
• in the future we might want to adjust the model to ensure easy filtering by issuer / key version, but I don't. think this would be a priority from the beginning

Open questions

• shall we change the stamp schema and add a version field for the stamp?
  • probably not, if we also change the signing key, and we will have different issuers
• what is the impact on data analysis
• what is the impact on on-chain data (score & stamps)

[nutrina]

Other notes:

• introduce the notion of epochs
• do we start with a clean score every epoch (do all scores expire at epoch end)?
• need to link all hashes of a provider & address ???
  • to track the 2 overlapping hashes in an epoch it probably makes most sense to convert the hash field to an array and store both hashes there, for example: hashes: ["v0.0.1:12324...","v0.0.1:56789..."]
• we'll need to support rotating issuers in our onchain Verifier