Epic: safekeepers dynamic membership change (Phase 2)

[koivunej]

Phase 1 enabled runtime changes to walproposer configuration, and a semi-manual routine involving pullign timeline data between safekeepers (#4782)

In this phase, we will implement joint consensus for the safekeeper

Tasks

Tasks

Preview Give feedback

1. safekeeper membership change rfc #8455
   +0
2. TLA+ spec for changes #8699 1-2w
3. Make safekeepers registered in storage controller (deploy scripts) #8698
   3 of 3
   +0
   
4. Safekeeper listing API, storcon commands, deletion API, deleted state #9981
   +0
   
5. Store timeline locations on storage controller (schema, insert, timeline creation and passthrough to the majority) #9011 1-2w
6. add configurations to safekeepers, including API calls for switching into joint consensus state #9965
   c/storage/safekeeper +1
7. compute-safekeeper protocol changes 1w #10326
   c/storage/safekeeper +1
   
8. walproposer changes to support configurations and joint consensus 1-2w
9. storage controller procedure to drive migration from old set to new set (includes API+CLI to evacuate SK) 1-2w
10. --- Milestone: demonstrate (neon_local) full migration process with running compute (python tests) 1w
11. Infrastructure to make storcon/SK bits work within the event simulator (i.e. not use a real network, or impl generic read/write traits and wire them into in-memory stuff) 1-2w
12. Event simulator tests for changes (the primary tests for covering all the new code paths) 1w
13. cplane parts: https://github.com/neondatabase/cloud/issues/16796
14. --- Milestone: demonstrate migration in staging including control plane parts, on selected timelines with SK generations enabled in their computes.
15. Migrate all staging tenants' timelines into storage controller, enable SK generations for all computes in staging (1w)
16. Add background migration chaos for SK timelinesn in storage controller + enable in staging (1w)
17. --- Milestone: drill for SK node failure response, recovering using new APIs
18. Migrate all prod tenants' timelines into storage controller
19. Await compute restarts to pick up generation enablement: TODO coordinate with compute team on dates
20. --- Milestone: demonstrate migration on prod

Later

Tasks

Preview Give feedback

1. Make storage controller scan safekeepers (heartbeats)