Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Store timeline locations on storage controller (schema, insert, timeline creation and passthrough to the majority) #9011

Open
4 of 11 tasks
jcsp opened this issue Sep 16, 2024 · 0 comments
Open
4 of 11 tasks

Store timeline locations on storage controller (schema, insert, timeline creation and passthrough to the majority) #9011

jcsp opened this issue Sep 16, 2024 · 0 comments
Assignees
@arssher @arpad-m

Comments

@jcsp
Copy link
Collaborator

jcsp commented Sep 16, 2024 •
edited by arpad-m
Loading

Tasks
Preview Give feedback
  4. Extract public sk types to safekeeper_api #10137
  5. Extract safekeeper http client to separate crate. #10140
  9. Add safekeeper utilization endpoint #10429
  10. Expose safekeeper APIs for creation and deletion #10478
  11. storcon: timeline table, deletion and creation #10440

There is a subtlety in handling timeline creations & deletions on safekeepers. We don't want to block op if one safekeeper is down, but neither want to have uncreated / undeleted timelines left behind. So seems like we should track these in tables like sk_pending_timeline_creations and sk_pending_timeline_deletions, and have background task working on these.
@jcsp jcsp mentioned this issue Sep 16, 2024
Open
@arssher arssher changed the title Store timeline locations on storage controller (schema, insert, timeline creation passthrough to the majority) Store timeline locations on storage controller (schema, insert, timeline creation and passthrough to the majority) Sep 30, 2024
@arssher arssher self-assigned this Dec 2, 2024
@arssher arssher mentioned this issue Dec 13, 2024
Merged
github-merge-queue bot pushed a commit that referenced this issue Dec 13, 2024
@arssher
Extract public sk types to safekeeper_api (#10137)
ce8eb08 
## Problem

We want to extract safekeeper http client to separate crate for use in
storage controller and neon_local. However, many types used in the API
are internal to safekeeper.

## Summary of changes

Move them to safekeeper_api crate. No functional changes.

ref #9011
@jcsp jcsp assigned arpad-m and unassigned koivunej Jan 6, 2025
@arpad-m arpad-m mentioned this issue Jan 16, 2025
Merged
github-merge-queue bot pushed a commit that referenced this issue Jan 17, 2025
@arpad-m
Add safekeeper utilization endpoint (#10429)
b0f3409 
Add an endpoint to obtain the utilization of a safekeeper. Future
changes to the storage controller can use this endpoint to find the most
suitable safekeepers for newly created timelines, analogously to how
it's done for pageservers already.

Initially we just want to assign by timeline count, then we can iterate
from there.

Part of #9011
This was referenced Jan 18, 2025
Draft
Merged
github-merge-queue bot pushed a commit that referenced this issue Jan 22, 2025
@arpad-m
Expose safekeeper APIs for creation and deletion (#10478)
c60b913 
Add APIs for timeline creation and deletion to the safekeeper client
crate. Going to be used later in #10440.

Split off from #10440.

Part of #9011
@arpad-m arpad-m mentioned this issue Jan 30, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@arssher arssher

@arpad-m arpad-m

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jcsp @koivunej @arssher @arpad-m