Need info on how to handle 1 billion keys #6441
-
My use case if to reference person's first name, last name & SSN to uniquely identify .Insert that record in key value on if thats unique.If my records grow to 1 billion I want to search unique person and insert only if not existing |
Beta Was this translation helpful? Give feedback.
Answered by
derekcollison
Feb 3, 2025
Replies: 1 comment
-
Currently subject based indexes for lookup are held in memory. So I would suggest using a hashing approach to use multiple streams / KVs and size accordingly with memory. We are looking at improving the memory footprint for very large key/subject spaces. |
Beta Was this translation helpful? Give feedback.
0 replies
Answer selected by
derekcollison
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Currently subject based indexes for lookup are held in memory.
So I would suggest using a hashing approach to use multiple streams / KVs and size accordingly with memory.
We are looking at improving the memory footprint for very large key/subject spaces.