invoices: migrate KV invoices to native SQL for users of KV SQL backends #8831
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Important Review skippedAuto reviews are limited to specific labels. 🏷️ Labels to auto review (1)
Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
bhandras
force-pushed
the
sql-invoice-migration
branch
3 times, most recently
from
6682b50 to
338e1f0
Compare
bhandras
force-pushed
the
sql-invoice-migration
branch
3 times, most recently
from
d2a329f to
6379a8b
Compare
bhandras
force-pushed
the
sql-invoice-migration
branch
2 times, most recently
from
5fe92e2 to
a7bf598
Compare
aakselrod
reviewed
Sep 5, 2024
bhandras
force-pushed
the
sql-invoice-migration
branch
5 times, most recently
from
b6f0ac8 to
b983851
Compare
bhandras
changed the title
bhandras
force-pushed
the
sql-invoice-migration
branch
from
b983851 to
706b444
Compare
bhandras
force-pushed
the
sql-invoice-migration
branch
7 times, most recently
from
96f0cbe to
bfe4ad5
Compare
bhandras
force-pushed
the
sql-invoice-migration
branch
from
998b544 to
d4fb931
Compare
bhandras
force-pushed
the
sql-invoice-migration
branch
from
d4fb931 to
350cd2d
Compare
bhandras
force-pushed
the
sql-invoice-migration
branch
from
350cd2d to
36ac31b
Compare
This commit adds the migration_tracker table which we'll use to track if a custom migration has already been done.
This commit introduces support for custom, in-code migrations, allowing a specific Go function to be executed at a designated database version during sqlc migrations. If the current database version surpasses the specified version, the migration will be skipped.
This commit separates the execution of SQL and in-code migrations from their construction. This change is necessary because, currently, the SQL schema is migrated during the construction phase in the lncfg package. However, migrations are typically executed when individual stores are constructed within the configuration builder.
Previously we intentially did not set settled_at and settle_index when inserting a new invoice as those fields are set when we settle an invoice through the usual invoice update. As migration requires that we set these nullable fields, we can safely add them.
Certain invoices may not have a deterministic payment hash. For such invoices we still store the payment hashes in our KV database, but we do not have a sufficient index to retrieve them. This PR adds such index to the SQL database that will be used during migration to retrieve payment hashes.
…a hash The current sqlc GetInvoice query experiences incremental slowdowns during the migration of large invoice databases, primarily due to its complex predicate set. For this specific use case, a streamlined GetInvoiceByHash function provides a more efficient solution, maintaining near-constant lookup times even with extensive table sizes.
This commit runs the invoice migration if the user has a KV SQL backend configured.
Previously we'd recalculate the paid amount by summing amounts of settled HTLCs. This approach while correct would stop the SQL migration process as some KV invoices may have incorrectly stored paid amounts.
Previously, we applied replacements to our schema definitions to make them compatible with both SQLite and Postgres backends, as the files were not fully compatible with either. With this change, the only replacement required for SQLite has been moved to the generator script. This adjustment ensures compatibility by enabling auto-incrementing primary keys that are treated as 64-bit integers by sqlc.
bhandras
force-pushed
the
sql-invoice-migration
branch
from
36ac31b to
b1a462d
Compare
|
The "Check commits" CI step succeeded locally, probably just too many commits for GitHub. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Change Description
This pull request adds the migration of old key-value (KV) invoices to the new native SQL schema when the --db.use-native-sql flag is set, unless the --db.skip-sql-invoice-migration flag is also specified.
Please note that since we currently do not support running on mixed database backends for users of
bboltoretcd, an additional step is required to migrate their KV database to SQL first. For more context, please see lightninglabs/lndinit#21.