-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Introducing a memory control mechanism during the query planning stage #12573
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #12573 +/- ##
============================================
- Coverage 43.50% 43.37% -0.14%
Complexity 69 69
============================================
Files 3125 3164 +39
Lines 194301 196674 +2373
Branches 23168 23425 +257
============================================
+ Hits 84536 85308 +772
- Misses 109765 111366 +1601 ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Think about the query retry logic and the last remaining memory(less than 1MB), you can reserve that before dispatch.
LOCAL_EXECUTION_PLANNER.releaseToFreeMemoryForOperators(reservedBytesInTotalForFrontEnd); | ||
reservedBytesInTotalForFrontEnd = 0; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LOCAL_EXECUTION_PLANNER.releaseToFreeMemoryForOperators(reservedBytesInTotalForFrontEnd); | |
reservedBytesInTotalForFrontEnd = 0; | |
if (reservedBytesInTotalForFrontEnd != 0) { | |
LOCAL_EXECUTION_PLANNER.releaseToFreeMemoryForOperators(reservedBytesInTotalForFrontEnd); | |
reservedBytesInTotalForFrontEnd = 0; | |
} |
…re retry of QueryExectuion & reserverMemoryImmediately after dispatching the FIs for the last batch of memory occupied by the FE.
Quality Gate failedFailed conditions |
This PR mainly introduces a memory control mechanism during the query planning stage. Currently, the IoTDB query engine does not implement memory control at the FE (Frontend) stage. In scenarios with massive series queries (e.g., select * from root.**), the query plans generated at the FE stage can become excessively large. Roughly estimating, the size of a single SeriesScanNode is about 1/2 KB, which means that two million series corresponding to two million SeriesScanNodes would occupy 1 GB, posing a potential risk of Out-Of-Memory (OOM). In high concurrency scenarios, even if a single query plan is not large, the total memory occupied by multiple query plans can still lead to OOM.
Therefore, it is now desired to introduce memory size control for FE query plans within the query engine.
Related docs:https://apache-iotdb.feishu.cn/docx/F7nidsysKofhO4xTi35cuDhynCh