🐛 fix knowledge king oom bug #150
Conversation
- upgrade jtokkit version from 0.4.0 to 0.5.0 - use lazy load register that the new method of the Encodings class, which can resolve the issue of initializing all encoding types problem
|
🤖 OpenAI High-level SummaryThe code changes in this pull request involve upgrading the File Changes
Please review the specific file changes for more detailed information on each modification. Chat with 🤖 OpenAI Bot (
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| fun measureNumOfTokens(string: String): Int { | ||
| val encoding: Encoding = registry.getEncoding(encodingType) | ||
| val encoded: List<Int> = encoding.encode(string) | ||
| return encoded.size | ||
| return encoding.countTokens(string) | ||
| } |
There was a problem hiding this comment.
🤖 OpenAI
The measureNumOfTokens function has been modified to use the countTokens method of the encoding object. This change improves the efficiency of token counting by utilizing the built-in functionality provided by the jtokkit library.
| fun measureNumOfTokens(string: String): Int { | |
| val encoding: Encoding = registry.getEncoding(encodingType) | |
| val encoded: List<Int> = encoding.encode(string) | |
| return encoded.size | |
| return encoding.countTokens(string) | |
| } | |
| fun measureNumOfTokens(string: String): Int { | |
| return encoding.countTokens(string) | |
| } |
| fun measureNumOfTokens(string: String): Int { | ||
| val encoding: Encoding = registry.getEncoding(encodingType) | ||
| val encoded: List<Int> = encoding.encode(string) | ||
| return encoded.size | ||
| return encoding.countTokens(string) | ||
| } |
There was a problem hiding this comment.
fun measureNumOfTokens(string: String): Int = encoding.countTokens(string)
| companion object { | ||
| private val registry = Encodings.newLazyEncodingRegistry() | ||
| private lateinit var encoding: Encoding | ||
| } | ||
|
|
||
| init { | ||
| encoding = registry.getEncodingForModel(ModelType.GPT_3_5_TURBO) | ||
| } |
There was a problem hiding this comment.
Q：這裡不能直接初始化嗎？
companion object {
private val registry = Encodings.newLazyEncodingRegistry()
private val encoding = registry.getEncodingForModel(ModelType.GPT_3_5_TURBO)
}
|
Kudos, SonarCloud Quality Gate passed!
|
Why need this change? / Root cause:
Changes made:
Test Scope / Change impact:
Summary by OpenAI
Release Notes:
jtokkitversion from 0.4.0 to 0.5.1.JTokkitclass.JTokkitclass.measureNumOfTokensfunction to use thecountTokensmethod of the encoding object.