diff options
| author | Tatsuya Kyushima <49891479+kyu08@users.noreply.github.com> | 2024-05-02 18:17:50 +0900 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2024-05-02 17:17:50 +0800 |
| commit | ebad4bb88842812c9a2633cb9e06071114519314 (patch) | |
| tree | 7c747ee9356a22f9168370ddb76d7764320f3763 /src/utils/tiktoken.rs | |
| parent | b6957ae3af07608d92b7dda1709997a8da77c62b (diff) | |
| download | aichat-ebad4bb88842812c9a2633cb9e06071114519314.tar.gz | |
chore: fix typos (#476)
Diffstat (limited to 'src/utils/tiktoken.rs')
| -rw-r--r-- | src/utils/tiktoken.rs | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/src/utils/tiktoken.rs b/src/utils/tiktoken.rs index 8b7e712..cd83ed3 100644 --- a/src/utils/tiktoken.rs +++ b/src/utils/tiktoken.rs @@ -128,7 +128,7 @@ pub fn byte_pair_split<'a>(piece: &'a [u8], ranks: &HashMap<Vec<u8>, usize>) -> // Originally, we had one too! Without it, we were only vaguely faster than Python. // I used an RWLock to protect the cache. This didn't seem to hurt single threaded performance // noticeably, but it did affect multi-threaded performance. Weirdly, it seemed to affect -// multi-threaded performance even when I only had readers (maybed I messed something up?). +// multi-threaded performance even when I only had readers (maybe I messed something up?). // Anyway, I realised that we could get rid of the cache, if we treat the set of tokens as a cache! // These are exactly the set or merges that are likely to be hot. And now we don't have to think // about interior mutability, memory use, or cloning. |