Skip to content

Commit e2d8a24

Browse files
Quentin-Anthonygithub-actions
andauthored
Move conditional tiktoken import to the __init__ func (#842)
* Move conditional tiktoken import to the __init__ func * Update NeoXArgs docs automatically --------- Co-authored-by: github-actions <[email protected]>
1 parent 91b72d9 commit e2d8a24

File tree

2 files changed

+7
-7
lines changed

2 files changed

+7
-7
lines changed

configs/neox_arguments.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -111,7 +111,7 @@ Logging Arguments
111111

112112
- **git_hash**: str
113113

114-
Default = 7d682df
114+
Default = 6dd0344
115115

116116
current git hash of repository
117117

megatron/tokenizer/tokenizer.py

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -353,13 +353,13 @@ def eod(self):
353353
class TiktokenTokenizer(AbstractTokenizer):
354354
"""Tokenizer from OpenAI's tiktoken implementation"""
355355

356-
try:
357-
import tiktoken
358-
except ModuleNotFoundError:
359-
print("Please install tiktoken: (https://github.com/openai/tiktoken)")
360-
raise Exception
361-
362356
def __init__(self, vocab_file):
357+
try:
358+
import tiktoken
359+
except ModuleNotFoundError:
360+
print("Please install tiktoken: (https://github.com/openai/tiktoken)")
361+
raise Exception
362+
363363
name = "TiktokenTokenizer"
364364
super().__init__(name)
365365

0 commit comments

Comments
 (0)