Skip to content

Instantly share code, notes, and snippets.

@islem-esi

islem-esi/tokenize_code.py

Created April 23, 2022 11:21
Show Gist options
  • Save islem-esi/eada733bb90adf8eab2001cbd0ab6866 to your computer and use it in GitHub Desktop.
Save islem-esi/eada733bb90adf8eab2001cbd0ab6866 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
tokenize_code.py
rootdir = "PATH TO THE DIRECTORY OF THE PROJECT YOU CLONED FROM GITHUB"
files_code = load_all_files(root_dir)
tokenized_data = []
for d in files_code:
try:
tokenized_data.append(tokenize_python(d))
except:
print('error')
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment