Skip to content

Instantly share code, notes, and snippets.

@kabirahuja2431

kabirahuja2431/indexify.py

Last active October 8, 2019 14:39
Show Gist options
  • Save kabirahuja2431/33a14281f6ef6cf0f1bf9caa9e069907 to your computer and use it in GitHub Desktop.
Save kabirahuja2431/33a14281f6ef6cf0f1bf9caa9e069907 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
indexify.py
# Obtaining indices for each token
sent_ids = tokenizer.convert_tokens_to_ids(padded_tokens)
print(sent_ids)
# Out: [101, 1045, 2428, 5632, 2023, 3185, 1037, 2843, 1012, 102, 0, 0]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment