Skip to content

Instantly share code, notes, and snippets.

@creyesp

creyesp/tokenization_of_transformer_models.py

Created February 21, 2021 21:38
Show Gist options
  • Save creyesp/4e309e322d462ea234e2d73203544d7a to your computer and use it in GitHub Desktop.
Save creyesp/4e309e322d462ea234e2d73203544d7a to your computer and use it in GitHub Desktop.
Download ZIP
Raw
tokenization_of_transformer_models.py
from transformers import AutoTokenizer, AutoModelForSequenceClassification
model_name = "tr3cks/3LabelsSentimentAnalysisSpanish"
tokenizer_sent_esp = AutoTokenizer.from_pretrained(model_name)
model_sent_esp = AutoModelForSequenceClassification.from_pretrained(model_name)
# The output is ['ja', '##ja', '##ja', 'que', 'risa', 'me', 'da']
tokenizer_sent_esp.tokenize('jajaja que risa me da')
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment