Created
March 29, 2019 13:40
-
-
Save ManivannanMurugavel/7e61eecbfb37bbc073ae58e7c07f2a6a to your computer and use it in GitHub Desktop.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import spacy | |
| import random | |
| TRAIN_DATA = [('what is the price of polo?', {'entities': [(21, 25, 'PrdName')]}), | |
| ('what is the price of ball?', {'entities': [(21, 25, 'PrdName')]}), | |
| ('what is the price of jegging?', {'entities': [(21, 28, 'PrdName')]}), | |
| ('what is the price of t-shirt?', {'entities': [(21, 28, 'PrdName')]}), | |
| ('what is the price of jeans?', {'entities': [(21, 26, 'PrdName')]}), | |
| ('what is the price of bat?', {'entities': [(21, 24, 'PrdName')]}), | |
| ('what is the price of shirt?', {'entities': [(21, 26, 'PrdName')]}), | |
| ('what is the price of bag?', {'entities': [(21, 24, 'PrdName')]}), | |
| ('what is the price of cup?', {'entities': [(21, 24, 'PrdName')]}), | |
| ('what is the price of jug?', {'entities': [(21, 24, 'PrdName')]}), | |
| ('what is the price of plate?', {'entities': [(21, 26, 'PrdName')]}), | |
| ('what is the price of glass?', {'entities': [(21, 26, 'PrdName')]}), | |
| ('what is the price of moniter?', {'entities': [(21, 28, 'PrdName')]}), | |
| ('what is the price of desktop?', {'entities': [(21, 28, 'PrdName')]}), | |
| ('what is the price of bottle?', {'entities': [(21, 27, 'PrdName')]}), | |
| ('what is the price of mouse?', {'entities': [(21, 26, 'PrdName')]}), | |
| ('what is the price of keyboad?', {'entities': [(21, 28, 'PrdName')]}), | |
| ('what is the price of chair?', {'entities': [(21, 26, 'PrdName')]}), | |
| ('what is the price of table?', {'entities': [(21, 26, 'PrdName')]}), | |
| ('what is the price of watch?', {'entities': [(21, 26, 'PrdName')]})] | |
| def train_spacy(data,iterations): | |
| TRAIN_DATA = data | |
| nlp = spacy.blank('en') # create blank Language class | |
| # create the built-in pipeline components and add them to the pipeline | |
| # nlp.create_pipe works for built-ins that are registered with spaCy | |
| if 'ner' not in nlp.pipe_names: | |
| ner = nlp.create_pipe('ner') | |
| nlp.add_pipe(ner, last=True) | |
| # add labels | |
| for _, annotations in TRAIN_DATA: | |
| for ent in annotations.get('entities'): | |
| ner.add_label(ent[2]) | |
| # get names of other pipes to disable them during training | |
| other_pipes = [pipe for pipe in nlp.pipe_names if pipe != 'ner'] | |
| with nlp.disable_pipes(*other_pipes): # only train NER | |
| optimizer = nlp.begin_training() | |
| for itn in range(iterations): | |
| print("Statring iteration " + str(itn)) | |
| random.shuffle(TRAIN_DATA) | |
| losses = {} | |
| for text, annotations in TRAIN_DATA: | |
| nlp.update( | |
| [text], # batch of texts | |
| [annotations], # batch of annotations | |
| drop=0.2, # dropout - make it harder to memorise data | |
| sgd=optimizer, # callable to update weights | |
| losses=losses) | |
| print(losses) | |
| return nlp | |
| prdnlp = train_spacy(TRAIN_DATA, 20) | |
| # Save our trained Model | |
| modelfile = input("Enter your Model Name: ") | |
| prdnlp.to_disk(modelfile) | |
| #Test your text | |
| test_text = input("Enter your testing text: ") | |
| doc = prdnlp(test_text) | |
| for ent in doc.ents: | |
| print(ent.text, ent.start_char, ent.end_char, ent.label_) |
ykarray
commented
Sep 15, 2021
via email
any update please. !
Le mar. 14 sept. 2021 à 12:51, yasmine karray ***@***.***> a
écrit :
… hi
yes sure this is my data :
[('nom du client sami adresse client 75000 paris france\xa0', {'entities':
[(34, 52, 'adresse_client', 1, 'rgb(167, 248, 246)'), (14, 18,
'nom_client', 0, 'rgb(45, 50, 168)')]}), ('N de facture RE-103020170731
date de facturetion 31 juil.2017 reference 12345 date de livraison 26
juil.2017 votre numero client 100020170731 votre interlocuteur Anne-laure
Enault\xa0', {'entities': [(49, 61, 'date', 1, 'rgb(125, 180, 233)'), (13,
28, 'num_facture', 0, 'rgb(63, 10, 198)')]}), ('description Ecran
plat\xa0', {'entities': [(12, 23, 'description', 0, 'rgb(138, 58,
123)')]}), ('quantité 1 pcs\xa0', {'entities': [(9, 15, 'quantite', 0,
'rgb(23, 210, 137)')]}), ('prix unitaire -699.00 EUR\xa0', {'entities':
[(14, 26, 'prix_unitaire', 0, 'rgb(19, 91, 227)')]}), ('prix unitaire
-689.00 EUR\xa0', {'entities': [(14, 25, 'prix_unitaire', 0, 'rgb(19, 91,
227)')]}), ('prix total -699.00 EUR', {'entities': [(11, 22, 'prix_totale',
0, 'rgb(146, 188, 197)')]}), ('Montant total -699.00 EUR hors taxes 21.2%
-148.19 EUR Montant de la facture -847.19 EUR\xa0', {'entities': [(77, 89,
'montant_de_la_facture', 0, 'rgb(139, 118, 5)'), (37, 50,
'montant_hors_taxe', 1, 'rgb(7, 41, 47)')]}), ('nom du client : mohamed
adresse client : 75000, paris, france\xa0', {'entities': [(41, 62,
'adresse_client', 1, 'rgb(167, 248, 246)'), (16, 23, 'nom_client', 0,
'rgb(45, 50, 168)')]}), ('nom client yasmine adresse client 123 sfax
tunis', {'entities': [(34, 48, 'adresse_client', 1, 'rgb(167, 248, 246)'),
(11, 18, 'nom_client', 0, 'rgb(45, 50, 168)')]}), ('nom du client : ahmed
adresse client : route tunis 4.5 Km\xa0', {'entities': [(39, 57,
'adresse_client', 1, 'rgb(167, 248, 246)'), (16, 21, 'nom_client', 0,
'rgb(45, 50, 168)')]}), ('nom du client yosra adresse client ariana',
{'entities': [(35, 41, 'adresse_client', 1, 'rgb(167, 248, 246)'), (14, 19,
'nom_client', 0, 'rgb(45, 50, 168)')]}), ('N de facture RE-103020170731
date de facturetion 31 juil.2017 reference 12345 date de livraison 26
juil.2017 votre numero client 100020170731 votre interlocuteur Anne-laure
Enault\xa0', {'entities': [(49, 62, 'date', 1, 'rgb(125, 180, 233)'), (13,
28, 'num_facture', 0, 'rgb(63, 10, 198)')]}), ('N de facture 123ef54545
date de facturetion 01/01/2000\xa0', {'entities': [(44, 55, 'date', 1,
'rgb(125, 180, 233)'), (13, 23, 'num_facture', 0, 'rgb(63, 10, 198)')]}),
('N de facture 456879 date de facturetion 6-juin-2003 reference 12345 date
de livraison 26 juil.2017 votre numero client 100020170731\xa0',
{'entities': [(40, 51, 'date', 1, 'rgb(125, 180, 233)'), (13, 19,
'num_facture', 0, 'rgb(63, 10, 198)')]}), ('Num de facture 50010 date de
facturation 1 janvier 2021\xa0', {'entities': [(41, 56, 'date', 1,
'rgb(125, 180, 233)'), (15, 20, 'num_facture', 0, 'rgb(63, 10, 198)')]}),
('description Ecran plat\xa0', {'entities': [(12, 23, 'description', 0,
'rgb(138, 58, 123)')]}), ('description: ordinateur television\xa0',
{'entities': [(13, 35, 'description', 0, 'rgb(138, 58, 123)')]}),
('description radio macbook imprimante', {'entities': [(12, 36,
'montant_de_la_facture', 0, 'rgb(139, 118, 5)')]}), ('description: Ecran
plat télévision radio plaque chauffante ordinateur potable\xa0',
{'entities': [(12, 78, 'description', 0, 'rgb(138, 58, 123)')]}),
('quantité 1 pcs\xa0', {'entities': [(8, 15, 'quantite', 0, 'rgb(23, 210,
137)')]}), ('quantité: 1 pcs\xa0', {'entities': [(10, 16, 'quantite', 0,
'rgb(23, 210, 137)')]}), ('quantité 1 pcs 2 pcs 50 pcs 6000 pcs\xa0',
{'entities': [(9, 37, 'quantite', 0, 'rgb(23, 210, 137)')]}), ('quantité:
50 92 63 78 56\xa0\xa0', {'entities': [(10, 26, 'quantite', 0, 'rgb(23,
210, 137)')]}), ('quantité 1 10.5 52.36', {'entities': [(9, 21, 'quantite',
0, 'rgb(23, 210, 137)')]}), ('prix unitaire -699.00 EUR\xa0', {'entities':
[(14, 26, 'prix_unitaire', 0, 'rgb(19, 91, 227)')]}), ('prix unitaire 600
DT 50 DT 60 DT 90 DT', {'entities': [(14, 38, 'prix_unitaire', 0, 'rgb(19,
91, 227)')]}), ('prix unitaire: 60.00 DT 90.50 DT 89.36 DT', {'entities':
[(15, 41, 'prix_unitaire', 0, 'rgb(19, 91, 227)')]}), ('prix unitaire:
-699.00 EUR 93.80 EUR 150.7 EUR\xa0', {'entities': [(15, 47,
'prix_unitaire', 0, 'rgb(19, 91, 227)')]}), ('prix total -699.00 EUR',
{'entities': [(11, 22, 'prix_totale', 1, 'rgb(146, 188, 197)')]}), ('prix
total: 36.95 EUR 50.00 EUR 80.12 EUR', {'entities': [(12, 41,
'prix_totale', 0, 'rgb(146, 188, 197)')]}), ('prix total 50.00 DT 86.35 DT
20.78 DT\xa0', {'entities': [(11, 38, 'prix_totale', 0, 'rgb(146, 188,
197)')]}), ('prix total: 50 DT', {'entities': [(11, 17, 'prix_totale', 0,
'rgb(146, 188, 197)')]}), ('prix total 2 DT 92.2 DT 519.454.22 DT
600.060.123.09 DT', {'entities': [(11, 55, 'prix_totale', 0, 'rgb(146, 188,
197)')]}), ('Montant total -832.55 EUR hors taxes 21.2% -148.19 EUR Montant
de la facture -847.19 EUR', {'entities': [(77, 88, 'montant_de_la_facture',
0, 'rgb(139, 118, 5)'), (37, 50, 'montant_hors_taxe', 1, 'rgb(7, 41,
47)')]}), ('Montant total 596.30 DT hors taxes 10% -352.50 DT Montant de la
facture 500.96 DT', {'entities': [(72, 81, 'montant_de_la_facture', 1,
'rgb(139, 118, 5)'), (35, 50, 'montant_hors_taxe', 0, 'rgb(7, 41, 47)')]}),
('Montant total : 500 DT montant hors taxes 30% 50 DT Montant total de la
facture 250 DT', {'entities': [(80, 86, 'montant_de_la_facture', 0,
'rgb(139, 118, 5)'), (42, 51, 'montant_hors_taxe', 1, 'rgb(7, 41, 47)')]}),
('à mhamed masmoudi ***@***.*** rue 65 dhbhg 66
telephone:(789)654123', {'entities': [(63, 74, 'telephone', 4, 'rgb(199,
47, 32)'), (37, 52, 'adresse_client', 3, 'rgb(167, 248, 246)'), (18, 36,
'email_client', 1, 'rgb(62, 11, 159)'), (2, 17, 'nom_client', 0, 'rgb(45,
50, 168)')]}), ('à yasmine karray\xa0 ***@***.*** route
bosten\xa0 telephone:(789)12345678', {'entities': [(63, 76, 'telephone', 3,
'rgb(199, 47, 32)'), (39, 53, 'adresse_client', 2, 'rgb(167, 248, 246)'),
(19, 39, 'email_client', 1, 'rgb(62, 11, 159)'), (1, 16, 'nom_client', 0,
'rgb(45, 50, 168)')]}), ('à sami sallemi\xa0 ***@***.*** rue taniour
telephone:(789)25893614', {'entities': [(53, 66, 'telephone', 4, 'rgb(199,
47, 32)'), (31, 42, 'adresse_client', 3, 'rgb(167, 248, 246)'), (16, 31,
'email_client', 2, 'rgb(62, 11, 159)'), (2, 14, 'nom_client', 1, 'rgb(45,
50, 168)')]}), ('à omar karray ***@***.*** rue tunis
telephone:(213)36521478', {'entities': [(50, 63, 'telephone', 7, 'rgb(199,
47, 32)'), (30, 39, 'adresse_client', 6, 'rgb(167, 248, 246)'), (14, 29,
'email_client', 5, 'rgb(62, 11, 159)'), (2, 13, 'nom_client', 4, 'rgb(45,
50, 168)')]}), ('nombre INV0001', {'entities': [(7, 14, 'num_facture', 0,
'rgb(63, 10, 198)')]}), ('nombre 23f548e4', {'entities': [(7, 15,
'num_facture', 0, 'rgb(63, 10, 198)')]}), ('numero INV45454', {'entities':
[(7, 15, 'num_facture', 0, 'rgb(63, 10, 198)')]}), ('nombre: e5445555',
{'entities': [(7, 16, 'num_facture', 0, 'rgb(63, 10, 198)')]}), ('numero:
g5556e554f', {'entities': [(7, 18, 'num_facture', 1, 'rgb(63, 10,
198)')]}), ('numéro facture INV652', {'entities': [(15, 21, 'num_facture',
0, 'rgb(63, 10, 198)')]}), ('numéro facture: 5454g654e454', {'entities':
[(15, 28, 'num_facture', 0, 'rgb(63, 10, 198)')]}), ('description: thé vert
biologique miel\xa0', {'entities': [(13, 38, 'description', 0, 'rgb(138,
58, 123)')]}), ('quantité 5 3\xa0', {'entities': [(9, 13, 'quantite', 0,
'rgb(23, 210, 137)')]}), ('unité kg 1\xa0', {'entities': [(6, 11, 'unité',
0, 'rgb(161, 156, 249)')]}), ('unité: 1 kg 1.5 litre 3.25 mètre\xa0\xa0',
{'entities': [(6, 34, 'unité', 0, 'rgb(161, 156, 249)')]}), ('unité 1 kg
1.5 kg 3.25 g\xa0', {'entities': [(6, 25, 'unité', 0, 'rgb(161, 156,
249)')]}), ('unité: 1\xa0m3\xa01.5 m2 3.25 m\xa0', {'entities': [(7, 26,
'unité', 0, 'rgb(161, 156, 249)')]})]
Le lun. 13 sept. 2021 à 18:52, ManivannanMurugavel <
***@***.***> a écrit :
> ***@***.**** commented on this gist.
> ------------------------------
>
> Hi,
> Can you send me the TRAIN_DATA?
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://gist.github.com/7e61eecbfb37bbc073ae58e7c07f2a6a#gistcomment-3891607>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AVA5FFSN56IAKJ3V2BQ7ID3UBYT57ANCNFSM4HQJPMMA>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
>
Hi,
Use this python script to convert correct TRAIN_DATA. The input of the this script is json file which is you downloaded from annotation tool.
You are using the format is:
('nom du client sami adresse client 75000 paris france\xa0', {'entities': [(34, 52, 'adresse_client', 1, 'rgb(167, 248, 246)'), (14, 18, 'nom_client', 0, 'rgb(45, 50, 168)')]})
Original format is:
('nom du client sami adresse client 75000 paris france\xa0', {'entities': [(34, 52, 'adresse_client', 1), (14, 18, 'nom_client', 0)]})
hi
got it ! thank you so much
Le jeu. 16 sept. 2021 à 10:16, ManivannanMurugavel ***@***.***>
a écrit :
… ***@***.**** commented on this gist.
------------------------------
Hi,
Use this python
<https://github.com/ManivannanMurugavel/spacy-ner-annotator/blob/master/convert_spacy_train_data.py>
script to convert correct TRAIN_DATA. The input of the this script is json
file which is you downloaded from annotation tool.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<https://gist.github.com/7e61eecbfb37bbc073ae58e7c07f2a6a#gistcomment-3894989>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AVA5FFURJKJXZJTA34IRM3TUCGRVZANCNFSM4HQJPMMA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment