SETIADEEPANSHU · February 9, 2022 09:01
diff --git a/retriever.ipynb b/retriever.ipynb
 {
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/gist/SETIADEEPANSHU/c168ca1a86cafa9592d8f05abcb518f7/retriever.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IQbPtRyMpvE7"
      },
      "outputs": [],
      "source": [
        "from haystack.document_store.elasticsearch import ElasticsearchDocumentStore"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0qUqpatFpvFB",
        "outputId": "eed6d6cd-71e6-48f2-aeef-94cb3972c476"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "04/10/2021 15:16:48 - INFO - elasticsearch -   HEAD http://localhost:9200/aurelius [status:200 request:0.005s]\n",
            "04/10/2021 15:16:48 - INFO - elasticsearch -   GET http://localhost:9200/aurelius [status:200 request:0.001s]\n",
            "04/10/2021 15:16:48 - INFO - elasticsearch -   PUT http://localhost:9200/aurelius/_mapping [status:200 request:0.007s]\n",
            "04/10/2021 15:16:48 - INFO - elasticsearch -   HEAD http://localhost:9200/label [status:200 request:0.001s]\n"
          ]
        }
      ],
      "source": [
        "doc_store = ElasticsearchDocumentStore(\n",
        "    host='localhost',\n",
        "    username='', password='',\n",
        "    index='aurelius'\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "e-EvVV7fpvFD"
      },
      "outputs": [],
      "source": [
        "from haystack.retriever.dense import DensePassageRetriever"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WyfUA0UApvFE"
      },
      "outputs": [],
      "source": [
        "retriever = DensePassageRetriever(\n",
        "    document_store=doc_store,\n",
        "    query_embedding_model='facebook/dpr-question_encoder-single-nq-base',\n",
        "    passage_embedding_model='facebook/dpr-ctx_encoder-single-nq-base',\n",
        "    use_gpu=False,\n",
        "    embed_title=True\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CE4m_Y2mpvFF",
        "outputId": "811149b3-f8e2-485f-eaf2-3adae25a7b0c"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "04/10/2021 15:30:08 - INFO - elasticsearch -   POST http://localhost:9200/aurelius/_count [status:200 request:0.005s]\n",
            "04/10/2021 15:30:08 - INFO - haystack.document_store.elasticsearch -   Updating embeddings for 508 docs ...\n",
            "04/10/2021 15:30:08 - INFO - elasticsearch -   POST http://localhost:9200/aurelius/_search?scroll=1d&size=10000 [status:200 request:0.038s]\n",
            "04/10/2021 15:30:08 - INFO - elasticsearch -   POST http://localhost:9200/_search/scroll [status:200 request:0.010s]\n",
            "04/10/2021 15:30:08 - INFO - elasticsearch -   DELETE http://localhost:9200/_search/scroll [status:200 request:0.003s]\n",
            "Creating Embeddings: 100%|███████████████████████████████████████████████████████| 32/32 [00:02<00:00, 12.66 Batches/s]\n",
            "04/10/2021 15:30:12 - INFO - elasticsearch -   POST http://localhost:9200/_bulk?refresh=wait_for [status:200 request:1.189s]\n",
            "04/10/2021 15:30:13 - INFO - elasticsearch -   POST http://localhost:9200/_bulk?refresh=wait_for [status:200 request:1.032s]\n"
          ]
        }
      ],
      "source": [
        "doc_store.update_embeddings(retriever=retriever)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "outputs_hidden": true
        },
        "id": "m6INLqXDpvFF",
        "outputId": "0894bac0-1b7d-4e03-9e16-cb03e70dcfd5"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Creating Embeddings: 100%|█████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 62.79 Batches/s]\n",
            "04/10/2021 15:33:20 - INFO - elasticsearch -   POST http://localhost:9200/aurelius/_search [status:200 request:0.012s]\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "[{'text': 'No man can rob us of our free will.  Epictetus also said, A man must discover an art (or rules) with respect to giving his assent; and in respect to his movements he must be careful that they be made with regard to circumstances, that they be consistent with social interests, that they have regard to the value of the object; and as to sensual desire, he should altogether keep away from it; and as to avoidance (aversion) he should not show it with respect to any of the things which are not in our power.', 'id': 'cbb8929a-6914-4874-a525-111885176e9f', 'score': 71.9670000000001, 'probability': 0.6725343446791505, 'question': None, 'meta': {'source': 'meditations'}, 'embedding': None},\n",
              " {'text': 'In the application of thy principles thou must be like the pancratiast, not like the gladiator; for the gladiator lets fall the sword which he uses and is killed; but the other always has his hand, and needs to do nothing else than use it.', 'id': 'fb64da3e-c1e3-4da7-ab94-d1daf0cfc480', 'score': 71.72199999999998, 'probability': 0.6719945486176593, 'question': None, 'meta': {'source': 'meditations'}, 'embedding': None},\n",
              " {'text': 'Remember that all is opinion. For what was said by the Cynic Monimus is manifest: and manifest too is the use of what was said, if a man receives what may be got out of it as far as it is true.', 'id': 'f52fda27-c36a-45c3-9cb6-cf4a5fd3b218', 'score': 70.28770000000009, 'probability': 0.6688253311652296, 'question': None, 'meta': {'source': 'meditations'}, 'embedding': None},\n",
              " {'text': 'From Apollonius I learned freedom of will and undeviating steadiness of purpose; and to look to nothing else, not even for a moment, except to reason; and to be always the same, in sharp pains, on the occasion of the loss of a child, and in long illness; and to see clearly in a living example that the same man can be both most resolute and yielding, and not peevish in giving his instruction; and to have had before my eyes a man who clearly considered his experience and his skill in expounding philosophical principles as the smallest of his merits; and from him I learned how to receive from friends what are esteemed favours, without being either humbled by them or letting them pass unnoticed.', 'id': 'b5cfcf8a-ca22-4d02-be81-b8fe0fa1172f', 'score': 70.15329999999994, 'probability': 0.6685275703255888, 'question': None, 'meta': {'source': 'meditations'}, 'embedding': None},\n",
              " {'text': 'The ruling principle is that which rouses and turns itself, and while it makes itself such as it is and such as it wills to be, it also makes everything which happens appear to itself to be such as it wills.', 'id': '4ce9795e-977d-44a0-a363-1fbd28e2795a', 'score': 68.97330000000011, 'probability': 0.6659075286406019, 'question': None, 'meta': {'source': 'meditations'}, 'embedding': None},\n",
              " {'text': 'How cruel it is not to allow men to strive after the things which appear to them to be suitable to their nature and profitable! And yet in a manner thou dost not allow them to do this, when thou art vexed because they do wrong. For they are certainly moved towards things because they suppose them to be suitable to their nature and profitable to them.- But it is not so.- Teach them then, and show them without being angry.', 'id': 'c9d303b4-38f3-40bd-83a4-56ecab626bfa', 'score': 68.4994999999999, 'probability': 0.6648526162888172, 'question': None, 'meta': {'source': 'meditations'}, 'embedding': None},\n",
              " {'text': \"Remember these nine rules, as if thou hadst received them as a gift from the Muses, and begin at last to be a man while thou livest. But thou must equally avoid flattering men and being veied at them, for both are unsocial and lead to harm. And let this truth be present to thee in the excitement of anger, that to be moved by passion is not manly, but that mildness and gentleness, as they are more agreeable to human nature, so also are they more manly; and he who possesses these qualities possesses strength, nerves and courage, and not the man who is subject to fits of passion and discontent. For in the same degree in which a man's mind is nearer to freedom from all passion, in the same degree also is it nearer to strength: and as the sense of pain is a characteristic of weakness, so also is anger. For he who yields to pain and he who yields to anger, both are wounded and both submit.\", 'id': '8dc4159d-6b76-48f9-85a7-a123224fa95d', 'score': 68.47119999999995, 'probability': 0.6647895542641697, 'question': None, 'meta': {'source': 'meditations'}, 'embedding': None},\n",
              " {'text': \"The prime principle then in man's constitution is the social. And the second is not to yield to the persuasions of the body, for it is the peculiar office of the rational and intelligent motion to circumscribe itself, and never to be overpowered either by the motion of the senses or of the appetites, for both are animal; but the intelligent motion claims superiority and does not permit itself to be overpowered by the others. And with good reason, for it is formed by nature to use all of them. The third thing in the rational constitution is freedom from error and from deception. Let then the ruling principle holding fast to these things go straight on, and it has what is its own.\", 'id': 'a5cb4af5-2574-4340-9730-ac94745805c9', 'score': 68.40820000000008, 'probability': 0.6646491477184014, 'question': None, 'meta': {'source': 'meditations'}, 'embedding': None},\n",
              " {'text': 'After tragedy the old comedy was introduced, which had a magisterial freedom of speech, and by its very plainness of speaking was useful in reminding men to beware of insolence; and for this purpose too Diogenes used to take from these writers.', 'id': '1617dc5a-c13a-4e4e-b20b-9bbfbd498d9f', 'score': 68.08290000000011, 'probability': 0.6639236964928045, 'question': None, 'meta': {'source': 'meditations'}, 'embedding': None},\n",
              " {'text': 'To him who is penetrated by true principles even the briefest precept is sufficient, and any common precept, to remind him that he should be free from grief and fear. For example-', 'id': '14064a78-a866-4eef-8393-13ba4dd4868a', 'score': 67.95980000000009, 'probability': 0.6636489692645229, 'question': None, 'meta': {'source': 'meditations'}, 'embedding': None}]"
            ]
          },
          "execution_count": 8,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "retriever.retrieve('Who taught the freedom of will?')"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "ML",
      "language": "python",
      "name": "ml"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.5"
    },
    "colab": {
      "name": "retriever.ipynb",
      "provenance": [],
      "include_colab_link": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
 }
	{
	"cells": [
	{
	"cell_type": "markdown",
	"metadata": {
	"id": "view-in-github",
	"colab_type": "text"
	},
	"source": [
	"<a href=\"https://colab.research.google.com/gist/SETIADEEPANSHU/c168ca1a86cafa9592d8f05abcb518f7/retriever.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
	]
	},
	{
	"cell_type": "code",
	"execution_count": null,
	"metadata": {
	"id": "IQbPtRyMpvE7"
	},
	"outputs": [],
	"source": [
	"from haystack.document_store.elasticsearch import ElasticsearchDocumentStore"
	]
	},
	{
	"cell_type": "code",
	"execution_count": null,
	"metadata": {
	"id": "0qUqpatFpvFB",
	"outputId": "eed6d6cd-71e6-48f2-aeef-94cb3972c476"
	},
	"outputs": [
	{
	"name": "stderr",
	"output_type": "stream",
	"text": [
	"04/10/2021 15:16:48 - INFO - elasticsearch - HEAD http://localhost:9200/aurelius [status:200 request:0.005s]\n",
	"04/10/2021 15:16:48 - INFO - elasticsearch - GET http://localhost:9200/aurelius [status:200 request:0.001s]\n",
	"04/10/2021 15:16:48 - INFO - elasticsearch - PUT http://localhost:9200/aurelius/_mapping [status:200 request:0.007s]\n",
	"04/10/2021 15:16:48 - INFO - elasticsearch - HEAD http://localhost:9200/label [status:200 request:0.001s]\n"
	]
	}
	],
	"source": [
	"doc_store = ElasticsearchDocumentStore(\n",
	" host='localhost',\n",
	" username='', password='',\n",
	" index='aurelius'\n",
	")"
	]
	},
	{
	"cell_type": "code",
	"execution_count": null,
	"metadata": {
	"id": "e-EvVV7fpvFD"
	},
	"outputs": [],
	"source": [
	"from haystack.retriever.dense import DensePassageRetriever"
	]
	},
	{
	"cell_type": "code",
	"execution_count": null,
	"metadata": {
	"id": "WyfUA0UApvFE"
	},
	"outputs": [],
	"source": [
	"retriever = DensePassageRetriever(\n",
	" document_store=doc_store,\n",
	" query_embedding_model='facebook/dpr-question_encoder-single-nq-base',\n",
	" passage_embedding_model='facebook/dpr-ctx_encoder-single-nq-base',\n",
	" use_gpu=False,\n",
	" embed_title=True\n",
	")"
	]
	},
	{
	"cell_type": "code",
	"execution_count": null,
	"metadata": {
	"id": "CE4m_Y2mpvFF",
	"outputId": "811149b3-f8e2-485f-eaf2-3adae25a7b0c"
	},
	"outputs": [
	{
	"name": "stderr",
	"output_type": "stream",
	"text": [
	"04/10/2021 15:30:08 - INFO - elasticsearch - POST http://localhost:9200/aurelius/_count [status:200 request:0.005s]\n",
	"04/10/2021 15:30:08 - INFO - haystack.document_store.elasticsearch - Updating embeddings for 508 docs ...\n",
	"04/10/2021 15:30:08 - INFO - elasticsearch - POST http://localhost:9200/aurelius/_search?scroll=1d&size=10000 [status:200 request:0.038s]\n",
	"04/10/2021 15:30:08 - INFO - elasticsearch - POST http://localhost:9200/_search/scroll [status:200 request:0.010s]\n",
	"04/10/2021 15:30:08 - INFO - elasticsearch - DELETE http://localhost:9200/_search/scroll [status:200 request:0.003s]\n",
	"Creating Embeddings: 100%\|███████████████████████████████████████████████████████\| 32/32 [00:02<00:00, 12.66 Batches/s]\n",
	"04/10/2021 15:30:12 - INFO - elasticsearch - POST http://localhost:9200/_bulk?refresh=wait_for [status:200 request:1.189s]\n",
	"04/10/2021 15:30:13 - INFO - elasticsearch - POST http://localhost:9200/_bulk?refresh=wait_for [status:200 request:1.032s]\n"
	]
	}
	],
	"source": [
	"doc_store.update_embeddings(retriever=retriever)"
	]
	},
	{
	"cell_type": "code",
	"execution_count": null,
	"metadata": {
	"jupyter": {
	"outputs_hidden": true
	},
	"id": "m6INLqXDpvFF",
	"outputId": "0894bac0-1b7d-4e03-9e16-cb03e70dcfd5"
	},
	"outputs": [
	{
	"name": "stderr",
	"output_type": "stream",
	"text": [
	"Creating Embeddings: 100%\|█████████████████████████████████████████████████████████\| 1/1 [00:00<00:00, 62.79 Batches/s]\n",
	"04/10/2021 15:33:20 - INFO - elasticsearch - POST http://localhost:9200/aurelius/_search [status:200 request:0.012s]\n"
	]
	},
	{
	"data": {
	"text/plain": [
	"[{'text': 'No man can rob us of our free will. Epictetus also said, A man must discover an art (or rules) with respect to giving his assent; and in respect to his movements he must be careful that they be made with regard to circumstances, that they be consistent with social interests, that they have regard to the value of the object; and as to sensual desire, he should altogether keep away from it; and as to avoidance (aversion) he should not show it with respect to any of the things which are not in our power.', 'id': 'cbb8929a-6914-4874-a525-111885176e9f', 'score': 71.9670000000001, 'probability': 0.6725343446791505, 'question': None, 'meta': {'source': 'meditations'}, 'embedding': None},\n",
	" {'text': 'In the application of thy principles thou must be like the pancratiast, not like the gladiator; for the gladiator lets fall the sword which he uses and is killed; but the other always has his hand, and needs to do nothing else than use it.', 'id': 'fb64da3e-c1e3-4da7-ab94-d1daf0cfc480', 'score': 71.72199999999998, 'probability': 0.6719945486176593, 'question': None, 'meta': {'source': 'meditations'}, 'embedding': None},\n",
	" {'text': 'Remember that all is opinion. For what was said by the Cynic Monimus is manifest: and manifest too is the use of what was said, if a man receives what may be got out of it as far as it is true.', 'id': 'f52fda27-c36a-45c3-9cb6-cf4a5fd3b218', 'score': 70.28770000000009, 'probability': 0.6688253311652296, 'question': None, 'meta': {'source': 'meditations'}, 'embedding': None},\n",
	" {'text': 'From Apollonius I learned freedom of will and undeviating steadiness of purpose; and to look to nothing else, not even for a moment, except to reason; and to be always the same, in sharp pains, on the occasion of the loss of a child, and in long illness; and to see clearly in a living example that the same man can be both most resolute and yielding, and not peevish in giving his instruction; and to have had before my eyes a man who clearly considered his experience and his skill in expounding philosophical principles as the smallest of his merits; and from him I learned how to receive from friends what are esteemed favours, without being either humbled by them or letting them pass unnoticed.', 'id': 'b5cfcf8a-ca22-4d02-be81-b8fe0fa1172f', 'score': 70.15329999999994, 'probability': 0.6685275703255888, 'question': None, 'meta': {'source': 'meditations'}, 'embedding': None},\n",
	" {'text': 'The ruling principle is that which rouses and turns itself, and while it makes itself such as it is and such as it wills to be, it also makes everything which happens appear to itself to be such as it wills.', 'id': '4ce9795e-977d-44a0-a363-1fbd28e2795a', 'score': 68.97330000000011, 'probability': 0.6659075286406019, 'question': None, 'meta': {'source': 'meditations'}, 'embedding': None},\n",
	" {'text': 'How cruel it is not to allow men to strive after the things which appear to them to be suitable to their nature and profitable! And yet in a manner thou dost not allow them to do this, when thou art vexed because they do wrong. For they are certainly moved towards things because they suppose them to be suitable to their nature and profitable to them.- But it is not so.- Teach them then, and show them without being angry.', 'id': 'c9d303b4-38f3-40bd-83a4-56ecab626bfa', 'score': 68.4994999999999, 'probability': 0.6648526162888172, 'question': None, 'meta': {'source': 'meditations'}, 'embedding': None},\n",
	" {'text': \"Remember these nine rules, as if thou hadst received them as a gift from the Muses, and begin at last to be a man while thou livest. But thou must equally avoid flattering men and being veied at them, for both are unsocial and lead to harm. And let this truth be present to thee in the excitement of anger, that to be moved by passion is not manly, but that mildness and gentleness, as they are more agreeable to human nature, so also are they more manly; and he who possesses these qualities possesses strength, nerves and courage, and not the man who is subject to fits of passion and discontent. For in the same degree in which a man's mind is nearer to freedom from all passion, in the same degree also is it nearer to strength: and as the sense of pain is a characteristic of weakness, so also is anger. For he who yields to pain and he who yields to anger, both are wounded and both submit.\", 'id': '8dc4159d-6b76-48f9-85a7-a123224fa95d', 'score': 68.47119999999995, 'probability': 0.6647895542641697, 'question': None, 'meta': {'source': 'meditations'}, 'embedding': None},\n",
	" {'text': \"The prime principle then in man's constitution is the social. And the second is not to yield to the persuasions of the body, for it is the peculiar office of the rational and intelligent motion to circumscribe itself, and never to be overpowered either by the motion of the senses or of the appetites, for both are animal; but the intelligent motion claims superiority and does not permit itself to be overpowered by the others. And with good reason, for it is formed by nature to use all of them. The third thing in the rational constitution is freedom from error and from deception. Let then the ruling principle holding fast to these things go straight on, and it has what is its own.\", 'id': 'a5cb4af5-2574-4340-9730-ac94745805c9', 'score': 68.40820000000008, 'probability': 0.6646491477184014, 'question': None, 'meta': {'source': 'meditations'}, 'embedding': None},\n",
	" {'text': 'After tragedy the old comedy was introduced, which had a magisterial freedom of speech, and by its very plainness of speaking was useful in reminding men to beware of insolence; and for this purpose too Diogenes used to take from these writers.', 'id': '1617dc5a-c13a-4e4e-b20b-9bbfbd498d9f', 'score': 68.08290000000011, 'probability': 0.6639236964928045, 'question': None, 'meta': {'source': 'meditations'}, 'embedding': None},\n",
	" {'text': 'To him who is penetrated by true principles even the briefest precept is sufficient, and any common precept, to remind him that he should be free from grief and fear. For example-', 'id': '14064a78-a866-4eef-8393-13ba4dd4868a', 'score': 67.95980000000009, 'probability': 0.6636489692645229, 'question': None, 'meta': {'source': 'meditations'}, 'embedding': None}]"
	]
	},
	"execution_count": 8,
	"metadata": {},
	"output_type": "execute_result"
	}
	],
	"source": [
	"retriever.retrieve('Who taught the freedom of will?')"
	]
	}
	],
	"metadata": {
	"kernelspec": {
	"display_name": "ML",
	"language": "python",
	"name": "ml"
	},
	"language_info": {
	"codemirror_mode": {
	"name": "ipython",
	"version": 3
	},
	"file_extension": ".py",
	"mimetype": "text/x-python",
	"name": "python",
	"nbconvert_exporter": "python",
	"pygments_lexer": "ipython3",
	"version": "3.8.5"
	},
	"colab": {
	"name": "retriever.ipynb",
	"provenance": [],
	"include_colab_link": true
	}
	},
	"nbformat": 4,
	"nbformat_minor": 0
	}
No results found