-
-
Save patrickloeber/4e09660692390c2b35417aa4f792c988 to your computer and use it in GitHub Desktop.
Structured Output Question.ipynb
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "nbformat": 4, | |
| "nbformat_minor": 0, | |
| "metadata": { | |
| "colab": { | |
| "provenance": [], | |
| "authorship_tag": "ABX9TyNDd6b/wKZoFeKkUw2BLkQt", | |
| "include_colab_link": true | |
| }, | |
| "kernelspec": { | |
| "name": "python3", | |
| "display_name": "Python 3" | |
| }, | |
| "language_info": { | |
| "name": "python" | |
| } | |
| }, | |
| "cells": [ | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "view-in-github", | |
| "colab_type": "text" | |
| }, | |
| "source": [ | |
| "<a href=\"https://colab.research.google.com/gist/j-jayes/b03403b55006039abe4cc65338ad6f48/structured-output-question.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "source": [ | |
| "# Purpose\n", | |
| "\n", | |
| "Patrick Löber posted on LinkedIn last week about using Pydantic data models to get structured information using Google's gemini flash 2.0. This method performs far better than providing a JSON schema in terms of reliable output.\n", | |
| "\n", | |
| "A screenshot of the example he gave is below:" | |
| ], | |
| "metadata": { | |
| "id": "6d2Ry2hIlXXr" | |
| } | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "source": [ | |
| "\n", | |
| "\n" | |
| ], | |
| "metadata": { | |
| "id": "8iM4ZrCTgWqf" | |
| } | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "source": [ | |
| "In his [accompanying article](https://patloeber.com/gemini-multimodal/), he linked to [a blog post by Phil Schmid](https://www.philschmid.de/gemini-pdf-to-data) on how to extract data from PDF documents with gemini 2.0.\n", | |
| "\n", | |
| "I want to follow this use case, and use the structred output format to extract information from PDFs of some length e.g. the [South African treasury's budget review](https://www.treasury.gov.za/documents/national%20budget/2024/review/FullBR.pdf).\n", | |
| "\n", | |
| "I have attempted the task of extracting information from these kinds of budget documents before, using a [hacky RAG system in mid-2023](https://interludeone.com/posts/2023-07-05-langchain-demo/langchain-demo)\n", | |
| "\n", | |
| "Gemini flash 2.0 has such a large context window that creating a RAG system is unnecessary. However, it would be ideal if I can include keep metadata about where in the document the extracted information was sourced, in order to perform some validation on the extracted output.\n", | |
| "\n", | |
| "Ideally I would be able to pass more than one PDF to the LLM and then keep track of which one was used to source the answer, as well as where in the document the answer came from." | |
| ], | |
| "metadata": { | |
| "id": "a7xpUMfShgKD" | |
| } | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "source": [ | |
| "NotebookLM, that uses Gemini under the hood, has a way to display which part of the PDF the information in response to a click on the source button. I demonstrate this in the screenshot below, after uploading the budget review and asking what CPI inflation was in 2023.\n", | |
| "\n", | |
| "Clearly here the PDF is presented just as text, rather than a table in e.g. HTML or Markdown format. It could be that NotebookLM is doing RAG in the background on the raw text chunks, or it could be doing a more complex tokenization process that keeps table structure, but when clicking on the source button, just showing the user the raw text rather than the table representation.\n" | |
| ], | |
| "metadata": { | |
| "id": "9XcacUhcmGEe" | |
| } | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "source": [ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment