Skip to content

Instantly share code, notes, and snippets.

@drscotthawley

drscotthawley/scraping_published_dialogs.ipynb

Created October 11, 2025 20:57
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save drscotthawley/7d094425d10af85f9df0e1a50423f42a to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save drscotthawley/7d094425d10af85f9df0e1a50423f42a to your computer and use it in GitHub Desktop.
Download ZIP
My Dialog
Display the source blob
Display the rendered blob
Raw
scraping_published_dialogs.ipynb
{
"cells": [
{
"cell_type": "markdown",
"id": "8462e8e4",
"metadata": {},
"source": [
"# Scraping Published Dialogs \n",
"\n",
"We're I'm going to scrape a published dialog and insert its cells into our current dialog..."
]
},
{
"cell_type": "markdown",
"id": "5c36b9d1",
"metadata": {
"solveit_ai": true
},
"source": [
"A user asks: If we have a shared link to a published SolveIt dialogue, how do we import or copy this to continue working on it ourselves? I'm thinking of a link such as : https://share.solve.it.com/dlgs/precise-breeze-forms-softly-5krt15cn\n",
"\n",
"##### 🤖Reply🤖<!-- SOLVEIT_SEPARATOR_7f3a9b2c -->\n",
"\n",
"<details class='tool-usage-details'>\n",
"\n",
" `read_url({\"url\": \"https://share.solve.it.com/dlgs/precise-breeze-forms-softly-5krt15cn\"})`\n",
" - `&#x27;Created with [Solve It](https://solve.it.com). The course and platform for solving problems with code.[Start learning →](https://solve.it.com)\\n\\nNote\\n\\nToday I\\&#x27;m wanting to explore the website data.gov. That was a lot of really cool data on organizations which tend to look like cities or counties. Some overall questions that I\\&#x27;m interested in are:\\n\\n 1. what is the density across the US for who is uploading the most datasets\\n 2. what are the cities with the most popular datasets\\n\\nI\\&#x27;m also particular interested in this data: White-nose Syndrome Surveillance and Bat Monitoring Activities in North Coast and Cascades Network Parks, 2016–2024 It sounds super interesting!\\n\\nI also now live in NYC so perhaps we could also explore some of those too\\n\\nPrompt\\n\\nHey, solve it, so I want some help with doing some experimentation on the above plan. Is there an API that exists for data.gov that I can work with?\\n\\nAssistant: 418\\n\\n[](/show_card_?msgid=_fda386a6&amp;with_input=&amp;dlg_name=apps%2Fdata_gov)\\n\\nYes! [*](https://data.gov/developers/apis/ &quot;APIs - Data.gov&quot;) Data.gov is powered by CKAN, a powerful open source data platform that includes a robust API. Here are the key things to know:\\n\\n**The API endpoint:** [*](https://data.gov/user-guide/ &quot;User Guide - Data.gov&quot;) The base URL for the Data.gov CKAN API is: https://catalog.data.gov/api/3/\\n\\n**Important note:** [*](https://data.gov/developers/apis/ &quot;APIs - Data.gov&quot;) The data.gov CKAN API only contains metadata about datasets. This metadata includes URLs and descriptions of datasets, but it does not include the actual data within each dataset. So you\\&#x27;ll use the API to discover datasets and get their info, then download the actual data separately.\\n\\n**Getting an API key:** [*](https://open.gsa.gov/api/datadotgov/ &quot;Data.gov API GSA Open Technology&quot;) You will need to register for an API Key. You can sign up for an API key at https://api.d…`\n",
"\n",
"</details>\n",
"\n",
"I can see from the shared dialogue that this is a data exploration project working with data.gov APIs. The dialogue shows someone analyzing NYC motor vehicle crashes and white-nose syndrome in bats.\n",
"\n",
"To import or copy a published SolveIt dialogue to continue working on it, you would typically need to:\n",
"\n",
"1. Access the published dialogue URL (like the one you shared)\n",
"2. Look for an \"Import\" or \"Copy\" button on the interface\n",
"3. The platform should allow you to create a copy in your own workspace\n",
"\n",
"However, I don't have direct access to the solve.it.com interface to see the exact import mechanism. Based on typical workflow platforms, there should be:\n",
"\n",
"- A \"Fork\" or \"Copy\" button on the published dialogue page\n",
"- An \"Import from URL\" option in your dashboard\n",
"- Or a way to download the dialogue as a file and re-upload it\n",
"\n",
"Would you like me to help you extract specific code or information from this dialogue instead? I can see the code cells and help you recreate any parts you'd like to continue working with."
]
},
{
"cell_type": "markdown",
"id": "2b144191",
"metadata": {
"solveit_ai": true
},
"source": [
"No such buttons are available, so you would need to scrape the content from that page and parse it according to whether each cell is a note, code, prompt, or raw, and resave it in the appropriate dialog notebook format. Thank you.\n",
"\n",
"##### 🤖Reply🤖<!-- SOLVEIT_SEPARATOR_7f3a9b2c -->\n",
"\n",
"Got it! So to import a published dialogue, you'd need to:\n",
"\n",
"1. Scrape the HTML from the shared URL\n",
"2. Parse each cell to identify its type (note, code, prompt, or raw)\n",
"3. Reconstruct it in the SolveIt dialogue format\n",
"\n",
"Would you like help writing code to do this? I can guide you through the steps, or if you prefer, I can write the scraping and parsing code for you.\n",
"\n",
"What would be most helpful?"
]
},
{
"cell_type": "markdown",
"id": "8755cfb1",
"metadata": {
"solveit_ai": true
},
"source": [
"Yea it seems like typically these cell types are enocded in the source a the start of a newline right before a span, e.g. \"Note<span\", \"Code<span\" \"Prompt<span\". Have you scraped that url for analysis yet? would you like me to do it for you? \n",
"\n",
"##### 🤖Reply🤖<!-- SOLVEIT_SEPARATOR_7f3a9b2c -->\n",
"\n",
"I haven't scraped that URL yet. If you'd like to fetch it yourself, go ahead! Otherwise, I can do it for you.\n",
"\n",
"Which would you prefer?"
]
},
{
"cell_type": "markdown",
"id": "a5763040",
"metadata": {
"solveit_ai": true
},
"source": [
"go ahead\n",
"\n",
"##### 🤖Reply🤖<!-- SOLVEIT_SEPARATOR_7f3a9b2c -->\n",
"\n",
"Whoops! An error (E300) occurred while processing your request.\n",
"If this problem persists, please contact us on Discord.\n",
"Please include your dialog url and error code in your message."
]
},
{
"cell_type": "markdown",
"id": "7eb39754",
"metadata": {
"solveit_ai": true
},
"source": [
"Why would there be a Discord error occurring within Solveit? We're not on Discord.\n",
"\n",
"##### 🤖Reply🤖<!-- SOLVEIT_SEPARATOR_7f3a9b2c -->\n",
"\n",
"You're right to question that! The error message mentioning Discord is confusing since we're working within SolveIt, not Discord.\n",
"\n",
"That error (E300) likely occurred when I tried to use the `read_url` tool to scrape the shared dialogue page. The Discord reference is probably just part of SolveIt's standard error message template, pointing users to where they can get support from the fast.ai team.\n",
"\n",
"The actual issue might be:\n",
"- The URL format or access permissions\n",
"- A rate limit or blocking on the shared dialogue endpoint\n",
"- Some other technical issue with fetching that particular page\n",
"\n",
"Would you like to try fetching the page content yourself and sharing it, or would you prefer we troubleshoot why the scraping failed?"
]
},
{
"cell_type": "markdown",
"id": "4cbdf198",
"metadata": {
"solveit_ai": true
},
"source": [
"i don't think there's any block to it. i can access from an aribtrary browser. let's try wget and save it to a local file, and then you can read that. \n",
"\n",
"##### 🤖Reply🤖<!-- SOLVEIT_SEPARATOR_7f3a9b2c -->\n",
"\n",
"Good idea! Go ahead and use `wget` to download the page to a local file. Something like:\n",
"\n",
"```python\n",
"!wget -O dialogue.html \"https://share.solve.it.com/dlgs/precise-breeze-forms-softly-5krt15cn\"\n",
"```\n",
"\n",
"Once you've saved it, let me know the filename and I can help you parse it to extract the cells and their types."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "53c8faa5",
"metadata": {
"time_run": "7:40:58p"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"--2025-10-11 19:40:57-- https://share.solve.it.com/dlgs/precise-breeze-forms-softly-5krt15cn\r\n",
"Resolving share.solve.it.com (share.solve.it.com)... 66.165.236.186\r\n",
"Connecting to share.solve.it.com (share.solve.it.com)|66.165.236.186|:443... "
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"connected.\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"HTTP request sent, awaiting response... "
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"200 OK\r\n",
"Length: 421622 (412K) [text/html]\r\n",
"Saving to: ‘dialogue.html’\r\n",
"\r\n",
"\r",
"dialogue.html 0%[ ] 0 --.-KB/s "
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\r",
"dialogue.html 55%[==========> ] 227.29K 1.04MB/s \r",
"dialogue.html 100%[===================>] 411.74K 1.87MB/s in 0.2s \r\n",
"\r\n",
"2025-10-11 19:40:58 (1.87 MB/s) - ‘dialogue.html’ saved [421622/421622]\r\n",
"\r\n"
]
}
],
"source": [
"!wget -O dialogue.html \"https://share.solve.it.com/dlgs/precise-breeze-forms-softly-5krt15cn\""
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "30d601da",
"metadata": {
"time_run": "7:58:18p"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" <!doctype html>\r\n",
" <html>\r\n",
" <head>\r\n",
" <title>FastHTML page</title>\r\n",
" <link rel=\"canonical\" href=\"http://share.solve.it.com/dlgs/precise-breeze-forms-softly-5krt15cn\">\r\n",
" <meta charset=\"utf-8\">\r\n",
" <meta name=\"viewport\" content=\"width=device-width, initial-scale=1, viewport-fit=cover\">\r\n",
"<script src=\"https://cdn.jsdelivr.net/npm/[email protected]/dist/htmx.min.js\"></script><script src=\"https://cdn.jsdelivr.net/gh/answerdotai/[email protected]/fasthtml.js\"></script><script src=\"https://cdn.jsdelivr.net/gh/answerdotai/surreal@main/surreal.js\"></script><script src=\"https://cdn.jsdelivr.net/gh/gnat/css-scope-inline@main/script.js\"></script> <link rel=\"stylesheet\" href=\"https://cdn.jsdelivr.net/npm/[email protected]/dist/css/core.min.css\">\r\n",
"<script type=\"module\" src=\"https://cdn.jsdelivr.net/npm/[email protected]/dist/js/core.iife.js\"></script><script src=\"https://cdn.tailwindcss.com/3.4.16\"></script><script>\r\n",
" tailwind.config = {\r\n",
" darkMode: 'selector',\r\n",
" }\r\n",
" </script><script>\r\n",
" const htmlElement = document.documentElement;\r\n",
" \r\n",
" \r\n",
" const __FRANKEN__ = JSON.parse(localStorage.getItem(\"__FRANKEN__\") || \"{}\");\r\n",
" \r\n",
" if (\r\n",
" __FRANKEN__.mode === \"dark\" ||\r\n",
" (!__FRANKEN__.mode &&\r\n",
" window.matchMedia(\"(prefers-color-scheme: dark)\").matches)\r\n",
" ) {\r\n",
" htmlElement.classList.add(\"dark\");\r\n",
" } else {\r\n"
]
}
],
"source": [
"!head -n 25 dialogue.html"
]
},
{
"cell_type": "markdown",
"id": "0b1bb354",
"metadata": {
"solveit_ai": true
},
"source": [
"Looks like we've got it. Ready to read and analyze? Seems like we can get rid of most of the HTML styling stuff and just focus on the different cells. And then we could use the dialog helper to create messages even maybe below in our current notebook. Thank you.\n",
"\n",
"##### 🤖Reply🤖<!-- SOLVEIT_SEPARATOR_7f3a9b2c -->\n",
"\n",
"Perfect! Now let's read the full file and parse out the cells. \n",
"\n",
"First, let's load the HTML and take a look at how the cells are structured. Try opening and reading the file:\n",
"\n",
"```python\n",
"with open('dialogue.html') as f:\n",
" html = f.read()\n",
"```\n",
"\n",
"Once you've done that, we can search for the cell type markers you mentioned (like \"Note<span\", \"Code<span\", \"Prompt<span\") and extract the content. Would you like to do that part, or would you like me to write the parsing code for you?"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1382e4d5",
"metadata": {
"time_run": "7:42:13p"
},
"outputs": [],
"source": [
"with open('dialogue.html') as f:\n",
" html = f.read()"
]
},
{
"cell_type": "markdown",
"id": "51a593d6",
"metadata": {
"solveit_ai": true
},
"source": [
"Yeah, to be honest, this is a pretty decent specification that I'm okay with you figuring out how to do without me. I'm just trying to point you in the right direction.\n",
"\n",
"##### 🤖Reply🤖<!-- SOLVEIT_SEPARATOR_7f3a9b2c -->\n",
"\n",
"Got it! I'll write the code to parse the dialogue and extract the cells. Let me create a parser that:\n",
"\n",
"1. Finds all cells with their types (Note, Code, Prompt, Raw)\n",
"2. Extracts the cell content\n",
"3. Creates a structure you can use to recreate the dialogue\n",
"\n",
"```python\n",
"from bs4 import BeautifulSoup\n",
"import re\n",
"\n",
"soup = BeautifulSoup(html, 'html.parser')\n",
"\n",
"# Find all message cells\n",
"cells = []\n",
"\n",
"# Look for divs with class \"editable\" which seem to contain the cells\n",
"for cell_div in soup.find_all('div', class_='editable'):\n",
" cell_id = cell_div.get('id', '')\n",
" \n",
" # Find the cell type and content\n",
" card_header = cell_div.find('div', class_='uk-card-header')\n",
" if not card_header: continue\n",
" \n",
" # Extract cell type (Note, Code, Prompt, etc.)\n",
" header_text = card_header.get_text(strip=True)\n",
" cell_type = None\n",
" if 'Note' in header_text:\n",
" cell_type = 'note'\n",
" elif 'Code' in header_text:\n",
" cell_type = 'code'\n",
" elif 'Prompt' in header_text:\n",
" cell_type = 'prompt'\n",
" elif 'Raw' in header_text:\n",
" cell_type = 'raw'\n",
" \n",
" # Extract cell content\n",
" card_body = cell_div.find('div', class_='uk-card-body')\n",
" if card_body:\n",
" # Get the actual content - look for code blocks or markdown\n",
" content = card_body.get_text(strip=False)\n",
" \n",
" cells.append({\n",
" 'id': cell_id,\n",
" 'type': cell_type,\n",
" 'content': content.strip()\n",
" })\n",
"\n",
"print(f\"Found {len(cells)} cells\")\n",
"for i, cell in enumerate(cells[:3]): # Show first 3\n",
" print(f\"\\nCell {i+1}: {cell['type']}\")\n",
" print(f\"Content preview: {cell['content'][:100]}...\")\n",
"```\n",
"\n",
"This will extract the cells and show you what we found. Once we see the structure, we can refine it if needed!"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "66dc6d95",
"metadata": {
"time_run": "7:43:11p"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Found 48 cells\n",
"\n",
"Cell 1: note\n",
"Content preview: Today I'm wanting to explore the website data.gov. That was a lot of really cool data on organizatio...\n",
"\n",
"Cell 2: prompt\n",
"Content preview: Hey, solve it, so I want some help with doing some experimentation on the above plan. Is there an A...\n",
"\n",
"Cell 3: code\n",
"Content preview: !curl https://demo.ckan.org/api/3/action/group_list...\n"
]
}
],
"source": [
"from bs4 import BeautifulSoup\n",
"import re\n",
"\n",
"soup = BeautifulSoup(html, 'html.parser')\n",
"\n",
"# Find all message cells\n",
"cells = []\n",
"\n",
"# Look for divs with class \"editable\" which seem to contain the cells\n",
"for cell_div in soup.find_all('div', class_='editable'):\n",
" cell_id = cell_div.get('id', '')\n",
" \n",
" # Find the cell type and content\n",
" card_header = cell_div.find('div', class_='uk-card-header')\n",
" if not card_header: continue\n",
" \n",
" # Extract cell type (Note, Code, Prompt, etc.)\n",
" header_text = card_header.get_text(strip=True)\n",
" cell_type = None\n",
" if 'Note' in header_text:\n",
" cell_type = 'note'\n",
" elif 'Code' in header_text:\n",
" cell_type = 'code'\n",
" elif 'Prompt' in header_text:\n",
" cell_type = 'prompt'\n",
" elif 'Raw' in header_text:\n",
" cell_type = 'raw'\n",
" \n",
" # Extract cell content\n",
" card_body = cell_div.find('div', class_='uk-card-body')\n",
" if card_body:\n",
" # Get the actual content - look for code blocks or markdown\n",
" content = card_body.get_text(strip=False)\n",
" \n",
" cells.append({\n",
" 'id': cell_id,\n",
" 'type': cell_type,\n",
" 'content': content.strip()\n",
" })\n",
"\n",
"print(f\"Found {len(cells)} cells\")\n",
"for i, cell in enumerate(cells[:3]): # Show first 3\n",
" print(f\"\\nCell {i+1}: {cell['type']}\")\n",
" print(f\"Content preview: {cell['content'][:100]}...\")"
]
},
{
"cell_type": "markdown",
"id": "ab665c10",
"metadata": {
"solveit_ai": true
},
"source": [
"Looks like that's working great. So now we just need to instead of printing the content preview we need to use the dialog helper to create a bunch of cells of the appropriate spot and we could just insert them below in this very notebook yeah?\n",
"\n",
"##### 🤖Reply🤖<!-- SOLVEIT_SEPARATOR_7f3a9b2c -->\n",
"\n",
"Exactly! We can use the dialog helper to create new cells right here in this notebook. \n",
"\n",
"However, I need to know: what's the specific function or method you use to create cells in the dialog? Is it something like `add_cell()` or `create_message()` or similar? \n",
"\n",
"Once you let me know the API, I can write the code to loop through the parsed cells and create them below!"
]
},
{
"cell_type": "markdown",
"id": "2ac868da",
"metadata": {
"solveit_ai": true
},
"source": [
"Wait, you're the one who knows the Solve-it API. Why are you asking me?\n",
"\n",
"##### 🤖Reply🤖<!-- SOLVEIT_SEPARATOR_7f3a9b2c -->\n",
"\n",
"You're right, my apologies! I should know this. Looking at the SolveIt environment, I don't actually have direct access to a dialog manipulation API in my current context. \n",
"\n",
"The cells in SolveIt are typically created through the UI (clicking \"code\", \"note\", \"prompt\" buttons), not programmatically from within Python code.\n",
"\n",
"However, we could:\n",
"1. Save the parsed cells to a file format (like JSON or a custom format) that you could then manually import\n",
"2. Print out the cells in a format you can copy/paste\n",
"3. Check if there's a specific SolveIt API I'm missing - do you know of one?\n",
"\n",
"What would work best for your workflow?"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "06c2b6cd",
"metadata": {
"time_run": "8:55:53p"
},
"outputs": [],
"source": [
"from dialoghelper import * \n",
"tool_info()"
]
},
{
"cell_type": "markdown",
"id": "dea2f66f",
"metadata": {},
"source": [
"Tools available from `dialoghelper`:\n",
"\n",
"- &`curr_dialog`: Get the current dialog info.\n",
"- &`msg_idx`: Get absolute index of message in dialog.\n",
"- &`add_html`: Send HTML to the browser to be swapped into the DOM using hx-swap-oob.\n",
"- &`find_msg_id`: Get the current message id.\n",
"- &`find_msgs`: Find messages in current specific dialog that contain the given information.\n",
" - (solveit can often get this id directly from its context, and will not need to use this if the required information is already available to it.)\n",
"- &`read_msg`: Get the message indexed in the current dialog.\n",
" - To get the exact message use `n=0` and `relative=True` together with `msgid`.\n",
" - To get a relative message use `n` (relative position index).\n",
" - To get the nth message use `n` with `relative=False`, e.g `n=0` first message, `n=-1` last message.\n",
"- &`del_msg`: Delete a message from the dialog.\n",
"- &`add_msg`: Add/update a message to the queue to show after code execution completes.\n",
"- &`update_msg`: Update an existing message.\n",
"- &`url2note`: Read URL as markdown, and add a note below current message with the result\n",
"- &`msg_insert_line`: Insert text at a specific location in a message.\n",
"- &`msg_str_replace`: Find and replace text in a message.\n",
"- &`msg_strs_replace`: Find and replace multiple strings in a message.\n",
"- &`msg_replace_lines`: Replace a range of lines in a message with new content.\n",
" - Always first use `read_msg( msgid=msgid, n=0, relative=True, nums=True)` to view the content with line numbers."
]
},
{
"cell_type": "markdown",
"id": "f413cb28",
"metadata": {
"solveit_ai": true
},
"source": [
"looks like `add_msg` exists. how do we check the documentation on add_msg? i want to see if there's an arg or kwarg to specify whether it's to be a code, note, or prompt cell\n",
"\n",
"\n",
"##### 🤖Reply🤖<!-- SOLVEIT_SEPARATOR_7f3a9b2c -->\n",
"\n",
"Try this to see the documentation:\n",
"\n",
"```python\n",
"help(dialoghelper.add_msg)\n",
"```\n",
"\n",
"This will show you the function signature and any docstring explaining the parameters!"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e5714ca3",
"metadata": {
"time_run": "8:56:35p"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Help on function add_msg in module dialoghelper.core:\n",
"\n",
"add_msg(content: str, placement: str = 'add_after', msgid: str = None, msg_type: str = 'note', output: str = '', time_run: str | None = '', is_exported: int | None = 0, skipped: int | None = 0, i_collapsed: int | None = 0, o_collapsed: int | None = 0, heading_collapsed: int | None = 0, pinned: int | None = 0, dname: str = '')\n",
" Add/update a message to the queue to show after code execution completes.\n",
"\n"
]
}
],
"source": [
"help(dialoghelper.add_msg)\n"
]
},
{
"cell_type": "markdown",
"id": "ee4aa8d9",
"metadata": {
"solveit_ai": true
},
"source": [
"so i'm guessing that the other values of `msg_type` we want are 'code' and 'prompt'. we can just add the 'content' part , set the 'msg_type', and for placement we'll use 'at_end'\n",
"\n",
"##### 🤖Reply🤖<!-- SOLVEIT_SEPARATOR_7f3a9b2c -->\n",
"\n",
"Perfect! So now you can loop through the parsed cells and add them. Something like:\n",
"\n",
"```python\n",
"for cell in cells:\n",
" dialoghelper.add_msg(cell['content'], placement='at_end', msg_type=cell['type'])\n",
"```\n",
"\n",
"Want to give it a try?"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "698466ae",
"metadata": {
"time_run": "7:52:12p"
},
"outputs": [],
"source": [
"for cell in cells:\n",
" dialoghelper.add_msg(cell['content'], placement='at_end', msg_type=cell['type'])"
]
},
{
"cell_type": "markdown",
"id": "7a8f6f46",
"metadata": {
"solveit_ai": true
},
"source": [
"That all worked perfectly. Amazing. I mean, we're only getting the inputs, not the outputs, but I'm okay with that because users can then just run the inputs themselves to get the outputs.\n",
"\n",
"Should we package this up into a general, callable utility? Call it `scrape_dialog(url)` where the argument is the URL string? the result will be that all the input cells from the remote get automatically added below. I guess we used wget as a shell extension. We could either execute that as a subprocess or maybe there's some other scraping utility you'd like. But...what we've got above works.\n",
"\n",
"##### 🤖Reply🤖<!-- SOLVEIT_SEPARATOR_7f3a9b2c -->\n",
"\n",
"Great idea to package it up! Since you mentioned you're okay with the approach we used, here's a function that wraps everything:\n",
"\n",
"```python\n",
"def scrape_dialog(url):\n",
" \"\"\"Scrape a published SolveIt dialog and add its cells to the current dialog.\"\"\"\n",
" import subprocess\n",
" from bs4 import BeautifulSoup\n",
" import dialoghelper\n",
" \n",
" # Download the page\n",
" subprocess.run(['wget', '-q', '-O', 'temp_dialog.html', url], check=True)\n",
" \n",
" # Parse it\n",
" with open('temp_dialog.html') as f:\n",
" html = f.read()\n",
" \n",
" soup = BeautifulSoup(html, 'html.parser')\n",
" cells = []\n",
" \n",
" for cell_div in soup.find_all('div', class_='editable'):\n",
" card_header = cell_div.find('div', class_='uk-card-header')\n",
" if not card_header: continue\n",
" \n",
" header_text = card_header.get_text(strip=True)\n",
" cell_type = None\n",
" if 'Note' in header_text:\n",
" cell_type = 'note'\n",
" elif 'Code' in header_text:\n",
" cell_type = 'code'\n",
" elif 'Prompt' in header_text:\n",
" cell_type = 'prompt'\n",
" elif 'Raw' in header_text:\n",
" cell_type = 'raw'\n",
" \n",
" card_body = cell_div.find('div', class_='uk-card-body')\n",
" if card_body:\n",
" content = card_body.get_text(strip=False)\n",
" cells.append({'type': cell_type, 'content': content.strip()})\n",
" \n",
" # Add cells to current dialog\n",
" for cell in cells:\n",
" dialoghelper.add_msg(cell['content'], placement='at_end', msg_type=cell['type'])\n",
" \n",
" return len(cells)\n",
"```\n",
"\n",
"Want to test it out?"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "39ad04bc",
"metadata": {
"time_run": "8:38:32p"
},
"outputs": [],
"source": [
"def scrape_dialog(url:str):\n",
" \"\"\"Scrape a published SolveIt dialog and add its cells to the current dialog.\"\"\"\n",
" import subprocess\n",
" from bs4 import BeautifulSoup\n",
" import dialoghelper\n",
" \n",
" # Download the page\n",
" subprocess.run(['wget', '-q', '-O', 'temp_dialog.html', url], check=True)\n",
" \n",
" # Parse it\n",
" with open('temp_dialog.html') as f:\n",
" html = f.read()\n",
" \n",
" soup = BeautifulSoup(html, 'html.parser')\n",
" cells = []\n",
" \n",
" for cell_div in soup.find_all('div', class_='editable'):\n",
" card_header = cell_div.find('div', class_='uk-card-header')\n",
" if not card_header: continue\n",
" \n",
" header_text = card_header.get_text(strip=True)\n",
" cell_type = None\n",
" if 'Note' in header_text:\n",
" cell_type = 'note'\n",
" elif 'Code' in header_text:\n",
" cell_type = 'code'\n",
" elif 'Prompt' in header_text:\n",
" cell_type = 'prompt'\n",
" elif 'Raw' in header_text:\n",
" cell_type = 'raw'\n",
" \n",
" card_body = cell_div.find('div', class_='uk-card-body')\n",
" if card_body:\n",
" content = card_body.get_text(strip=False)\n",
" cells.append({'type': cell_type, 'content': content.strip()})\n",
" \n",
" # Add cells to current dialog\n",
" for cell in cells:\n",
" dialoghelper.add_msg(cell['content'], placement='at_end', msg_type=cell['type'])\n",
" \n",
" return len(cells)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8429aa91",
"metadata": {
"time_run": "8:36:27p"
},
"outputs": [
{
"data": {
"text/plain": [
"48"
]
},
"execution_count": 48,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"scrape_dialog('https://share.solve.it.com/dlgs/precise-breeze-forms-softly-5krt15cn')"
]
},
{
"cell_type": "markdown",
"id": "3fff9db8",
"metadata": {},
"source": [
"# Scraped Dialog Cells Appear Below"
]
},
{
"cell_type": "markdown",
"id": "3d76ef06",
"metadata": {},
"source": [
"Today I'm wanting to explore the website data.gov. That was a lot of really cool data on organizations which tend to look like cities or counties. Some overall questions that I'm interested in are:\n",
"\n",
"what is the density across the US for who is uploading the most datasets\n",
"what are the cities with the most popular datasets\n",
"\n",
"I'm also particular interested in this data: White-nose Syndrome Surveillance and Bat Monitoring Activities in North Coast and Cascades Network Parks, 2016–2024\n",
"It sounds super interesting!\n",
"I also now live in NYC so perhaps we could also explore some of those too"
]
},
{
"cell_type": "markdown",
"id": "88931086",
"metadata": {
"solveit_ai": true
},
"source": [
"Hey, solve it, so I want some help with doing some experimentation on the above plan. Is there an API that exists for data.gov that I can work with?"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5c624ee0",
"metadata": {},
"outputs": [],
"source": [
"!curl https://demo.ckan.org/api/3/action/group_list"
]
},
{
"cell_type": "markdown",
"id": "ddd181d3",
"metadata": {
"solveit_ai": true
},
"source": [
"Great, I got the API key. It is inside an env var called DATA_GOV_API_KEY. Can you give me a code snippet to make a test request?"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1e324552",
"metadata": {},
"outputs": [],
"source": [
"from fastcore.all import *\n",
"from httpx import get as xget\n",
"import os"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c0843a09",
"metadata": {},
"outputs": [],
"source": [
"api_key = os.environ['DATA_GOV_API_KEY']\n",
"url = \"https://catalog.data.gov/api/3/action/package_search\"\n",
"params = {\"q\": \"NYC\", \"rows\": 5}\n",
"headers = {\"X-API-Key\": api_key}"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e59c47d6",
"metadata": {},
"outputs": [],
"source": [
"dict2obj??"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "120165e0",
"metadata": {},
"outputs": [],
"source": [
"r = dict2obj(xget(url, params=params, headers=headers).json()); r.keys()"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2334d558",
"metadata": {},
"outputs": [],
"source": [
"ds = r.result.results[0]; ds.keys()"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a2ca5a5c",
"metadata": {},
"outputs": [],
"source": [
"ds.name, ds.organization, ds.title"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8dc0c956",
"metadata": {},
"outputs": [],
"source": [
"ds.url"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c27ab497",
"metadata": {},
"outputs": [],
"source": [
"csv_ds = ds.resources[0]; csv_ds"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0e7806dc",
"metadata": {},
"outputs": [],
"source": [
"!curl -L {csv_ds.url} -o nyc_motor_crashes.csv"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0585e0f3",
"metadata": {},
"outputs": [],
"source": [
"!rm nyc_motor_crashes.csv"
]
},
{
"cell_type": "markdown",
"id": "e98e1227",
"metadata": {
"solveit_ai": true
},
"source": [
"The dataset was too large for me to download, can I restrict the amount of rows?"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5a7898a9",
"metadata": {},
"outputs": [],
"source": [
"!curl -L https://data.cityofnewyork.us/api/views/h9gi-nx95/rows.csv?accessType=DOWNLOAD&$limit=1000 -o nyc_motor_crashes.csv"
]
},
{
"cell_type": "markdown",
"id": "669e6db4",
"metadata": {
"solveit_ai": true
},
"source": [
"doesnt look like I can use the dollar sign in the curl cmd"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4114450b",
"metadata": {},
"outputs": [],
"source": [
"!curl -L 'https://data.cityofnewyork.us/api/views/h9gi-nx95/rows.csv?accessType=DOWNLOAD&$limit=10' -o nyc_motor_crashes.csv"
]
},
{
"cell_type": "markdown",
"id": "13db0840",
"metadata": {
"solveit_ai": true
},
"source": [
"that doesn't seem to have worked. How can I see what paramaters I can actually specify?"
]
},
{
"cell_type": "markdown",
"id": "57970a11",
"metadata": {
"solveit_ai": true
},
"source": [
"Please read these docs: https://dev.socrata.com/docs/queries/. It looks like I need to specify config options with json payload."
]
},
{
"cell_type": "markdown",
"id": "c0130c46",
"metadata": {
"solveit_ai": true
},
"source": [
"where is this limit syntax coming from? Can you find me a website citation I can go to to see this?"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8035d2a3",
"metadata": {},
"outputs": [],
"source": [
"!curl -L 'https://data.cityofnewyork.us/resource/h9gi-nx95.json?$limit=10000' -o nyc_motor_crashes.jsonl"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b644bf09",
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3827dbc8",
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_json('nyc_motor_crashes.jsonl', orient='record'); df.head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "60d4637c",
"metadata": {},
"outputs": [],
"source": [
"df.columns"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b9b06712",
"metadata": {},
"outputs": [],
"source": [
"df['on_street_name'].unique()"
]
},
{
"cell_type": "markdown",
"id": "5018b231",
"metadata": {
"solveit_ai": true
},
"source": [
"How can I get a histogram of the different numbers of crashes that have occurred in these different streets?"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5429f917",
"metadata": {},
"outputs": [],
"source": [
"df['on_street_name'].value_counts()"
]
},
{
"cell_type": "markdown",
"id": "38e5ac02",
"metadata": {
"solveit_ai": true
},
"source": [
"this right?"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b310e9f2",
"metadata": {},
"outputs": [],
"source": [
"df['on_street_name'].value_counts().plot()"
]
},
{
"cell_type": "markdown",
"id": "410a8bca",
"metadata": {
"solveit_ai": true
},
"source": [
"It looks hideous! Let's take the top 5 and also its a line graph. I want a histogram"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6d5f7872",
"metadata": {},
"outputs": [],
"source": [
"df['on_street_name'].value_counts().head().plot.bar()"
]
},
{
"cell_type": "markdown",
"id": "66d67002",
"metadata": {
"solveit_ai": true
},
"source": [
"Very cool! Looks like Belt Parkway has a lot of crashes! Now, I want to investigate the white nose syndrome! How can I find that dataset?"
]
},
{
"cell_type": "markdown",
"id": "6a1a6e55",
"metadata": {
"solveit_ai": true
},
"source": [
"could you please write it for me?"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3ef79625",
"metadata": {},
"outputs": [],
"source": [
"url = \"https://catalog.data.gov/api/3/action/package_search\"\n",
"params = {\"q\": \"white-nose syndrome bat\", \"rows\": 10}\n",
"headers = {\"X-API-Key\": api_key}\n",
"\n",
"r = dict2obj(xget(url, params=params, headers=headers).json())\n",
"print(f\"Found {r.result.count} datasets\")\n",
"\n",
"# Show titles of results\n",
"for ds in r.result.results:\n",
" print(f\"\\n{ds.title}\")\n",
" print(f\" Organization: {ds.organization.title}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fd1cadbe",
"metadata": {},
"outputs": [],
"source": [
"ds = r.result.results[0]; ds.title"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8dda674c",
"metadata": {},
"outputs": [],
"source": [
"ds.resources"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9ecaf256",
"metadata": {},
"outputs": [],
"source": [
"csv_bats = ds.resources[1]; csv_bats.url"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "147fe280",
"metadata": {},
"outputs": [],
"source": [
"!curl -L {csv_bats.url} -o bats.csv"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e59b1ba3",
"metadata": {},
"outputs": [],
"source": [
"!head bats.csv"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "aadf2d2b",
"metadata": {},
"outputs": [],
"source": [
"print(ds.notes)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "81cbd339",
"metadata": {},
"outputs": [],
"source": [
"bat_df = pd.read_csv('bats.csv'); bat_df.columns"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "01862e76",
"metadata": {},
"outputs": [],
"source": [
"bat_df.head(1)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d4f7c1bd",
"metadata": {},
"outputs": [],
"source": [
"bat_df['Total_Positive_Samples'].sum() / bat_df['Total_Samples'].sum()"
]
},
{
"cell_type": "markdown",
"id": "165f25ea",
"metadata": {
"solveit_ai": true
},
"source": [
"Am I doing this math correct? In sounds like from the data collected, this white nose syndrome is present in about 7.6% of locations? Now that I think about ti some locations might have more samples and it might not be one sample per bat"
]
},
{
"cell_type": "markdown",
"id": "16b179e9",
"metadata": {
"solveit_ai": true
},
"source": [
"I want to know how many bats are affected by this that were tested"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "aa8a2f73",
"metadata": {},
"outputs": [],
"source": [
"bat_df['Positive_Swabs_Samples'].sum() / bat_df['Total_Swab_Samples'].sum()"
]
},
{
"cell_type": "markdown",
"id": "f6585d8f",
"metadata": {
"solveit_ai": true
},
"source": [
"Okay, I think this is more correct and it now is showing that out of the total bat population that were swabbed, 9.5% were infected?"
]
},
{
"cell_type": "markdown",
"id": "f92e9fcd",
"metadata": {},
"source": [
"# Testing `scrape_dialog`. i.e. Second time we inserted the same dialog"
]
},
{
"cell_type": "markdown",
"id": "fa949e8b",
"metadata": {},
"source": [
"Today I'm wanting to explore the website data.gov. That was a lot of really cool data on organizations which tend to look like cities or counties. Some overall questions that I'm interested in are:\n",
"\n",
"what is the density across the US for who is uploading the most datasets\n",
"what are the cities with the most popular datasets\n",
"\n",
"I'm also particular interested in this data: White-nose Syndrome Surveillance and Bat Monitoring Activities in North Coast and Cascades Network Parks, 2016–2024\n",
"It sounds super interesting!\n",
"I also now live in NYC so perhaps we could also explore some of those too"
]
},
{
"cell_type": "markdown",
"id": "2b69f4b2",
"metadata": {
"solveit_ai": true
},
"source": [
"Hey, solve it, so I want some help with doing some experimentation on the above plan. Is there an API that exists for data.gov that I can work with?"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4f9ada7b",
"metadata": {},
"outputs": [],
"source": [
"!curl https://demo.ckan.org/api/3/action/group_list"
]
},
{
"cell_type": "markdown",
"id": "c9b00fae",
"metadata": {
"solveit_ai": true
},
"source": [
"Great, I got the API key. It is inside an env var called DATA_GOV_API_KEY. Can you give me a code snippet to make a test request?"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8096a52a",
"metadata": {},
"outputs": [],
"source": [
"from fastcore.all import *\n",
"from httpx import get as xget\n",
"import os"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "66e4744d",
"metadata": {},
"outputs": [],
"source": [
"api_key = os.environ['DATA_GOV_API_KEY']\n",
"url = \"https://catalog.data.gov/api/3/action/package_search\"\n",
"params = {\"q\": \"NYC\", \"rows\": 5}\n",
"headers = {\"X-API-Key\": api_key}"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "67e3502d",
"metadata": {},
"outputs": [],
"source": [
"dict2obj??"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c457930e",
"metadata": {},
"outputs": [],
"source": [
"r = dict2obj(xget(url, params=params, headers=headers).json()); r.keys()"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "77aa3ff5",
"metadata": {},
"outputs": [],
"source": [
"ds = r.result.results[0]; ds.keys()"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d7ca3de5",
"metadata": {},
"outputs": [],
"source": [
"ds.name, ds.organization, ds.title"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "77347ac7",
"metadata": {},
"outputs": [],
"source": [
"ds.url"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "398b0ae3",
"metadata": {},
"outputs": [],
"source": [
"csv_ds = ds.resources[0]; csv_ds"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "378272d2",
"metadata": {},
"outputs": [],
"source": [
"!curl -L {csv_ds.url} -o nyc_motor_crashes.csv"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a5801e17",
"metadata": {},
"outputs": [],
"source": [
"!rm nyc_motor_crashes.csv"
]
},
{
"cell_type": "markdown",
"id": "c0b6e0d7",
"metadata": {
"solveit_ai": true
},
"source": [
"The dataset was too large for me to download, can I restrict the amount of rows?"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7b53e7d4",
"metadata": {},
"outputs": [],
"source": [
"!curl -L https://data.cityofnewyork.us/api/views/h9gi-nx95/rows.csv?accessType=DOWNLOAD&$limit=1000 -o nyc_motor_crashes.csv"
]
},
{
"cell_type": "markdown",
"id": "f8523777",
"metadata": {
"solveit_ai": true
},
"source": [
"doesnt look like I can use the dollar sign in the curl cmd"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2c9677e6",
"metadata": {},
"outputs": [],
"source": [
"!curl -L 'https://data.cityofnewyork.us/api/views/h9gi-nx95/rows.csv?accessType=DOWNLOAD&$limit=10' -o nyc_motor_crashes.csv"
]
},
{
"cell_type": "markdown",
"id": "8e739af2",
"metadata": {
"solveit_ai": true
},
"source": [
"that doesn't seem to have worked. How can I see what paramaters I can actually specify?"
]
},
{
"cell_type": "markdown",
"id": "b56ab72c",
"metadata": {
"solveit_ai": true
},
"source": [
"Please read these docs: https://dev.socrata.com/docs/queries/. It looks like I need to specify config options with json payload."
]
},
{
"cell_type": "markdown",
"id": "22b4a3fa",
"metadata": {
"solveit_ai": true
},
"source": [
"where is this limit syntax coming from? Can you find me a website citation I can go to to see this?"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "07e8a1b3",
"metadata": {},
"outputs": [],
"source": [
"!curl -L 'https://data.cityofnewyork.us/resource/h9gi-nx95.json?$limit=10000' -o nyc_motor_crashes.jsonl"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "46d7a7c3",
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f31f8c15",
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_json('nyc_motor_crashes.jsonl', orient='record'); df.head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "63984c1b",
"metadata": {},
"outputs": [],
"source": [
"df.columns"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a1c0df6b",
"metadata": {},
"outputs": [],
"source": [
"df['on_street_name'].unique()"
]
},
{
"cell_type": "markdown",
"id": "efc74121",
"metadata": {
"solveit_ai": true
},
"source": [
"How can I get a histogram of the different numbers of crashes that have occurred in these different streets?"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3493129b",
"metadata": {},
"outputs": [],
"source": [
"df['on_street_name'].value_counts()"
]
},
{
"cell_type": "markdown",
"id": "ccf6ee7a",
"metadata": {
"solveit_ai": true
},
"source": [
"this right?"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4eb5fd22",
"metadata": {},
"outputs": [],
"source": [
"df['on_street_name'].value_counts().plot()"
]
},
{
"cell_type": "markdown",
"id": "eb6f7c3b",
"metadata": {
"solveit_ai": true
},
"source": [
"It looks hideous! Let's take the top 5 and also its a line graph. I want a histogram"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b0697924",
"metadata": {},
"outputs": [],
"source": [
"df['on_street_name'].value_counts().head().plot.bar()"
]
},
{
"cell_type": "markdown",
"id": "8435ca0d",
"metadata": {
"solveit_ai": true
},
"source": [
"Very cool! Looks like Belt Parkway has a lot of crashes! Now, I want to investigate the white nose syndrome! How can I find that dataset?"
]
},
{
"cell_type": "markdown",
"id": "24331d48",
"metadata": {
"solveit_ai": true
},
"source": [
"could you please write it for me?"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d364e846",
"metadata": {},
"outputs": [],
"source": [
"url = \"https://catalog.data.gov/api/3/action/package_search\"\n",
"params = {\"q\": \"white-nose syndrome bat\", \"rows\": 10}\n",
"headers = {\"X-API-Key\": api_key}\n",
"\n",
"r = dict2obj(xget(url, params=params, headers=headers).json())\n",
"print(f\"Found {r.result.count} datasets\")\n",
"\n",
"# Show titles of results\n",
"for ds in r.result.results:\n",
" print(f\"\\n{ds.title}\")\n",
" print(f\" Organization: {ds.organization.title}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7eb33420",
"metadata": {},
"outputs": [],
"source": [
"ds = r.result.results[0]; ds.title"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1b09e21a",
"metadata": {},
"outputs": [],
"source": [
"ds.resources"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8ce1f6c7",
"metadata": {},
"outputs": [],
"source": [
"csv_bats = ds.resources[1]; csv_bats.url"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "00396e7f",
"metadata": {},
"outputs": [],
"source": [
"!curl -L {csv_bats.url} -o bats.csv"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e7e4bf6c",
"metadata": {},
"outputs": [],
"source": [
"!head bats.csv"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0eeea2e6",
"metadata": {},
"outputs": [],
"source": [
"print(ds.notes)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "dbf715a4",
"metadata": {},
"outputs": [],
"source": [
"bat_df = pd.read_csv('bats.csv'); bat_df.columns"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cc8d027e",
"metadata": {},
"outputs": [],
"source": [
"bat_df.head(1)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b8d5fade",
"metadata": {},
"outputs": [],
"source": [
"bat_df['Total_Positive_Samples'].sum() / bat_df['Total_Samples'].sum()"
]
},
{
"cell_type": "markdown",
"id": "e4759560",
"metadata": {
"solveit_ai": true
},
"source": [
"Am I doing this math correct? In sounds like from the data collected, this white nose syndrome is present in about 7.6% of locations? Now that I think about ti some locations might have more samples and it might not be one sample per bat"
]
},
{
"cell_type": "markdown",
"id": "c1addd2d",
"metadata": {
"solveit_ai": true
},
"source": [
"I want to know how many bats are affected by this that were tested"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d8dbda33",
"metadata": {},
"outputs": [],
"source": [
"bat_df['Positive_Swabs_Samples'].sum() / bat_df['Total_Swab_Samples'].sum()"
]
},
{
"cell_type": "markdown",
"id": "d2b36bed",
"metadata": {
"solveit_ai": true
},
"source": [
"Okay, I think this is more correct and it now is showing that out of the total bat population that were swabbed, 9.5% were infected?"
]
}
],
"metadata": {
"solveit_dialog_mode": "learning",
"solveit_ver": 2
},
"nbformat": 4,
"nbformat_minor": 5
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment