Skip to content

Instantly share code, notes, and snippets.

@UnaNancyOwen

UnaNancyOwen/interactive-select-rect-in-image-using-tensorflow-colab-utils-on-google-colab.ipynb

Created May 1, 2023 09:15
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save UnaNancyOwen/c08fc1c6b5092a2eb8ba2e512aaf809a to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save UnaNancyOwen/c08fc1c6b5092a2eb8ba2e512aaf809a to your computer and use it in GitHub Desktop.
Download ZIP
Interactive Select Rect in Image using TensorFlow Colab Utils on Google Colab.ipynb
Display the source blob
Display the rendered blob
Raw
interactive-select-rect-in-image-using-tensorflow-colab-utils-on-google-colab.ipynb
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"provenance": [],
"authorship_tag": "ABX9TyNOwRe1ljHPTZ7JxV9b7+6x",
"include_colab_link": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"language_info": {
"name": "python"
}
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/gist/UnaNancyOwen/c08fc1c6b5092a2eb8ba2e512aaf809a/interactive-select-rect-in-image-using-tensorflow-colab-utils-on-google-colab.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "markdown",
"source": [
"This is sample program of simple interactive-selecter rect in image using matplotlib on Google Colab. This is referring to [this code](https://github.com/tensorflow/models/blob/master/research/object_detection/utils/colab_utils.py). "
],
"metadata": {
"id": "Q-p3xyq9bMv_"
}
},
{
"cell_type": "markdown",
"source": [
"Download Sample Image\n"
],
"metadata": {
"id": "oR56Y7xfzaa9"
}
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "RgzpJ3J1ohlt",
"outputId": "e6a057e3-10e0-4400-95f3-4a23ad35194c"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"--2023-05-01 08:35:06-- https://raw.githubusercontent.com/AlexeyAB/darknet/yolov4/data/dog.jpg\n",
"Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.108.133, 185.199.110.133, ...\n",
"Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected.\n",
"HTTP request sent, awaiting response... 200 OK\n",
"Length: 163759 (160K) [image/jpeg]\n",
"Saving to: ‘images/dog.jpg’\n",
"\n",
"\rdog.jpg 0%[ ] 0 --.-KB/s \rdog.jpg 100%[===================>] 159.92K --.-KB/s in 0.01s \n",
"\n",
"2023-05-01 08:35:06 (11.1 MB/s) - ‘images/dog.jpg’ saved [163759/163759]\n",
"\n"
]
}
],
"source": [
"# Download Sample Image\n",
"!wget https://raw.githubusercontent.com/AlexeyAB/darknet/yolov4/data/dog.jpg -P images"
]
},
{
"cell_type": "markdown",
"source": [
"Download Colab Utils from TensorFlow Object Detection API"
],
"metadata": {
"id": "oJkDM9WazfC6"
}
},
{
"cell_type": "code",
"source": [
"# Download Colab Utils\n",
"!wget https://raw.githubusercontent.com/tensorflow/models/master/research/object_detection/utils/colab_utils.py -P object_detection/utils"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "rYOGlxFMHcZL",
"outputId": "b59a5acc-4647-4643-f47b-8b614eb3a36c"
},
"execution_count": 2,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"--2023-05-01 08:35:08-- https://raw.githubusercontent.com/tensorflow/models/master/research/object_detection/utils/colab_utils.py\n",
"Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
"Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
"HTTP request sent, awaiting response... 200 OK\n",
"Length: 19472 (19K) [text/plain]\n",
"Saving to: ‘object_detection/utils/colab_utils.py’\n",
"\n",
"\rcolab_utils.py 0%[ ] 0 --.-KB/s \rcolab_utils.py 100%[===================>] 19.02K --.-KB/s in 0s \n",
"\n",
"2023-05-01 08:35:09 (78.7 MB/s) - ‘object_detection/utils/colab_utils.py’ saved [19472/19472]\n",
"\n"
]
}
]
},
{
"cell_type": "markdown",
"source": [
"Import"
],
"metadata": {
"id": "cMdOMQW_zwD0"
}
},
{
"cell_type": "code",
"source": [
"# Import\n",
"from object_detection.utils import colab_utils"
],
"metadata": {
"id": "4ZcwNODypDaq"
},
"execution_count": 3,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Import\n",
"import cv2 as cv\n",
"import numpy as np\n",
"from google.colab.patches import cv2_imshow"
],
"metadata": {
"id": "w6EO9eMTLt0r"
},
"execution_count": 32,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Load Image\n",
"image = cv.imread(\"images/dog.jpg\")\n",
"image = cv.cvtColor(image, cv.COLOR_BGR2RGB)"
],
"metadata": {
"id": "P5ZYHc_GK0Pd"
},
"execution_count": 5,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"Interactive Select for Rect"
],
"metadata": {
"id": "MW5i-QozMsiE"
}
},
{
"cell_type": "code",
"source": [
"boxes = []\n",
"colab_utils.annotate([image], box_storage_pointer=boxes)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 734
},
"id": "aA9vbSjiwb2W",
"outputId": "2c6484e8-fc97-423d-e631-398ac0a11f4d"
},
"execution_count": 6,
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": [
"<IPython.core.display.Javascript object>"
],
"application/javascript": [
"\n",
" async function load_image(imgs, callbackId) {\n",
" //init organizational elements\n",
" const div = document.createElement('div');\n",
" var image_cont = document.createElement('div');\n",
" var errorlog = document.createElement('div');\n",
" var crosshair_h = document.createElement('div');\n",
" crosshair_h.style.position = \"absolute\";\n",
" crosshair_h.style.backgroundColor = \"transparent\";\n",
" crosshair_h.style.width = \"100%\";\n",
" crosshair_h.style.height = \"0px\";\n",
" crosshair_h.style.zIndex = 9998;\n",
" crosshair_h.style.borderStyle = \"dotted\";\n",
" crosshair_h.style.borderWidth = \"2px\";\n",
" crosshair_h.style.borderColor = \"rgba(255, 0, 0, 0.75)\";\n",
" crosshair_h.style.cursor = \"crosshair\";\n",
" var crosshair_v = document.createElement('div');\n",
" crosshair_v.style.position = \"absolute\";\n",
" crosshair_v.style.backgroundColor = \"transparent\";\n",
" crosshair_v.style.width = \"0px\";\n",
" crosshair_v.style.height = \"100%\";\n",
" crosshair_v.style.zIndex = 9999;\n",
" crosshair_v.style.top = \"0px\";\n",
" crosshair_v.style.borderStyle = \"dotted\";\n",
" crosshair_v.style.borderWidth = \"2px\";\n",
" crosshair_v.style.borderColor = \"rgba(255, 0, 0, 0.75)\";\n",
" crosshair_v.style.cursor = \"crosshair\";\n",
" crosshair_v.style.marginTop = \"23px\";\n",
" var brdiv = document.createElement('br');\n",
"\n",
"\n",
" //init control elements\n",
" var next = document.createElement('button');\n",
" var prev = document.createElement('button');\n",
" var submit = document.createElement('button');\n",
" var deleteButton = document.createElement('button');\n",
" var deleteAllbutton = document.createElement('button');\n",
"\n",
" //init image containers\n",
" var image = new Image();\n",
" var canvas_img = document.createElement('canvas');\n",
" var ctx = canvas_img.getContext(\"2d\");\n",
" canvas_img.style.cursor = \"crosshair\";\n",
" canvas_img.setAttribute('draggable', false);\n",
" crosshair_v.setAttribute('draggable', false);\n",
" crosshair_h.setAttribute('draggable', false);\n",
"\n",
" // bounding box containers\n",
" const height = 600\n",
" var allBoundingBoxes = [];\n",
" var curr_image = 0\n",
" var im_height = 0;\n",
" var im_width = 0;\n",
"\n",
" //initialize bounding boxes\n",
" for (var i = 0; i < imgs.length; i++) {\n",
" allBoundingBoxes[i] = [];\n",
" }\n",
" //initialize image view\n",
" errorlog.id = 'errorlog';\n",
" image.style.display = 'block';\n",
" image.setAttribute('draggable', false);\n",
"\n",
" //load the first image\n",
" img = imgs[curr_image];\n",
" image.src = \"data:image/png;base64,\" + img;\n",
" image.onload = function() {\n",
" // normalize display height and canvas\n",
" image.height = height;\n",
" image_cont.height = canvas_img.height = image.height;\n",
" image_cont.width = canvas_img.width = image.naturalWidth;\n",
" crosshair_v.style.height = image_cont.height + \"px\";\n",
" crosshair_h.style.width = image_cont.width + \"px\";\n",
"\n",
" // draw the new image\n",
" ctx.drawImage(image, 0, 0, image.naturalWidth, image.naturalHeight, 0, 0, canvas_img.width, canvas_img.height);\n",
"\n",
" };\n",
"\n",
" // move to next image in array\n",
" next.textContent = \"next image\";\n",
" next.onclick = function(){\n",
" if (curr_image < imgs.length - 1){\n",
" // clear canvas and load new image\n",
" curr_image += 1;\n",
" errorlog.innerHTML = \"\";\n",
" }\n",
" else{\n",
" errorlog.innerHTML = \"All images completed!!\";\n",
" }\n",
" resetcanvas();\n",
" }\n",
"\n",
" //move forward through list of images\n",
" prev.textContent = \"prev image\"\n",
" prev.onclick = function(){\n",
" if (curr_image > 0){\n",
" // clear canvas and load new image\n",
" curr_image -= 1;\n",
" errorlog.innerHTML = \"\";\n",
" }\n",
" else{\n",
" errorlog.innerHTML = \"at the beginning\";\n",
" }\n",
" resetcanvas();\n",
" }\n",
" // on delete, deletes the last bounding box\n",
" deleteButton.textContent = \"undo bbox\";\n",
" deleteButton.onclick = function(){\n",
" boundingBoxes.pop();\n",
" ctx.clearRect(0, 0, canvas_img.width, canvas_img.height);\n",
" image.src = \"data:image/png;base64,\" + img;\n",
" image.onload = function() {\n",
" ctx.drawImage(image, 0, 0, image.naturalWidth, image.naturalHeight, 0, 0, canvas_img.width, canvas_img.height);\n",
" boundingBoxes.map(r => {drawRect(r)});\n",
" };\n",
" }\n",
" // on all delete, deletes all of the bounding box\n",
" deleteAllbutton.textContent = \"delete all\"\n",
" deleteAllbutton.onclick = function(){\n",
" boundingBoxes = [];\n",
" ctx.clearRect(0, 0, canvas_img.width, canvas_img.height);\n",
" image.src = \"data:image/png;base64,\" + img;\n",
" image.onload = function() {\n",
" ctx.drawImage(image, 0, 0, image.naturalWidth, image.naturalHeight, 0, 0, canvas_img.width, canvas_img.height);\n",
" //boundingBoxes.map(r => {drawRect(r)});\n",
" };\n",
" }\n",
"\n",
" // on submit, send the boxes to display\n",
" submit.textContent = \"submit\";\n",
" submit.onclick = function(){\n",
" errorlog.innerHTML = \"\";\n",
"\n",
" // send box data to callback fucntion\n",
" google.colab.kernel.invokeFunction(callbackId, [allBoundingBoxes], {});\n",
" }\n",
"\n",
" // init template for annotations\n",
" const annotation = {\n",
" x: 0,\n",
" y: 0,\n",
" w: 0,\n",
" h: 0,\n",
" };\n",
"\n",
" // the array of all rectangles\n",
" let boundingBoxes = allBoundingBoxes[curr_image];\n",
"\n",
" // the actual rectangle, the one that is being drawn\n",
" let o = {};\n",
"\n",
" // a variable to store the mouse position\n",
" let m = {},\n",
"\n",
" // a variable to store the point where you begin to draw the\n",
" // rectangle\n",
" start = {};\n",
"\n",
" // a boolean variable to store the drawing state\n",
" let isDrawing = false;\n",
" var elem = null;\n",
"\n",
" function handleMouseDown(e) {\n",
" // on mouse click set change the cursor and start tracking the mouse position\n",
" start = oMousePos(canvas_img, e);\n",
"\n",
" // configure is drawing to true\n",
" isDrawing = true;\n",
" }\n",
"\n",
" function handleMouseMove(e) {\n",
" // move crosshairs, but only within the bounds of the canvas\n",
" if (document.elementsFromPoint(e.pageX, e.pageY).includes(canvas_img)) {\n",
" crosshair_h.style.top = e.pageY + \"px\";\n",
" crosshair_v.style.left = e.pageX + \"px\";\n",
" }\n",
"\n",
" // move the bounding box\n",
" if(isDrawing){\n",
" m = oMousePos(canvas_img, e);\n",
" draw();\n",
" }\n",
" }\n",
"\n",
" function handleMouseUp(e) {\n",
" if (isDrawing) {\n",
" // on mouse release, push a bounding box to array and draw all boxes\n",
" isDrawing = false;\n",
"\n",
" const box = Object.create(annotation);\n",
"\n",
" // calculate the position of the rectangle\n",
" if (o.w > 0){\n",
" box.x = o.x;\n",
" }\n",
" else{\n",
" box.x = o.x + o.w;\n",
" }\n",
" if (o.h > 0){\n",
" box.y = o.y;\n",
" }\n",
" else{\n",
" box.y = o.y + o.h;\n",
" }\n",
" box.w = Math.abs(o.w);\n",
" box.h = Math.abs(o.h);\n",
"\n",
" // add the bounding box to the image\n",
" boundingBoxes.push(box);\n",
" draw();\n",
" }\n",
" }\n",
"\n",
" function draw() {\n",
" o.x = (start.x)/image.width; // start position of x\n",
" o.y = (start.y)/image.height; // start position of y\n",
" o.w = (m.x - start.x)/image.width; // width\n",
" o.h = (m.y - start.y)/image.height; // height\n",
"\n",
" ctx.clearRect(0, 0, canvas_img.width, canvas_img.height);\n",
" ctx.drawImage(image, 0, 0, image.naturalWidth, image.naturalHeight, 0, 0, canvas_img.width, canvas_img.height);\n",
" // draw all the rectangles saved in the rectsRy\n",
" boundingBoxes.map(r => {drawRect(r)});\n",
" // draw the actual rectangle\n",
" drawRect(o);\n",
" }\n",
"\n",
" // add the handlers needed for dragging\n",
" crosshair_h.addEventListener(\"mousedown\", handleMouseDown);\n",
" crosshair_v.addEventListener(\"mousedown\", handleMouseDown);\n",
" document.addEventListener(\"mousemove\", handleMouseMove);\n",
" document.addEventListener(\"mouseup\", handleMouseUp);\n",
"\n",
"\n",
" function resetcanvas(){\n",
" // clear canvas\n",
" ctx.clearRect(0, 0, canvas_img.width, canvas_img.height);\n",
" img = imgs[curr_image]\n",
" image.src = \"data:image/png;base64,\" + img;\n",
"\n",
" // onload init new canvas and display image\n",
" image.onload = function() {\n",
" // normalize display height and canvas\n",
" image.height = height;\n",
" image_cont.height = canvas_img.height = image.height;\n",
" image_cont.width = canvas_img.width = image.naturalWidth;\n",
" crosshair_v.style.height = image_cont.height + \"px\";\n",
" crosshair_h.style.width = image_cont.width + \"px\";\n",
"\n",
" // draw the new image\n",
" ctx.drawImage(image, 0, 0, image.naturalWidth, image.naturalHeight, 0, 0, canvas_img.width, canvas_img.height);\n",
"\n",
" // draw bounding boxes\n",
" boundingBoxes = allBoundingBoxes[curr_image];\n",
" boundingBoxes.map(r => {drawRect(r)});\n",
" };\n",
" }\n",
"\n",
" function drawRect(o){\n",
" // draw a predefined rectangle\n",
" ctx.strokeStyle = \"red\";\n",
" ctx.lineWidth = 2;\n",
" ctx.beginPath(o);\n",
" ctx.rect(o.x * image.width, o.y * image.height, o.w * image.width, o.h * image.height);\n",
" ctx.stroke();\n",
" }\n",
"\n",
" // Function to detect the mouse position\n",
" function oMousePos(canvas_img, evt) {\n",
" let ClientRect = canvas_img.getBoundingClientRect();\n",
" return {\n",
" x: evt.clientX - ClientRect.left,\n",
" y: evt.clientY - ClientRect.top\n",
" };\n",
" }\n",
"\n",
"\n",
" //configure colab output display\n",
" google.colab.output.setIframeHeight(document.documentElement.scrollHeight, true);\n",
"\n",
" //build the html document that will be seen in output\n",
" div.appendChild(document.createElement('br'))\n",
" div.appendChild(image_cont)\n",
" image_cont.appendChild(canvas_img)\n",
" image_cont.appendChild(crosshair_h)\n",
" image_cont.appendChild(crosshair_v)\n",
" div.appendChild(document.createElement('br'))\n",
" div.appendChild(errorlog)\n",
" div.appendChild(prev)\n",
" div.appendChild(next)\n",
" div.appendChild(deleteButton)\n",
" div.appendChild(deleteAllbutton)\n",
" div.appendChild(document.createElement('br'))\n",
" div.appendChild(brdiv)\n",
" div.appendChild(submit)\n",
" document.querySelector(\"#output-area\").appendChild(div);\n",
" return\n",
" }"
]
},
"metadata": {}
},
{
"output_type": "display_data",
"data": {
"text/plain": [
"<IPython.core.display.Javascript object>"
],
"application/javascript": [
"window[\"25416412-e7fb-11ed-964d-0242ac1c000c\"] = google.colab.output.getActiveOutputArea();\n",
"//# sourceURL=js_2af1b1eea9"
]
},
"metadata": {}
},
{
"output_type": "display_data",
"data": {
"text/plain": [
"<IPython.core.display.Javascript object>"
],
"application/javascript": [
"window[\"2541b8d6-e7fb-11ed-964d-0242ac1c000c\"] = document.querySelector(\"#errorlog\");\n",
"//# sourceURL=js_8e9ffcb726"
]
},
"metadata": {}
},
{
"output_type": "display_data",
"data": {
"text/plain": [
"<IPython.core.display.Javascript object>"
],
"application/javascript": [
"window[\"25420070-e7fb-11ed-964d-0242ac1c000c\"] = google.colab.output.setActiveOutputArea(window[\"2541b8d6-e7fb-11ed-964d-0242ac1c000c\"]);\n",
"//# sourceURL=js_50e3bdb27b"
]
},
"metadata": {}
},
{
"output_type": "display_data",
"data": {
"text/plain": [
"'--boxes array populated--'"
],
"application/vnd.google.colaboratory.intrinsic+json": {
"type": "string"
}
},
"metadata": {}
},
{
"output_type": "display_data",
"data": {
"text/plain": [
"<IPython.core.display.Javascript object>"
],
"application/javascript": [
"window[\"2542730c-e7fb-11ed-964d-0242ac1c000c\"] = google.colab.output.setActiveOutputArea(window[\"25416412-e7fb-11ed-964d-0242ac1c000c\"]);\n",
"//# sourceURL=js_a26b878f33"
]
},
"metadata": {}
}
]
},
{
"cell_type": "markdown",
"source": [
"Print Rect"
],
"metadata": {
"id": "Dt0dJWGWMwBw"
}
},
{
"cell_type": "code",
"source": [
"# Print Rect(YXYX) in Normalised Coordinate System\n",
"box = boxes[0][0]\n",
"print(box)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "imgWl-V0xGee",
"outputId": "0207b660-8adc-42ac-ccd1-fdc03b7b30b9"
},
"execution_count": 35,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"[0.38106771 0.17317708 0.93773438 0.40625 ]\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"# Print Rect(YXYX) in Image Coordinate System\n",
"box = (np.array(boxes[0][0]) * np.tile([image.shape[0], image.shape[1]], 2))\n",
"print(box)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "CQwrGIGJOxC2",
"outputId": "a58afb3b-95a0-4042-f686-bf78a934c02d"
},
"execution_count": 36,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"[219.495 133. 540.135 312. ]\n"
]
}
]
},
{
"cell_type": "markdown",
"source": [
"Show Rect"
],
"metadata": {
"id": "rj6DT650i6Gz"
}
},
{
"cell_type": "code",
"source": [
"# Show Image with Draw Rect\n",
"box = (np.array(boxes[0][0]) * np.tile([image.shape[0], image.shape[1]], 2))\n",
"point1 = (int(box[1]), int(box[0]))\n",
"point2 = (int(box[3]), int(box[2]))\n",
"draw_image = cv.cvtColor(image, cv.COLOR_RGB2BGR)\n",
"draw_image = cv.rectangle(draw_image, point1, point2, (0, 0, 255), 2, cv.LINE_AA)\n",
"cv2_imshow(draw_image)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 593
},
"id": "gO-VkaEii1Ls",
"outputId": "854346d8-a2a5-43c6-acec-142381ca2fe7"
},
"execution_count": 34,
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": [
"<PIL.Image.Image image mode=RGB size=768x576 at 0x7F8291DCCD00>"
],
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment