Kunal Grover kunalgrover05
kunalgrover05
/ google-crawl.js
Last active
March 6, 2017 15:01
NodeJS Osmosis based crawler script for Google search results. Read: https://crondev.wordpress.com/2017/03/05/web-crawling-with-nodejs-its-an-interesting-world/
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| var nextLink; | |
| function open_page(url) { | |
| console.log("Opening " + url); | |
| osmosis.get(url) | |
| .find('#nav td:last a') | |
| .set({ | |
| 'nextLink': '@href' | |
| }) | |
| .find('.g') | |
| .set({ |
kunalgrover05
/ Retry.java
Created
February 17, 2017 18:08
Example function for doing retries with Java8 CompletableFuture API
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import java.util.List; | |
| import java.util.Optional; | |
| import java.util.concurrent.CompletableFuture; | |
| import java.util.concurrent.Executor; | |
| import java.util.concurrent.RejectedExecutionException; | |
| import java.util.function.Supplier; | |
| /** | |
| * A class which implements a Retry API for a Supplier function asynchronously on a new executor | |
| * A retry is triggered when any of the exceptions which are specified is thrown, and retryCount is lower |