Sebastian Nagel sebastian-nagel
sebastian-nagel
/ saturn.pretty.wat
Last active
December 17, 2024 11:41
— forked from pjox/saturn.pretty.wat
Common Crawl format example for https://en.wikipedia.org/wiki/Saturn
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| WARC/1.0 | |
| WARC-Type: metadata | |
| WARC-Target-URI: https://en.wikipedia.org/wiki/Saturn | |
| WARC-Date: 2024-12-11T20:20:04Z | |
| WARC-Record-ID: <urn:uuid:74b1614e-97bb-4a19-b02f-defc603ab81c> | |
| WARC-Refers-To: <urn:uuid:90f1a666-d5ba-4e8d-806d-4d848e77a0f8> | |
| Content-Type: application/json | |
| Content-Length: 1910 | |
| { |
sebastian-nagel
/ common-crawl-cdx.py
Last active
May 16, 2016 13:20
— forked from tfmorris/common-crawl-cdx.py
Analyze Common Crawl index - http://index.commoncrawl.org/
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # -*- coding: utf-8 -*- | |
| """ | |
| common-crawl-cdx.py | |
| A simple example program to analyze the Common Crawl index. | |
| This is implemented as a single stream job which accesses S3 via HTTP, | |
| so that it can be easily be run from any laptop, but it could easily be | |
| converted to an EMR job which processed the 300 index files in parallel. |