This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
// 1) the story starts with more than 500 mil. compressed XML files stored in S3 (various sizes: 10KB - 200MB) | |
// 2) the "small" files are grouped, decompressed, cleaned and stored as Parquet files | |
// 3) the next piece of code converts the blob column to a column with a complex schema (more or less equivalent with the XML structure) | |
// 4) the result can be persisted and afterwards queried in an efficient way | |
case class OrderReference(ID: String, | |
SalesOrderID: String, | |
UUID: String, | |
IssueDate: String) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
license: gpl-3.0 | |
height: 510 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
numbers = (1, 2, 3) | |
chars = ('a', 'b', 'c') | |
numbers_chars = list(zip(numbers, chars)) | |
# [(1, 'a'), (2, 'b'), (3, 'c')] | |
unzipped_numbers, unzipped_chars = zip(*numbers_chars) | |
# ((1, 2, 3), ('a', 'b', 'c')) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Kafka - Getting Started | |
https://kafka.apache.org/quickstart | |
1. get | |
wget http://mirrors.m247.ro/apache/kafka/0.10.2.0/kafka_2.11-0.10.2.0.tgz | |
tar -xzf kafka_2.11-0.10.2.0.tgz |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#include <stdio.h> | |
#include <stdlib.h> | |
#include <assert.h> | |
#include <math.h> | |
#define NDEBUG | |
int size(int n) { | |
int c = 0; | |
while (n) { | |
n /= 10; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
<!DOCTYPE html> | |
<html> | |
<head> | |
<title>OSM PBF to GeoJSON example (osm-read + osmtogeojson)</title> | |
<meta charset="utf-8"/> | |
</head> | |
<body> | |
<pre id="log" style="max-height: 480px; overflow-y: auto;"></pre> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
/** | |
* An implementation of microKanren (and probably most of miniKanren), with | |
* a few extras. Currently, it supports: | |
* | |
* - The essential core of microKanren: Unify, Fresh, Disjunction, Conjunction | |
* - Standard terms: Variables, Atoms, TermCons, EmptyTerm. | |
* - An implicit conversion from type T to Atom[T]. This makes writing programs | |
* much easier. | |
* - A decent reifier, which converts terms to strings. | |
* |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
case class Way(id: Long, nodes: Array[Long], tags: Map[String, String]) | |
val w1 = Way(1, Array(100, 200, 13, 20), Map("type"-> "highway")) | |
val w2 = Way(2, Array(30, 13, 500), Map("type"-> "ulita")) | |
val wayRDD = sc.parallelize(Array(w1, w2)) | |
val wayDF = wayRDD.toDF | |
wayDF.write.parquet("path/to/mighty_map") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#include <stdio.h> | |
#include <stdlib.h> | |
#include <assert.h> | |
#define NDEBUG | |
typedef struct matrix { | |
int n; | |
int m; | |
int *data; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#include <stdio.h> | |
#include <stdlib.h> | |
#include <assert.h> | |
#define NDEBUG | |
typedef enum { | |
false, true | |
} bool; |
NewerOlder