Argenis Leon argenisleon

GIS with pySpark : A not-so-easy journey

Today, many datas are geolocalised (meaning that they have a position in space). They're named GIS datas.

It's not rare that we need to do operations on those, such as aggregations, and there are many optimisations existing to do that.

A "Best of the Best Practices" (BOBP) guide to developing in Python.

	const crypto = require('crypto');
	const base64url = require('base64url');
	const cbor = require('cbor');
	const asn1 = require('@lapo/asn1js');
	const jsrsasign = require('jsrsasign');

	/* Android Keystore Root is not published anywhere.
	* This certificate was extracted from one of the attestations
	* The last certificate in x5c must match this certificate
	* This needs to be checked to ensure that malicious party wont generate fake attestations

	package org.anish.spark.mostcommonvalue

	import org.apache.spark.sql.Row
	import org.apache.spark.sql.expressions.{MutableAggregationBuffer, UserDefinedAggregateFunction}
	import org.apache.spark.sql.types._

	import scalaz.Scalaz._

	/**
	* Spark User Defined Aggregate Function to calculate the most frequent value in a column. This is similar to

	import pandas as pd

	def _map_to_pandas(rdds):
	""" Needs to be here due to pickling issues """
	return [pd.DataFrame(list(rdds))]

	def toPandas(df, n_partitions=None):
	"""
	Returns the contents of `df` as a local `pandas.DataFrame` in a speedy fashion. The DataFrame is
	repartitioned if `n_partitions` is passed.

	'''
	A python script which starts celery worker and auto reload it when any code change happens.
	I did this because Celery worker's "--autoreload" option seems not working for a lot of people.
	'''

	import time
	from watchdog.observers import Observer ##pip install watchdog
	from watchdog.events import PatternMatchingEventHandler
	import psutil ##pip install psutil
	import os

	#!/usr/bin/python

	import argparse
	from pymongo import MongoClient as Client
	from bson import BSON
	from bson import json_util
	import json
	import os

	# mongo client

	sudo openvpn --config *.opvn
	apt-get update
	apt-get install vim
	wget http://d3kbcqa49mib13.cloudfront.net/spark-1.3.0-bin-hadoop2.4.tgz \| tar zxf
	hadoop fs -mkdir /spark
	hadoop fs -put spark-1.3.0-bin-hadoop2.4.tgz /spark
	hadoop fs -du -h /spark
	cp spark-env.sh.template spark-env.sh

	def get_count(q):
	count_q = q.statement.with_only_columns([func.count()]).order_by(None)
	count = q.session.execute(count_q).scalar()
	return count

	q = session.query(TestModel).filter(...).order_by(...)

	# Slow: SELECT COUNT(*) FROM (SELECT ... FROM TestModel WHERE ...) ...
	print q.count()