djamelz’s gists

djamelz / custom_deser.scala

Created September 29, 2016 12:28

custom deserialization with json4s for date in string format to timestamp (long)

	import org.joda.time.DateTime
	import org.joda.time.format.DateTimeFormat
	import org.json4s._
	import org.json4s.native.JsonMethods._


	val datePattern = "yyyy-MM-dd hh:mm:ss.SSSSSS"
	val s = """{"timestamp": "2016-09-29 11:31:13.247772", "domain": "d1", "filePath": "s3://..."}"""
	case class Event(domain : String, filePath : String, timestamp : Long)

djamelz / emr_wait.py

Created October 12, 2015 21:35

Sync waiting for the TERMINATED status of an EMR cluster use as : python emr_wait.py us-east-1 j-XXXXXXXXXXXX

	#!/usr/bin/python

	import boto.emr
	import time
	import sys

	region = sys.argv[1]
	clusterId = sys.argv[2]

	def getStatus(cluster_id):

djamelz / aliasSwitching.py

Created October 12, 2015 21:09

Elasticsearch Alias switching in python => restore the new one, close the latest and delete the olders (based on name versioning index_name_xxx, index_name_xxy for index name and index_name for alias)

	from elasticsearch import Elasticsearch
	es = Elasticsearch(hosts=['es-host:9200'], timeout=600) #long timeout for the restore operation

	snapshotName='index_name_20151012140502'
	alias = es.indices.get_alias('index_name').keys()[0] if es.indices.exists_alias(name='index_name') else ''
	indices = filter(lambda x: x.startswith('index_name'), es.indices.get_aliases())
	indices.sort()
	print 'restore repository'
	es.snapshot.restore(repository='repository_name', snapshot=snapshotName, body ='{ "ignore_unavailable": "true", "include_global_state": false}', wait_for_completion=True)
	if (len(alias) > 1):

djamelz / Ec2Utils.scala

Created October 7, 2015 22:28

Play with ec2 and java aws api.... Not the best way to do with scala (no future, tail rec vs. while...) but useful to have sync way to pop ec2 instances and kill them

	import com.amazonaws.auth.{BasicAWSCredentials, InstanceProfileCredentialsProvider}
	import com.amazonaws.services.ec2.AmazonEC2Client
	import com.amazonaws.services.ec2.model._
	import com.amazonaws.util.Base64

	trait Ec2Utils extends Logging {
	val AWS_ACCESS_KEY_ID: String
	val AWS_SECRET_ACCESS_KEY: String
	val instanceProfile = false

djamelz / es.sh

Last active October 6, 2015 12:25

Script to install es 1.1 on ec2 ubuntu ami (sudo es.sh)

	#!/bin/sh

	cd /mnt/
	apt-get -y update
	apt-get -y install openjdk-7-jre
	apt-get -y install zip
	wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.1.1.zip && unzip elasticsearch-1.1.1.zip && /mnt/elasticsearch-1.1.1/bin/plugin -install mobz/elasticsearch-head
	/mnt/elasticsearch-1.1.1/bin/plugin -install elasticsearch/elasticsearch-cloud-aws/2.1.1
	echo " network.bind_host: $(ip addr \| grep 'state UP' -A2 \| tail -n1 \| awk '{print $2}' \| cut -f1 -d'/')" >> /mnt/elasticsearch-1.1.1/config/elasticsearch.yml
	screen -dmS new_screen sh

djamelz / FunWithAwsCli.txt

Last active October 6, 2015 07:45

Fun with aws-cli

	#pop emr cluster with spark job to execute

	aws emr create-cluster --name "<cluster name>"\
	--region us-east-1\
	--log-uri s3://<bucket>/logs/<cluster name>/\
	--enable-debugging\
	--release-label emr-4.0.0\
	--applications Name=Spark\
	--ec2-attributes KeyName=<keyname>\
	--use-default-roles\

djamelz / spark-ec2.py

Created November 16, 2014 00:27 — forked from amar-analytx/spark-ec2.py

	#!/usr/bin/env python
	# -- coding: utf-8 --

	#
	# Licensed to the Apache Software Foundation (ASF) under one
	# or more contributor license agreements. See the NOTICE file
	# distributed with this work for additional information
	# regarding copyright ownership. The ASF licenses this file
	# to you under the Apache License, Version 2.0 (the
	# "License"); you may not use this file except in compliance

Djamel Zouaoui djamelz