Rufus Pollock rufuspollock

Currently Data Packagese must have a name attribute but do not have an id attribute.

There has been debate about both the semantics (e.g. uniqueness) of the name field and its usability for certain cases (e.g. importing datasets into a new catalog) - see #220 for extensive discussions.

Proposal

Two identifier fields:

name: SHOULD be present (and certainly required for installation etc). Name is human meaningful and is designed to support both resolution (protocol to be determined) and easy use by humans e.g. in data dependencies
- (?) Have this as a MUST?

Challenge 1

Your task: write a script to get a nice CSV file of natural gas prices.

Please publish your results in a git repo or a gist. Please include both script and your resulting data -- so the CSV files should be stored in the repo too!

More detail:

Prices should be Henry Hub gas prices. Use EIA data here: http://www.eia.gov/dnav/ng/hist/rngwhhdm.htm
Hint: you can get the data from any data source on the page ...

HDX Common Humanitarian Dataset data into CKAN instance (we used datahub.io for convenience).

http://datahub.io/dataset/hdx-common-humanitarian-dataset

We've loaded (indicator) value table and indicator table separately in the CKAN DataStore (we have not bothered loading dataset table for the present) and we've also created a python script to automate this (which can also serve as an example of how to work with CKAN API).

Setting this up was pretty fast (most of the work was actually tidying up the data and then making some scripts to make this repeatable and testable).

[notitle] [fullwidth]

Get your local open data census here!
Local data is often the most relevant to citizens on a daily basis - from rubbish collection times to local tax rates.
At the moment it’s hard to know what local open data is available.

This is a short introduction to how to administer an Open Data Census.

Note: it assumes that a census instance has been booted for you (and is not about the technical side of deploying a census instance)

[toc]

Overview of How a Census is Structured

A Census is a survey built around 4 axes:

	# python 3+
	def convert_data_package_to_ckan_package(data_package):
	'''
	Documentation of CKAN metadata structure ...

	https://docs.ckan.org/en/2.8/api/index.html#ckan.logic.action.create.package_create
	https://docs.ckan.org/en/2.8/api/index.html#ckan.logic.action.create.resource_create
	'''
	out = dict(data_package)
	out['extras'] = []

	name: opsd-time-series

	title: Time series

	description: Load, wind and solar, prices in hourly resolution

	long_description: This data package contains different kinds of time series data relevant for power system modelling, namely electricity consumption (load) for 36 European countries as well as...

	homepage: http://data.open-power-system-data.org/time_series/2016-07-14/

	year,admin1,admin2,admin3,admin4,admin5,admin6,func1,func2,econ1,econ2,fin_source,exp_type,transfer,approved,adjusted,executed
	2009,Central,101 Parliament,0101 Parliament,,,010 Central apparatus (office) of ministries and other administrative authorities,01 General purpose state services,01.01 Legislative authorities,111 Remuneration of work,111.00 Remuneration of work,Base component,Personnel,Excluding transfers,30269300,30269300,27849186
	2009,Central,101 Parliament,0101 Parliament,,,010 Central apparatus (office) of ministries and other administrative authorities,01 General purpose state services,01.01 Legislative authorities,112 Mandatory state social insurance premiums,112.00 Mandatory state social insurance premiums,Base component,Personnel,Excluding transfers,5564000,5564000,5401021
	2009,Central,101 Parliament,0101 Parliament,,,010 Central apparatus (office) of ministries and other administrative authorities,01 General purpose state services,01.01 Legislative authorities,113 Payment for goods and servic

	// replace this with your CKAN website
	var ckanSite = 'http://datahub.io'

	var sql = 'Your SQL goes here';

	// =================
	// Using jQuery only
	// =================

	var data = encodeURIComponent(JSON.stringify({sql: sql}));

Rufus Pollock rufuspollock

Proposal

Challenge 1

Get your local open data census here! Local data is often the most relevant to citizens on a daily basis - from rubbish collection times to local tax rates. At the moment it’s hard to know what local open data is available.

Overview of How a Census is Structured

Get your local open data census here!
Local data is often the most relevant to citizens on a daily basis - from rubbish collection times to local tax rates.
At the moment it’s hard to know what local open data is available.