mzakaria mzakariaCERN

Grouping results and removing unwanted ones

Here we want to scrape product name, price and rating from ebay product pages:

url = 'https://www.ebay.com/itm/Sony-PlayStation-4-PS4-Pro-1TB-4K-Console-Black/203084236670' 

wanted_list = ['Sony PlayStation 4 PS4 Pro 1TB 4K Console - Black', 'US $349.99', '4.8'] 

scraper.build(url, wanted_list)

R to python data wrangling snippets

The dplyr package in R makes data wrangling significantly easier. The beauty of dplyr is that, by design, the options available are limited. Specifically, a set of key verbs form the core of the package. Using these verbs you can solve a wide range of data problems effectively in a shorter timeframe. Whilse transitioning to Python I have greatly missed the ease with which I can think through and solve problems using dplyr in R. The purpose of this document is to demonstrate how to execute the key dplyr verbs when manipulating data using Python (with the pandas package).

dplyr is organised around six key verbs:

	# upload a data frame to google drive, make it shareable, and
	# copy the shareable link into the clipboard

	# if direct = TRUE, the link can be used immediately to read in the file
	# if direct = FALSE, the link takes users to a nice preview of the file instead
	get_shareable_link_to_data <- function(data, path, direct = TRUE) {
	readr::write_csv(data, path)
	df <- googledrive::drive_upload(path, path)
	df <- googledrive::drive_share(df, role = "reader", type = "anyone")

	# From https://github.com/tpope/vim-pathogen with .vim replaced with vimfiles
	mkdir -p ~/vimfiles/autoload ~/vimfiles/bundle; \
	curl -so ~/vimfiles/autoload/pathogen.vim \
	https://raw.github.com/tpope/vim-pathogen/HEAD/autoload/pathogen.vim

	# Install some plugins
	cd ~/vimfiles/bundle
	git clone git://github.com/scrooloose/nerdtree.git
	git clone git://github.com/msanders/snipmate.vim.git