jjesusfilho

UPDATE: I have baked the ideas in this file inside a Python CLI tool called pyds-cli. Please find it here: https://github.com/ericmjl/pyds-cli

How to organize your Python data science project

Having done a number of data projects over the years, and having seen a number of them up on GitHub, I've come to see that there's a wide range in terms of how "readable" a project is. I'd like to share some practices that I have come to adopt in my projects, which I hope will bring some organization to your projects.

Disclaimer: I'm hoping nobody takes this to be "the definitive guide" to organizing a data project; rather, I hope you, the reader, find useful tips that you can adapt to your own projects.

Disclaimer 2: What I’m writing below is primarily geared towards Python language users. Some ideas may be transferable to other languages; others may not be so. Please feel free to remix whatever you see here!

jjesusfilho

// save to windows-user directory

linters: with_defaults(object_name_linter = NULL,
                       object_length_linter(50),
                       commented_code_linter = NULL,
                       object_usage_linter = NULL,
                       line_length_linter(120),
                       cyclocomp_linter = cyclocomp_linter(50))

jjesusfilho

import openai
import tiktoken
from scipy import spatial 
import pandas as pd

df=pd.read_csv('./data/oscars.csv')
print(df.head())

df=df.loc[df['year_ceremony'] == 2023]
df=df.dropna(subset=['film'])

jjesusfilho

Python Zeep SOAP Client

First, we'll look at the SOAP URL and see what Prefixes, Global Elements, Global Types, Bindings, and Services are provided by the SOAP Service.

You can do this by running zeep as a CLI tool.

export WSDL_URL="http://www.dneonline.com/calculator.asmx?WSDL"
python -m zeep $WSDL_URL

jjesusfilho

• Aeroporto
• Alameda
• Área
• Avenida
• Campo
• Chácara
• Colônia
• Condomínio
• Conjunto
• Distrito

jjesusfilho

PSQL

Magic words:

psql -U postgres

Some interesting flags (to see all, use -h or --help depending on your psql version):

• -E: will describe the underlaying queries of the \ commands (cool for learning!)
• -l: psql will list all databases and then exit (useful if the user you connect with doesn't has a default database, like at AWS RDS)

jjesusfilho

DROP TABLE IF EXISTS matrices;
CREATE TABLE matrices (
  matrix_id int NOT NULL, -- unique identifier for each matrix
  i int NOT NULL,         -- row number
  j int NOT NULL,         -- column number
  val decimal NOT NULL,   -- the value in this cell
  PRIMARY KEY (matrix_id, i, j)
);

-- insert sparse representation of

jjesusfilho

-- show running queries (pre 9.2)
SELECT procpid, age(clock_timestamp(), query_start), usename, current_query 
FROM pg_stat_activity 
WHERE current_query != '<IDLE>' AND current_query NOT ILIKE '%pg_stat_activity%' 
ORDER BY query_start desc;

-- show running queries (9.2)
SELECT pid, age(clock_timestamp(), query_start), usename, query 
FROM pg_stat_activity 
WHERE query != '<IDLE>' AND query NOT ILIKE '%pg_stat_activity%' 

jjesusfilho

Importing/Indexing database (MySQL or SQL Server) in Solr using Data Import Handler

Install Solr

download and install Solr from http://lucene.apache.org/solr/.

you can access Solr admin from your browser: http://localhost:8983/solr/

use the port number used in installation.

jjesusfilho

### Libraries
library(shiny)
library(dplyr)
library(DT)

### Data
input_data <- data.frame(Brand = c("Brand1", "Brand2","Brand3"),
                         ratio = c (.5, .5, .5),
                         cost = c(2000, 3000, 4000),
                         stringsAsFactors = FALSE) %>%