2019 update

Author: cmaimone

.gitignore

@@ -6,6 +6,15 @@ python/.ipynb_checkpoints/*
 
 .idea/
 
-handout.*
+handout*
+
+PostgreSQL-Cheat-Sheet.pdf
+
+datafiles/
+
+db_setup.txt
+
+*.tar
+
+netids.txt
 
-PostgreSQL-Cheat-Sheet.pdf

PITCHME.md

@@ -106,33 +106,7 @@ Tables are linked together with *keys* (relational model of data)
 
 ---
 
-# When to Use SQL/Relational Databases
-
----
-
-![good reason](https://imgs.xkcd.com/comics/algorithms.png)
-
----
-
-![bad reason](https://s-media-cache-ak0.pinimg.com/736x/8d/91/18/8d9118b4ffae7881453f34a645b66264--web-images-mauve.jpg)
-
----
-
-## When to Consider
-<hr>
-
-Current system is unmanageable
-
-A lot of duplicated information 
-
-Different types of *related* observations 
-
-Access or analyze a subset of observations
-
----
-
-## Other Factors
-<hr>
+## When are Databases Useful?
 
 Multiple people need access to the same, current data
 
@@ -142,16 +116,6 @@ Enforce constraints about data values, structure, and relationships
 
 Automatically maintain attributes of data like "last modified," "created date"
 
----
-
-## Maybe look elsewhere when
-<hr>
-
-Entire dataset is generated together
-
-Data is static and manageable size
-
-Data that would be in a table is used as a complete unit
 
 ---
 
@@ -171,21 +135,10 @@ Data that would be in a table is used as a complete unit
 
 ---
 
-## Relational data conventions
-<hr>
-
-**Primary keys** serve as ID to your primary data entities
-
-**Foreign keys** serve as ID in one table that identifies a row in another table (e.g. children -> parents)
-
-**Relationship tables** express many-to-many type relationships (e.g. Netflix account <-> movie data)
-
-
----
 
 ## Entity-relationship diagram example
 
-<img src="http://www.postgresqltutorial.com/wp-content/uploads/2013/05/PostgreSQL-Sample-Database.png" 
+<img src="http://www.postgresqltutorial.com/wp-content/uploads/2018/03/dvd-rental-sample-database-diagram.png" 
 width="500">
 
 

explore_db.pdf

@@ -269,7 +269,7 @@ R Markdown lets you execute SQL queries directly.  You first set up a `DBI` conn
 
 ````r
 `r ''````{r}
-library(RPostgreSQL)
+library(RPostgres)
 con <- dbConnect(RPostgres::Postgres(), host="localhost", dbname="dvdrental")
 ```
 ````

presentation_assets/joins.png

@@ -1,14 +1,42 @@
 # Databases
 
-This workshop uses PostgreSQL.  Much of the material applies generically to SQL and other relational database systems, but some of it is specific to PostgreSQL.
+There are four databases workshops that cover parts of the material in this repo.
 
-This workshop uses the database discussed in, and follows much of the content of, the [PostgreSQL Tutorial](http://www.postgresqltutorial.com/).
+The workshops use PostgreSQL.  Much of the material applies generically to SQL and other relational database systems, but some of it is specific to PostgreSQL.
 
-The workshop starts with the presentation below, then [SQL Part 1](sql/part1.md).
+Some of the workshops use the database discussed in, and follow much of the content of, the [PostgreSQL Tutorial](http://www.postgresqltutorial.com/), which makes use of the [Skalia database](https://www.jooq.org/sakila).
 
-## Presentation Materials
 
-[![GitPitch](https://gitpitch.com/assets/badge.svg)](https://gitpitch.com/nuitrcs/databases_workshop/)
+## Selecting and Joining Data
+
+[Intro presentation](https://gitpitch.com/nuitrcs/databases_workshop/)
+
+The workshop follows the files below, which have links to the exercises.
+
+[Part 1](sql/part1.md)
+
+[Part 2](sql/part2.md)
+
+
+
+## Exploring Data
+
+See [Exploring Data](sql/exploring.md) for materials and exercises.
+
+This workshop uses data files from the [nycflights13 R package](https://github.com/hadley/nycflights13) and open data from the city of Evanston.
+
+## Updating and Changing Data
+
+Uses materials from [Part 4](sql/part4.md), which are in the process of being updated/expanded.
+
+## Creating and Designing Databases
+
+[Part 3](sql/part3.md) materials are relevant, but materials for this workshop are currently being updated/expanded.  
+
+## R and Python
+
+The [R](/r) and [Python](/python) materials may also be of interest to those taking any of the above workshops.
+
 
 ## Note
 
@@ -22,21 +50,19 @@ See Section 1. Getting started with PostgreSQL, from the [PostgreSQL Tutorial](h
 
 While in-person workshop participants do not need to install PostgreSQL, you will need a terminal program capable of creating an SSH connection to a remote server.  On a Mac, the built-in Terminal program will work.  On Windows, we suggest [PuTTY](http://www.putty.org/) if you don't already have another program installed.
 
-Typing long commands in a terminal can be tedious.  We also recommend you install [DataGrip](https://www.jetbrains.com/datagrip/) for working with databases.  It has a free 30 day trial or you can apply for a JetBrains academic license for free.
-
-This repository also includes materials for connecting to a database using Python or R.  For Python, you will need to install the `psycopg2` package.  For R, you will need the package `RPostgreSQL`.
+This repository also includes materials for connecting to a database using Python or R.  For Python, you will need to install the `psycopg2` package.  For R, you will need the package `RPostgres`.
 
-## Resources
+# Resources
 
-### Background
+## Background
 
 [Basic Explanation of Relational Databases](http://www.bbc.co.uk/education/guides/ztsvb9q/revision/1): from the BBC, a quick explanation of relational databases
 
-### Software
+## Software
 
 [DataGrip Tutorial](https://www.youtube.com/watch?v=Xb9K8IAdZNg): video on how to use the DataGrip program; it even uses the same database we use in this workshop.
 
-### Reference
+## Reference
 
 [PostgreSQL cheat sheet](http://www.postgresqltutorial.com/wp-content/uploads/2018/03/PostgreSQL-Cheat-Sheet.pdf): a list of basic commands and patterns for statements
 
@@ -46,11 +72,11 @@ This repository also includes materials for connecting to a database using Pytho
 
 [psql commands cheat sheet](http://www.postgresonline.com/downloads/special_feature/postgresql83_psql_cheatsheet.pdf): describe commands, other slash commands
 
-### Additional Exercises/Tutorials
+## Additional Exercises/Tutorials
 
 _These resources use PostgreSQL or SQL generally._
 
-[Mode SQL Tutorials](https://mode.com/sql-tutorial/)
+[Mode SQL Tutorials](https://mode.com/sql-tutorial/): good introduction, and you can try running queries on their platform
 
 [PostgreSQL Exercises](https://pgexercises.com/): interactive, online exercises to practice SQL skills in a PostgreSQL environment.  
 
@@ -70,17 +96,4 @@ _These resources use PostgreSQL or SQL generally._
 
 [SQL for Data Analysis](https://www.udacity.com/course/sql-for-data-analysis--ud198#) from Udacity - an online self-paced course
 
-### SQLite
-
-The workshop uses the PostgreSQL database system, but if you're working on your own projects, [SQLite](https://www.sqlite.org/) may be a good option.  SQLite doesn't require running a server and it creates a database in a single, portable file locally on your computer.
-
-[Intro to SQL with SQLite](https://github.com/tthibo/SQL-Tutorial)
-
-[Software Carpentry Databases and SQL](http://swcarpentry.github.io/sql-novice-survey/): introductory workshop using SQLite
-
-[Databases with Python and Pandas](https://www.dataquest.io/blog/python-pandas-databases/): from Data Quest.  Examples of using a SQLite database with Python and pandas too.
-
-
-### Other
 
-[Programming for Biologists](http://www.programmingforbiologists.org/exercises/): includes a section on databases; it uses MS Access instead of PostgreSQL or SQLite, but many of the concepts should be the same.

r/r_databases.Rmd

@@ -0,0 +1,366 @@
+# Exercise Answers
+
+Answers to some of the exercises in [explore.md](explore.md).
+
+
+## Exploring a database
+
+### WITH
+
+Select all of the flights made by the oldest plane in the data.  
+
+Hint: Get the year of the oldest plane with
+
+```sql
+SELECT min(year) 
+FROM planes;
+```
+
+Then get the tailnum for this plane.  Then get the flights.  Use a WITH clause in your query.
+
+```sql
+WITH minyear AS 
+  (SELECT min(year) AS year FROM planes),
+oldplane AS 
+  (SELECT tailnum FROM planes, minyear WHERE planes.year = minyear.year)
+SELECT * 
+FROM flights, oldplane
+WHERE flights.tailnum = oldplane.tailnum;
+```
+
+
+### More exploring
+
+Write a query to find the names of all businesses that had a violation of type `'(6) FOOD PROTECTION: Potentially hazardous food properly thawed.'`.
+
+```sql
+SELECT name
+FROM business
+INNER JOIN violations
+ON business.license = violations.license
+WHERE violation = '(6) FOOD PROTECTION: Potentially hazardous food properly thawed.';
+```
+
+Select the business name, violation type, and comment for only violations that were found on the most recent inspection of a business.
+
+```sql
+SELECT name, violation, comments
+FROM violations
+INNER JOIN business
+ON business.license = violations.license
+WHERE date = last_inspection;
+```
+
+Select violations for inspections where the inspection score was below 80.  Select an appropriate/informative subset of the columns.  Order the results in an meaningful order.  Hint: you need to use both the business license and inspection date information to join the tables properly.  
+
+```sql
+SELECT name, inspections.date, score, violation
+FROM inspections
+INNER JOIN business
+ON inspections.license = business.license
+INNER JOIN violations
+ON violations.license = business.license AND violations.date = inspections.date
+WHERE score < 80
+ORDER BY name, date;
+```
+
+Bonus: What is the most common violation type in inspections with total scores less than 80? 
+
+```sql
+SELECT violation, count(*)
+FROM inspections
+INNER JOIN business
+ON inspections.license = business.license
+INNER JOIN violations
+ON violations.license = business.license AND violations.date = inspections.date
+WHERE score < 80
+GROUP BY violation
+ORDER BY count DESC;
+```
+
+
+## Data Types
+
+Select `time_hour` and `time_hour` cast as date from `weather`; limit to a few rows.
+
+```sql
+SELECT time_hour, time_hour::date 
+FROM weather
+LIMIT 20;
+```
+
+## Missing Rows
+
+Are there days for any airports where there are no weather observations at all?
+
+Hint: one way to do this is to see what the normal number of days per airport is in the data (if there is a normal value), and then look for abnormal values.
+
+Hint 2: Look at the subquery section above for an example query that might be useful.
+
+
+```sql
+WITH DISTINCT_days AS 
+  (SELECT DISTINCT origin, year, month, day
+   FROM weather)
+SELECT origin, count(*) 
+FROM DISTINCT_days
+GROUP BY origin
+ORDER BY count DESC;
+```
+
+Bonus exercise: Which hour of the day is most likely to be missing a weather observation?
+
+```sql
+SELECT hour, count(*)
+FROM weather
+GROUP BY hour
+ORDER BY count;
+```
+
+The midnight (hour 0) hour has the fewest observations in the data, so is therefore the most likely to be missing.
+
+Bonus exercise 2: Are there any duplicate measurements (same airport and time) in the weather data?
+
+```sql
+SELECT origin, year, month, day, hour, count(*) 
+FROM weather
+GROUP BY origin, year, month, day, hour
+HAVING count(*) > 1;
+```
+
+
+
+## LIKE
+
+How many Starbucks are in the Evanston inspection data?
+
+```sql
+SELECT count(*) 
+FROM business
+WHERE name LIKE 'Starbucks%';
+```
+
+How many distinct violation types have FOOD in the violation name?
+
+```sql
+SELECT count(DISTINCT violation)
+FROM violations
+WHERE violation LIKE '%FOOD%';
+```
+
+Which violation type is most likely to be a critical violation (see the comments)?
+
+```sql
+SELECT violation, count(*) 
+FROM violations
+WHERE comments LIKE '%CRITICAL VIOLATION%'
+GROUP BY violation
+ORDER BY count; 
+```
+
+Find any violation entries that do not conform to the pattern of `(#) VIOLATION TITLE: violation description`
+
+```sql
+SELECT DISTINCT violation
+FROM violations
+WHERE violation NOT LIKE '(%) %: %';
+```
+
+Challenge: Get all addresses where the street name starts with A (and only those addresses).
+
+
+```sql
+SELECT address
+FROM business
+WHERE address LIKE '% A% AVE';
+```
+
+Note that the solution above is specific to the values in the data.  There are no "Streets" that start with A, only "Avenues",  We'd have to change the query if the data was different.
+
+
+
+
+
+
+
+
+
+
+
+## String Splitting
+
+Get just the street name from the business address (the result won't be perfect at this stage -- just do what you can with a simple query).  Group and count to see which street has the most food businesses.
+
+Let's take this in steps
+
+```sql
+SELECT split_part(address, ' ', 2)
+FROM business;
+```
+
+What are the unique values?  (To check)
+
+```sql
+SELECT DISTINCT split_part(address, ' ', 2)
+FROM business;
+```
+
+Which are most common?
+
+```sql
+SELECT DISTINCT split_part(address, ' ', 2) AS street, count(*)
+FROM business
+GROUP BY street
+ORDER BY count DESC;
+```
+
+## Numerical Functions
+
+How much total precipitation did each airport get by month?  Hint: Use `sum()`, and you'll need to group by several columns.  Order the results in a reasonable order.
+
+```sql
+SELECT origin, month, sum(precip) 
+FROM weather
+GROUP BY origin, month
+ORDER BY origin, month;
+```
+
+You could include year above, but since everything is 2013, it's not necessary.
+
+```sql
+SELECT month, round(avg(precip), 2)
+FROM (SELECT origin, month, sum(precip) AS precip
+      FROM weather
+      GROUP BY origin, month) AS monthly
+GROUP BY month
+ORDER BY month;
+```
+
+## Numerical Distributions
+
+Examine the distribution of temperature like we did with humidity.  Then do it by airport (origin).  
+
+```sql
+SELECT trunc(temp, -1), count(*)
+FROM weather
+GROUP BY trunc(temp, -1)
+ORDER BY trunc(temp, -1);
+```
+
+```sql
+SELECT origin, trunc(temp, -1), count(*)
+FROM weather
+GROUP BY origin, trunc(temp, -1)
+ORDER BY origin, trunc(temp, -1);
+```
+
+
+## Date Part Functions
+
+Which day of the week had the most rain at JFK airport?
+
+```sql
+SELECT date_part('dow', time_hour) AS dow, sum(precip) 
+FROM weather
+WHERE origin='JFK'
+GROUP BY dow
+ORDER BY sum(precip) DESC;
+```
+
+What week of the year in 2018 were the most food inspections done?
+
+```sql
+SELECT date_part('week', date) AS week, count(*)
+FROM inspections
+WHERE date >= '2018-01-01' AND date <= '2018-12-31'
+GROUP BY week
+ORDER BY count DESC;
+```
+
+Bonus (Hard, because we didn't learn everything you need): What days had no inspections?  The `generate_series()` function can be used to create a series of dates.  
+
+Example:
+
+```sql
+SELECT generate_series('2018-01-01', '2019-05-14', '1 day'::interval) AS days;
+```
+
+The above are timestamps, but they can be compared to dates.
+
+Can you use the above to help you find dates with no inspections?  
+
+
+There are a few options:
+
+```sql
+SELECT date 
+FROM generate_series('2018-01-01', '2019-05-14', '1 day'::interval) AS days
+LEFT JOIN inspections
+ON days = inspections.date
+WHERE inspections.date IS NULL;
+```
+
+OR 
+
+```sql
+WITH days AS 
+     (SELECT * 
+      FROM generate_series('2018-01-01', 
+         '2019-05-14', '1 day'::interval) AS day)
+SELECT day
+FROM days 
+WHERE day NOT IN (SELECT date FROM INSPECTIONS);
+```
+
+What days of the week are days without inspections?  Any that aren't on the weekend?
+
+These use the WITH option, but you can do this as well with other versions of the query.
+
+```sql
+WITH days AS (SELECT * FROM generate_series('2018-01-01', '2019-05-14', '1 day'::interval) AS day)
+SELECT day, date_part('dow', day)
+FROM days 
+WHERE day NOT IN (SELECT date FROM INSPECTIONS);
+```
+
+```sql
+WITH days AS (SELECT * FROM generate_series('2018-01-01', '2019-05-14', '1 day'::interval) AS day)
+SELECT day, date_part('dow', day)
+FROM days 
+WHERE day NOT IN (SELECT date FROM INSPECTIONS) 
+AND date_part('dow', day) NOT IN (0,6);
+```
+
+
+
+## Lag
+
+We don't just have to lag dates themselves -- we can also lag other columns by their date values.  Let's get the changes in temperature from one hour to the next at JFK.  And the find the biggest change.  
+
+Remember to limit to just rows for JFK, and order by time_hour overall.  And you'll want the absolute value of the difference with function `abs`.
+
+```sql
+WITH changes AS 
+   (SELECT time_hour, temp,
+       lag(temp) OVER (ORDER BY time_hour) AS lagged,
+       abs(temp-lag(temp) OVER (ORDER BY time_hour)) AS diff
+    FROM weather
+    WHERE origin='JFK'
+    ORDER BY time_hour)
+SELECT time_hour, temp, lagged, diff 
+FROM changes
+WHERE diff = (SELECT max(diff) FROM changes);
+```
+
+This seems impossible, so let's take a look:
+
+```sql
+SELECT *
+FROM weather
+WHERE origin='JFK'
+AND date_trunc('day', time_hour) = '2013-05-09';
+```
+
+Looks like bad data entry.
+
+

r/r_databases.html

@@ -1,5 +1,23 @@
 # SQL Part 1
 
+* [Connecting](#connecting)
+* [Select basics](#select)
+* [`LIMIT`](#limit)
+* [`OFFSET`](#offset)
+* [`WHERE`](#where)
+    - [`BETWEEN`](#between)
+    - [`IN`](#in)
+    - [`LIKE`](#like)
+    - [`IS NULL`](#is-null)
+* [`ORDER BY`](#order-by)
+* [`DISTINCT`](#distinct)
+* [Functions and Arithmetic](#functions-and-arithmetic)
+* [`GROUP BY`](#group-by)
+* [`HAVING`](#having)
+* [Dates](#dates)
+
+
+
 ## Connecting
 
 To connect to a database, you need the following information:
@@ -10,10 +28,9 @@ To connect to a database, you need the following information:
 * Password: 
 * Port: 5432 (default for PostgreSQL)
 
-You also need a client program to connect to the database.  It's suggested that you start with `psql`, which is a command-line client.  Output below is from `psql`.
-
-In the workshop, you only have permission to read data from the dvdrental database, not add, update, or delete data, or modify the database.  When we get to changing databases, you'll have access to a database you have permissions to modify.
+You also need a client program to connect to the database.  Output below is from a command line client called `psql`.  
 
+In the introductory workshop, you only have permission to read data from the dvdrental database, not add, update, or delete data, or modify the database.  
 
 # Database Schema
 
@@ -23,7 +40,9 @@ The schema (the set of tables, their columns and types, and the relationships be
 
 We can also get information directly from the database itself.  These commands are specific to each database system.  The commands below are for PostgreSQL.  
 
-First, use `\d` to get a list of relations (tables, views, sequences) -- we'll talk about what views and sequences are later.  
+## `psql` describe commands
+
+With `psql`, you can use `\d` to get a list of relations (tables, views, sequences).
 
 ```sql
 \dvdrental=# \d
@@ -127,15 +146,51 @@ There are also other `\d` describe functions, among them:
 
 You can get a complete list of backslash commands with `\?`.
 
+
+## Other programs
+
+Each database client will have its own way to show you information about the tables in the database.  
+
+## Useful Notes
+
+A useful setting for psql is:
+
+```sql
+\pset format wrapped
+```
+
+Comments in SQL files:
+
+```sql
+/* this is a comment; it can 
+span multiple lines */
+
+SELECT * FROM actor; /* this is also a comment */
+
+-- this is a single line comment
+
+SELECT * from actor; -- another single line comment
+```
+
+
+
+
 # `SELECT`
 
-Select is the command we use most often in SQL.  It let's us select data (specified rows and columns) from one or more tables.  Columns are selected by name, rows are selected with conditional statements (values of a particular column meeting some criteria).  
+Select is the command we use most often in SQL.  It lets us select data (specified rows and columns) from one or more tables.  Columns are selected by name, rows are selected with conditional statements (values of a particular column meeting some criteria).  
 
 The basic format of a `SELECT` command is 
 
 ```sql
-SELECT column_1, column_2 
-FROM table1;
+SELECT <columns>
+FROM <table>;
+```
+
+For example:
+
+```sql
+SELECT actor_id, first_name
+FROM actor;
 ```
 
 `SELECT` and `FROM` are reserved keywords.  SQL is case-insensitive, but many times you'll see the key terms in all caps.  Note that you use a semicolon `;` to end the statement.  You can also split a SQL statement across multiple lines -- the space between the terms doesn't matter (a new line counts as space).  
@@ -285,7 +340,7 @@ SELECT * FROM customer WHERE store_id=2;
 
 (_Note: Going forward, output will only be included when there's something about it to discuss._)
 
-You can combined conditions together with `AND` and `OR`:
+You can combine conditions together with `AND` and `OR`:
 
 ```sql
 SELECT * FROM customer WHERE store_id=2 AND customer_id=400;
@@ -346,61 +401,13 @@ SELECT * FROM film WHERE film_id IN (3,5,7,9);
 
 Select rows from the actor table where the first name is Angela, Angelina, or Audrey using `IN`.
 
-### `LIKE`
+---
 
-`LIKE` lets you do pattern matching on strings.  The only two pattern characters are `_` for a single character and `%` for any number of characters (including none).  In some implementations of SQL, `LIKE` is case-insensitive.  In PostgreSQL, it is case-sensitive; `ILIKE` is the PostgreSQL case-insensitive version.
+Break for exercises: [part1_exercises.md](part1_exercises.md) - sections Describe Commands and Select.
 
-Get names of actors that start with A.
-
-```sql
-SELECT * FROM actor WHERE first_name LIKE 'A%';
-```   
+---
 
-Note that the following will yield no results:
 
-```sql
-SELECT * FROM actor WHERE first_name LIKE 'a%';
-``` 
-
-But the following is ok:
-
-```sql
-SELECT * FROM actor WHERE first_name ILIKE 'a%';
-``` 
-
-Get 4 letter names starting with A:
-
-```sql
-SELECT * FROM actor WHERE first_name LIKE 'A___';
-```
-
-Get names that end with y:
-
-```sql
-SELECT * FROM actor WHERE first_name LIKE '%y';
-```
-
-Any names with a z in them?
-
-```sql
-SELECT * FROM actor WHERE first_name ILIKE '%z%';
-```
-
-```sql
-dvdrental=# SELECT * FROM actor WHERE first_name ILIKE '%z%';
- actor_id | first_name | last_name |      last_update       
-----------+------------+-----------+------------------------
-       11 | Zero       | Cage      | 2013-05-26 14:47:57.62
-       89 | Charlize   | Dench     | 2013-05-26 14:47:57.62
-      121 | Liza       | Bergman   | 2013-05-26 14:47:57.62
-(3 rows)
-```
-
-See that we have results where Z is the first letter (since % can match 0 characters) as well as results where there's a z in the middle of the name.
-
-### Exercise
-
-Select rows from the city table where city starts with a B.
 
 ### `IS NULL`
 
@@ -427,6 +434,8 @@ SELECT * FROM address WHERE address2 = NULL;
 `NULL` values are omitted from the results of comparison tests. 
 
 
+
+
 ## `ORDER BY`
 
 We can determine the order that our results are shown in:
@@ -477,6 +486,11 @@ SELECT DISTINCT customer_id, staff_id FROM payment;
 SELECT DISTINCT amount FROM payment ORDER BY amount;
 ```
 
+---
+
+Break for exercises: [part1_exercises.md](part1_exercises.md) - sections for Like, Distinct, and Order by exercises
+
+---
 
 
 ## Functions and Arithmetic
@@ -653,38 +667,18 @@ dvdrental=# select rental_date from rental where rental_date<'2005-05-25';
 (8 rows)
 ```
 
-This will get you everything before 2005-05-25 00:00:00.  If you want just the date part of a datetime:
+This will get you everything before 2005-05-25 00:00:00.  
 
-```sql
-SELECT rental_date FROM rental 
-WHERE date_trunc('day', rental_date) = '2005-05-24';
-```
+TODO: check date equality
 
-or
+---
 
-```sql
-SELECT rental_date FROM rental
-WHERE cast(rental_date as date) = '2005-05-24';
-```
+Break for exercises: [part1_exercises.md](part1_exercises.md) - remaining sections
 
-or 
+---
 
-```sql
-SELECT rental_date FROM rental 
-WHERE rental_date::date = '2005-05-24';
-```
 
-# Comments
 
-Comments in SQL files:
 
-```sql
-/* this is a comment; it can 
-span multiple lines */
 
-SELECT * FROM actor; /* this is also a comment */
 
--- this is a single line comment
-
-SELECT * from actor; -- another single line comment
-```

readme.md

@@ -23,31 +23,34 @@ Select the row from customer for customer named Jamie Rice.
 
 Select amount and payment_date from payment where the amount paid was less than $1.  
 
-What are the different rental durations that the store allows?
 
-## Exercise: Counting
 
-How many films are rated NC-17?  How many are rated PG or PG-13?
+## Exercise: Distinct
+
+What are the different rental durations that the store allows?
+
 
 
-Challenge: How many different customers have entries in the rental table?  [Hint](http://www.w3resource.com/sql/aggregate-functions/count-with-distinct.php)
 
 ## Exercise: Order By
 
 What are the IDs of the last 3 customers to return a rental?
 
 
-## Exercise: Like
 
-Select film title that have "Dragon" in them.
+## Exercise: Counting
+
+How many films are rated NC-17?  How many are rated PG or PG-13?
+
+
+Challenge: How many different customers have entries in the rental table?  [Hint](http://www.w3resource.com/sql/aggregate-functions/count-with-distinct.php)
+
 
-Challenge: only select titles that have just the word "Dragon" (not "Dragonfly") in them.
 
 ## Exercise: Group By
 
 Does the average replacement cost of a film differ by rating?
 
-Which store (`store_id`) has the most customers whose first name starts with M?
 
 Challenge: Are there any customers with the same last name? 
 

sql/explore.md

@@ -42,7 +42,7 @@ Select the row from customer for customer named Jamie Rice.
 
 Select amount and payment_date from payment where the amount paid was less than $1.  
 
-What are the different rental durations that the store allows?
+
 
 #### Solution
 
@@ -75,43 +75,15 @@ WHERE amount < 1;
 ```
 
 
-```sql
-SELECT DISTINCT rental_duration FROM film;
-```
-
-
-## Exercise: Counting
-
-How many films are rated NC-17?  How many are rated PG or PG-13?
-
-
-Challenge: How many different customers have entries in the rental table?  [Hint](http://www.w3resource.com/sql/aggregate-functions/count-with-distinct.php)
-
-#### Solution
-
-```sql
-SELECT count(*) FROM film 
-WHERE rating = 'NC-17';
-```
-
-
-```sql
-SELECT count(*) FROM film 
-WHERE rating in ('PG', 'PG-13');
-```
+## Exercise: Distinct
 
-or 
+What are the different rental durations that the store allows?
 
 ```sql
-SELECT count(*) FROM film 
-WHERE rating = 'PG' OR rating = 'PG-13';
+SELECT DISTINCT rental_duration FROM film;
 ```
 
-Challenge:
 
-```sql
-SELECT COUNT(DISTINCT customer_id) FROM rental;
-```
 
 
 
@@ -131,41 +103,49 @@ LIMIT 3;
 
 
 
-## Exercise: Like
 
-Select film title that have "Dragon" in them.
 
-Challenge: only select titles that have just the word "Dragon" (not "Dragonfly") in them.
+## Exercise: Counting
+
+How many films are rated NC-17?  How many are rated PG or PG-13?
+
+
+Challenge: How many different customers have entries in the rental table?  [Hint](http://www.w3resource.com/sql/aggregate-functions/count-with-distinct.php)
 
 #### Solution
 
 ```sql
-SELECT title FROM film 
-WHERE title like '%Dragon%';
+SELECT count(*) FROM film 
+WHERE rating = 'NC-17';
 ```
 
-Challenge:
 
-This solution only works because the titles are all two words, so we can just check the beginning or end of the title:
+```sql
+SELECT count(*) FROM film 
+WHERE rating in ('PG', 'PG-13');
+```
+
+or 
 
 ```sql
-SELECT title FROM film 
-WHERE title like '% Dragon' 
-OR title like 'Dragon %';
+SELECT count(*) FROM film 
+WHERE rating = 'PG' OR rating = 'PG-13';
 ```
 
-For a more general purpose solution, you would need to use [regular expressions](https://www.postgresql.org/docs/current/static/functions-matching.html) to match word boundaries, but we're not going to cover that here.
+Challenge:
 
 ```sql
-SELECT title FROM film                                                                                         WHERE title ~ '.*\mDragon\M.*';
+SELECT COUNT(DISTINCT customer_id) FROM rental;
 ```
 
 
+
+
+
 ## Exercise: Group By
 
 Does the average replacement cost of a film differ by rating?
 
-Which store (`store_id`) has the most customers whose first name starts with M?
 
 Challenge: Are there any customers with the same last name? 
 
@@ -176,11 +156,6 @@ SELECT rating, avg(replacement_cost) FROM film
 GROUP BY rating;
 ```
 
-```sql
-SELECT store_id, count(*) FROM customer
-WHERE first_name LIKE 'M%' GROUP BY store_id;
-```
-
 Challenge:
 
 ```sql

sql/explore_answers.md

@@ -2,6 +2,14 @@
 
 This section first covers the topics of aliasing and subqueries, then we get to joining tables, which is the real power of relational databases.
 
+* [Aliasing](#aliasing)
+* [Subqueries](#subqueries)
+* [Joins](#joins)
+    - [`INNER JOIN`](#inner-join)
+    - [`LEFT JOIN`](#left-join)
+    - [`FULL OUTER JOIN`](#full-outer-join)
+
+
 # Aliasing
 
 You can rename columns and tables in queries.  This will mostly be useful when we're joining tables together, but it can also be useful when you're working with functions.
@@ -22,6 +30,8 @@ In the output above, the name of the column is the alias.
 
 One important note is that _column_ aliases can't be used in where or having clauses:
 
+TODO: check
+
 ```sql
 SELECT title, rating AS rate
 FROM film 
@@ -58,7 +68,7 @@ WHERE rental_rate < (SELECT avg(rental_rate) FROM film)
 ORDER BY rental_rate DESC;
 ```
 
-The subquery is executed first, and the the result is used the broader query.
+The subquery is executed first, and then the result is used the broader query.
 
 We can also use subqueries with `IN`.  Find customers with an address in `postal_code` 35200
 
@@ -78,9 +88,11 @@ SELECT count(customer_id) FROM
  HAVING count(*) > 30) AS foo;
 ```
 
+`foo` is a common throwaway name that gets used -- you can pick any name you want for the alias though.
+
 ## Exercise
 
-Find the title of movies that have the maximum replacement fee.
+Find the titles of movies that have the maximum replacement fee.
 
 
 # Joins
@@ -100,7 +112,8 @@ The first and most common type of join is called an inner join.  You specify the
 Let's start with the example we just used above: customers with postal code 52137. To start with, how do we join the tables generally:
 
 ```sql
-SELECT * FROM customer INNER JOIN address 
+SELECT * FROM customer 
+INNER JOIN address 
 ON customer.address_id = address.address_id;
 ```
 
@@ -109,12 +122,26 @@ This matches up the customer to the full address information.
 Then we can select a specific postal code if we want:
 
 ```sql
-SELECT * FROM customer INNER JOIN address 
+SELECT * FROM customer 
+INNER JOIN address 
 ON customer.address_id = address.address_id
 WHERE postal_code='52137';
 ```
 
-Note that both tables have a column called `address_id`.  We add the table name to the front of the column name when referencing them.  You can do this anytime, but typically only do it when you're joining and there's ambiguity.  
+Note that both tables have a column called `address_id`.  We add the table name to the front of the column name when referencing them.  You can do this anytime, but typically only do it when you're joining and there's ambiguity. 
+
+We can also group by, order by, and use other where clause conditions on the joined tables.  For example, we can count the customers in each postal code.
+
+```sql
+SELECT postal_code, count(*) 
+FROM customer 
+INNER JOIN address 
+ON customer.address_id = address.address_id
+GROUP BY postal_code
+ORDER BY count;
+``` 
+
+TODO: check above
 
 ### Alternative Syntax
 
@@ -150,24 +177,26 @@ LIMIT 10;
 
 Join the store table to the address table to add the address information to the store information.
 
-Select film\_id, category\_id, and name from joining the film\_category and category tables, only where the category\_id is less than 10.
+
 
 ### Table Names and Aliases
 
 We can alias tables as well as columns.  If a column name appears in both tables, then we have to specify the table name when selecting it.
 
 ```sql 
 SELECT first_name, last_name, customer.address_id, postal_code 
-FROM customer, address
-WHERE customer.address_id = address.address_id;
+FROM customer
+INNER JOIN address
+ON customer.address_id = address.address_id;
 ```
 
 If we don't put a table name in front of `address_id` we get an error:
 
 ```sql
 dvdrental=# SELECT first_name, last_name, address_id, postal_code 
-dvdrental-# FROM customer, address
-dvdrental-# WHERE customer.address_id = address.address_id;
+dvdrental-# FROM customer
+dvdrental-# INNER JOIN address
+dvdrental-# ON customer.address_id = address.address_id;
 ERROR:  column reference "address_id" is ambiguous
 LINE 1: SELECT first_name, last_name, address_id, postal_code 
                                       ^
@@ -177,13 +206,31 @@ To make the references easier, it's common to alias table names
 
 ```sql 
 SELECT first_name, last_name, c.address_id, postal_code 
-FROM customer AS c, address AS a
-WHERE c.address_id = a.address_id;
+FROM customer AS c
+INNER JOIN address AS a
+ON c.address_id = a.address_id;
+```
+
+and we often drop the `AS`:
+
+```sql 
+SELECT first_name, last_name, c.address_id, postal_code 
+FROM customer c
+INNER JOIN address a
+ON c.address_id = a.address_id;
 ```
 
 The _table_ aliases can be used in the where clause as well as the select part of the statement.
 
-The 'AS' can also be dropped in an alias.  We'll do this below.
+---
+
+Break for exercises: [part2_exercises.md](part2_exercises.md) - Subqueries, Inner Joins, and Joining and Grouping: Customer Spending
+
+---
+
+
+
+
 
 ### More than 2 Tables
 
@@ -192,7 +239,8 @@ We can join more than 2 tables together.  Let's match the names of actors with t
 
 ```sql
 SELECT title, first_name, last_name 
-FROM film f INNER JOIN film_actor fa ON f.film_id=fa.film_id 
+FROM film f 
+INNER JOIN film_actor fa ON f.film_id=fa.film_id 
 INNER JOIN actor a ON fa.actor_id=a.actor_id;
 ```
 
@@ -211,21 +259,29 @@ Join store, address, and city tables to show the store\_id, address, and city na
 
 ## `LEFT JOIN`
 
-With an inner join, we only get the results that are in both tables.  But sometimes we want to know which rows in a table don't have a match in the other table.  For this we can use a `LEFT JOIN` or `RIGHT JOIN` (depending on which table you want all of the results from).
+With an inner join, we only get the results that are in both tables.  But there are other types of joins.
+
+![](presentation_assets/joins.png)
+
+
+
+If we want to know which rows in a table don't have a match in the other table, we use a `LEFT JOIN` or `RIGHT JOIN` (depending on which table you want all of the results from).
 
 In the dvd database, there can be films that don't have an inventory record.  We don't want these to be dropped from our results of joining the film and inventory tables.  Start with the join.  
 
 ```sql
 SELECT f.film_id, title, inventory_id, store_id 
-FROM film f LEFT JOIN inventory i
+FROM film f 
+LEFT JOIN inventory i
 ON f.film_id=i.film_id;
 ```
 
 Now find the rows where there isn't matching inventory:
 
 ```sql
 SELECT f.film_id, title, inventory_id, store_id 
-FROM film f LEFT JOIN inventory i
+FROM film f 
+LEFT JOIN inventory i
 ON f.film_id=i.film_id
 WHERE i.film_id IS NULL;
 ```
@@ -242,29 +298,10 @@ A `FULL OUTER JOIN` is like doing a left and right join at the same time: you ge
 There aren't any tables with this type of relationship to each other in the dvdrental database, so we aren't going to do an example here.  The syntax is the same as the other joins.  
 
 
-# Views 
-
-A view is a virtual table that has the results of a query in it (a result set).  You give it a name like you would a table, and you can use it like a table.  It's useful when you have common views of the data that you need to access often.  It's a way to save common queries, particularly ones that are long or complicated.
 
-We can list views with 
 
-```sql
-\dv
-```
-
-And then select from them like a table
-
-```sql
-select * from actor_info limit 5;
-```
-
-You can create views with `CREATE VIEW` and a select query: 
-
-```sql
-CREATE VIEW named_film_actor AS 
-SELECT f.film_id, title, a.actor_id, first_name, last_name 
-FROM film f, film_actor fa, actor a
-WHERE f.film_id=fa.film_id AND fa.actor_id=a.actor_id;
-```
+---
 
+Break for exercises: [part2_exercises.md](part2_exercises.md) - Remaining Sections
 
+---

sql/part1.md

@@ -1,7 +1,7 @@
 # SQL Part 2: Exercises
 ----
 
-There may be other ways to achieve the same result.  Remember that SQL commands are not case sensitive (but data values are).
+There may be other ways to achieve the same result.  Remember that SQL commands are not case sensitive (but data values and table and column names are).
 
 All of these exercises use the `dvdrental` database.  
 
@@ -15,28 +15,36 @@ What films are actors with ids 129 and 195 in together?
 Challenge: How many actors are in more films than actor id 47?  Hint: this takes 2 subqueries (one nested in the other).  Work inside out: 1) how many films is actor 47 in; 2) which actors are in more films than this? 3) Count those actors.
 
 
+## Exercise: Inner Joins
 
-## Exercise: Joining Customers, Payments, and Staff
+Select `first_name`, `last_name`, `amount`, and `payment_date` by joining the customer and payment tables.  
+
+Select film\_id, category\_id, and name from joining the film\_category and category tables, only where the category\_id is less than 10.
+
+
+## Exercise: Joining and Grouping: Customer Spending
+
+Get a list of the names of customers who have spent more than $150, along with their total spending.
+
+Who is the customer with the highest average payment amount?
 
-Join the customer and payment tables together with an inner join; select customer id, name, amount, and date and order by customer id.  Then join the staff table to them as well to add the staff's name.  
 
 ## Exercise: Joining for Better Addresses
 
 Create a list of addresses that includes the name of the city instead of an ID number and the name of the country as well.   
 
+## Exercise: Joining Customers, Payments, and Staff
 
-## Exercise: Joining and Grouping
+Join the customer and payment tables together with an inner join; select customer id, name, amount, and date and order by customer id.  Then join the staff table to them as well to add the staff's name.  
 
-Repeating an exercise from Part 1, but adding in information from additional tables:  Which film (_by title_) has the most actors?  Which actor (_by name_) is in the most films?
 
-Challenge: Which two actors have been in the most films together?  Hint: You can join a table to itself by including it twice with different aliases.  Hint 2: Try writing the query first to find the answer in terms of actor ids (not names); then for a super challenge (it takes a complicated query), rewrite it to get the actor names instead of the IDs.  Hint 3: make sure not to count pairs twice (a in the movie with b and b in the movie with a) and avoid counting cases of an actor being in a movie with themselves.
+## Exercise: Joining and Grouping: Films and Actors
 
+Repeating an exercise from Part 1, but adding in information from additional tables:  Which film (_by title_) has the most actors?  Which actor (_by name_) is in the most films?
 
-## Exercise: Joining and Grouping 2
+Challenge: Which two actors have been in the most films together?  Hint: You can join a table to itself by including it twice with different aliases.  Hint 2: Try writing the query first to find the answer in terms of actor ids (not names); then for a super challenge (it takes a complicated query), rewrite it to get the actor names instead of the IDs.  Hint 3: make sure not to count pairs twice (a in the movie with b and b in the movie with a) and avoid counting cases of an actor being in a movie with themselves.
 
-Get a list of the names of customers who have spent more than $150, along with their total spending.
 
-Who is the customer with the highest average payment amount?
 
 
 

sql/part1_exercises.md

@@ -38,12 +38,76 @@ SELECT count(actor_id) FROM
 ```
 
 
+## Exercise: Inner Joins
+
+Select `first_name`, `last_name`, `amount`, and `payment_date` by joining the customer and payment tables.  
+
+
+Select film\_id, category\_id, and name from joining the film\_category and category tables, only where the category\_id is less than 10.
+
+
+#### Solutions
+
+
+```sql
+SELECT first_name, last_name, amount, payment_date
+FROM customer c 
+INNER JOIN payment p
+ON c.customer_id=p.customer_id;
+```
+
+```sql
+SELECT film_id, c.category_id, name
+FROM film_category fc
+INNER JOIN category c
+ON fc.category_id = c.category_id
+WHERE c.category < 10;
+```
+
+TODO: check above
+
+
+
+## Exercise: Joining and Grouping: Customer Spending
+
+Get a list of the names of customers who have spent more than $150, along with their total spending.
+
+Who is the customer with the highest average payment amount?
+
+
+#### Solution
+
+```sql
+SELECT first_name, last_name, sum(amount)
+FROM customer c 
+INNER JOIN payment p
+ON c.customer_id=p.customer_id
+GROUP BY first_name, last_name
+HAVING sum(amount) > 150;
+```
+
+```sql
+SELECT c.customer_id, first_name, last_name, avg(amount)
+FROM customer c 
+INNER JOIN payment p
+ON c.customer_id=p.customer_id
+GROUP BY c.customer_id, first_name, last_name
+ORDER BY avg(amount) DESC 
+LIMIT 1;
+```
+
+
+
+
+
 
 ## Exercise: Joining Customers, Payments, and Staff
 
+
+
 Join the customer and payment tables together with an inner join; select customer id, name, amount, and date and order by customer id.  Then join the staff table to them as well to add the staff's name.  
 
-#### Solution
+#### Solutions
 
 ```sql
 SELECT
@@ -83,6 +147,16 @@ Create a list of addresses that includes the name of the city instead of an ID n
 
 #### Solution
 
+
+```sql
+SELECT address, address2, district, postal_code, city, country 
+FROM address
+INNER JOIN city ON address.city_id=city.city_id
+INNER JOIN country ON city.country_id = country.country_id;
+```
+
+or
+
 ```sql
 SELECT address, address2, district, postal_code, city, country 
 FROM address, city, country
@@ -93,7 +167,7 @@ AND city.country_id = country.country_id;
 
 
 
-## Exercise: Joining and Grouping
+## Exercise: Joining and Grouping: Films and Actors
 
 Repeating an exercise from Part 1, but adding in information from additional tables:  Which film (_by title_) has the most actors?  Which actor (_by name_) is in the most films?
 
@@ -102,6 +176,27 @@ Challenge: Which two actors have been in the most films together?  Hint: You can
 
 #### Solution
 
+
+```sql
+SELECT title, count(actor_id) 
+FROM film, film_actor
+WHERE film.film_id=film_actor.film_id
+GROUP BY title
+ORDER BY count(actor_id) DESC 
+LIMIT 1;
+```
+
+```sql
+SELECT first_name, last_name, count(film_id) 
+FROM actor, film_actor
+WHERE actor.actor_id=film_actor.actor_id
+GROUP BY first_name, last_name
+ORDER BY count(film_id) DESC 
+LIMIT 1;
+```
+
+** Alternative Syntax:**
+
 ```sql
 SELECT title, count(actor_id) 
 FROM film, film_actor
@@ -124,9 +219,9 @@ Challenge:
 
 ```sql
 SELECT a.actor_id, b.actor_id, count(*)
-FROM film_actor a, film_actor b -- join the table to itself
-WHERE a.film_id=b.film_id -- on the film id
-      AND a.actor_id > b.actor_id -- avoid duplicates and matching to the same actor
+FROM film_actor a, film_actor b  -- join the table to itself
+WHERE a.film_id=b.film_id  -- on the film id
+      AND a.actor_id > b.actor_id  -- avoid duplicates and matching to the same actor
 GROUP BY a.actor_id, b.actor_id
 ORDER BY count(*) DESC 
 LIMIT 1;
@@ -138,10 +233,10 @@ Super Challenge:
 SELECT c.first_name, c.last_name, d.first_name, d.last_name, fcount
 FROM
 (SELECT a.actor_id AS a1, b.actor_id AS a2, count(*) AS fcount
-FROM film_actor a, film_actor b -- join the table to itself
-WHERE a.film_id=b.film_id -- on the film id
-      AND a.actor_id > b.actor_id -- avoid duplicates and matching to the same actor
-GROUP BY a.actor_id, b.actor_id) foo -- this is the query from above
+FROM film_actor a, film_actor b  -- join the table to itself
+WHERE a.film_id=b.film_id  -- on the film id
+      AND a.actor_id > b.actor_id  -- avoid duplicates and matching to the same actor
+GROUP BY a.actor_id, b.actor_id) foo  -- this is the query from above
 INNER JOIN actor c ON c.actor_id=a1
 INNER JOIN actor d ON d.actor_id=a2
 ORDER BY fcount DESC LIMIT 1;
@@ -150,32 +245,5 @@ ORDER BY fcount DESC LIMIT 1;
 There are other ways to accomplish the above.
 
 
-## Exercise: Joining and Grouping 2
-
-Get a list of the names of customers who have spent more than $150, along with their total spending.
-
-Who is the customer with the highest average payment amount?
-
-
-#### Solution
-
-```sql
-SELECT first_name, last_name, sum(amount)
-FROM customer c INNER JOIN payment p
-ON c.customer_id=p.customer_id
-GROUP BY first_name, last_name
-HAVING sum(amount) > 150;
-```
-
-```sql
-SELECT c.customer_id, first_name, last_name, avg(amount)
-FROM customer c INNER JOIN payment p
-ON c.customer_id=p.customer_id
-GROUP BY c.customer_id, first_name, last_name
-ORDER BY avg(amount) DESC 
-LIMIT 1;
-```
-
-