Module 01

Intro to ArcGIS Pro & ArcPy

Yesterday in the Great Smokies
Webcam archive

GEO 409: Advanced GIS

Virtual field trip to

Scuttlehole Gap, Rockcastle River

36.963583,-84.346543

Goals

Set up our local workspace.
Fire up ArcGIS Pro.
Run a few Python scripts.
Learn basic Python
- data types
- assign and access variables
- if/else statements

Our lab

Find GNIS points near selected locations.
- Geographic Names Information System (GNIS).
- Use Python in ArcGIS Pro to code solution.
- Do it in class and submit as assignment.

Getting started

Set up GitHub.com user account (while mindful of the username).
Accept Lesson Invitation in Module 01.
Lesson and lab located at github.com/uky-gis/geo409-mod-01-username

I. Our workspace

I got data but

but where is it on my computer?

Perennial challenge

Data hygiene
concerns keeping data organized on your computer.

Your local computer

Create top-level directory for all projects
We call this the root directory
C:\YodasGIS (Windows OS)
/Users/yoda/YodasGIS (macOS)
Put projects inside this directory
Separate downloaded from created assets

Example directory structure

Repo?

A repository ('repo') is a project stored as a folder that is synchronized to a remote server. (More soon)

Toaster problem

Why does it have a cancel button?
Because we don't trust it automatically
and so goes your computer.

At the very least

Be able to find the absolute file path and file extension
of an asset on your Windows computer.
C:\YodasGIS\data\N042E254_2020.tif

Windows File Explorer

Right click on the file
click Properties.

ArcGIS Pro

Locate asset in Catalog window
hover mouse cursor over asset.

Folder & file naming conventions

Semantic name, e.g., describe contents.
No spaces; replaces with hyphen, underscore, or use 'camelCase'.
Always use a file extension for file names.
Always case-sensitive.
How the Smithsonian does it

Most common errors

Not knowing where the asset is located.
Not providing the correct name.

In-class task

Create root GIS and data folders.
Download and extract this database.
Move ky_small_scale.gdb to data folder:
C:\YodasGIS\data\ky_small_scale.gdb

II. ArcGIS Pro

CHECK YOUR EMAIL:

Look for “An invitation to join an ArcGIS Online organization, University of Kentucky.” to find your ArcGIS Pro login.

ArcGIS Pro

Released by Esri 2015 as flagship desktop GIS software
Combines 2D and 3D capabilities
and replaces ArcGIS Desktop
- a collection of desktop GIS software.
Explore interface

Before visualizing data

explore its:
- file path
- data type
- CRS
- attributes

When visualizing data

understand its:
- extent
- scale
- error

In-class task

Fire up ArcGIS Pro and customize app for your workspace.
Create a new map project called, "module-01".
Inspect ky_small_scale.gdb.
Visualize and symbolize a few layers.

III. Python & ArcPy

Python ✊🐍

Created by Guido van Rossum in 1990s.
One of the most popular programming languages.
Designed to be easy to read and write.
Named after Monty Python; not a constrictor.

ArcPy

ArcGIS Pro's Python API (application programing interface)
exposes ArcGIS Pro's functionality to Python.

Python in ArcGIS Pro

We'll use Jupyter Notebook (or just Notebook)
- which creates cells of Markdown content and Python code
- with interactive execution cell by cell
- and saved as a text file with the .ipynb file extension.

Programming goals

We'll cover enough to work in ArcGIS Pro
to script/automate geoprocessing workflows.
Let's give Python a spin in an ArcGIS Pro Notebook.

In-class task

Create a Notebook called 'practice.ipynb' in module-01.
Copy the code in the next slide into the first Notebook cell.
Run the cell; output and a new cell should appear.
Each slide is a new cell.

# first script 😱
					---
					Speak!
					```python
					# This is comment. What can I say?
					# Anything!
					print("Geography, y'all!")
					# The print function prints whatever is in the parentheses.
					```
					---
					Assign a variable
					```python
					# We printed a string literal earlier. 
					# Let's assign a string to a variable.
					phrase = "Geography, y'all!"
					print(phrase)
					```
					---
					The `print()` function is your ✨Friend✨
					```python
					# Often, we don't know the contents of a variable.
					# Print it to see what's inside.
					aNumber = 42**3
					print(aNumber)
					```

---
					Print an expression
					```python
					# The expression is evaluated before printing.
					# Remember order of operations?
					print(aNumber / 2 + 1)
					```

---
					Operations on strings
					```python
					# We can strings together with the + operator.
					# This is called concatenation.
					a = "Geography"
					b = "y'all!"
					c = a + " " + b
					print(c)
					```

---
					Comparing values
					```python
					# How can we check if values are the same?
					# Comparison operators!
					check = c == phrase
					print(check)
					```

Want to learn more?

Jump down to the Learning to code 🧐 section and continue on your own in this Notebook.

How much of x is near y?

How many arches are withing one mile of Miguel's Pizza?
Let's find out with ArcPy.

In-class task

Create a Notebook called 'arches-near-pizza.ipynb' in module-01.
Copy the code in the next slides into the Notebook.
Run the cell-by-cell.

Pseudocode

Describing code without using the language.
Helps understand the flow of the code.

Our Pseudocode I

Import needed tools.
Connect our input/output database.
Create our input/output coordinate systems.
We call this setting up our environment.

Our Pseudocode II

Create point for desired location.
Copy it to our database.
Buffer point to desired distance.
Clip features from another layer within this distance.
We call this our analysis.

Step 1: Set up the environment
					```python
					# import ArcGIS Pro's Python API
					import arcpy
					```
					---
					```python
					# Set the workspace to the ky_small_scale.gdb
					arcpy.env.workspace = "C:\\rootGIS\\data\\ky_small_scale.gdb"
					```
					---
					```python
					# function to create CRS for Kentucky
					ky = arcpy.SpatialReference(3089) # search spatialreference.org
					# CRS for WGS84, the standard for GPS
					wgs84 = arcpy.SpatialReference(4326)
					```

---
					```python
					# set the output CRS to Kentucky State Plane
					arcpy.env.outputCoordinateSystem = ky
					```

---
					```python
					# overwrite existing data
					arcpy.env.overwriteOutput = True
					```

---
					Step 2: Analysis. 
					Create a point geometry
					```python
					# set the location of Miguel's Pizza from GPS coordinates
					lon = -83.682834
					lat = 37.783013 
					# create a single point using GPS coordinates
					point = arcpy.Point(lon, lat) 
					# create the point geometry using the WGS84 CRS
					ptGeometry = arcpy.PointGeometry(point, wgs84)
					# show point in the Notebook
					ptGeometry
					```

---
					```python
					# save the point to a feature class in our workspace
					arcpy.management.CopyFeatures(ptGeometry, "miguels")
					# input is ptGeometry, output is miguels
					```

---
					Buffer point and clip GNIS
					```python
					# create a one-mile buffer around the point
					arcpy.analysis.Buffer("miguels", "one_mi_miguels", "1 mile")
					# clip the GNIS layer with the buffer
					arcpy.analysis.Clip("gnis", "one_mi_miguels", "gnis_miguels")
					```

What could go wrong? 💣

TYPOS!
Misspelled string for the workspace property.
We'll discover a way list contents of a workspace.
Check all strings.

What could go wrong? 💣

SELECTIONS!
If features are selected (highlighted in cyan) in ArcGIS Pro, they will be used in the script.
Clear all selections before running a script.

Learning to code 🧐

First steps

Use code editor to highlight syntax
distinguish data, keywords, and comments
always case-sensitive
spacing matters.

Python I

Practice along

Create/open practice.ipynb notebook in ArcGIS Pro
copy and paste examples from lesson README.md.
TIP: keystroke SHIFT + ENTER runs the cell.
TIP: have only one notebook open at a time in ArcGIS Pro.

Required viewing

Variables, expressions, and statements P4E
- Part 1 and Part 2
Conditional Execution P4E
- Part 1 and Part 2
Videos embedded in next four slides 🛎️

Value

A value is data that we use in our program
All values have a type
The type of value determines what can be done in the program.

Basic types

String (of characters)
Number types
- Integer (1)
- Float (1.0)
Boolean (True/False)
None (null)

Levels of measurement

A number doesn't always represent a quantity
Nominal (zipcode)
Ordinal (basketball ranking)
Interval (temperature of gym)
Ratio (weight of basketball)

### Constants
					* Literals: the syntax tells us what type it is.
					---
					```python
					"I'm literally a string of characters (the quotes fence me in)"
					1 # an Integer
					1.0 # a Float
					True # speak it loud!
					None # don't know what it is
					```
					---
					### Objects
					* Once a value is created, it is an **object** with special powers ⚡ (actually the powers are called **methods**)

---
					```python
					"I am a literally a string of characters".upper()
					# returns 'I AM A LITERALLY A STRING OF CHARACTERS'
					```

---
					### Variables
					* Assign value to an arbitrary and limited set of characters. 
					* variable <=> value 
					* use meaningful names, can't start with number, uses spaces, etc. 
					
					---
					```python
					myZipcode = "40588"
					# type() is a function
					# Functions have () where we can accept input.
					type(myZipcode)
					# returns str because it is a string
					```

---
					### Tip 💡
					* `print()` your variable to see its value 
					* `type()` your variable to see its type 
					* Do this all of the time 💯

---
					### Type
					* Knowing the type of value tells you methods 🔌 the value has. 
					* View a list of [string methods](https://www.w3schools.com/python/python_ref_string.asp)

---
					```python
					myZipcode = "40588"
					# .find() returns the position of a character in a string
					# .count() returns the number of times a character is in a string
					print(myZipcode.find("4")) # outputs 0
					print(myZipcode.find("5")) # outputs 2
					print(myZipcode.count("8")) # outputs 2
					print(myZipcode.count("88")) # outputs 1
					```

---
					### Challenge
					* Let's place with some text from a classic Hunter S. Thompson quote 
					* Count how many times he says "I"

---
					I am more than just a serious basketball fan. I am a life-long addict. I was addicted from birth, in fact, because I was born in Kentucky and I learned, early on, that Habitual Domination was a natural way of life. (Hunter S. Thompson)

---
					### Operators
					* Perform [operations](https://www.w3schools.com/python/python_operators.asp) on variables and values 
					* Context of operation depends on value type, e.g., concatenation vs. math addition 
					* Operation returns either a value or True/False

---
					### Assignment
					* <code class='icode'>=</code> assigns value right to left 
					* <code class='icode'>+=</code> adds value right to left

---
					```python [1-3|4-7|8-11|12|13-14]
					# assingment 
					x = 10
					y = 20 
					print(x, y) # returns 10, 20
					x += y
					# same as x = x + y
					print(x, y) # returns 30, 20
					a = "Greef"
					b = 'Karga'
					a += b 
					print(a) # returns GreefKarga
					a += x # throws an error because of mixed types
					a += str(x) # convert x to a String
					print(a) # returns GreefKarga10
					```

---
					### Comparison and identity
					* <code class='icode'>==</code> True if same value 
					* <code class='icode'>!=</code> True if different value 
					* <code class='icode'>is</code> True if same object (value and type) 
					* <code class='icode'>is not</code> True if different object

---
					```python [1-4|5-7|8-10]
					# assignment, then comparison
					x = 10
					y = 10.0
					b = "10"
					print(x == y) # True
					print(x == b) # False
					print(x == int(b)) # True
					# identity
					x is y # False
					print(x is int(b)) # True
					```

---
					### When to use == vs. !=
					* Use == when you know what you are looking for 
					* Find only apples in a basket of mixed fruit 
					* Use != when you want find a unique value, but you don't know what it is
					* A basket of apples has a different fruit, find it

---
					### 🔭 into the future!
					* Combining operators and **looping** through data is an essential concept

---
					```python
					# Assignment operator
					x = 0
					# Comparison operator
					while x != 7:
						print(x)
						# Addition and reassignment operator
						x += 1
					# prints x until it adds up to 7
					```

---
					### Membership
					* <code class='icode'>in</code> True if substring in string
					* <code class='icode'>not in</code> True if substring not in string

---
					```python
					place = "Grays Arch"
					print("Gray" in place) # True
					print("gray" in place) # False
					print("gray" not in place) # True
					```

---
					### Logical operators
					* Combine operators
					* <code class='icode'>and</code> True if all operations are True
					* <code class='icode'>or</code> True if any operation is True

---
					```python
					place = "Grays Arch"
					print("Gray" in place or "gray" in place) # True
					print("Gray" in place and "arch" in place) # False
					print("Gray" in place and "Arch" in place) # True
					```

---
					### Challenge
					It's 2020 Census &  your department needs to check if a two-letter abbreviation for a state is correct. Hint in next slide.

---
					states = "AL AK AZ AR CA CO CT DC DE FL GA HI ID IL IN IA KS KY LA ME MD MA MI MN MS MO MT NE NV NH NJ NM NY NC ND OH OK OR PA RI SC SD TN TX UT VT VA WA WV WI WY"

---
					### Challenge
					Post in [Gitter](https://gitter.im/uky-maps/geo409) a statement that prints True if the state "Ut" is in the list of states.

---
					# STOP

---
					```python
					print("WY" in states) # True
					print("wy" in states) # False
					print("wy".upper() in states) # True
					```

---
					### if statement
					* Controls flow of execution 
					* if this, then do that. 
					* Focus of our module's application

---
					### if statement format
					* <code class='icode'>if</code> expression <code class='icode'>True</code>, execute this block of code
					* condition ends with a colon 
					* block indented four spaces 
					* Use pseudocode to write our logic

---
					```txt
					# Example of pseudocode
					if expression is true:
						execute this block of indented code
					```

---
					```python
					fruit = "banana"
					if fruit == "apple":
					    print("You have an apple!")
					```

---
					```python
					nameOfState = "Wy"
					if nameOfState.upper() in states:
					    print("Your state abbreviation exists!")
					```

---
					### f-Strings
					* a string proceeded with a <code class='icode'>f</code>
					* expands variables and expressions enclosed with <code class='icode'>{ }</code> curly brackets

---
					```python
					nameOfState = "Wy"
					if nameOfState.upper() in states:
					    print(f"Your state abbreviation of {nameOfState} exists!")
					```

---
					### else statement
					* <code class='icode'>else</code> condition comes after <code class='icode'>if</code> statement
					* condition ends with a colon
					* body indented four spaces
					* executes when <code class='icode'>if</code> statement is not True

---
					```python
					nameOfState = "Wy"
					if nameOfState.upper() in states:
						print(f"Your state abbreviation of {nameOfState} exists!")
					else:
						print(f"Your state abbreviation of {nameOfState} does not exist!!")
					```

---
					### elif statement
					* else if statement
					* <code class='icode'>elif</code> condition comes after <code class='icode'>if</code> statement
					* condition ends with a colon
					* body indented four spaces
					* executes when <code class='icode'>if</code> statement is not True

---
					```python
					nameOfState = "Wy"
					if nameOfState in states:
						print(f"Your state abbreviation of {nameOfState} exists spelled exactly.")
					elif nameOfState.upper() in states:
						print(f"Your state abbreviation of {nameOfState} is there but not in uppercase.")
					else:
						print(f"Your state abbreviation of {nameOfState} does not exist!!")
					```

---
					### if/elif/else statement
					* Only one branch is true and executes

---
					### if/if/if statement
					* All branches could be true

---
					### Challenge
					Write an if/elif/else statement that finds if you are taller or shorter than UKy's starting five.

---
					```python
					x = 60 # your height in inches
					hi = 81 # Oscar's height in inches
					lo = 69 # Sahvir's height in inches
					```

---
					### Challenge
					Post solution in [Gitter](https://gitter.im/uky-maps/geo409)

Errors

Anticipate and prevent
Look at last line in Python error message
Use print() function on variables

Syntax errors

Incorrect sequence of characters
Most common when beginning to code
Check spelling, everything is case-sensitive

Exceptions

Syntactically correct but still flawed.
"John is a married bachelor." Can't be true!
Division by zero

Logic errors

Syntactically correct and executes properly.
Result is not correct, though not immediately apparent
Order of operations in math calculations
(10 + 20 + 30) / 3 vs. 10 + 20 + 30/3

Lab 2

Review

Open Lab-02.ipynb Jupyter Notebook in ArcGIS Pro
Five code exercises
Application to filter GNIS data
- US Place name database (GNIS) 2.5 M records Download National File
- Import Python csv module and work on real data

Python modules

Import additional functions
Extends built-in functions
csv module from the standard library (no need to install anything!)

Namespace

Uniquely identify resources
Access functions in module with dot notation
E.g., csv.reader() to access the read-in function.

### Add location of data
					---
					```python
					# Make variable to data location
					myData = 'C:\\MyGIS\\data\\NationalFile_20210101.txt'
					```
					---
					### Inspect data in VS 
					* look at first row for column names
					---
					![Columns in file](images/CSV-diagram.png)
					---
					### Open and create CSV object
					---
					
					```python
					with open(myData, encoding="utf8") as csvFile:
						# Create the CSV object. 
						reader = csv.reader(csvFile, delimiter='|')
						# Loop through records and print them.
						for row in reader:
							# Each row is a list of values.
							print(row[2], row[3])
							# Use if block to search records
					```

---

```python
					if row[3] == 'KY' and row[2] == 'Arch':
						print(row)
					```

---
					### Write out data

---
					```py
					# Write out file location
					myOutput = 'C:\\MyGIS\\data\\utah_arches.csv'
					# Open data
					with open(myData, encoding="utf8") as csvFile:
						# Open empty file to write output
						with open(myOutput, "w", encoding="utf8", newline='') as csvOutput:
							# Create reader object
							reader = csv.reader(csvFile, delimiter='|')
							# Create writer object
							writer = csv.writer(csvOutput)
							# Loop through records
							for row in reader:
								if row[3] == 'UT':
									# Write out selected columns
									writer.writerow(row)
					```
					---
					### Add to map