This is a quick introduction to Python. Since Python is a full programming language with many scientific computing libraries, a short introduction can only scratch the surface. These notes will help you get started, and you can use the web and the references below for even more.
The documentation for individual packages, which usually include a tutorial in addition to more detailed references (I have linked to the webpage for each package referenced below)
Preliminaries
Installing and interacting with Python
There are many ways to install and run Python. You can install Python using the official installer or the Anaconda distribution. You can run Python directly from the terminal (macOS) or command line (Windows), using an IDE like VSCode or Positron, using JupyterLab Notebooks in a browser-based interface, or online using services like Google Collab.
Getting Positron/VSCode up and running
I’ll briefly describe how to get set up with the official installer in Positron (the process for VSCode is similar, since Positron is based on VSCode). Download and install both. Now you can create or open either a Python (.py) or Jupyter Notebook (.ipynb) file. When you do this, you’ll need to select a Python interpreter from the menu in the upper right corner, just above the document, or by typing cmd+shift+p (ctrl+shift+p on Windows) to open the command palette, then typing to search for Python: Select Interpreter. My system also has another version of Python installed, which I don’t want to use or modify, so I select the version that says Python 3.15.0 (saved in /usr/local/bin/python3), which came from the official installer. It’s also possible to set up virtual environments, which are more advanced, but useful for working with projects that may have conflicting dependencies.
To work with Jupyter Notebook file, click +Code or +Markdown to add a code or a text cell. To work with a Python script, you can run a line or selection by hitting cmd+enter (ctrl+enter), and the entire script by clicking on the arrow in the upper left of the document window (in VSCode, you will need to type Jupyter: Create Interactive Window first, and the shortcut for running a line is shift+enter).
Packages
Packages add functionality to Python, and can be installed using the package manager pip. You can install a package by clicking on the Terminal tab in the bottom window, then typing path/python3 -m pip install packagename, replacing path with the path to your Python distribution (mine is in /usr/local/bin/) and packagename with the name of your package (you can skip the path part if Python has been added to your path, but you might want to use it just to be safe in case you have multiple Python installations). In VSCode, the process is the same, but you may need to go to Terminal > New terminal in the menu (or using the command palette) if it isn’t already open.
Alternatively, once you have a Python environment running, you can use %pip install packagename in a Jupyter Notebook code cell, or pip install packagename in a Python script:
%pip install scipy
Requirement already satisfied: scipy in /Library/Frameworks/Python.framework/Versions/3.15/lib/python3.15/site-packages (1.18.1)
Requirement already satisfied: numpy<2.8,>=2.0.0 in /Library/Frameworks/Python.framework/Versions/3.15/lib/python3.15/site-packages (from scipy) (2.5.4)
Note: you may need to restart the kernel to use updated packages.
Once a package is installed, you can use import to load it. For example, we can import the numpy package using
import numpy as np
Language basics
Data types
Python is an object-oriented language, which means that objects are associated with attributes and methods. You can create an integer or string, and check their type using (note that the print statement is necessary in a notebook but not an interactive script):
Anything after # is a comment. It’s a good idea to use comments to document and structure your code.
Lists are another useful type of object. They can contain numbers, strings, or a mix of the two:
c = [1, 2, 3]d = ['foo', 'bar', 'baz', 4]print(d)print(type(d))
['foo', 'bar', 'baz', 4]
<class 'list'>
You can use slice notation to access elements of lists:
print(d[0])print(d[1:3])print(d[3:])print(d[-1])
foo
['bar', 'baz']
[4]
4
Python starts counting at zero, so d[0] is the first element. d[1:3] gives the sublist starting at the second element (element 1…) and ending before the fourth element (element 3). d[-1] returns the last element.
List comprehensions give us a way to automatically generate lists (similar to the slice notation, the range function gives us numbers stopping before the specified point):
e = [i**2for i inrange(4)]print(e)
[0, 1, 4, 9]
Tuples are similar to lists, but they are specified with parentheses instead of brackets, and they are immutable, which means that their elements can’t be modified after they are assigned.
One thing to watch out for with mutable objects like lists is that changes to their elements propagate to objects that are created from them. If we want these new objects to be independent, we have to use .copy(), which is our first example of a method attached to an object:
f = eg = e.copy()print(f)e[0] =10print(f)print(g)
[0, 1, 4, 9]
[10, 1, 4, 9]
[0, 1, 4, 9]
You can get help on a function by typing help(function) or function?. You can see all of the attributes and methods associated with an object by typing dir(object), and you can get help on attributes/methods using object.method?.
Functions
Functions allow you create outputs from inputs. Here are two simple examples:
def fct(x):return x**2-16print(fct(5))def fcts(x, y):''' This function returns x^2 - 16 and x^2 - 8 ''' f1 = x**2-16 f2 = x**2-8return f1, f2h, i = fcts(5, 6)print(h, i)j = fcts(5, 6)print(type(j))
9
9 17
<class 'tuple'>
Notes:
Python uses indentation (rather than brackets or semicolons) to structure code
Python uses ** rather than ^ for exponentiation
A function can optionally include a description using '''
If we assign the output of a function with multiple returns to a single object, the result is a tuple
Loops and conditionals
Loops are useful for doing things iteratively. Here are some basic examples:
for i in c:print(i)sum=0for i inrange(len(c)):sum+= c[i]print(sum)
1
2
3
6
The syntax sum += c[i] is shorthand for sum = sum + c[i].
Two additional looping patterns are zip and enumerate:
for i, j inzip(c, d):print(f"i={i}, j={j}")
i=1, j=foo
i=2, j=bar
i=3, j=baz
for i, j inenumerate(d):print(f"i={i}, j={j}")
i=0, j=foo
i=1, j=bar
i=2, j=baz
i=3, j=4
We can also use while loops to iterate while a condition holds, and if, else and elseif to check whether conditions hold. The following illustrates this by implementing the bisection method to find the positive root of \(x^2 - 16\):
def bisect(f): eps, low, high = [1, 0, 5] i =1while eps>(10**(-9)): mid =.5*(low + high) eps =abs(f(mid))if f(low)* f(mid) <0: high = midelse: low = mid i +=1return mid, iresult = bisect(fct)print(result)
(3.9999999999417923, 35)
NumPy
The NumPy package provides numerical computing functionality, including linear algebra.
x is a flat array (it only has one dimension), while y is a 3x3 matrix. shape is an example of an attribute. If we want, we can turn x into a column or row vector using x.reshape(3, 1) or x.reshape(1, 3).
You can create an \(n\)-vector of zeros using np.zeros(n) and an \(n \times k\) matrix of zeros using np.zeros([n, k]). Replacing zeros with ones creates a vector/matrix of ones. You can create an an \(n\)-dimensional identity matrix using np.eye(n).
We can use @ for matrix multiplication and + for matrix addition. Python intelligently treats x as a row vector when we premultiply y by x, but a column vector when we postmultiply y by x:
print(x @ y)print(y @ x)
[ 8 10 24]
[14 13 20]
You can also slice arrays. For example:
print(y[:, 0])print(y[2, 0:2])
[1 2 1]
[1 2]
Other useful matrix algebra tools are the .T method for transpose and np.linalg.inv for inversion:
Two useful functions for array creation are np.linspace, which creates a specified number of points, and np.arange, which creates points at a specified distance:
NumPy is also useful for generating random numbers. First, we have to choose a random number generator (we’ll use the default) and seed it (to ensure reproducible results):
As far as DataFrame hygiene goes, you can use rename to change column names and sort_values to re-order rows. Below, columns gives the column names of a DataFrame, and we create a test DataFrame using Python’s dictionary syntax:
You can use .value_counts() to get distributions of discrete variables, and crosstabs for frequency tables. To get proportions, we can use normalize='index' for column percentages and normalize='columns' for row percentages:
All of the following allow you to choose rows according to conditions:
data.loc[data['y1']<0].head()data.loc[data['z1'].isin([1, 2])].head()data.query('y1 < 0 and z1 in [1,2]')[['y1', 'z1']].head()print(data.query('y1<0 and z1 in [1,2]')[['y1', 'z1']].head())
y1 x1 z1 y2 x2 z2 x_bin
0 0.853312 1.848289 1 0.904750 1.170736 2 1.0
1 -0.032010 1.899953 0 NaN -0.937843 2 1.0
2 NaN -0.098425 0 -3.976422 0.343267 3 0.0
3 -0.102921 0.813445 0 NaN 0.050898 2 1.0
4 NaN 0.392494 1 NaN 0.517560 3 1.0
/Library/Frameworks/Python.framework/Versions/3.15/lib/python3.15/site-packages/pandas/core/arraylike.py:402: RuntimeWarning: invalid value encountered in log
result = getattr(ufunc, method)(*inputs, **kwargs)
The syntax lambda x: np.log(x) is a lambda function, which allows you to define a function inline (without using def). Note that taking logs produces missing values.
We can combine this with a string trick to add new variables by transforming existing ones, although a loop might be more intuitive:
We can use merge to combine two DataFrames. For example, instead of using transform to add summary statistics, we could merge the statistics we created using agg:
how='left' performs a left merge (keep all observations from the first dataset, drop observations from the second that don’t match to the first). There is also right (keep all observations from the second dataset), inner (only keep observations in both), and outer (keep all).
Suppose x1 and x2 represent the same variable at different points in time. To go from a wide form (time periods in columns) to a long form (time periods in rows), we can use:
# copy the data so we don't mess it updata4 = data.copy()# add the row index as an "id" variabledata4 = data4.reset_index() print(data4.head())# reshapedata_long = pd.wide_to_long(data4, ['x', 'y', 'z'], i='index', j='time')data_longprint(data_long.head())
index time x_bin x y z
0 0 1 1.0 1.848289 2.347410 3
1 1 1 1.0 1.899953 0.968497 2
2 2 1 0.0 -0.098425 -0.759387 2
3 3 1 1.0 0.813445 0.902198 2
4 4 1 1.0 0.392494 -0.466953 3
x y z
time 1 2 1 2 1 2
index
0 1.848289 1.170736 2.347410 2.471313 3.0 2.0
1 1.899953 -0.937843 0.968497 -1.019885 2.0 2.0
2 -0.098425 0.343267 -0.759387 0.018753 2.0 3.0
3 0.813445 0.050898 0.902198 -1.894264 2.0 2.0
4 0.392494 0.517560 -0.466953 -0.755002 3.0 3.0
We could use the method discussed previously to flatten the column indices if we wanted, or we can just work with the hierarchical indices. We can also accomplish the above using unstack (or we could go back again using the inverse, stack), as long as we keep the indices set:
import matplotlib.pyplot as plt# use default theme/make figures smallerplt.style.use('default')plt.rcParams['figure.figsize'] = (4, 3)plt.hist(data["y1"])plt.xlabel('y1')plt.ylabel('frequency')plt.title('Frequency of y1')
In a Python script, we can use plt.figure() to open a new figure, clearing the existing one.
Matplotlib integrates with pandas, so we can also just type:
data['y1'].hist()
To create a scatter plot, we can use
plt.scatter(data['y1'], data['x1'])
We can also create line plots. For example, to plot a normal cdf, we can use norm from the SciPy package, which provides scientific computing tools for statistics, optimization, numerical integration, and more:
from scipy.stats import normx = np.linspace(-2, 2, 50)y = norm.pdf(x, 0, 1)y2 = norm.pdf(x, 0, 2)plt.plot(x, y, 'b')plt.plot(x, y2, 'r-')
Note that additional plots are added to the existing figure (if we want a new plot, we can use plt.clf() or plt.figure()).
You can save a plot using (the type of file is inferred from the extension)
plt.savefig('name.pdf')
Matplotlib also has an object-oriented API which is useful for more complex graphs. We could recreate the above using
fig, ax = plt.subplots()ax.plot(x, y, 'b-')ax.plot(x, y2, 'r-')
fig, ax sets up a figure (the figure itself) and an axes (the area where the plot goes). Then each ax.plot statement draws a picture on the axes. “axes” isn’t a typo, it’s matplotlib’s terminology for the collection of objects that make up a plot.
To make a scatter plot with points colored by a group variable, we can use the following (the loop works because data.groupby provides a group name and the associated data):
For multiple plots side by side (or in a grid), we can use
# binary group indicatordata['gp'] =1*(data['z2']<3)# gen list of groups and number of groupsgps =sorted(data['gp'].unique())num_gps =len(gps)# iterate axes across groups and colors across z1 valuesfig, axes = plt.subplots(1, num_gps, figsize=(5, 3))for gp in gps:for z1_gp_name, z1_gp_data in data[data['gp']==gp].groupby('z1'): axes[gp].scatter(z1_gp_data['x1'], z1_gp_data['y1'], label = z1_gp_name, ) axes[gp].set_title(f"group = {gp}") axes[gp].legend(title='z1', loc='lower left')fig.supxlabel('x1')fig.supylabel('y1')
Seaborn
Seaborn is another plotting package, built on top of matplotlib, which makes it easier to produce certain more complicated graphs.
For example, seaborn makes it easy to get a scatter plot with points colored by group, a scatter plot with a regression line, or side-by-side plots for different groups:
import seaborn as sns# use the default theme (instead of inheriting it from the previous graph)sns.set_theme()sns.scatterplot(data = data, x="x1", y="y1", hue="z1", )plt.figure()sns.regplot(data, x='x1', y='y1')plt.figure()sns.relplot(data=data, x="x1", y="y1", hue="z1", col="gp", height=3)
<Figure size 800x600 with 0 Axes>
We can also get stacked bar charts, or overlapping density plots:
For some kinds of computing, speed matters. Python is an interpreted language, which means that it has to translate commands before it runs them, and this can make it slow. One way to avoid this is to use precompiled and vectorized functions, which don’t have to be translated, and can operate on entire vectors at once.
For example, it’s much slower to create the log of every element in a vector using a loop than it is to use NumPy’s vectorize np.log function:
import timex = np.arange(2, 1_000_000)# create logs in loop, record start and end timet0 = time.perf_counter()def slow_log(z): log_x_1 = np.zeros(len(x))for i inrange(len(x)): log_x_1[i] = np.log(x[i])return log_x_1%timeit -r 5-n 1 slow_log(x)# create logs with numpy%timeit -r 5-n 1 np.log(x)
632 ms ± 5.46 ms per loop (mean ± std. dev. of 5 runs, 1 loop each)
3.66 ms ± 170 µs per loop (mean ± std. dev. of 5 runs, 1 loop each)
Some algorithms can’t be vectorized, however, which is where Numba comes in. Numba lets you compile a function, so it doesn’t have to be interpreted before it’s run (this is called just-in-time compilation).
Here’s how it works in the example above:
from numba import njit@njitdef new_log(z): log_z = np.zeros(len(z))for i inrange(len(z)): log_z[i] = np.log(z[i])return log_zx = np.arange(2, 1_000_000)# run once to warm uptest = new_log(x)%timeit -r 5-n 1 new_log(x)
4.15 ms ± 377 µs per loop (mean ± std. dev. of 5 runs, 1 loop each)
The version compiled with numba is much faster, and all we had to do was add @njit before it (in a Python script, rather than the Jupyter notebook used to create this document, it is nearly as fast as the NumPy function).
Notes:
The function won’t be compiled until it’s run, which may make the initial run slow (this won’t add much to overall time on functions that are called repeatedly)
Numba only works with a subset of Python, although this subset includes the key numerical computing commands
Statistics and econometrics
In addition to letting you do your own programming, Python also has packages that run common statistical and econometric procedures. One of the main packages is statsmodels.
If you prefer, you can use Patsy to use R-style formulas. You can use C() to include categorical variables, I() to include transformations of variables, * to include main effects and interactions, and : to get interactions:
Statsmodels supports many more estimators, including binary choice models, count data models, and survival models. The linearmodels package supports panel data models, instrumental variables, and GMM estimation. The marginaleffects package provides marginal effects for many different types of models.
Basic programming examples
In addition to pre-programmed estimators, Python makes it (relatively) easy to program your own estimator.
As a simple example, here’s how we can get the OLS estimate “by hand”: