Pandas Python Guide — Data Analysis for Beginners

If you work with tables of data in Python, you will meet pandas within the first hour. It is the library that turns a CSV file, a spreadsheet, or a database query into an object you can filter, reshape, summarize, and plot with a few lines of code. Learning it is less about memorizing hundreds of functions and more about understanding a handful of ideas: labeled data, vectorized operations, and the split-apply-combine pattern.

This guide covers the core data structures, loading and exploring files, selecting rows and columns without tripping over the common traps, cleaning missing data, grouping and merging, plotting, and a full pipeline from raw CSV to a finished chart. The code targets pandas 3.0, released on January 21, 2026, and flags the places where older versions behave differently.

What pandas is and why it became the default

Pandas is an open-source library built on top of NumPy, Python's array-computing package. According to the project's own history page, development began in 2008 at the investment firm AQR Capital Management, and the code was open sourced by the end of 2009. Wes McKinney, who started the project, described its design in a 2010 paper presented at the Python in Science Conference. Since 2015 pandas has been a NumFOCUS sponsored project, and it is maintained by volunteers around the world.

The name is usually traced to "panel data," a term from econometrics for observations of the same units over time, and the library's early emphasis on financial time series fits that origin.

The reason it spread is practical. Before pandas, tabular data in Python meant choosing between raw NumPy arrays, which are fast but have no column names and handle mixed types awkwardly, and lists of dictionaries, which are readable but slow and tedious to aggregate. Pandas offered something closer to a programmable spreadsheet: named columns, an index for row labels, built-in handling of missing values, and fast operations that run in compiled code underneath.

Its reach is large. PyPI Stats, which counts downloads of the package from the Python Package Index, showed roughly 547 million downloads of pandas over a recent 30-day window. That figure includes automated installs from build servers and mirrors, so it overstates the number of people using it, but it signals how deeply the library is embedded in the Python data ecosystem.

Install it with:

pip install pandas matplotlib

Pandas 3.0 requires Python 3.11 or higher and NumPy 1.26.0 or newer, according to its release notes.

Core data structures: Series and DataFrame

Series: one labeled column

A Series is a one-dimensional array with an index, which is a set of labels attached to each value. Think of a single column in a spreadsheet where the row headers are part of the data.

import pandas as pd

temperatures = pd.Series([22.1, 19.8, 25.3, 21.0], index=["Mon", "Tue", "Wed", "Thu"])
print(temperatures)
# Mon    22.1
# Tue    19.8
# Wed    25.3
# Thu    21.0
# dtype: float64

Arithmetic on a Series applies to every element at once. temperatures * 1.8 + 32 converts all four readings to Fahrenheit without a loop. This is called vectorization: the loop still happens, but inside optimized compiled code instead of in slow, interpreted Python.

DataFrame: a table of Series

A DataFrame is a two-dimensional table with labeled rows and labeled columns. It is the object you will use for nearly everything.

data = {
    "city": ["London", "Paris", "Berlin", "Madrid"],
    "population_millions": [9.0, 2.1, 3.7, 3.3],
    "avg_temp_celsius": [11, 12, 10, 15],
}
df = pd.DataFrame(data)
print(df)
#      city  population_millions  avg_temp_celsius
# 0  London                  9.0                11
# 1   Paris                  2.1                12
# 2  Berlin                  3.7                10
# 3  Madrid                  3.3                15

Each column is a Series, and df.index holds the row labels, which default to the integers 0, 1, 2 and so on. A useful mental model is a dictionary of Series that all share one index. When you add two columns, pandas matches rows by index label instead of by position, which is what lets it combine data from different sources without you counting rows.

Data types and what they mean for memory

Every column has a dtype, such as int64, float64, bool, datetime64, category, or a string type. In pandas 3.0 a dedicated string dtype is the default for text columns. According to the release notes it is backed by PyArrow when that is installed and falls back to NumPy object storage otherwise, and it can hold only strings or missing values. Older code that checks for object dtype to find text columns may need updating.

Dtypes matter for two reasons. They determine which operations are allowed (you cannot average a column of strings), and they determine how much memory the table uses, which becomes important once files grow past a few hundred megabytes.

Loading data from files and databases

Real data rarely starts as a Python dictionary. Pandas has readers for many formats, including CSV, Excel, JSON, HTML tables, XML, Parquet, ORC, Feather, HDF5, Stata, SAS, SPSS, and SQL.

# CSV (the most common case)
df = pd.read_csv("data.csv")

# Excel (needs an Excel engine such as openpyxl installed)
df = pd.read_excel("report.xlsx", sheet_name="Sheet1")

# JSON
df = pd.read_json("records.json")

# Parquet, a compressed columnar format that preserves dtypes
df = pd.read_parquet("data.parquet")

# SQL database, here with Python's built-in SQLite driver
import sqlite3
conn = sqlite3.connect("shop.db")
df = pd.read_sql("SELECT * FROM sales WHERE year = 2025", conn)

For other databases, pandas accepts a SQLAlchemy connection, which is a common way to reach PostgreSQL or MySQL.

The read_csv parameters worth knowing

read_csv has dozens of options, and a handful solve most real-world problems:

Parameter	What it does	Typical use
sep	Sets the field delimiter (default comma)	sep="\t" for tab-separated files
encoding	Sets the text encoding	encoding="utf-8" or "latin-1" for accented characters
parse_dates	Parses named columns into datetimes	parse_dates=["order_date"]
index_col	Uses a column as the row labels	index_col="id"
usecols	Reads only the listed columns	usecols=["date", "revenue"]
dtype	Forces column types	dtype={"zip": "string"} to keep leading zeros
nrows	Reads only the first n rows	nrows=1000 to peek at a huge file
chunksize	Returns an iterator of DataFrames	Streaming a file too big for memory
na_values	Adds extra strings to treat as missing	na_values=["n/a", "--"]

The dtype option deserves a note. Postal codes, phone numbers, and product IDs look like numbers but are really labels. If pandas guesses int64, the leading zero in "02134" disappears. Telling it up front that the column is a string avoids a bug that is easy to miss and hard to trace later.

Exploring a new dataset in five commands

When you open a file for the first time, five commands tell you most of what you need to know before you do any analysis:

df.shape        # (rows, columns), for example (10000, 15)
df.dtypes       # data type of each column
df.head(10)     # first 10 rows
df.info()       # column names, dtypes, non-null counts, memory use
df.describe()   # count, mean, std, min, quartiles, max for numeric columns

describe() is the single most informative line. A maximum far above the 75th percentile hints at outliers. A count lower than the number of rows reveals missing values. A minimum of zero or a negative number in a column that should be positive points to bad data.

Two more commands help with text and category columns. df["region"].value_counts() lists each distinct value with its frequency, which exposes typos such as "North" and "north " counted as separate groups. df.nunique() shows how many distinct values each column has, which helps you decide whether a column is an identifier, a category, or free text.

Spend real time here. Most analysis mistakes come from assumptions about the data that a two-minute look would have contradicted. Our explainer on what data cleaning actually involves describes the kinds of problems this step uncovers, and why data cleaning takes so much of a data scientist's time covers why it is worth doing carefully.

Selecting data without the common traps

Pandas gives you several ways to pull out a subset, and the differences between them are a frequent source of confusion.

# One column returns a Series
df["city"]

# A list of columns returns a DataFrame
df[["city", "population_millions"]]

# Rows by label with loc
df.loc[0]          # the row whose index label is 0
df.loc[0:2]        # labels 0, 1, and 2

# Rows by integer position with iloc
df.iloc[0]         # the first row
df.iloc[0:3]       # positions 0, 1, and 2 (stop excluded)

# Boolean indexing: keep rows that satisfy a condition
big_cities = df[df["population_millions"] > 3.0]

# Combine conditions with & and |, with each one in parentheses
big_and_warm = df[(df["population_millions"] > 3.0) & (df["avg_temp_celsius"] > 11)]

# query() offers a readable string syntax
big_and_warm = df.query("population_millions > 3.0 and avg_temp_celsius > 11")

loc versus iloc

The pandas documentation states the slicing difference plainly. With loc, "both the start bound and the stop bound are included, if present in the index." With iloc, the start is included and the stop is excluded, like ordinary Python slicing. So df.loc[0:2] returns three rows and df.iloc[0:2] returns two.

The two diverge in practice as soon as the index is not the default 0, 1, 2. After you filter rows, the surviving labels might be 4, 9, and 15. Then df.loc[9] finds the row labeled 9 and df.iloc[1] finds the second row. A rule that prevents most errors: use loc when you know the label or are filtering with a condition, and iloc when you truly mean "the nth row."

Accessor	Selects by	Slice end	Best for
loc[row, col]	Labels or a boolean array	Inclusive	Filtering and named columns
iloc[row, col]	Integer positions	Exclusive	"First five rows," "last column"
at[row, col]	One label pair	Not applicable	Fast single-cell read or write by label
iat[row, col]	One position pair	Not applicable	Fast single-cell read or write by position

The documentation calls at and iat the fastest way to reach a single scalar. Reach for them only inside tight loops, and consider whether a vectorized operation could replace the loop altogether.

Boolean masks and the parentheses rule

Pandas uses & for and, | for or, and ~ for not, and each condition must be wrapped in parentheses because of how Python orders operators. Writing df[df["a"] > 1 & df["b"] < 5] fails or gives wrong answers. A related detail: missing values in a boolean mask propagate as False, so rows with NaN in the tested column silently drop out of the result. That is usually what you want, but it is worth knowing when row counts look lower than expected.

Writing values: one step, not two

Assignments that select and set in two separate steps are the source of the infamous SettingWithCopyWarning in older pandas. Under pandas 3.0 the rules are stricter and simpler. Writing df[df["bar"] > 5]["foo"] = 100 is chained assignment and does nothing useful. Write it in a single step instead:

df.loc[df["avg_temp_celsius"] > 12, "climate"] = "warm"

The next section explains why.

Copy-on-Write: the biggest behavior change in pandas 3.0

Before pandas 3.0, whether a selected subset was a copy or a view of the original depended on internal details that users could not easily predict. Modifying the subset sometimes changed the original and sometimes did not, which is why the warning existed.

Pandas 3.0 makes Copy-on-Write the default and only behavior. The documentation states the rule: any DataFrame or Series derived from another behaves as a copy, and you change an object only by modifying that object directly. The release notes add that chained assignment "will stop working" and that SettingWithCopyWarning is removed.

An analogy helps. Imagine a shared document where each person who opens it gets their own private copy the moment they start typing, but until then everyone reads the same file, so nothing is duplicated needlessly. That is the lazy-copy idea behind Copy-on-Write: data is shared until someone modifies it, and only then is it copied. The documentation says this improves both average speed and memory use, and it makes results predictable, because no statement can update more than one object.

The practical consequences for everyday code:

• Use df.loc[mask, "col"] = value for conditional updates, in one step.
• Reassign columns instead of using inplace=True on a selected column. Writing df["a"].fillna(0, inplace=True) operates on a temporary object, so df stays unchanged under Copy-on-Write. Write df["a"] = df["a"].fillna(0) instead.
• Treat results as new objects. Methods return a new DataFrame, and the common idiom df = df.dropna() is both clear and efficient.

If you are working on a project pinned to pandas 2.x, the same patterns still work and will not break when you upgrade.

Handling missing data

Missing data is a fact of life, and pandas has several ways to represent it. According to the user guide, NaN is used for NumPy float columns, NaT for datetime and timedelta columns, and NA for pandas' nullable dtypes such as Int64, Float64, boolean, and the string dtype. One quirk matters: a NumPy integer column that gains a missing value is converted to floats, while a nullable Int64 column keeps its type and shows <NA>.

Another quirk: np.nan == np.nan is False, so you cannot find missing values with ==. Use isna(), which detects NaN, NaT, None, and NA alike.

Finding it

df.isna()                    # True where a value is missing
df.isna().sum()              # missing count per column
df.isna().mean()             # proportion missing per column

Dropping it

df_clean = df.dropna()                        # drop rows with any missing value
df_clean = df.dropna(how="all")               # drop rows that are entirely missing
df_clean = df.dropna(subset=["order_id"])     # only look at specific columns

# Keep columns that have at least half their values present
df_clean = df.dropna(axis="columns", thresh=int(len(df) * 0.5))

According to the reference documentation, thresh is the minimum number of non-missing values a row or column needs to be kept, and it cannot be combined with how. The count of non-missing values is the opposite of what some people expect when they first read "threshold."

Filling it

df["score"] = df["score"].fillna(0)                      # a constant
df["score"] = df["score"].fillna(df["score"].mean())     # the column mean
df["price"] = df["price"].ffill()                        # carry the last value forward

Forward fill with ffill() and backward fill with bfill() replaced the old fillna(method="ffill") form, which was deprecated in pandas 2.1. Using the dedicated methods works in every recent version.

The judgment call

Whether to drop or fill is a statistical question, not a coding one. Dropping rows is safe when the missing values are scattered at random and make up a small share of the data. If missingness is systematic, say, high-income respondents skipping the income question, then both dropping and filling with the mean can bias results. Filling with the mean also shrinks the apparent variation in a column, because it adds many identical values. Before choosing, look at which rows are missing and ask why. Our deeper treatment of missing-data imputation strategies covers the options beyond these basics.

Transforming data: vectorized operations, map, and apply

Prefer column arithmetic and built-in accessors

The fastest and clearest way to transform data is to operate on whole columns:

df["population"] = df["population_millions"] * 1_000_000
df["name_upper"] = df["city"].str.upper()           # string methods via .str
df["year"] = df["order_date"].dt.year               # datetime parts via .dt

The .str accessor applies string methods to every element and skips missing values gracefully. The .dt accessor does the same for dates. Both avoid writing a loop.

map for lookups, apply for custom logic

map() on a Series replaces values using a dictionary or a function, which makes it a natural fit for lookup tables:

city_codes = {"London": "LDN", "Paris": "PAR", "Berlin": "BER", "Madrid": "MAD"}
df["code"] = df["city"].map(city_codes)

apply() runs an arbitrary Python function over each element, or over each row if you pass axis=1:

df["size"] = df["population_millions"].apply(lambda x: "large" if x > 5 else "small")

Because apply calls a Python function once per element or row, it is generally slower than the equivalent vectorized expression, which operates on the whole column in compiled code. Use it when no built-in method expresses what you need. For the example above, np.where(df["population_millions"] > 5, "large", "small") does the same job as one vectorized call.

Pandas 3.0 also adds pd.col(), which lets you refer to columns by name inside assign: df.assign(c=pd.col("a") + pd.col("b")). It replaces the more awkward lambda form and is handy in method chains.

Grouping: the split-apply-combine pattern

groupby is the operation that makes pandas feel powerful. The documentation describes it as three steps: split the data into groups by some key, apply a function to each group independently, and combine the results into a new structure. If you have ever built a pivot table in a spreadsheet, you have done this by hand.

# Average temperature by region
df.groupby("region")["avg_temp_celsius"].mean()

# Several summaries at once using named aggregation
df.groupby("region").agg(
    avg_temp=("avg_temp_celsius", "mean"),
    total_pop=("population_millions", "sum"),
    city_count=("city", "count"),
)

Named aggregation, where each keyword becomes an output column and each value is a (column, function) pair, gives clean column names without a cleanup step afterward.

agg, transform, and apply return different shapes

Method	Result	Use it to
agg	One row per group	Produce summary statistics
transform	Same index as the input	Add a group-level number back onto every row
apply	Whatever the function returns	Handle custom logic that the others cannot express

transform solves a common need: you want each row's share of its group total, or each value minus its group mean. For example, df["share"] = df["revenue"] / df.groupby("region")["revenue"].transform("sum") computes each order's share of its region's revenue while keeping the table at its original length. The documentation cautions that apply has to infer whether it should act as a reducer, a transformer, or a filter, so it can behave unexpectedly, and recommends built-in methods when they can do the job.

Two defaults that surprise people

First, groupby drops rows whose group key is missing. With the default dropna=True, a row with NaN in the key column vanishes from the result. Pass dropna=False to keep those rows as their own group. A sum computed over the groups can then fail to match the total of the column, and this is usually why.

Second, when grouping by a categorical column, the observed argument controls whether categories with no rows appear in the output. Setting observed=False includes empty categories with zero counts, while observed=True shows only categories present in the data. Passing it explicitly makes your intent clear.

Combining tables: merge, join, and concat

Real analysis usually involves more than one table: orders in one, customers in another. Pandas provides three tools, and the documentation distinguishes them clearly.

Function	What it does	Matches rows by
pd.concat()	Stacks objects along an axis	Position along one axis, with the other axis aligned by label
DataFrame.join()	Combines columns from two frames	The index by default
pd.merge()	SQL-style join on keys	Columns and/or index levels you name

# SQL-style join on a key column
merged = pd.merge(orders, customers, on="customer_id", how="left")

# Stack tables with the same columns vertically
combined = pd.concat([df_2024, df_2025], ignore_index=True)

# Place tables side by side, aligned on the index
combined = pd.concat([features, labels], axis=1)

The how argument maps directly to SQL. A left merge keeps every row of the left table and fills NaN where there is no match. An inner merge keeps only matching keys. An outer merge keeps all keys from both tables. A right merge is the mirror of left, and cross produces every pairing of rows.

Two safeguards that catch silent errors

The most dangerous merge bug is a many-to-many join that multiplies rows. If a key appears twice in both tables, the result contains every pairing of those rows, and the documentation warns that this "can cause a memory overflow." Totals computed afterward are inflated and nothing raises an error.

Two arguments protect you. validate="many_to_one" (or "one_to_one", "one_to_many") checks key uniqueness before merging and raises a MergeError if your assumption is wrong. indicator=True adds a _merge column with the values left_only, right_only, or both, so you can count how many rows failed to match before you filter them away.

checked = pd.merge(
    orders, customers,
    on="customer_id", how="left",
    validate="many_to_one",   # each order has one customer; customers are unique
    indicator=True,
)
checked["_merge"].value_counts()   # how many orders found no customer?

A good habit: compare row counts before and after every merge. If a left join on a unique key changes the number of rows, something is wrong with the keys.

Reshaping with pivot_table

A pivot table turns long data (one row per transaction) into a cross-tabulation (regions down the side, categories across the top).

pivot = df.pivot_table(
    values="sales",
    index="region",
    columns="product_category",
    aggfunc="sum",
    fill_value=0,
)

This is groupby followed by a reshape in one call. It is especially useful for preparing data for a heat map or a report table. For a statistical angle on cross-tabulated data, see our explainer on how correlation differs from causation, since a neat table of group totals is often where people first draw conclusions it cannot support.

Visualization with pandas and Matplotlib

Pandas has a built-in .plot() method that wraps Matplotlib. It is meant for fast exploratory charts, not publication graphics.

import matplotlib.pyplot as plt

# Line plot
df.set_index("date")["sales"].plot(title="Daily Sales", figsize=(10, 4))
plt.xlabel("Date")
plt.ylabel("Sales (GBP)")
plt.tight_layout()
plt.savefig("sales.png", dpi=150)
plt.show()

# Bar chart of grouped totals
df.groupby("region")["sales"].sum().plot(kind="bar", color="steelblue")
plt.title("Total Sales by Region")
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()

# Histogram of one numeric column
df["order_value"].plot(kind="hist", bins=30, edgecolor="white")
plt.title("Distribution of Order Values")
plt.show()

# Scatter plot of two columns
df.plot(kind="scatter", x="population_millions", y="avg_temp_celsius", alpha=0.7)
plt.title("Population vs Average Temperature")
plt.show()

Choose the chart to match the question. A histogram shows the shape of one variable, a bar chart compares groups, a line chart shows change over time, and a scatter plot shows the relationship between two variables. When you need statistical plots or interactive charts, seaborn and Plotly read DataFrames directly. And because a chart can mislead through its axes or scale choices as easily as through its data, it is worth reading how data visualization can mislead before sharing one. If you want to put a chart in front of other people without building a website, Streamlit turns a script into an interactive app.

A complete analysis, from raw CSV to chart

The following walkthrough strings the pieces together on a hypothetical sales file. The column names are invented for illustration.

import pandas as pd
import matplotlib.pyplot as plt

# 1. LOAD
df = pd.read_csv(
    "sales_2025.csv",
    parse_dates=["order_date"],
    dtype={"customer_id": "string"},
)

# 2. EXPLORE
print(df.shape)            # How big is it?
print(df.dtypes)           # Are dates parsed correctly?
print(df.isna().sum())     # Where are the gaps?
print(df.describe())       # Any suspicious extremes?

# 3. CLEAN
# Drop rows missing fields the analysis cannot do without
df = df.dropna(subset=["order_id", "customer_id", "revenue"])

# Label missing categories instead of discarding the rows
df["category"] = df["category"].fillna("Unknown")

# Remove impossible values, and count what was removed
before = len(df)
df = df[df["revenue"] >= 0]
print(f"Removed {before - len(df)} rows with negative revenue")

# Standardize text so "north " and "North" become one group
df["region"] = df["region"].str.strip().str.title()

# Remove exact duplicate orders
df = df.drop_duplicates(subset=["order_id"])

# 4. ENGINEER FEATURES
df["year"] = df["order_date"].dt.year
df["month"] = df["order_date"].dt.month
df["day_of_week"] = df["order_date"].dt.day_name()

# 5. ANALYZE
monthly = df.groupby(["year", "month"])["revenue"].sum().reset_index()
monthly["period"] = pd.to_datetime(monthly[["year", "month"]].assign(day=1))

top_regions = (
    df.groupby("region")["revenue"].sum().sort_values(ascending=False).head(5)
)

pivot = df.pivot_table(
    values="revenue",
    index="category",
    columns="day_of_week",
    aggfunc="sum",
    fill_value=0,
)

# 6. VISUALIZE
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
monthly.plot(x="period", y="revenue", ax=axes[0], title="Monthly Revenue 2025")
top_regions.plot(kind="barh", ax=axes[1], title="Top 5 Regions by Revenue")
plt.tight_layout()
plt.savefig("sales_analysis.png", dpi=150)
plt.show()

The six steps (load, explore, clean, engineer features, analyze, visualize) apply to almost any tabular project. Two habits in this script are worth copying. It prints how many rows each cleaning step removed, so a filter that deletes half your data does not pass unnoticed. And it keeps cleaning rules in code instead of editing the source file by hand, which makes the whole analysis repeatable when next quarter's file arrives.

Checking your own work

Before you trust a result, try three sanity checks. Recompute one number a second way, for example the total revenue with and without groupby, and confirm they match. Look at a handful of raw rows from each group to see that the labels mean what you think. And check that row counts before and after each merge and filter are what you expected.

Working with larger datasets

Pandas holds a table in memory, so a file larger than your available RAM will not load in one piece. Before reaching for another tool, the pandas scaling guide lists four tactics.

1. Load less data. Read only the columns you need. The guide's example, using columns= with read_parquet on a 40-column file, used about one-tenth of the memory. For CSV files, use usecols.
2. Use efficient dtypes. Convert text columns with few distinct values to category, and downcast numbers. In the guide's example, converting a name column to category cut it from about 13.7 million bytes to about 1.05 million, and downcasting integers and floats halved or quartered those columns. Total memory fell to about 42% of the original.
3. Process in chunks. With chunksize, read_csv returns pieces you can handle one at a time. This suits operations that need little coordination between chunks, such as counting values. The guide warns that operations such as groupby are much harder to do chunk by chunk.
4. Use another library. For operations that do not chunk cleanly, switch to a tool designed for out-of-core or parallel work.

# Load only what you need, with compact types
df = pd.read_csv(
    "big.csv",
    usecols=["date", "region", "revenue"],
    dtype={"region": "category"},
    parse_dates=["date"],
)

# Stream a huge file in chunks, keeping only a running total
totals = pd.Series(dtype="float64")
with pd.read_csv("big.csv", usecols=["region", "revenue"], chunksize=100_000) as reader:
    for chunk in reader:
        totals = totals.add(chunk.groupby("region")["revenue"].sum(), fill_value=0)

One note on query(): pandas' indexing guide says that using the numexpr engine with query() is only faster for frames with more than about 100,000 rows, so do not expect it to speed up small tables.

When the data genuinely outgrows one machine's memory, options include Polars, DuckDB, Dask, and Apache Spark. Our comparisons of Polars versus pandas and DuckDB versus pandas cover the single-machine choices, and Apache Spark addresses cluster-scale work. Under the hood, pandas builds on NumPy, so understanding arrays helps when performance matters.

Common mistakes and how to avoid them

Most beginner problems fall into a few patterns.

Looping over rows. Writing for i, row in df.iterrows() to compute a new column is slow and usually unnecessary. Try a vectorized expression first.

Ignoring the index after filtering. After df[df["x"] > 5], the original labels survive, so df.loc[0] may fail or pick an unexpected row. Call reset_index(drop=True) when you want fresh 0-based labels.

Trusting dtypes after loading. A column of dates that arrives as text, or numbers stored as strings because of one stray "N/A", will break arithmetic or sorting. Check df.dtypes and convert with pd.to_datetime or pd.to_numeric(errors="coerce").

Chained assignment. As covered above, df[mask]["col"] = value does not work in pandas 3.0. Use loc.

Merging without checking keys. Use validate= and compare row counts.

Treating a group total as the whole story. A table of averages hides spread and sample size. Always look at counts next to means, because a mean of three observations and a mean of three thousand are not equally trustworthy.

Forgetting that NaN skips silently. Aggregations such as sum and mean skip missing values by default. That is convenient, but a column that is 60% missing can still produce a confident-looking average of the remainder.

Where pandas fits in a data science workflow

Pandas typically sits in the middle of a pipeline: after you have extracted data from a file, API, or database, and before modeling or reporting. Upstream, you might pull raw tables with SQL. Downstream, DataFrames feed directly into scikit-learn for machine learning, into Matplotlib, seaborn, or Plotly for charts, and into notebooks and reports. For a broader view of the surrounding libraries, see our Python data science guide and the Python data libraries cheat sheet.

Frequently Asked Questions

Q: What is the difference between loc and iloc in pandas? A: loc selects by label (index values and column names) and includes both ends of a slice. iloc selects by integer position and excludes the end, like ordinary Python slicing. They match only when the index is the default 0, 1, 2 range. Use loc for labels and conditions, and iloc for "the nth row."

Q: How do I handle duplicate rows in a DataFrame? A: Use df.duplicated() to flag duplicates and df.drop_duplicates() to remove them. By default both compare every column. Pass subset to compare only some, for example df.drop_duplicates(subset=["order_id"]). Decide which copy to keep with the keep argument, and check why duplicates exist before deleting them.

Q: When should I use apply() instead of vectorized operations? A: Prefer vectorized column arithmetic, the .str and .dt accessors, and built-in methods, since they run in compiled code. Use apply() when the logic has no vectorized equivalent, such as a custom function involving several columns or an external library that works on single values. Expect it to be slower on large tables.

Q: What does inplace=True mean, and should I use it? A: inplace=True modifies an object directly instead of returning a new one. It often does not save memory, and it prevents method chaining. Under Copy-on-Write in pandas 3.0, calling it on a column selected from a DataFrame, such as df["a"].fillna(0, inplace=True), does not update df. Assign the result back instead: df["a"] = df["a"].fillna(0).

Q: How do I read only certain columns from a large CSV? A: Pass usecols to read_csv, for example pd.read_csv("data.csv", usecols=["date", "revenue", "region"]). Pandas then skips the other columns, which reduces memory use and often speeds up loading. Combine it with dtype to store low-variety text columns as category.

Q: What is the difference between merge() and join()? A: merge() is the general tool: it joins on any columns or index levels and supports inner, left, right, outer, and cross joins. join() is a convenience method that matches on the index by default and does a left join unless told otherwise. Most people use merge() for clarity and add validate= to catch unexpected duplicate keys.

Q: Is pandas still worth learning with Polars and DuckDB around? A: Yes, for most people. Pandas remains the common language of Python data work, most tutorials and libraries assume it, and its DataFrame concepts carry over to the alternatives. Alternatives can be faster or leaner for large data, so many analysts learn pandas first and add one later when a project needs it.

The takeaway

Pandas rewards a small set of habits more than a large vocabulary. Look at the data before you transform it. Prefer whole-column operations to loops. Make selections and assignments in one explicit step, which pandas 3.0 now requires. Group with groupby when the question is about categories, merge with validate= when it spans tables, and count rows after every step that can change them. The library will do exactly what you ask, so most of the skill is in asking about the right thing and checking that the answer is the size you expected.

Sources

• McKinney W (2010). Data structures for statistical computing in Python. Proceedings of the 9th Python in Science Conference, 51-56.
• pandas development team, "About pandas" (project history): pandas.pydata.org/about
• pandas development team, "What's new in 3.0.0 (January 21, 2026)": pandas.pydata.org/docs/whatsnew/v3.0.0.html
• pandas user guide, "Copy-on-Write (CoW)": pandas.pydata.org/docs/user_guide/copy_on_write.html
• pandas user guide, "Indexing and selecting data": pandas.pydata.org/docs/user_guide/indexing.html
• pandas user guide, "Working with missing data": pandas.pydata.org/docs/user_guide/missing_data.html
• pandas user guide, "Group by: split-apply-combine": pandas.pydata.org/docs/user_guide/groupby.html
• pandas user guide, "Merge, join, concatenate and compare": pandas.pydata.org/docs/user_guide/merging.html
• pandas user guide, "Scaling to large datasets": pandas.pydata.org/docs/user_guide/scale.html
• pandas user guide, "IO tools": pandas.pydata.org/docs/user_guide/io.html
• pandas API reference, DataFrame.dropna: pandas.pydata.org/docs/reference/api/pandas.DataFrame.dropna.html
• PyPI Stats, download counts for pandas (read October 2026): pypistats.org/packages/pandas