When a grouping column is categorical, observed=True includes only categories present in the data; observed=False also includes unused categories. Set this option explicitly when a report needs a stable market list. A category appearing with zero rows does not, by itself, prove that the market had zero real-world activity.
Why did a country disappear after a pandas upgrade?
The official pandas documentation records a change to the groupby default: observed became True in version 3.0. A report that previously showed every declared category may therefore display fewer groups if the code relied on the default. Check the column dtype and the explicit parameter before assuming records were deleted.
This behavior applies to categorical groupers. A plain string column does not contain a declared list of unused categories to display. It is also separate from dropna, which controls grouping of missing keys. Adding observed=False will not automatically solve every missing-market problem.
Run a US, UK and Germany reporting example
These are synthetic order records. US, GB and DE identify the US, UK and Germany; all amounts are already expressed in USD for this teaching example. The categorical column declares three supported markets, but the extract contains records for only two. No currency conversion is performed.
import pandas as pd
orders = pd.DataFrame({
"market": pd.Categorical(
["US", "US", "GB"],
categories=["US", "GB", "DE"]
),
"amount_usd": [40, 60, 80]
})
seen = orders.groupby("market", observed=True).agg(
order_rows=("amount_usd", "size"),
revenue_usd=("amount_usd", "sum")
)
all_categories = orders.groupby("market", observed=False).agg(
order_rows=("amount_usd", "size"),
revenue_usd=("amount_usd", "sum")
)
print(seen)
print(all_categories)
assert seen.index.tolist() == ["US", "GB"]
assert all_categories.loc["DE", "order_rows"] == 0
assert all_categories["revenue_usd"].sum() == 180
With observed=True, the result contains US: 2 rows and $100; GB: 1 row and $80. With observed=False, the same results are accompanied by DE: 0 rows and $0. Both totals are $180. The difference is the displayed category coverage, not a change to the underlying orders.
Does the zero for Germany mean no sales?
Only if you know the extract is complete for Germany and the reporting period. A group created from category metadata is not an independently measured zero. If the German source failed to arrive, the zero is an aggregation artifact that should not be presented as confirmed revenue.
Keep source completeness as separate evidence. For example, assume the US and UK exports were received but Germany is still pending. Preserve the raw grouped totals for reconciliation, then mask the published revenue for incomplete markets:
complete = pd.Series({"US": True, "GB": True, "DE": False})
report = all_categories.copy()
report["source_complete"] = complete.reindex(report.index)
report["reported_revenue_usd"] = report["revenue_usd"].where(
report["source_complete"].eq(True)
)
assert pd.isna(report.loc["DE", "reported_revenue_usd"])
print(report[["source_complete", "reported_revenue_usd"]])
The completeness flags here are explicit scenario assumptions, not something inferred from the order rows. The published German revenue is now missing, while US and UK remain $100 and $80. Describe $180 as the known subtotal, not a complete three-market revenue total. Do not fill that missing value back to zero for chart convenience.
What if a new market arrives?
Validate raw market codes before converting them to a restricted categorical dtype. A code outside the declared categories can become missing during conversion, which can then be dropped from a default groupby. Maintain a market reference table and report unexpected codes rather than silently forcing every new country into an existing category.
For a dashboard requiring only specific market/product combinations, define that reporting grid deliberately. Expanding categorical combinations is not a substitute for business rules about where products are actually offered. Test the expected output labels as well as totals; a correct grand total can coexist with an incomplete breakdown.
How do you test this reporting change?
Check both row coverage and numeric reconciliation. This example asserts the observed labels, the unused category’s row count, the $180 subtotal and the missing published amount for an incomplete source. The explicit-option example was executed with pandas 2.2.3; the version-3.0 default change is sourced from the current official reference, not claimed as a local runtime test.
For related issues, read groupby with missing keys and why reindex does not fill existing missing amounts. Those address missing identifiers and missing values, rather than unused categorical labels.
Source and interview takeaway
Source: official pandas DataFrame.groupby reference. In an interview, explain the three separate decisions: which categories belong in the report, which records were observed, and whether each source is complete. Set observed explicitly, then communicate unavailable measurements honestly.