Polars DataFrames skill for Claude Code

Good Polars code is lazy by default, expresses every transformation as expressions (no Python row loops), lets the optimizer push filters and column selection into the scan, declares schemas where inference is risky, and collects once at the end.

Ground rules

• Start from pl.scan_* (LazyFrame), not pl.read_*, for anything non-trivial.
• Express logic with pl.col(...) expressions inside select/with_columns/filter/agg. map_elements (Python UDF) is a last resort; it is single-threaded and opaque.
• Collect once. Multiple .collect() calls on the same plan recompute it.
• Polars has no index. Order is not guaranteed after group_by unless maintain_order=True or an explicit sort.
• null (missing) and NaN (float not-a-number) are different; fill_null does not touch NaN.
• Check the installed version (pl.__version__); this skill assumes Polars 1.x.

Workflow

- [ ] 1. Inspect schema and size (scan + collect_schema, head)
- [ ] 2. Build the lazy query with expressions
- [ ] 3. Check the optimized plan (explain)
- [ ] 4. Collect (in-memory or streaming) or sink to Parquet
- [ ] 5. Validate results: counts, nulls, totals
- [ ] 6. Benchmark against the old version if this is a migration

1. Inspect

import polars as pl

lf = pl.scan_parquet("data/events/*.parquet")
print(lf.collect_schema())
print(lf.head(5).collect())
print(lf.select(pl.len()).collect().item())

CSV with explicit types:

lf = pl.scan_csv(
    "orders.csv",
    schema_overrides={"order_id": pl.String, "customer_id": pl.String, "amount": pl.Float64},
    try_parse_dates=True,
    null_values=["", "NA", "null"],
)

Convert large, repeatedly read CSVs to Parquet once: pl.scan_csv("big.csv").sink_parquet("big.parquet").

2. Expressions

result = (
    lf
    .filter(pl.col("status") != "test")
    .with_columns(
        pl.col("email").str.strip_chars().str.to_lowercase(),
        pl.col("created_at").dt.truncate("1mo").alias("month"),
        pl.when(pl.col("amount") >= 100).then(pl.lit("large"))
          .when(pl.col("amount") >= 20).then(pl.lit("medium"))
          .otherwise(pl.lit("small")).alias("size_band"),
    )
    .group_by("region", "month")
    .agg(
        pl.len().alias("orders"),
        pl.col("amount").sum().alias("revenue"),
        pl.col("customer_id").n_unique().alias("customers"),
        pl.col("amount").filter(pl.col("size_band") == "large").sum().alias("large_revenue"),
    )
    .sort("region", "month")
)
df = result.collect()

Window functions (keep row count, compute per group):

lf.with_columns(
    (pl.col("amount") / pl.col("amount").sum().over("customer_id")).alias("share_of_customer"),
    pl.col("amount").rank("dense", descending=True).over("region").alias("rank_in_region"),
    pl.col("created_at").sort().diff().over("customer_id").alias("gap"),   # sort inside the window
)

Time series:

daily = (
    lf.sort("ts")
      .group_by_dynamic("ts", every="1d", group_by="device_id")
      .agg(pl.col("value").mean().alias("avg_value"))
)
rolling = lf.sort("ts").with_columns(
    pl.col("value").rolling_mean(window_size=7).over("device_id").alias("value_7")
)

Joins

joined = orders.join(
    customers, on="customer_id", how="left",
    validate="m:1",          # error if customers has duplicate keys
    coalesce=True,
)

Need	Join
Enrich, keep all left rows	how="left"
Rows with/without a match	how="semi" / how="anti" (no columns added)
Nearest earlier timestamp	join_asof(other, on="ts", by="symbol", strategy="backward") (both sorted)
Cartesian	how="cross" (check sizes first)

Key dtypes must match exactly (String vs Int64 raises); cast before joining.

3. Check the plan

print(result.explain())          # optimized plan

Look for the filter and the column projection appearing inside the scan node (SELECTION / PROJECT n/m columns). If a filter sits above a map_elements or a non-pushdown-able step, move it earlier.

4. Collect

Data size vs RAM	Do
Comfortably fits	.collect()
Larger than memory, aggregations/filters	.collect(engine="streaming")
Output is large	.sink_parquet("out.parquet") (streams without materializing)
NVIDIA GPU available, big aggregations	.collect(engine="gpu") with the GPU extra installed

Not every operation supports streaming; unsupported parts fall back to in-memory.

5. Validate

orders = lf.filter(pl.col("status") != "test").select("order_id", "amount").collect()
assert orders["order_id"].is_unique().all(), "duplicate order_id in source"
print(df.null_count())                                  # nulls per output column
assert abs(df["revenue"].sum() - orders["amount"].sum()) < 1e-6 * max(1.0, orders["amount"].sum())

Compare a headline total with the source system or the old pandas output (polars.testing.assert_frame_equal for exact migration checks, with check_dtypes=False when dtype mapping differs intentionally).

6. Migration from pandas

Read references/pandas-to-polars.md for the API mapping and semantic differences before converting a pandas script. Migrate function by function, keep the pandas version as an oracle, and compare outputs.

Interop is cheap when needed: df.to_pandas() (use use_pyarrow_extension_array=True to avoid copies) and pl.from_pandas(pdf).

Gotchas

• pl.read_csv then .lazy() already loaded everything; scan instead.
• with_columns expressions run in parallel and cannot see each other’s new columns within the same call; chain another with_columns to use a column just created.
• pl.col("a").sum().over("g") broadcasts; group_by("g").agg(...) collapses. Pick deliberately.
• pivot exists only on DataFrame (eager); collect first, or aggregate with group_by + conditional sums for a lazy equivalent. melt was renamed unpivot.
• String to number casts with strict=True (default) raise on bad values; use cast(pl.Float64, strict=False) and count the resulting nulls.
• is_in and joins on Categorical columns from different sources can mismatch; prefer pl.Enum with fixed categories or plain String.
• Row order after joins and group_by is not stable unless requested.
• Deprecated names from 0.x (groupby, apply, with_column, dtypes=) fail on 1.x.

Output

**Query:** monthly revenue by region from data/events/*.parquet (2.1B rows, 48 files)
**Plan:** filter and 6/41 columns pushed into scan; streaming engine
**Runtime / peak memory:** 38 s / 3.2 GB (pandas version: OOM at 64 GB)
**Checks:** order_id unique; total revenue matches warehouse within 0.01%
**Output:** out/revenue_by_region.parquet (1,404 rows)