Working with Data

Imagine a small chain of breakfast kedai with branches in Kuala Lumpur, Penang and Johor Bahru. Each branch sends in its sales every evening. One manager types “KL”, another “Kuala Lumpur”; one leaves a box empty when the till crashed; one writes “RM2.20” where everyone else writes “2.20”. None of this looks serious in a spreadsheet. Feed it to a model, though, and the model will learn — faithfully and confidently — from the mess. This page is about getting data into shape before anything learns from it.

NumPy: maths on whole arrays

Machine learning is arithmetic on large tables of numbers. Plain Python lists can hold numbers, but adding two lists glues them together rather than adding them, and looping over a million items one at a time is slow. NumPy solves both problems with the array: a block of numbers of one type, stored compactly, with arithmetic that applies to every element at once.

import numpy as np

prices = np.array([3.50, 1.80, 2.20, 1.90])   # nasi lemak, roti canai, teh tarik, kopi o
sold = np.array([120, 340, 410, 95])

revenue = prices * sold          # multiplies item by item, no loop
print(revenue)
print(revenue.sum(), revenue.mean())
print(prices > 2)                # a True/False mask
print(sold[prices > 2])          # use the mask to pick values
print(prices.dtype, sold.dtype, revenue.shape)

[420.  612.  902.  180.5]
2114.5 528.625
[ True False  True False]
[120 410]
float64 int64 (4,)

prices * sold multiplies the first price by the first quantity, the second by the second, and so on — no loop in sight. This style is called vectorisation, and it is the single most important habit in numerical Python: shorter to write, easier to read, and typically many times faster than a Python loop because the work happens in compiled code. prices > 2 produces an array of True/False values called a mask, and sold[prices > 2] uses the mask to keep only the matching values: the quantities of the two items costing more than RM2. Every array also has a dtype (the type of all its elements) and a shape.

Arrays can have more than one dimension. A table of sales — days down the side, items across the top — is a two-dimensional array.

import numpy as np

# rows = days (Mon, Tue, Wed), columns = the same four items
sold = np.array([
    [120, 340, 410,  95],
    [ 98, 310, 380, 110],
    [150, 400, 450, 130],
])
prices = np.array([3.50, 1.80, 2.20, 1.90])

print(sold.shape)
print(sold.sum(axis=0))          # total per item (down the columns)
print(sold.sum(axis=1))          # total per day (across the rows)
daily_revenue = (sold * prices).sum(axis=1)   # prices stretch to fit each row
print(daily_revenue.round(2))

(3, 4)
[ 368 1050 1240  335]
[ 965  898 1130]
[2114.5 1946.  2482. ]

The shape (3, 4) means 3 rows and 4 columns. The axis argument says which direction to add along: axis=0 collapses the rows and gives one total per column (per item); axis=1 collapses the columns and gives one total per row (per day). The last line uses a trick called broadcasting: prices has four values and sold has three rows of four, so NumPy stretches the prices across every row before multiplying. Images, audio, word embeddings and the weights inside a neural network are all just arrays with more dimensions; the same rules apply.

pandas: tables with names

NumPy arrays are fast but anonymous — columns are numbered, not named, and every value must share one type. Real business data has dates, text and numbers side by side. pandas adds the DataFrame: a table with named columns, each with its own type, built on NumPy underneath. Think of it as a spreadsheet you control entirely with code, so every step is repeatable.

Normally you would load a file with pd.read_csv("sales.csv"). To keep the example self-contained, we put the CSV text in a string and wrap it in io.StringIO, which makes a string behave like an open file.

import io
import pandas as pd

csv_text = """date,branch,item,qty,price
2026-09-01,Kuala Lumpur,Nasi lemak,120,3.50
2026-09-01,Kuala Lumpur,Teh tarik,200,2.20
2026-09-01,Penang,Nasi lemak,80,3.50
2026-09-01,Penang,Teh tarik,140,2.20
2026-09-01,Johor Bahru,Roti canai,90,1.80
2026-09-02,Kuala Lumpur,Nasi lemak,130,3.50
2026-09-02,Kuala Lumpur,Roti canai,160,1.80
2026-09-02,Penang,Roti canai,70,1.80
2026-09-02,Johor Bahru,Teh tarik,150,2.20
2026-09-02,Johor Bahru,Nasi lemak,60,3.50
"""
df = pd.read_csv(io.StringIO(csv_text))   # StringIO makes a string behave like a file

print(df.shape)
print(df.head(3))
print(df.dtypes)

(10, 5)
         date        branch        item  qty  price
0  2026-09-01  Kuala Lumpur  Nasi lemak  120    3.5
1  2026-09-01  Kuala Lumpur   Teh tarik  200    2.2
2  2026-09-01        Penang  Nasi lemak   80    3.5
date          str
branch        str
item          str
qty         int64
price     float64
dtype: object

Three habits to build on any new dataset: check shape (10 rows, 5 columns — is that what you expected?), look at the first few rows with head, and read the dtypes. Here pandas has correctly guessed that qty holds whole numbers (int64) and price decimals (float64), and that the other columns are text (str). Note that date is still text, not a real date; we will convert it during cleaning. The numbers down the left (0, 1, 2) are the index, pandas’ row labels.

Examples on this page were run with Python 3.13, NumPy 2.5, pandas 3.1 and matplotlib 3.11 As of 2026. pandas 3 shows text columns as str; older versions show object — same meaning.

Select, filter and group

Most questions you ask of a table are combinations of three moves: pick some columns, keep some rows, and summarise by group.

import io
import pandas as pd

csv_text = """date,branch,item,qty,price
2026-09-01,Kuala Lumpur,Nasi lemak,120,3.50
2026-09-01,Kuala Lumpur,Teh tarik,200,2.20
2026-09-01,Penang,Nasi lemak,80,3.50
2026-09-01,Penang,Teh tarik,140,2.20
2026-09-01,Johor Bahru,Roti canai,90,1.80
2026-09-02,Kuala Lumpur,Nasi lemak,130,3.50
2026-09-02,Kuala Lumpur,Roti canai,160,1.80
2026-09-02,Penang,Roti canai,70,1.80
2026-09-02,Johor Bahru,Teh tarik,150,2.20
2026-09-02,Johor Bahru,Nasi lemak,60,3.50
"""
df = pd.read_csv(io.StringIO(csv_text))

df["revenue"] = df["qty"] * df["price"]          # a new column, computed for every row
print(df[["branch", "item", "revenue"]].head(3))   # select columns

busy = df[df["qty"] >= 140]                        # filter rows
print(busy[["branch", "item", "qty"]])

penang_nl = df[(df["branch"] == "Penang") & (df["item"] == "Nasi lemak")]
print(len(penang_nl), "row(s) for nasi lemak in Penang")

         branch        item  revenue
0  Kuala Lumpur  Nasi lemak    420.0
1  Kuala Lumpur   Teh tarik    440.0
2        Penang  Nasi lemak    280.0
         branch        item  qty
1  Kuala Lumpur   Teh tarik  200
3        Penang   Teh tarik  140
6  Kuala Lumpur  Roti canai  160
8   Johor Bahru   Teh tarik  150
1 row(s) for nasi lemak in Penang

• New columns are vectorised: df["qty"] * df["price"] multiplies the two columns row by row, exactly like the NumPy example.
• Selecting: one column name in square brackets gives a single column; a list of names (note the double brackets) gives a smaller table.
• Filtering: df["qty"] >= 140 is a mask with one True/False per row, and df[mask] keeps the True rows. The original row numbers survive, which helps you trace results back to the source.
• Combining conditions uses & (and) and | (or), with each condition in its own brackets. Python’s ordinary and/or do not work on whole columns — a classic first error.

import io
import pandas as pd

csv_text = """date,branch,item,qty,price
2026-09-01,Kuala Lumpur,Nasi lemak,120,3.50
2026-09-01,Kuala Lumpur,Teh tarik,200,2.20
2026-09-01,Penang,Nasi lemak,80,3.50
2026-09-01,Penang,Teh tarik,140,2.20
2026-09-01,Johor Bahru,Roti canai,90,1.80
2026-09-02,Kuala Lumpur,Nasi lemak,130,3.50
2026-09-02,Kuala Lumpur,Roti canai,160,1.80
2026-09-02,Penang,Roti canai,70,1.80
2026-09-02,Johor Bahru,Teh tarik,150,2.20
2026-09-02,Johor Bahru,Nasi lemak,60,3.50
"""
df = pd.read_csv(io.StringIO(csv_text))
df["revenue"] = df["qty"] * df["price"]

print(df.groupby("branch")["revenue"].sum().sort_values(ascending=False))
print(df.groupby("item")["qty"].agg(["sum", "mean"]))
print(df["qty"].describe().round(1))

branch
Kuala Lumpur    1603.0
Penang           714.0
Johor Bahru      702.0
Name: revenue, dtype: float64
            sum        mean
item                       
Nasi lemak  390   97.500000
Roti canai  320  106.666667
Teh tarik   490  163.333333
count     10.0
mean     120.0
std       44.7
min       60.0
25%       82.5
50%      125.0
75%      147.5
max      200.0
Name: qty, dtype: float64

groupby("branch") splits the table into one group per branch; ["revenue"].sum() adds up each group’s revenue. This is the same idea as a spreadsheet pivot table, written as one line you can rerun tomorrow on new data. agg applies several summaries at once. describe gives a quick statistical portrait of a numeric column: how many values, the average, the spread (std, the standard deviation), the smallest and largest, and the quartiles — 25% of the rows sold 82.5 or fewer, half sold 125 or fewer. Run describe on every new dataset: an impossible minimum (a negative quantity) or an absurd maximum (a kedai selling 50,000 roti canai in a day) is often the first sign of bad data.

Cleaning messy data

Now the real data. Here are the same kedai, but as the branch managers actually typed it.

import io
import pandas as pd

messy = """date,branch,item,qty,price
2026-09-01,KL,Nasi lemak,120,3.50
2026-09-01,Kuala Lumpur,Teh tarik,200,2.20
2026-09-01,Penang,nasi lemak ,80,3.50
2026-09-01,penang,Teh tarik,,2.20
2026-09-02,KL,Nasi lemak,130,3.50
2026-09-02,KL,Nasi lemak,130,3.50
2026-09-02,JB,Teh tarik,150,RM2.20
2026-09-02,Johor Bahru,Roti canai,90,1.80
"""
df = pd.read_csv(io.StringIO(messy))
print(df.dtypes)
print(df.isna().sum())
print(df["branch"].unique())

date          str
branch        str
item          str
qty       float64
price         str
dtype: object
date      0
branch    0
item      0
qty       1
price     0
dtype: int64
<StringArray>
['KL', 'Kuala Lumpur', 'Penang', 'penang', 'JB', 'Johor Bahru']
Length: 6, dtype: str

The diagnosis is already visible. price is text (str), not a number, because one row says RM2.20. qty has become a float64 because it contains one missing value — pandas marks missing numbers as NaN (“not a number”), which only floats can hold. And the branch column has six spellings of three places. Before fixing anything, look at what each kind of problem does to your results.

import io
import pandas as pd

messy = """branch,qty
KL,120
Kuala Lumpur,200
Penang,80
penang,140
KL,130
JB,150
Johor Bahru,90
"""
df = pd.read_csv(io.StringIO(messy))
print(df.groupby("branch")["qty"].sum())

branch
JB              150
Johor Bahru      90
KL              250
Kuala Lumpur    200
Penang           80
penang          140
Name: qty, dtype: int64

Kuala Lumpur sold 450 items, but no line of this report says so. A manager reading it would conclude that KL and “Kuala Lumpur” are two middling branches. A model trained on it would learn two separate patterns from half the data each. Nothing crashed, nothing warned. That silence is what makes data cleaning so important. Each problem type has a standard remedy; tap through them.

import io
import pandas as pd

messy = """date,branch,item,qty,price
2026-09-01,KL,Nasi lemak,120,3.50
2026-09-01,Kuala Lumpur,Teh tarik,200,2.20
2026-09-01,Penang,nasi lemak ,80,3.50
2026-09-01,penang,Teh tarik,,2.20
2026-09-02,KL,Nasi lemak,130,3.50
2026-09-02,KL,Nasi lemak,130,3.50
2026-09-02,JB,Teh tarik,150,RM2.20
2026-09-02,Johor Bahru,Roti canai,90,1.80
"""
df = pd.read_csv(io.StringIO(messy))

# 1. inconsistent labels: tidy spacing and case, then map short forms
df["item"] = df["item"].str.strip().str.capitalize()
df["branch"] = df["branch"].str.strip().str.title()
df["branch"] = df["branch"].replace({"Kl": "Kuala Lumpur", "Jb": "Johor Bahru"})

# 2. wrong types: remove the "RM" prefix, then convert text to numbers
df["price"] = pd.to_numeric(df["price"].str.replace("RM", ""))
df["date"] = pd.to_datetime(df["date"])

# 3. exact duplicates: keep the first copy
print("duplicates:", df.duplicated().sum())
df = df.drop_duplicates()

# 4. missing values: here we choose to drop the row (and say so)
print("missing qty:", df["qty"].isna().sum())
df = df.dropna(subset=["qty"])
df["qty"] = df["qty"].astype(int)

print(df)
print(df.dtypes)

duplicates: 1
missing qty: 1
        date        branch        item  qty  price
0 2026-09-01  Kuala Lumpur  Nasi lemak  120    3.5
1 2026-09-01  Kuala Lumpur   Teh tarik  200    2.2
2 2026-09-01        Penang  Nasi lemak   80    3.5
4 2026-09-02  Kuala Lumpur  Nasi lemak  130    3.5
6 2026-09-02   Johor Bahru   Teh tarik  150    2.2
7 2026-09-02   Johor Bahru  Roti canai   90    1.8
date      datetime64[us]
branch               str
item                 str
qty                int64
price            float64
dtype: object

Step by step: .str gives access to text methods for a whole column at once — strip removes spaces at the ends, capitalize and title fix the case (which is how “penang” became “Penang”, and “KL” became “Kl” before the mapping fixed it). replace with a dictionary maps the short forms to full names. str.replace("RM", "") removes the currency prefix so to_numeric can convert the column. One exact duplicate is found and dropped, the row with the missing quantity is dropped, and astype(int) turns the quantities back into whole numbers. The final dtypes show a proper date column, a numeric price and integer quantities. Eight rows went in; six honest rows came out, and the code records exactly why.

A first chart

Numbers in a table are hard to compare at a glance; a chart makes the pattern obvious. matplotlib is the standard Python plotting library, and pandas works with it directly. This script totals revenue per branch from the cleaned data and saves a horizontal bar chart as an image file.

import io
import matplotlib
matplotlib.use("Agg")            # draw to a file, no window needed
import matplotlib.pyplot as plt
import pandas as pd

csv_text = """branch,item,qty,price
Kuala Lumpur,Nasi lemak,250,3.50
Kuala Lumpur,Teh tarik,200,2.20
Kuala Lumpur,Roti canai,160,1.80
Penang,Nasi lemak,80,3.50
Penang,Teh tarik,140,2.20
Penang,Roti canai,70,1.80
Johor Bahru,Nasi lemak,60,3.50
Johor Bahru,Teh tarik,150,2.20
Johor Bahru,Roti canai,90,1.80
"""
df = pd.read_csv(io.StringIO(csv_text))
df["revenue"] = df["qty"] * df["price"]
totals = df.groupby("branch")["revenue"].sum().sort_values()

fig, ax = plt.subplots(figsize=(6, 3))
ax.barh(totals.index, totals.values, color="#4338CA")
ax.set_xlabel("Revenue (RM)")
ax.set_title("Two days of sales by branch")
fig.tight_layout()
fig.savefig("sales_by_branch.png", dpi=150)
print(totals)
print("Saved sales_by_branch.png")

branch
Johor Bahru      702.0
Penang           714.0
Kuala Lumpur    1603.0
Name: revenue, dtype: float64
Saved sales_by_branch.png

matplotlib.use("Agg") tells matplotlib to draw into a file rather than open a window, which also makes the script work on servers; in a notebook you can leave that line out and the chart appears under the cell. plt.subplots creates a figure (the whole image) and an axes (the plotting area). barh draws horizontal bars — easier to read than vertical ones when labels are words. savefig writes the PNG file next to your script.

The saved image shows three indigo bars on a white background under the title “Two days of sales by branch”, with revenue in RM along the bottom axis from 0 to 1,600. Kuala Lumpur’s bar is more than twice as long as the others; Penang and Johor Bahru are almost identical. The same comparison, drawn with this site’s bars:

• Kuala LumpurRM1,603
  Two days, three items. About 53% of the chain’s revenue.
• PenangRM714
  Just ahead of Johor Bahru.
• Johor BahruRM702
  Within RM12 of Penang — too close to call from two days of data.

Make a chart of every dataset early, before any modelling. Charts catch problems that summaries hide: a branch with no data on Sundays, a sudden jump when the till software was updated, a price that was entered in sen instead of ringgit.

Why data quality decides model quality

Consensus Practitioners agree that the quality of a model’s training data limits the quality of the model, a point often summarised as “garbage in, garbage out”. A model has no way to know that “KL” and “Kuala Lumpur” are the same place, that a missing value means “till crashed” rather than “sold nothing”, or that a duplicated row is a clerical slip rather than a real second sale. It learns whatever patterns the dataset contains — including the accidental ones.

Three consequences follow. First, many data problems produce no error message; they produce plausible numbers that are wrong, as the grouped report above showed. Second, cleaning decisions are modelling decisions: filling missing quantities with zero teaches a sales model that crashes mean no customers. Third, cleaning must be repeatable, because the same steps have to be applied to every future batch of data the model sees. The next page turns these steps into part of a machine learning workflow. Practitioners commonly report that preparing data takes more of their time than building models; the precise percentages often quoted come from informal surveys, so treat them loosely.

Data quality is also about who and what is in the data, not only how tidy it is. A sales model trained only on KL branches will misjudge a new branch in Kota Bharu. A health model trained mostly on one population can perform worse for others — one of the routes by which bias enters AI systems. Cleaning cannot fix missing people; only better collection can.

Key terms on this page

Sources & further reading

• NumPy developers, NumPy: the absolute basics for beginners (NumPy documentation).
• NumPy developers, Broadcasting (NumPy documentation).
• pandas development team, 10 minutes to pandas and Working with missing data (pandas documentation).
• Matplotlib development team, Quick start guide (Matplotlib documentation).
• Charles R. Harris et al., “Array programming with NumPy”, Nature 585 (2020).
• Wes McKinney, Python for Data Analysis, 3rd edition (O’Reilly, 2022; free to read online).