numpy: arrays, dtypes & whole-array operations

Level 3: Level 3: Patternsmedium18 mindata-structuresperformancetype-coercioniteration

Why a fixed-dtype array beats a list of ints, and what that promise costs you.

Optional, and for whom

This module and the pandas lesson after it are a detour, not a step on the main path. If you are heading for backend, platform or general software work, the Level 3 spine you already finished is what interviews will ask about. If you are heading for data engineering, analytics or anything with a pipeline in the job description, these two are the vocabulary every one of those interviews assumes you have.

A list of ints is not an array of ints

A Python list is a block of pointers. Each element points off to a full Python object somewhere else in memory, carrying a type pointer and a reference count around with it. That is why a list can hold an int, a str and a dict at once: every slot is the same size because every slot is just an address.

A numpy array is the opposite trade. It is one contiguous block of raw values, all the same type and all the same width. That is why it has a dtype, singular, for the entire array:

import numpy as np

a = np.array([1, 2, 3])       # dtype int64, shape (3,)
b = np.array([1, 2, 3.0])     # dtype float64: one float promotes them all
np.zeros(5)                   # five float64 zeros
np.arange(0, 10, 2)           # array([0, 2, 4, 6, 8])

Everything good and everything annoying about numpy follows from that one decision.

Check yourself

You have arr = np.array([1, 2, 3]), so its dtype is int64. You then run arr[0] = 'hello'. What happens?

The array switches to a mixed dtype and stores the string alongside the numbers.It raises, because 'hello' cannot be converted to int64.It stores the length of the string, since numpy converts whatever it is handed.

Vectorized operations

Because the whole array shares a dtype, one operation can apply to all of it at once. No loop in your code, and no loop in Python at all:

a * 2         # array([2, 4, 6])   scalar broadcast over every element
a + a         # array([2, 4, 6])   elementwise
a.sum()       # 6
a.mean()      # 2.0
a[a > 1]      # array([2, 3])      a boolean mask selects

That last line is the pattern the pandas lesson leans on: a > 1 builds an array of booleans, and indexing with it keeps the positions that are True.

Why the vectorized sum wins

Check yourself

Summing a million numbers with arr.sum() is far faster than a Python for loop over the same values in a list. What is the main reason?

numpy spreads the work across every CPU core.The values sit in one contiguous typed block, so the loop runs once in C with no Python object per element.The total is precomputed when the array is built and simply read back.

Count the work for a million values. The Python loop dereferences a pointer, unboxes an int object, dispatches __add__, allocates a result object, and does it again, a million times. arr.sum() walks a million adjacent 8-byte integers in a single compiled loop with none of that per-element overhead. The usual result is one to two orders of magnitude, and it holds for * 2, +, comparisons and every other whole-array operation.

The rule this gives you: if you are writing a Python for loop over a numpy array, you have probably lost the reason you reached for numpy.

Broadcasting

A scalar stretches to fit an array, which is why a * 2 works at all. The general rule compares shapes from the right: two dimensions are compatible when they are equal, or when one of them is 1, and a length-1 dimension is stretched to match.

np.array([[1], [2], [3]]) + np.array([[10, 20, 30, 40]])
# shape (3, 1) + shape (1, 4) -> shape (3, 4)

Check yourself

You add an array of shape (3, 1) to an array of shape (1, 4). What comes back?

An error, because the two shapes are not the same.An array of shape (3, 4), with each side stretched along its length-1 dimension.An array of shape (3, 1), because the left operand decides the result shape.

The cost of a fixed dtype

Check yourself

arr = np.array([1, 2, 3]) has dtype int64. You run arr[0] = 3.7 and then print arr. What is the first element?

3.7, because assigning into a slot simply replaces what was there.3, because the float is truncated to fit the int64 dtype.3.7, and the whole array is promoted to float64 to hold it.

The same edge shows up in width. np.array([100, 100], dtype=np.int8) + 100 wraps around instead of growing, because an int8 slot cannot hold 200. Python's own int has no such limit, so this is a habit you have to acquire rather than one you already have.

What runs where

numpy is not bundled into the Python build that powers this browser sandbox, so import numpy fails here. In a real environment it is one pip install numpy (or uv add numpy) away, using the setup from "Running Python & installing packages", and every snippet above runs as written. The exercises below build the same mechanics by hand over ordinary lists, so the model you take to the terminal is the right one.

Interview nuance: "why is numpy faster than a list?" is a memory-layout question wearing a library costume. The answer is not "it is written in C". Plenty of slow things are written in C. The answer is that the data is one contiguous typed block, so a single compiled loop touches adjacent bytes with no per-element Python object to unbox and no dynamic dispatch per step. Then name the price: the dtype is fixed, so an incompatible value raises and a convertible one is silently cast.

Worked example (Python)

# numpy is not bundled into this browser sandbox, so this demo builds the same
# mechanic by hand: one operation applied across a whole sequence.
def scale(values, factor):
    return [value * factor for value in values]


readings = [1, 2, 3, 4]
print("scaled:", scale(readings, 2))
print("mask:  ", [value for value in readings if value > 2])
print("sum:   ", sum(readings))

# The fixed-dtype cost, by hand: storing a float in an int column truncates it.
print("int column stores 3.7 as", int(3.7))

Apply

Your turn

The task this lesson builds to.

Warm-up: implement broadcast_add(values, addend), the pure-Python version of what arr + other does in numpy.

When addend is a list, add the two sequences elementwise. When it is a single number, broadcast it across every element. Return a new list either way: ([1, 2, 3], 10) gives [11, 12, 13], and ([1, 2, 3], [10, 20, 30]) gives [11, 22, 33].

3 hints and 4 automated checks are waiting in the workspace.

Practice

Make it stick

A second problem on the same idea, so it survives past today.

Your team's analytics job dropped every fractional reading last quarter, and nobody could say why until somebody printed the dtype. Build the model that explains it: implement Vector in nparray/vector.py so it infers a single dtype and casts every value to it, combines elementwise with another Vector (raising ValueError on a length mismatch), broadcasts a scalar, and sums its values. The operators are already written for you. Some tests are hidden.

3 hints and 2 automated checks are waiting in the workspace.