AI Foundations

One notebook, NumPy only, predicting median house value from the eight California Housing features. You write the closed-form solution and the gradient-descent solution, and prove they land in the same place.

1. Load and split. fetch_california_housing(as_frame=True), an 80/20 split with a fixed random_state, then standardise every feature using the training mean and standard deviation only.
2. Closed form first. Solve the normal equations with np.linalg.lstsq and record the test RMSE. That number is the target everything else has to reach.
3. Derive the gradient before you type it. Write it out in a markdown cell, then implement (2/n) * X.T @ (X @ w - y) as one vectorised expression — no Python loop over rows.
4. Train. Batch gradient descent, learning rate 0.1, 1,000 iterations, loss recorded every step. Plot it on a log y-axis; it must decrease monotonically.
5. Prove both agree. Assert your final weights are within 1e-3 of the closed-form weights, then gradient-check three random coordinates against a central difference at eps = 1e-5.
6. Break it deliberately. Re-run at learning rate 1.0, and again on unstandardised features. Two sentences on what diverged in each case and why.

Scope guard: no scikit-learn models, no regularisation, no feature engineering. If you cannot say what every line of the backward pass does, it is not finished.

Kaggle's Titanic set — 891 training rows — worked as a modelling exercise rather than a leaderboard chase. Every change has to earn its place against a cross-validated score.

1. Two dumb baselines first. Predict "nobody survives", then predict from Sex alone. Record both accuracies; every later model is measured against them.
2. Close the leak before you start. 5-fold StratifiedKFold, with every imputation and encoding step inside a scikit-learn Pipeline so fold statistics never touch the validation fold.
3. Deal with the three broken columns. Age (177 missing), Cabin (687 missing), Embarked (2 missing). Pick a strategy for each and write one line justifying it.
4. Add features one at a time. Title parsed from Name, family size from SibSp + Parch, deck letter from Cabin, fare per person. Re-run CV after each and record mean and standard deviation.
5. Stop when the gain is noise. Keep a feature only if the CV mean improves by more than the standard deviation across folds. Drop the rest and record that you did.
6. Submit once. Compare the public leaderboard score against your CV estimate and write down the gap and your explanation for it.

Scope guard: one model family, no ensembling, no tuning beyond a small grid. The table of what failed is worth more than half a percent of accuracy.

A complete pipeline on UCI Bank Marketing — 41,188 marketing calls, 20 features, and only 11% of customers subscribing. The imbalance is the point: accuracy is useless here, so you have to choose a metric and defend it.

1. Split by time, not at random. The rows are ordered by campaign date. Hold out the final 20% chronologically so the evaluation matches how the model would actually be used.
2. Remove the leaking column. duration is only known once the call has ended — the dataset documentation says so explicitly. Drop it, and record the accuracy you gave up by doing so.
3. One Pipeline, no exceptions. A ColumnTransformer doing median-impute plus scaling on the numeric columns and most-frequent-impute plus one-hot on the ten categorical ones. Nothing fitted outside it.
4. Choose the metric on purpose. Compare logistic regression against gradient boosting on PR-AUC rather than ROC-AUC or accuracy, and write two sentences on why that is right at an 11% positive rate.
5. Calibrate. Wrap the winner in CalibratedClassifierCV and plot a reliability curve before and after. State the operating threshold you would ship and the cost assumption behind it.
6. Slice the errors. Break test error down by age band, job and contact month. Name the single worst-served segment and say what you would do about it.

Scope guard: no stacking, no AutoML, no search beyond a five-point grid. A calibrated model with a named worst segment beats an uncalibrated model with a prettier score.

A two-layer network on MNIST, written twice: once with forward and backward passes you derive yourself in NumPy, once in PyTorch. The deliverable is the proof that the two agree.

1. Fix the architecture. 784 → 128 (ReLU) → 10, softmax with cross-entropy, batch size 64, SGD at 0.1, 10 epochs. Same seed and same initialisation in both implementations.
2. Derive first. Work out dL/dz for softmax plus cross-entropy on paper and confirm it collapses to (p - y). Nearly everyone gets this wrong once — do it before writing code.
3. NumPy version. Forward, backward and update loop by hand, caching activations on the way forward because you need them on the way back.
4. Gradient check. Compare your analytic gradients against central differences on ten random parameters. Relative error must be below 1e-6 before you train anything at all.
5. PyTorch version. Identical architecture and seed. Run one forward and one backward pass on the same batch and assert the gradients match your NumPy ones to 1e-5.
6. Report. Both should land around 97–98% test accuracy. Plot the two loss curves on one axis and note the wall-clock difference.

Scope guard: no convolutions, no Adam, no augmentation. This project is about gradients, and extra architecture only hides mistakes in them.

Retrieval over the HotpotQA distractor corpus, evaluated with real numbers. HotpotQA ships gold supporting paragraphs for every question, so retrieval recall is something you compute rather than something you claim.

1. Index. Take 1,000 dev questions and their paragraph pool. Chunk at roughly 300 tokens with 50 of overlap, embed, and store in one local vector database — FAISS, Chroma or pgvector. Pick one and stop shopping.
2. BM25 baseline. Keyword retrieval alone, top-5, scored as recall@5 against the gold supporting paragraphs. Dense retrieval has to beat this number to justify itself.
3. Dense, then hybrid. Add embedding search, then a hybrid of both, and put recall@1, @5 and @10 for all three approaches in one table.
4. Generate under constraint. The model must cite chunk IDs and must answer "not in the corpus" when retrieval misses. HotpotQA questions are multi-hop, so single-shot retrieval will fail — measure how often.
5. Score the answers. Exact match and token F1 against gold answers on 200 questions, plus a hand-checked faithfulness pass on 50: does every claim appear in a cited chunk?
6. Change one thing per run. Chunk size, a cross-encoder reranker, query rewriting — one variable at a time, each a row showing its effect on recall@5 and answer F1.

Scope guard: no agent framework, no UI beyond a script. The table of retrieval changes against measured recall is the entire portfolio value here.