AI Engineering

A small CLI that turns any job posting into a validated record: title, seniority, must-have skills, nice-to-haves and location.

1. Define the schema first. A Pydantic or Zod model with every field required and nullable where unknown.
2. Collect twenty postings from different industries and save them as text fixtures.
3. Prompt for structured output and validate every response against the schema; on failure, retry once with the validation error in the prompt.
4. Report the failure rate before and after the retry, and write down which postings broke it and why.

Answer questions over one real documentation site, with a link to the passage behind every claim.

1. Crawl and chunk one docs site; record chunk size and overlap as parameters.
2. Embed and index the chunks in a local vector store.
3. Write thirty questions with known answers before you build the prompt.
4. Answer with citations and refuse when the top passages do not support an answer.
5. Tune one parameter at a time and keep a table of accuracy against chunk size and number of passages retrieved.

An agent that answers a question by searching, reading and summarising, within a hard limit on steps and cost.

1. Give it three tools: search, fetch a page, and write a note.
2. Enforce a budget of steps and tokens in code, not in the prompt.
3. Log every step so a run can be replayed and inspected.
4. Compare against a fixed workflow (search, read the top three, summarise) on ten questions, and say honestly which one won.

Make every change to the assistant from the RAG topic prove itself.

1. Do error analysis first: read fifty real outputs and label what went wrong.
2. Turn each failure type into a test, with code checks where possible and a model-graded check only where you must.
3. Check the grader against your own labels and report how often it agrees.
4. Run the suite in CI so a pull request shows the score change.

Take any feature you built earlier and make it safe and affordable to run.

1. Measure the baseline: p50 and p95 latency and cost per request.
2. Add a cache and a cheaper fallback model, and measure again.
3. Red-team it with ten prompt-injection attempts from the OWASP list and fix what gets through.
4. Add tracing so any bad answer can be followed back to its inputs.