articles are also a way to hone my craft for me. counter intuitively, writing clearly has become probably THE most important skill in a world of slop.

propaganda im falling for these days

part 4 of the evals series. WRITING the evaluators. part 3 was question + delete. whatever survived gets an evaluator. this is step 3: simplify and optimize. the question now is WHICH kind of evaluator. 1. code first. always. if code can check it, then let it: valid json, required fields present, right tool called, refund row actually exists in the db. cheap, fast, and it will never let you down. 2. grade the OUTCOME, not the claim. when your agent says “refund processed”. great. did it though? don’t ask an llm to read the transcript. check the database. @AnthropicAI and @langfuse both make this point: the transcript is what the agent SAYS, the environment is what it DID. your eval harness should be able to independtly verify OUTCOMES. 3. llm judge ONLY for what actually needs reading. things like: is the answer grounded in the retrieved docs? did it answer what was asked? one judge per failure. also go for simple PASS/FAIL. i've found that a rating like 0-5 gives the llm a middleground (eg. 3). FORCE it to pick sides. @eugeneyan calls that the “God Evaluator”. don’t build god. 4. write the judge prompt yourself. do NOT OVER ENGINEER. i like to include: the context they need ONE criterion, and what to ignore examples from the traces you graded in part 3 reasoning first, verdict last let it say “unknown” instead of guessing 5. think about this question: your judge agrees with your labels 92% of the time. should you ship it? if only 10% of your traces are failures, a judge that ALWAYS says pass agrees with you 90% of the time. and catches NOTHING 😭 thats why you should measure two numbers separately: TPR: of the real failures, how many did it catch TNR: of the good outputs, how many did it pass and test on labels you did NOT use to write the prompt. otherwise you’re grading your own homework. @HamelHusain and @sh_reya split labels into train/dev/test for exactly this. 6. a judge needs 100+ labels and someone maintaining it every week. so before you build one ask: can code check this? can a prompt fix kill it? most of the time, yes.

how are AI engg interviews different from normal sde interviews? in a normal system design round you design for scale. how much traffic it takes, and what breaks under load. AI rounds have all of that, plus the harness. imo you should build it around how humans already work and find where you can speed up their loop. for me a generic harness that does a bit of everything is my WORST ENEMY and introduces all sorts of failure modes in the process.

bro is steering individual neuron weights in an LLM to jailbreak

been listening to @hubermanlab for yearsss now its so cool to see him distill down the work of his life to 663 pages. i can feel the value radiating out of this thing, just gotta absorb it properly now. goated gift by my goated gf <3

saw this on a @HamelHusain article this is peak