Classification Metrics That Matter: Confusion Matrix, Precision, Recall, F1, ROC/AUC, PR Curves, and Calibration

Let's start with the news, and the biggest item this week is OpenAI. Around July 9th, 2026, GPT-5.6 hit general availability, after a limited preview back around June 26th. The family ships in three tiers plus a high-effort mode: Sol the flagship, Terra the balanced middle, Luna the fastest and cheapest, and Sol Ultra on top. On pricing, per one million tokens, Sol is five dollars in and thirty out, Terra two fifty in and fifteen out, Luna one in and six out, with reportedly up to ninety percent off cached input. It landed in GitHub Copilot the same week. OpenAI published almost no standard benchmarks at launch. The one figure cited was Terminal-Bench 2.1, an agentic coding test, where Sol Ultra scored 91.9 percent and base Sol 88.8, against GPT-5.5 at 88.0. Altman reportedly touted about fifty-four percent greater token efficiency for coding. So read this as a price and efficiency play, not a raw capability jump.

The day before, on July 8th, xAI released Grok 4.5, positioned as their first model built for coding and agentic work. It's priced at two dollars in and six out per million tokens, reportedly more than sixty percent cheaper than Opus 4.8 and GPT-5.5. It's reportedly built on a 1.5-trillion-parameter foundation they call V9 and trained on real Cursor session data, though those specifics are unconfirmed. It reportedly ranks fourth on the Artificial Analysis Intelligence Index, and claims strong token efficiency, about fourteen thousand output tokens per index task versus roughly sixty-seven thousand for Opus 4.8. The benchmarks are mixed and self-reported, so treat the placement as a lead, not a verdict.

Google, meanwhile, reportedly delayed Gemini 3.5 Pro to around July 17th. Multiple outlets this week say DeepMind scrapped the 2.5 Pro base architecture for a full rebuild after early testers flagged token-efficiency and coding problems, plus weak multi-step math. Reported new features are a two-million-token context window and a Deep Think Reasoning Layer. That July 17th date is a rumor, not confirmed.

On the open-source side, around July 8th NVIDIA dropped new open Nemotron models for speech, multimodal retrieval-augmented generation, and safety, plus open datasets, training code, and blueprints, including the dataset and training code for the Llama Embed Nemotron 8B embedding model. Full open training data and code for a retrieval-embedding model is directly usable in a RAG pipeline.

In research, a paper on self-evolving agents argues deployed agents are frozen, weights and prompts and tools all static, and proposes three pillars: a vendor-neutral trajectory data protocol, a data proxy that turns real workloads into a replayable corpus, and an agent evolution control plane that decides whether to update memory, patch a skill, edit the harness, or update weights. It reframes improving an agent from retraining into a routing decision.

And in funding, the week of July 6th, Norm AI raised a hundred and twenty million dollar Series C led by Khosla Ventures, and Monogram, founded by Coursera co-founder Eren Bali, raised a forty million dollar seed led by DST Global and Lux Capital. Roughly four of every five dollars raised that week went to AI infrastructure.

Last episode we built logistic regression from the ground up. We covered the sigmoid, log-odds, cross-entropy or log loss as the thing it minimizes, the linear decision boundary, and we built a single neuron from scratch in NumPy. And right at the end, I made you a promise. I said accuracy has a trap on imbalanced data, and we'd unpack the full metrics story this episode. This is where I pay that off.

Here's the framing I want you to carry the whole way through. Remember the regression scorecard from the linear regression episode? R-squared, mean squared error, root mean squared error, mean absolute error. Those all answered one question: how far off is the number. This episode is the classification scorecard, the direct parallel. But classification asks a different pair of questions. Did you call the class right, and when you got it wrong, what did that mistake cost? Regression measures distance. Classification measures correctness and the price of being wrong. That shift, from distance to cost, is the whole reason we need more than one number.

And there's one idea I want you to hold onto above everything else, because nearly every evaluation mistake in the wild traces back to it. A classifier is really two things at once. It's a probability ranker, and it's a threshold. It ranks examples by how likely they are to be positive, and then it draws a line and says everything above this line is positive. Most of the metrics we'll talk about measure only one of those two things. Some measure the ranking, some measure the decisions at a particular threshold. Confuse the two and you'll draw the wrong conclusion almost every time. Keep that split in your head. Ranker, and threshold.

One more piece of setup, and then we build. Whatever model you swap in next episode, and next episode is the workhorses that win jobs, this episode is the ruler you measure it with. Metrics are model-agnostic. The ruler doesn't care what made the predictions. So learn it once, deeply, and you can evaluate anything.

Let's start with the object that everything else is built on. The confusion matrix.

For a binary problem there are exactly four things that can happen when your model makes a prediction. A true positive is when the actual answer is positive and you predicted positive. A true negative is when the actual answer is negative and you predicted negative. A false positive is when the actual answer is negative but you called it positive, that's a false alarm, and statisticians call it a Type One error. And a false negative is when the actual answer is positive but you called it negative, that's a miss, a Type Two error.

Here's a mnemonic that will save you every single time. In each of those two-word names, the second word is what the model said, and the first word is whether the model was right. So false positive means the model said positive, and it said so falsely. True negative means the model said negative, and it was true. Read them that way and you'll never scramble them.

Now here's why the confusion matrix matters so much. Everything downstream, accuracy, precision, recall, specificity, F1, the true positive rate, the false positive rate, every single one of them is just arithmetic on those four cells. If you own the four cells, you can derive the entire zoo from memory. You don't have to memorize a dozen formulas. You memorize four buckets and how to count them, and the rest falls out.

Let's talk about how scikit-learn actually lays this out, because there's a genuine trap here. The confusion matrix function in scikit-learn takes your true labels and your predicted labels. The rows are the true or actual class, and the columns are the predicted class. The entry in row i, column j, is the number of samples whose true label was i and whose predicted label was j. So the diagonal, top-left to bottom-right, is where true equals predicted. That's your correct predictions. For a standard binary problem with labels sorted as zero and one, the matrix comes out as true negatives in the top-left, false positives top-right, false negatives bottom-left, and true positives in the bottom-right. A common idiom is to flatten that matrix and unpack it in order as true negatives, false positives, false negatives, true positives.

And here's the pitfall, so mark it. This orientation is the opposite of what many textbooks and a lot of Wikipedia diagrams show you. Those often put predicted on the rows and actual on the columns, and they put the positive class in the top-left corner. Scikit-learn sorts labels in ascending order, so class zero, which is usually the negative class, lands in the top-left, and the true positives end up in the bottom-right. Neither convention is wrong, but they're mirror images, and if you assume the wrong one you'll read your own results backwards. So the rule is simple. Always state your convention out loud, to yourself and in your notebook.

Two more useful things on the confusion matrix. There's a normalize option, which can take the values true, pred, or all. If you normalize by true, each row sums to one, and the diagonal now shows you per-class recall. If you normalize by pred, each column sums to one, and the diagonal shows you precision. That's a nice way to eyeball those metrics straight off the picture. And scikit-learn gives you a display helper that plots the confusion matrix directly from an estimator and your data, or from a set of predictions, so you get the colored grid without hand-rolling a plot.

And this generalizes past binary. For a multiclass problem you get a K by K matrix. The diagonal is still correct predictions, and cell i, j, tells you how many class-i items got called class-j. Reading the off-diagonal cells is genuinely useful, because it tells you which classes your model confuses with which. The classic example is handwritten digits, MNIST, where you'll see fours getting misread as nines, because they look alike. The off-diagonal tells you exactly where the model is struggling. For multiclass, the per-class metrics are computed one class against all the others.

Okay. We have the four cells. Let's build the first metric, and it's the one everybody reaches for, and the one that lies to you.

Accuracy is true positives plus true negatives, divided by everything, all four cells. It's the fraction of predictions you got right. Scikit-learn's accuracy score computes it directly. Simple, intuitive, and dangerous on imbalanced data.

Let me show you the trap concretely. Imagine fraud detection, where one in a thousand transactions is actually fraud. That's zero point one percent positive. Now build the laziest model imaginable, one that predicts "not fraud" for every single transaction. What's its accuracy? Ninety-nine point nine percent. It looks spectacular. And it catches exactly zero fraud. It is completely useless and its accuracy is nearly perfect. Same story with a cancer screen where ninety-nine percent of scans are benign. Predict benign every time, score ninety-nine percent, miss every cancer.

Now connect this back to the workflow episode, where we met the DummyClassifier. Remember the most-frequent strategy, where the dummy just predicts the majority class for everything? Its accuracy equals the prevalence of the majority class. That's the point. On imbalanced data, accuracy's floor is already sky-high before your model does anything intelligent at all. So the real question is never "what's my accuracy," it's "how much did I beat the dummy." A metric with no baseline is a number with no meaning. Always stand your accuracy next to the DummyClassifier's accuracy, or it tells you nothing.

There's a name for this whole phenomenon: the accuracy paradox. A model with lower accuracy can be the more useful model. Why? Because accuracy weighs a missed fraud exactly the same as a false alarm. One count is one count. But in the real world those costs are wildly asymmetric. Missing a fraudulent charge costs you real money. A false alarm costs you a slightly annoyed customer. Accuracy is blind to that difference, and that blindness is the paradox.

There's a nice fix that's still simple: balanced accuracy. Scikit-learn has a balanced accuracy score, and it's just the mean of the per-class recall, the macro-averaged recall. Go back to that all-negative fraud model. Its recall on the positive class is zero, because it caught none of the real positives. Its recall on the negative class is one. Average those and you get zero point five. Balanced accuracy correctly exposes the useless model as no better than a coin flip, right where plain accuracy said ninety-nine point nine percent.

Now, I don't want you to over-correct here, so let me be fair to accuracy. Accuracy is genuinely fine when two conditions hold. When your classes are roughly balanced, and when the cost of a false positive and the cost of a false negative are roughly symmetric. In that world, accuracy is a perfectly good, honest summary. The failure is specifically imbalance and asymmetric cost. So don't walk away thinking "never use accuracy." Walk away thinking "check the balance and the costs first."

So if accuracy can lie, what do we replace it with? This is where precision and recall come in, and they're the heart of the episode.

Precision is true positives divided by true positives plus false positives. In words: of everything I flagged as positive, what fraction really was positive? It's also called the positive predictive value. The question precision answers is, can I trust a positive prediction from this model? When it raises its hand and says positive, how often is it telling the truth?

Recall is true positives divided by true positives plus false negatives. In words: of all the real positives out there, what fraction did I actually catch? Recall goes by several names, and you need to know all of them because they show up interchangeably: sensitivity, the true positive rate, the hit rate. The question recall answers is, how many did I miss?

There's a third one worth naming: specificity. Specificity is true negatives divided by true negatives plus false positives. Of all the real negatives, what fraction did I correctly clear? It's also called the true negative rate. And closely related is the false positive rate, which is false positives divided by false positives plus true negatives, and that's just one minus specificity. Hold onto the false positive rate, because it becomes the x-axis of the ROC curve later.

Scikit-learn has a precision score and a recall score. By default they use binary averaging with the positive label set to one. There's a subtlety with the zero-division parameter, because precision is undefined when the model never predicts positive at all, that's zero divided by zero, and scikit-learn will warn you and substitute a value rather than crash.

Now here's the single most important dynamic in this whole episode. Precision and recall trade off against each other as you move the decision threshold. Lower the threshold, and your model flags more things as positive. That catches more of the real positives, so recall goes up. But it also lets in more false alarms, so precision goes down. Raise the threshold, and the model becomes stingy, only flagging things it's very sure about. Precision goes up, but now you're missing more real positives, so recall goes down. You cannot improve both just by moving the threshold. To push both up at once you need a genuinely better model, better features, more data, and that moves the whole curve outward. The threshold just slides you along the curve you already have.

So which one do you optimize? That depends entirely on which mistake you can't afford, and let me give you several contrasting cases, because this is a judgment call, not a formula.

Cancer screening. A false negative means you missed a real cancer, and that's catastrophic. A false positive means a scare and a follow-up test, which is unpleasant but survivable. So you optimize recall. You would rather call in some healthy people than send one sick person home.

Spam filter. Flip it. A false positive means a real, important email got thrown in the spam folder and lost, maybe a job offer, maybe a contract. A false negative means one spam message reached your inbox and you delete it in two seconds. The false positive is far worse, so you optimize precision.

Fraud detection. This one's genuinely two-sided. A false negative is money lost to a fraudster. A false positive is a legitimate card getting declined at the checkout, which means friction and maybe a customer who leaves for good. Here the cost ratio between those two drives exactly where you set the threshold. There's no universal answer.

And it's worth knowing where precision and recall even come from. They were born in information retrieval, in search. The classic reference is van Rijsbergen. In search, precision is the fraction of the documents you returned that were actually relevant, and recall is the fraction of all the relevant documents that you managed to return. Same math, older field. Search engines were the original two-sided precision-recall problem.

So the takeaway is this. You pick your metric from the cost of the error you cannot afford. The choice of metric is a business decision dressed up as a math decision. Get the stakeholders to tell you which mistake hurts more, and the metric follows.

Now, sometimes you genuinely want a single number that balances precision and recall, and that's the F1 score. F1 is two times precision times recall, divided by precision plus recall. That's the harmonic mean of the two.

Why the harmonic mean, and not the plain arithmetic average? Because the harmonic mean is dominated by the smaller of the two values, and it punishes imbalance between them. Let me make that concrete. Take the all-positive model that flags everything. Its recall is one, it caught every positive, but its precision is tiny, because it flagged everything. The plain arithmetic mean of those rewards it, gives you something around zero point five one, which looks like a passing grade. The harmonic mean does not, it gives you something around zero point zero four. Or take the mirror case, precision one and recall zero. Arithmetic mean says zero point five. Harmonic mean says zero. F1 is only high when both precision and recall are high. It refuses to be fooled by one strong number papering over one terrible number.

There's an equivalent way to write F1 straight from the confusion cells: two times true positives, divided by two times true positives plus false positives plus false negatives. Notice what's missing from that formula. True negatives. F1 completely ignores true negatives. And that's exactly why it suits imbalanced problems, because on imbalanced data the true negatives are a huge, boring pile that dominate accuracy and tell you nothing. F1 throws them out and focuses on the class you care about.

Sometimes a plain balance of precision and recall isn't what you want, you want to tilt it. That's the F-beta score. F-beta is one plus beta-squared, times precision times recall, divided by beta-squared times precision, plus recall. The beta parameter controls how much you weight recall relative to precision. Beta equals one gives you plain F1. Beta equals two, the F2 score, weights recall twice as much as precision, which fits the disease case where misses hurt most. Beta equals zero point five, the F-0.5 score, weights precision more, which fits the spam case where false alarms hurt most. The clean way to remember it: beta is how many times more important recall is than precision.

A little history, because it shows up in interviews. The F-measure traces back to C. J. van Rijsbergen's 1979 book "Information Retrieval," where he actually defined a quantity called E, and F equals one minus E. The names "F1" and "F-score" got popularized later, through the MUC evaluations, the Message Understanding Conferences, in the early-to-mid 1990s.

Now, precision, recall, and F1 are defined for two classes, positive and negative. What happens with more than two classes? You have to decide how to average across the classes, and scikit-learn gives you an average parameter with a few options that you must understand, because they can tell very different stories.

Micro averaging pools all the classes' true positives, false positives, and false negatives into one global bucket, then computes the metric once. A quirk worth memorizing: in a standard single-label multiclass problem, micro precision, micro recall, and micro F1 all equal each other, and they all equal accuracy. Micro weights every sample equally, which means frequent classes dominate the number.

Macro averaging computes the metric separately for each class, then takes a plain unweighted mean. This weights every class equally, no matter how rare. That's what makes it valuable, because a class with three examples counts as much as a class with three thousand, so poor performance on a rare minority class actually shows up instead of getting drowned out. When the rare classes are the ones you care about, macro is usually the right call.

Weighted averaging computes the per-class metric and weights each by its support, its number of true instances. That gives a realistic overall summary, but the big classes dominate it, and one odd side effect is that the weighted F1 can come out to a value that isn't even between the overall precision and recall. And there's binary, the default for two classes, and None, which gives the per-class array with no averaging at all.

The single most useful one-call summary for all of this is scikit-learn's classification report. You hand it your true and predicted labels, and it prints per-class precision, recall, F1, and support, plus the overall accuracy, the macro average, and the weighted average, all in one block. You can set the number of digits and pass human-readable target names, and you can get it back as a dictionary if you want to compute on it. If you build one thing in your portfolio notebook, lead with the classification report. It's the fastest honest snapshot of a classifier.

Alright. We've mentioned the threshold a dozen times now. Let's make it the center of attention, because it's the pivot the whole episode turns on.

Callback to last episode. A classifier doesn't natively hand you a class. Logistic regression outputs a probability, by taking the sigmoid of the log-odds. The class only appears when you threshold that probability. When you call predict on a binary classifier, what's really happening under the hood is roughly: take the predicted probability of the positive class, and check whether it's at least zero point five. If it is, positive. If not, negative.

And here's the thing about that zero point five. It's a default, not a law. Zero point five is only the optimal threshold under a specific set of conditions: symmetric costs, a well-calibrated model, and balanced class priors. On an imbalanced or asymmetric-cost problem, and that's most real problems worth solving, the right threshold is almost never zero point five. Treating it as sacred is one of the most common and most expensive mistakes people make.

To move the threshold, you need the model's scores, not its hard class predictions. Two ways to get them. The predict-proba method gives you probabilities between zero and one, and for binary you take the second column, the probability of the positive class. The decision-function method gives you the raw signed score instead, which for logistic regression is the log-odds, and its natural threshold is zero rather than zero point five. Not every model exposes both. Trees and forests, for example, only give you predict-proba, not a decision function. So reach for predict-proba as your default.

Moving that threshold is the actual mechanism behind the precision-recall tradeoff we discussed. And more than that, sweeping the threshold from one extreme to the other is what generates the ROC curve and the precision-recall curve. Each single point on those curves is one specific threshold. Keep that in mind, it makes the curves click.

How do you choose a threshold in a principled way, rather than just leaving it at the default? A few honest approaches. You can fix a review budget: if your fraud team can only investigate a hundred alerts a day, pick the threshold that produces a hundred alerts and take the best you can get. You can target a minimum recall, say "I must catch at least ninety percent of cancers," and then take the best precision available at that recall. You can maximize F1 or F-beta over a grid of candidate thresholds. Or you can use Youden's J statistic, which is the maximum over all thresholds of the true positive rate minus the false positive rate. That's equivalent to sensitivity plus specificity minus one, and geometrically it's the point on the ROC curve that sits farthest above the diagonal. Youden's J, from W. J. Youden in 1950, is a good default when your costs are roughly symmetric.

Scikit-learn version 1.9 gives you real tooling for this. There's a tuned-threshold classifier that uses cross-validation to pick the threshold that optimizes a metric you name, and there's a fixed-threshold classifier that just applies a threshold you've already chosen at predict time, so your deployed model bakes in the right cutoff.

But here's a pitfall that trips up even experienced people, so burn it in. Choosing a threshold is model fitting. It is a parameter you are learning from data. So if you pick your threshold by looking at the test set, and then report your metrics on that same test set, you have leaked. You've fit to the test data and your reported numbers are optimistic lies. Tune the threshold on validation data or with cross-validation, and touch the held-out test set exactly once, at the end, to report.

Now let's talk about the curves, starting with ROC.

ROC stands for Receiver Operating Characteristic, and that name is a genuine historical artifact we'll get to. The ROC curve plots the true positive rate, which is recall, on the y-axis, against the false positive rate, which is one minus specificity, on the x-axis, as you sweep the threshold across its whole range. Each threshold gives you one point.

Walk the corners with me. At a threshold of positive infinity, the model calls nothing positive, so both rates are zero, and you're at the bottom-left corner, the origin. At a threshold of negative infinity, the model calls everything positive, so both rates are one, and you're at the top-right corner. A perfect classifier passes through the top-left corner, where the true positive rate is one and the false positive rate is zero, catch everything, false-alarm on nothing. The diagonal line from bottom-left to top-right is random guessing. Above the diagonal is better than chance. Below the diagonal is worse than chance, and if you ever find yourself there, you can just invert your predictions to beat it.

Scikit-learn's ROC curve function takes true labels and scores and returns three arrays: false positive rates, true positive rates, and the thresholds. One API note for the current version, the very first threshold it returns is infinity, which corresponds to that "predict nothing positive" point at the origin. There's also a display helper to plot the ROC curve straight from an estimator or from predictions.

Now collapse that whole curve into one number, and you get ROC AUC, the area under the ROC curve. It runs from zero to one. Zero point five is random, the area under the diagonal. One is perfect. Below zero point five is worse than random.

Here's the interpretation that turns up in interviews, so know it cold. ROC AUC equals the probability that your model gives a higher score to a randomly chosen positive example than to a randomly chosen negative example. It is literally a measure of ranking quality. It's the normalized version of the Mann-Whitney U statistic. And this is the punchline of that "ranker versus threshold" idea from the top of the episode. AUC measures how good your ranker is, entirely independent of any threshold. That's completely different from precision and recall, which measure how good your decisions are at one specific threshold. AUC is a ranker metric. Precision and recall are threshold metrics.

That threshold-independence is both the selling point and the catch. The selling point: AUC summarizes performance across every possible threshold, so it's excellent for comparing the raw ranking power of two models. The catch: it does not tell you how your model performs at the specific threshold you actually deploy. A great AUC does not guarantee a usable operating point.

For the AUC function, one requirement you must respect: it needs scores or probabilities, not hard zero-one labels. If you feed it predict output, you get garbage. There's a max-false-positive-rate parameter if you only care about a slice of the curve, a partial AUC. And for multiclass you have to specify a strategy, one-versus-rest or one-versus-one.

The history here is worth telling. ROC was born with radar operators in World War Two, around 1941 and after. An operator staring at a scope had to decide whether a blip was an enemy plane or just noise. The "operating characteristic" described how willing that operator was to call a blip a plane, trading real detections against false alarms by, in effect, moving a threshold in their head. Be trigger-happy and you catch every plane but panic at every bit of noise. Be cautious and you stay calm but miss real bombers. After the war this got formalized into signal detection theory, the classic reference is Green and Swets in 1966, and from there it spread into radiology, psychophysics, and eventually machine learning. So here's the hook. The precision-recall knob you turn on a fraud model is the same knob a radar operator turned in 1942, eighty years apart. And the link between AUC and the Mann-Whitney statistic was nailed down by Hanley and McNeil in 1982.

Now, ROC has a serious weakness, and it's the reason we need one more curve. Let's talk about the precision-recall curve.

The PR curve plots precision on the y-axis against recall on the x-axis as you sweep the threshold. Scikit-learn's precision-recall curve function returns precision, recall, and thresholds, with a small quirk: the precision and recall arrays are one longer than the thresholds array, because they tack on an endpoint where precision is one and recall is zero that has no corresponding threshold.

And here's a contrast you have to internalize. The baseline for a PR curve, the "random model" line, is not the diagonal. It's a horizontal line at a height equal to the prevalence of the positive class. If one percent of your data is positive, a random model sits at a precision of zero point zero one all the way across. That's completely different from ROC, where random is always the diagonal regardless of prevalence.

You collapse the PR curve into a single number too, and that's average precision. Average precision is the weighted mean of the precisions at each threshold, weighted by how much recall increased at that step. Written out, it's the sum over all thresholds of the change in recall from the previous step, times the precision at this step. It's deliberately a step-sum, not a trapezoidal area. And that's on purpose, because the PR curve is jagged, it sawtooths, and a trapezoid would optimistically interpolate across those teeth and overstate your performance. Scikit-learn specifically warns you against just calling its generic area function on recall and precision for this reason. There's a display helper for the PR curve too.

So why does the PR curve tell the truth on heavy imbalance when ROC lies? It comes down to that false positive rate on the ROC x-axis. The false positive rate is false positives divided by false positives plus true negatives. When negatives massively outnumber positives, the true negative count is enormous. So even a flood of false positives barely nudges the false positive rate, because it's swamped by that giant true-negative denominator. The result: the ROC curve hugs the top-left, and the AUC looks fantastic, even while your model is drowning you in false alarms relative to the handful of true positives you actually found. Precision has no true-negative term at all, it's true positives over true positives plus false positives. So precision feels that flood directly, and the PR curve tells you the ugly truth.

Here's the concrete number that makes it stick. On a data set that's one percent positive, a model can show a ROC AUC of around zero point ninety, which looks great, while its average precision is around zero point thirty, which is grim. Same predictions, same model, two completely different stories. So the rule is: when the positive class is rare and it's the class you care about, look at the PR curve and average precision, not ROC AUC. The references for this argument are Davis and Goadrich at ICML in 2006, and Saito and Rehmsmeier in PLOS ONE in 2015.

We've got one more family of metrics, and it's a different flavor entirely. Everything so far cared about classes or rankings. This last group cares about the actual probability numbers, whether they mean what they say.

Start with log loss, also called cross-entropy. And this is a beautiful callback, because log loss is literally the loss function logistic regression minimized last episode. Now we're reusing it as an evaluation metric. For a binary problem, log loss is the negative average, over all samples, of the true label times the log of the predicted probability, plus one minus the true label times the log of one minus the predicted probability. In plain terms, it rewards being confident and correct, and it punishes being confident and wrong, harshly. Lower is better, zero is perfect. Scikit-learn's log loss function needs probabilities, not hard labels. And the important property: log loss uses the actual magnitude of your probability, so it rewards calibrated confidence, not just getting the class or the ranking right.

Next, the Brier score. The Brier score is just the mean squared error between your predicted probability and the zero-or-one outcome. Sum up the squared differences between predicted probability and actual outcome, and average. It runs from zero to one, lower is better. It's from Glenn W. Brier in 1950, and it came out of weather forecasting, rating how good a "seventy percent chance of rain" forecast really was. Scikit-learn's Brier score function, in version 1.9, renamed its probability argument and added a parameter to optionally scale the score by a half. The contrast with log loss is worth holding: Brier is a bounded, quadratic penalty, so it's gentler on confident mistakes, while log loss is unbounded and logarithmic, so it's brutal on them. A confident wrong prediction costs you a lot under log loss and only a moderate amount under Brier. And both of them are what statisticians call proper scoring rules, meaning they're minimized when you report your true believed probability. They reward honesty about your own uncertainty. You can't game them by shading your numbers toward zero or one.

That brings us to calibration, which is the question underneath both of those. Do your probabilities mean what they say? If your model calls a batch of samples "seventy percent positive," then in a well-calibrated model, about seventy percent of them should actually turn out positive. And here's the thing that surprises people. A model can have a wonderful AUC, meaning it ranks beautifully, and still have terrible calibration, meaning the actual probability magnitudes are way off. Ranking and calibration are different properties, and a model can be great at one and awful at the other.

You inspect calibration with a reliability diagram. You bin the samples by predicted probability, then for each bin plot the mean predicted probability against the observed fraction that were actually positive. Perfect calibration is the diagonal line. Scikit-learn's calibration curve function does the binning, with a strategy of uniform bins or quantile bins. If your model isn't calibrated, there are fixers, and I'll just name them: the calibrated-classifier tool with either sigmoid calibration, which is Platt scaling, or isotonic calibration. As a rough guide, logistic regression is usually already fairly well calibrated, which makes sense given it optimizes log loss directly. Support vector machines, naive Bayes, and boosted trees often are not, and boosting in particular pushes its probabilities toward the extremes of zero and one.

Let me pull the common pitfalls together into one checklist, because knowing how to recognize each one in the wild is half the battle.

One. Optimizing accuracy on imbalanced data. You recognize it when your accuracy sits suspiciously close to the majority-class prevalence, or when the confusion matrix has an empty positive-prediction column. The fix is the DummyClassifier baseline, plus balanced accuracy, F1 or PR, and per-class recall.

Two. Reporting a single-threshold metric when the threshold is arbitrary. You recognize it when someone quotes one F1 or one precision number with no threshold and no curve behind it. The fix is to report the curve, the chosen threshold, and why you chose it.

Three. Trusting ROC AUC on extreme imbalance. You recognize it when the AUC is around zero point ninety-five but the precision at every threshold you'd actually use is dismal. The fix is PR and average precision, and precision at your operating recall.

Four. Test-set leakage through threshold tuning or metric-driven model selection. You recognize it when the threshold or the "best" model was chosen by peeking at test metrics. The fix is to tune on validation or cross-validation, and touch test once.

Five. Feeding hard labels where scores are required. The AUC function, the ROC curve function, the average precision function, and log loss all need probabilities or scores. Pass them zero-one predict output and you get a degenerate two-point curve and a meaningless AUC.

Six. Wrong positive label or label ordering. If your positive class is actually zero, or a string like "malignant," then the default positive label of one and the ascending sort will silently compute everything for the wrong class. Live example: in scikit-learn's breast cancer data set, target zero is malignant and target one is benign, which is probably the opposite of what you'd assume. Set the positive label explicitly.

Seven. Confusing the confusion matrix orientation, assuming top-left is true positives when scikit-learn puts it bottom-right for labels zero and one. Eight. Comparing F1 scores across different class distributions, or averaging without stating which average you used. And nine, reporting metrics with no measure of variability. A single test-set number carries sampling noise, wild on small or rare-positive data sets. Cross-validate or bootstrap so you know the error bars.

Let me close by sketching the worked example you should build, because this is the "build it, open it up, rebuild it from scratch" rhythm this show runs on.

Grab two data sets. The breast cancer set, five hundred sixty-nine samples, about thirty-seven percent malignant, and remember target zero is malignant. And a synthetic imbalanced set from the make-classification helper, ten thousand samples weighted ninety-nine to one, roughly one percent positive. Import the full metrics family.

The flow goes like this. Make the imbalanced data, and split it with stratification turned on, a callback to the workflow episode, because without stratify your test set might contain almost zero positives. Fit a DummyClassifier with the most-frequent strategy and watch it score around ninety-nine percent accuracy with zero recall, looks amazing, does nothing. Then fit logistic regression, and note that a balanced class-weight setting is a lever you can pull, though we'll defer it. Get hard predictions from predict and scores from predict-proba's positive column. Build the confusion matrix, unpack the four cells, plot it. Run the classification report with three digits and watch the accuracy sit near ninety-nine percent while the positive-class recall tells the real story. Compare ROC AUC against average precision and watch the gap open up, plot both curves, and draw the PR baseline at the prevalence. Then adjust the threshold: get the false positive rates, true positive rates, and thresholds from the ROC function, compute Youden's J as true positive rate minus false positive rate, pick the threshold at its maximum, remembering that first threshold is infinity, and recompute your confusion matrix, precision, and recall at that tuned threshold versus zero point five to see the move, tuning on validation not test. Compute log loss and the Brier score, and draw the reliability diagram from the calibration curve. Finally, rebuild it all in raw NumPy: compute the four cells with boolean masks, derive accuracy, precision, recall, specificity, false positive rate, and F1 by hand, and assert they're numerically close to scikit-learn's. Optionally rebuild ROC from scratch too, sort by score descending, sweep, integrate, and confirm that AUC really is the fraction of positive-negative pairs your model ordered correctly.

So that's the classification scorecard. The confusion matrix is the foundation, four cells, everything is arithmetic on them. Accuracy lies on imbalance, so beat the dummy. Precision and recall trade off along the threshold, and you pick between them by the cost of the error you can't afford. F1 balances them, F-beta tilts the balance. ROC AUC measures your ranker, PR and average precision tell the truth when positives are rare, and log loss, Brier, and calibration ask whether your probabilities mean what they say. Ranker, and threshold. Hold those two apart and you'll evaluate anything cleanly. Next episode, the workhorses: k-nearest neighbors, decision trees, random forests, and gradient boosting, and this is the ruler we'll measure them with.