A model coached a first-time solderer through a copper joint and then certified the result as clean and professional. A master plumber inspected the same joint: it would have leaked.
Design
One task with a hard, physical pass condition. Marc, with no soldering experience of any kind, was coached through sweating a copper joint by Grok over a live camera feed — clean the pipe, flux it, seat the fitting, light the torch, heat, feed the solder.
The premise, stated at 0:18: “AI is obviously coming after the white collar jobs. We're gonna see if it's coming after the blue collar jobs too.”
Ground truth was supplied by Bob, a master plumber, who inspected the finished joint without prompting and who also soldered a joint himself as a baseline. A soldered joint has an unambiguous pass condition — it either holds water or it does not — which is why a single-model trial is admissible here: the claim under test is the model's, not a comparison between models.
All findings are drawn from the published episode, Is AI coming for Plumbing?, and all timestamps refer to it.
The finding
False pass — certified work that failed inspection
| Source | Assessment of the same joint | At |
|---|---|---|
| Grok | “That joint looks good. Nice and shiny. Solder flowed all the way around. You did well.” | 5:14 |
| Grok | “You nailed it. Those joints look clean and professional.” | 5:24 |
| Bob — master plumber | “Yours didn’t flow all the way. You missed that. You would have had a leak at your house.” | 6:30 |
The model did not hedge, and it was not asked to guess — it volunteered a professional-grade verdict. A false pass is the most consequential error type this Index records, because it is the one a person acts on and stops checking. A model that says “I’m not sure, get it inspected” costs nothing. A model that says “you nailed it” ends the inspection.
The failure has a specific and instructive cause. Bob names it at 5:22: “Solder flowed well — she didn't see the bottom.” The model assessed the face of the joint that was pointed at the camera and issued a verdict about the whole of it.
Findings — each with a receipt
You nailed it. Those joints look clean and professional.
Expert inspection found the solder had not flowed all the way around. The joint would have leaked in service. The model certified work it had only partially seen, without qualifying the verdict to the part it could see.
Clean those copper surfaces… use the emery cloth… you got flux? … then slide the fitting on… light the torch, pull the solder away, and let it cool naturally…
The full procedure was delivered as one continuous block to a complete beginner holding a torch. Marc, immediately after: “It goes through that whole speech right and you just get lost.” The information was correct; the pacing made it unusable at the moment it was needed.
— “Okay, so I heat the pipe, not the fitting.”
— “No, actually do the opposite.”
Marc restated the instruction backwards and the model corrected him immediately and correctly. Heating the fitting rather than the pipe is the actual technique. Recorded because an index that logs only failures is not an instrument.
I can’t see anything in that shot. It’s all blurry. Is the flame lit now? Can you point the camera at the torch so I can see what’s happening?
The correct behaviour under uncertainty, and the direct counterpart to the false pass above. The same model that asked for a better view of the torch went on to certify a joint whose underside it had never been shown. The capability to say “I can’t see that” was present; it simply was not applied at the moment it mattered most.
Outcome & time
| Operator | Method | Time | Result |
|---|---|---|---|
| Marc | First time ever, AI-coached throughout | ~20 min | FAILED INSPECTION |
| Bob | Master plumber, no assistance | ~1 min | PASSED |
Times as stated on camera (Marc at 5:33, Bob at 6:24). Marc's own summary of what the coaching was worth, at 7:01: “It was good from an informational standpoint, but not an actual execution standpoint.”
That distinction is the substance of this trial. The instructions were substantially correct — clean to bright metal, flux both surfaces, heat the fitting not the pipe — and a beginner still produced a joint that would have failed in a wall. Knowing the procedure and being able to execute it are different things, and the model could not tell the difference from a camera feed.
Confounds & limits — disclosed
| Factor | Effect on attribution |
|---|---|
| Single model | Only one model was tested. This record supports no comparison against other models, and none is offered. |
| Camera framing | The model saw what a hand-held phone was pointed at. It could not have seen the underside of the joint — which is precisely why the unqualified verdict is the finding, rather than the missed defect itself. |
| Expert present | Bob was on camera throughout and gave his own guidance alongside the model's. Marc's execution cannot be attributed cleanly to the AI coaching alone. |
| One joint | A single attempt by a single beginner. It establishes that this failure happened, not how often it would. |
What this trial supports
Supported: the model issued an unqualified professional verdict on work it had only partially observed, and that verdict was wrong in a way that would have caused a water leak; the same model correctly refused to guess about the torch minutes earlier; procedure knowledge was largely sound; a coached beginner took roughly twenty times an expert's working time and still failed inspection.
Not supported: any claim about how often this occurs, any ranking against other models, or any conclusion about AI-assisted trades in general. One joint, one beginner, one model.
Cite this trial
Prompted Reality Index, Trial 002: The Solder Joint. Prompting LLC, 2026. promptedreality.ai/index/002/
Source footage: Is AI coming for Plumbing?, Prompted Reality. youtu.be/uaEo4uaIYC4
Run under The Prompted Reality Method, v1.1. Published under CC BY 4.0. This URL is permanent and will not be reused. If you believe a finding here is wrong, tell us.