Simple questions to an LLM like: “review this procedure, is it internally self consistent” will find issues, real issues - usually minor - but undeniable when you read the review, the problem it describes is absolutely present in the document, page 14 says “X” and page 23 says “not X” and the context is basically identical - so which are we really supposed to do: “X” or “not X”? Well, that depends. Depends on what? Additional context and detail that the already too long 50 page procedure doesn’t address.
lol and I find issues like this in LLM output of only 1 or 2 pages. LLMs can’t even be consistent across paragraphs, much less hundreds of pages of context window. what an insane claim.
lol and I find issues like this in LLM output of only 1 or 2 pages.
And there’s the distinction: nobody is perfect, especially not LLM agents, but… when a critic: human or LLM finds a flaw and points it out: there it is, no denying that is a flaw.
Asking LLMs to find flaws, then verifying those flaws are real, is more powerful than just looking for flaws without the tool.
lol and I find issues like this in LLM output of only 1 or 2 pages. LLMs can’t even be consistent across paragraphs, much less hundreds of pages of context window. what an insane claim.
And there’s the distinction: nobody is perfect, especially not LLM agents, but… when a critic: human or LLM finds a flaw and points it out: there it is, no denying that is a flaw.
Asking LLMs to find flaws, then verifying those flaws are real, is more powerful than just looking for flaws without the tool.