lol and I find issues like this in LLM output of only 1 or 2 pages.
And there’s the distinction: nobody is perfect, especially not LLM agents, but… when a critic: human or LLM finds a flaw and points it out: there it is, no denying that is a flaw.
Asking LLMs to find flaws, then verifying those flaws are real, is more powerful than just looking for flaws without the tool.
Asking LLMs to find flaws, then verifying those flaws are real, is more powerful than just looking for flaws without the tool.
No it isn’t. Your example is so contrived. What was happening before these documents were reviewed by LLMs? Were the lines that were incorrect causing issues? If so then a human would have found it years ago. If not then they weren’t causing problems and all the LLM has done is invent work that a human now needs to clean up. The LLM didn’t solve a problem, it created a new one!
Now you need to pay a human to make sure that the LLM isn’t actually making things worse, which wasn’t even a job before! Your company that won’t pay a human to review the document in the first place is now needing more humans to clean up a mess that wasn’t a problem in the first place.
Were the lines that were incorrect causing issues?
Mostly not because mostly people gloss over the text and say “yeah, yeah, I know what you’re getting at” and do what they think the page says instead of doing what the page actually says.
The LLM didn’t solve a problem, it created a new one!
If you have a road with mis-painted stripes, but it never causes a problem because everybody ignores the stripes anyway… yeah, in that sense fixing the stripes is un-necessary work…
Your company that won’t pay a human to review the document in the first place is now needing more humans to clean up a mess that wasn’t a problem in the first place.
Oh, the company pays plenty of people to create and review the documents. They have metrics for how many documents they review per quarter, and they do review those documents, and they do find these things, and other people are paid to revise the documents again with metrics for their performance, and we all get trained on the revisions as they come out - at a remarkably regular pace when you look across a scale of tens of years - and that training accounts for about 5% of my weekly chargeable time accounting.
The difference is: the LLMs are finding problems faster, all at once, and the “continuous improvement process” is a little bewildered about what to do with all these, valid, observations coming out so much faster than they used to. We’re just starting to think about how to come to terms with that, and in the meantime the process rolls on revising documents at about the same pace as always, because if we fixed them all at once it would put a kink in the system, like a python swallowing a buffalo.
And there’s the distinction: nobody is perfect, especially not LLM agents, but… when a critic: human or LLM finds a flaw and points it out: there it is, no denying that is a flaw.
Asking LLMs to find flaws, then verifying those flaws are real, is more powerful than just looking for flaws without the tool.
No it isn’t. Your example is so contrived. What was happening before these documents were reviewed by LLMs? Were the lines that were incorrect causing issues? If so then a human would have found it years ago. If not then they weren’t causing problems and all the LLM has done is invent work that a human now needs to clean up. The LLM didn’t solve a problem, it created a new one!
Now you need to pay a human to make sure that the LLM isn’t actually making things worse, which wasn’t even a job before! Your company that won’t pay a human to review the document in the first place is now needing more humans to clean up a mess that wasn’t a problem in the first place.
Mostly not because mostly people gloss over the text and say “yeah, yeah, I know what you’re getting at” and do what they think the page says instead of doing what the page actually says.
If you have a road with mis-painted stripes, but it never causes a problem because everybody ignores the stripes anyway… yeah, in that sense fixing the stripes is un-necessary work…
Oh, the company pays plenty of people to create and review the documents. They have metrics for how many documents they review per quarter, and they do review those documents, and they do find these things, and other people are paid to revise the documents again with metrics for their performance, and we all get trained on the revisions as they come out - at a remarkably regular pace when you look across a scale of tens of years - and that training accounts for about 5% of my weekly chargeable time accounting.
The difference is: the LLMs are finding problems faster, all at once, and the “continuous improvement process” is a little bewildered about what to do with all these, valid, observations coming out so much faster than they used to. We’re just starting to think about how to come to terms with that, and in the meantime the process rolls on revising documents at about the same pace as always, because if we fixed them all at once it would put a kink in the system, like a python swallowing a buffalo.