Getting AI agents or assistants to 50 or 60% accuracy with a standard RAG architecture may be fairly easy. However, that’s far from enough for most business use cases. Going up to 80% and beyond is a different order of difficulty, one that demands continuous, specialized, and heavy R&D investment that non-software companies and internal teams can’t always sustain. AODocs’ CEO Stéphane Donzé reflects on the path forward for knowledge managers, based on lessons from enterprise deployments of out-of-the-box solutions.
Reliable enterprise AI agents and assistants succeed or fail on the last 30-40% of accuracy. This reality demands continuous, specialized, and heavy investment in R&D that non-software companies and internal teams can’t always sustain. AODocs CEO Stephan Donze reflects on the path forward for knowledge managers.
Retrieval-augmented generation (RAG) has become deceptively easy to prototype. With available open source components, assembling an AI pipeline that gets to 50 or 60% of correct answers can be done in a matter of weeks. But for many business use cases, it’s also far from enough as reaching the must-have 80% accuracy and beyond is a different order of difficulty altogether.
That gap matters a lot. A chatbot answering casual internal questions with 6 out of 10 accurate responses may be a starting point. But an AI assistant that tells a technician which torque specification to apply to an aircraft component, which valve sequence to follow in a water treatment plant, or which configuration is approved for a data center is a different story altogether. In those environments, a wrong answer delivered by an overconfident chatbot may cost millions in lost productivity, trigger regulatory sanctions, or even lead to health and safety hazards.
But there is one category where this new “building is easy” logic breaks down completely: AI assistants that answer questions about your business-critical documents. A vacation tracker built in a quick sprint is a fine idea. An AI assistant that tells a technician which torque specification to apply to an aircraft component, which valve sequence to follow in a water treatment plant, or which configuration is approved for a data center is a different story altogether. In those environments, a confidently wrong answer can cost a fortune, a regulatory sanction, or worse.
So when your team asks, “Should we build our own AI assistant for knowledge management, or buy one?”, the honest answer depends on which of these two categories you’re in.
One prerequisite before going further: everything you read below assumes your document foundation is already in order. That means governed content, controlled access, and a trusted source of truth. If it isn’t, that’s the place to start (a topic we covered in depth in AI Needs Better Foundations, Not Faster Pilots).
The 60% precision plateau is actually a slippery slope
Here’s a pattern that repeats across large enterprises: a company with a capable internal AI team decides to build its own knowledge assistant. The plan sounds reasonable: use a document management system as the trusted source of information, but build the chatbot layer — the vector database, the semantic search, the prompts — in-house, because the assistant will eventually need to reach multiple sources beyond the DMS.
The first weeks tend to go well. With readily available open source components, a basic retrieval-augmented generation (RAG) pipeline can be assembled quickly, so that getting to 50 or 60% of correct answers comes relatively fast. The demo impresses everyone. And it’s precisely because a proof of concept at that level is so easy to reach that many teams assume the next step will be, too. Many decision-makers think that production-ready quality is just a matter of a bit more time and a reasonable amount of additional investment.
But then progress stops, as many organizations cannot get beyond that plateau. And for business-critical use cases, 60% accuracy is not a quality gap; it’s a disqualifying risk profile. “Mostly right” may be an acceptable bar for a weekend trip planning app. It is an unacceptable level of performance for aerospace documentation.
What lives between the demo and production
Closing the gap between a nice demo and a system you can deploy at enterprise scale requires solving problems that go well beyond basic RAG. The tricky part is that the difficulty doesn’t scale linearly. Above 60% accuracy, each additional five points of accuracy requires more work than the five before it. So getting from 80% to 85% is typically much harder than getting from 70% to 75%.
What lives in that space is hard, unglamorous engineering that rarely shows up in the initial project plan.
Understanding images and tables. A large share of enterprise knowledge doesn’t live in clean paragraphs but in tables, diagrams, and screenshots buried inside documentation. An assistant needs to extract that information in a way the search engine can actually find when it matches a user’s question. For example, answering questions about a mobile app by understanding the screenshots in its user documentation.
Knowing when to say “I don’t know.” It sounds trivial, but it isn’t. When a search query returns empty, the system has to decide between two very different situations. One is “the answer doesn’t exist in the sources”. The other is “my search query was wrong, I should try again with a different strategy.” Getting that judgment right means forcing the model to admit ignorance rather than improvise. This is crucial for trust, and it demands continuous tuning rather than a one-time fix.
Detecting and eliminating hallucinations. Even the most recent large language models still tend to make things up, because their training rewards producing text far more than declining to answer. A production-grade assistant needs built-in protections that double-check every answer, plus an ongoing process to find and fix hallucination patterns as they emerge.
Teaching the AI your company’s language. Every organization has its own acronyms, product names, and business slang, plus generic corporate knowledge like the org directory. Until an assistant understands that vocabulary, it misunderstands the questions themselves. Teams that incorporate this company-specific context consistently report an immediate drop in the number of answers users flag as bad. That’s not because the retrieval improved, but because the AI finally understands what is being asked.
None of this is a one-off effort. These capabilities represent many months of R&D by specialized AI engineers, and given how fast the AI landscape moves, the work never actually ends. For a software vendor, sustaining that heavy lift is the core job. For a company whose business is building planes or treating water, it’s a permanent distraction from the actual mission, one that internal teams, however talented, are structurally not set up to upkeep.
Build, buy, or both?
The resolution isn’t necessarily choosing a side; in some cases, it’s about drawing the line in the right place.
Buy the foundation: an AI-ready system of record that governs your critical documents, and the AI assistant machinery that solves the hard problems like multimodal extraction, honest “I don’t know” behavior, hallucination controls, corporate-context understanding, and that is continuously maintained as models evolve.
Build on top: in the context of RAG, this typically means the top-level chatbot. That would be a single entry point for employees that understands a question and routes it to the right specialized sub-agent, whether that handles HR policies, engineering specs, or general company knowledge.
When a question pertains to business-critical documents, that sub-agent should be the production-grade assistant your organization has bought rather than built, so your team can stay focused on routing and user experience rather than reinventing accuracy, controlling hallucination, and repeatedly searching from scratch.
That split gives you the best of both worlds: the speed and creativity of cheap building at the edges, and the accuracy, security, and governance of a dedicated platform at the core. With that, the explosion of custom apps becomes an asset instead of a liability.
AODocs AI-native document management platform and knowledge assistant are designed as a governed foundation you can trust.
Connect with us to find out more.
