Every week I get vendor pitches that promise to “revolutionize operations with proprietary AI algorithms” or “transform business processes using advanced machine learning.” It all sounds impressive until you start asking basic questions and realize they’re just running standard regression models with a fancy UI.

The AI vendor landscape is packed with companies that are 90% marketing and 10% substance. As IT leaders, we need to get better at separating the real capabilities from the promotional fluff. Here’s how I evaluate AI vendor proposals without getting sucked into buzzword bingo.

Ask for the Architecture Diagram First

This is my go-to filter. Before I’ll sit through a demo or read a thirty-page proposal, I ask for a technical architecture diagram. Not a marketing slide with clouds and arrows. A real diagram showing data flows, model types, infrastructure components, and integration points.

If they can’t produce this, or if what they send is vague and high-level, that tells me everything I need to know. Legitimate AI solutions have real technical architectures. Vendors who actually built the technology they’re selling can explain how it works.

Pay attention to what models they’re using. Are they training custom models on your data, or are they wrapping a GPT API call in a nice interface? Both can be valid approaches, but the pricing, performance, and data privacy implications are completely different.

The Data Question Separates Pretenders from Performers

Here’s my favorite question during vendor meetings: “What data quality and volume do you need to achieve the accuracy rates you’re claiming?”

Real AI vendors will give you specific answers. They’ll tell you they need at least 10,000 labeled examples for training, or that their model requires certain data fields to be populated at 95% completeness, or that accuracy degrades if you’ve got more than X% missing values.

Vendors who dodge this question or give vague answers like “we can work with whatever data you have” are bluffing. AI models are only as good as their training data. If they’re not asking hard questions about your data quality upfront, they’re either incompetent or planning to blame you when results disappoint.

Also ask about data labeling. If their solution requires labeled training data, who’s doing that work? If the answer is “you,” make sure you understand the effort involved. I’ve seen projects die because nobody realized they’d need to manually label 50,000 records before the AI could do anything useful.

Probe the “Proprietary Algorithm” Claims

Vendors love claiming their “proprietary algorithms” give them a competitive advantage. Sometimes this is legitimate. Often it’s nonsense.

Ask them to explain what makes their approach unique. If they’re using standard techniques (transformer models, gradient boosting, neural networks) applied to a specific use case, that’s fine—but it’s not proprietary in any meaningful sense. Every other vendor has access to the same techniques.

True proprietary value usually comes from one of three places: unique training data they’ve accumulated, domain-specific feature engineering that requires deep expertise, or genuinely novel model architectures (which is rare). If they can’t articulate which of these applies to them, their “proprietary” claim is marketing fluff.

Understand What “Accuracy” Actually Means

Vendors throw around accuracy numbers that sound impressive—“our model is 95% accurate!” But accuracy without context is meaningless.

First, accuracy at what? Predicting common cases or edge cases? I can build a spam filter that’s 99% accurate by marking everything as “not spam”—it’ll be right 99% of the time because 99% of email isn’t spam, but it’s completely useless.

Ask about precision, recall, and F1 scores for the specific use cases you care about. If they can’t explain these metrics or why they matter, you’re dealing with someone who doesn’t understand their own technology.

Also ask about accuracy in production vs. in testing. Models that perform great on curated test data often degrade significantly when exposed to messy real-world inputs. Any vendor worth working with will be upfront about this performance gap and explain how they plan to monitor and address it.

The Integration Conversation

This is where most vendor relationships fall apart. The AI model works great in isolation, but integrating it with your existing systems becomes a nightmare.

Ask specifically about integration patterns. How does data get into their system? How do you get results back out? What APIs do they expose? What authentication and authorization mechanisms do they support?

If they’re expecting you to build custom integrations, make sure you understand the development effort required. Get access to their API documentation before signing anything. I’ve reviewed API docs that were so poorly written that integration would’ve taken months of trial and error.

Also ask about latency requirements. If their AI takes 30 seconds to return a prediction, but your use case requires sub-second response times, that’s a deal-breaker no matter how accurate the model is.

The Explainability Problem

Depending on your use case and regulatory environment, you might need to explain how the AI arrived at its decisions. This is especially true in healthcare, finance, and any scenario involving legal compliance.

Ask vendors how their models handle explainability. Can they show which factors contributed most to a specific prediction? Can they provide reasoning that would satisfy an auditor or regulator?

Many modern AI techniques—especially deep learning—are essentially black boxes. That’s fine for some use cases (like image recognition) but unacceptable for others (like loan approvals). Make sure the vendor’s approach aligns with your explainability requirements.

Who Actually Built This?

Here’s a question that makes vendors uncomfortable: “Can I talk to the engineers who built this?”

Legitimate vendors will set up technical deep-dives with their engineering team. They want you to understand the technology because they’re proud of it and because technical buyers need technical answers.

Vendors who keep you at arm’s length from their technical team are hiding something. Either the technology is less impressive than marketed, or it doesn’t actually exist yet, or they’re reselling someone else’s solution with minimal customization.

Companies like those doing their consulting practice the right way will connect you with technical experts who can walk through implementation details, discuss edge cases, and explain architectural trade-offs. That’s the level of transparency you should demand.

The Total Cost Reality Check

Always ask for total cost of ownership over three years, not just the first-year license fee. AI solutions often have hidden costs that balloon over time:

• Model retraining frequency and associated costs
• Infrastructure scaling as usage grows
• Support and maintenance fees
• Professional services for updates and enhancements
• Data storage costs as you accumulate training data
• Compute costs for model inference at production scale

Get these numbers in writing. If the vendor can’t provide them, build in a substantial buffer because you’re going to encounter surprise costs.

The Reference Check That Actually Matters

Don’t just ask for reference customers—ask for reference customers with similar data characteristics and use cases. A vendor might have great success stories in retail but completely fail in manufacturing because the data patterns are different.

When you talk to references, ask about the implementation timeline versus what was promised. Ask about production accuracy versus pilot accuracy. Ask about ongoing operational costs versus initial estimates. These gaps tell you whether the vendor sets realistic expectations or overpromises to close deals.

Trust Your Technical Instincts

If a vendor pitch sounds too good to be true, it probably is. If they claim their AI will solve problems that entire research teams at Google and Microsoft haven’t figured out, be skeptical. If they dodge technical questions or fall back on buzzwords when pressed for specifics, walk away.

The best AI vendors are the ones who are honest about limitations, clear about requirements, and realistic about implementation timelines. They’re selling solutions to specific problems, not magic.

Your job as an IT leader isn’t to be an AI expert—it’s to ask the right questions, demand substantive answers, and cut through the marketing noise to evaluate what’s actually being offered. The vendors who get frustrated with your questions aren’t the ones you want to work with anyway.