After weeks of RFP response reviews, cost analysis, vendor presentations, and endless demos, it's tempting to feel like you know enough to make a decision. You've seen the software, met the sales team, and evaluated the features. .
Here's what you don't know yet: what working with this vendor will actually be like.
Why Reference Checks Matter More Than You Think
The strongest argument for taking reference checks seriously is a simple one: past behavior is the best predictor of future performance.
Demos show you what the software looks like today. Proposals tell you what the vendor is promising for tomorrow. But reference checks tell you how this company actually operates. That includes things like how they handle a difficult implementation, how they respond when something goes wrong, and how much of the workload lands on your staff versus theirs.
It's not information you can readily get anywhere else during the selection process. However, it's exactly the kind of information that determines whether a five-to-ten-year technology investment goes well or if it becomes a source of ongoing friction.
The Elephant in the Room: Reference Checks Start With Vendor-Provided Contacts
Let's name an obvious concern. When a vendor provides a reference list, they are putting their best foot forward. You are almost certainly talking to the clients they feel are most satisfied with their work and that they have served well.
Embrace that reality rather than fight it. It can actually be more useful than it might seem.
First, their best customers are not always positive. It happens more often than you'd expect. We have seen vendor finalists removed from consideration because reference checks surfaced real concerns. These reference checks have unearthed everything from unexpected upgrade costs running tens of thousands of dollars annually, to implementation experiences that didn't match what was promised, to support that didn't hold up after go-live. These things surface even in vendor-provided reference checks as long as you're asking the right questions.
Second, even genuinely positive reference checks tell you something critical: what does a good project with this vendor actually look like? What are the characteristics of their successful implementations? Who led the project on the client side? What role did staff have to play, and how much of the workload fell on the internal team versus the vendor?
That last question matters enormously. We have seen reference checks where every client contact was a superstar — highly capable, deeply involved, with the authority and bandwidth to manage a complex implementation. The project went beautifully. But when we looked at our client's organization, there was nobody positioned to play that role. In those cases, the vendor wasn't the problem. The fit was.
Understanding what a successful project requires from your side of the table is just as important as understanding what the vendor brings to it.
How to Actually Cut Through the Generic Answers in Your Reference Checks
Most reference check conversations follow a predictable pattern: the contact says positive things, mentions a few minor bumps, and wraps up quickly. Here's how to get past that and into what really matters.
- Ask how they handled it when something went wrong. Something always goes wrong in a complex technology implementation. The question isn't if there were problems. It's how the vendor responded when there were. A vendor who handles a crisis with accountability and transparency often builds more trust with clients than one who never had a problem to manage. The answer to this question tells you a great deal about how this company actually operates.
- Ask whether the project came in on time and on budget. This question surfaces patterns. If you ask it across three or four reference checks and each one says the project ran over — even if each contact frames it as their organization's fault for adding scope — that's a trend worth understanding. It may reflect how the vendor estimates, how they manage scope, or how they handle change orders. Ask the follow-up: was that typical in your experience with them?
- Pay attention to who you're talking to. Not just what they say, but who they are. A reference check conversation with someone from an organization that's twice your size, with a dedicated IT director and a full implementation team, will tell you a very different story than one from an organization that looks like yours. Try to find at least one reference that matches your organization in size, staff capacity, and the scope of the project you're undertaking.
Going Beyond the Vendor's Reference Check List
You should try to find at least one reference the vendor didn't give you. Not to say vendor-provided reference checks are worthless — they're not. But an independent reference gives you a different kind of data point.
The association community is remarkably willing to share. ASAE Collaborate, AWTC Slack, and peer CEO networks are all places where a straightforward question — has anyone worked with this vendor, and would you be willing to talk? — tends to get real answers quickly. Association professionals will talk. Use that.
If you're working with a consultant on your selection process, they're another resource. A good consultant has likely seen this vendor in action across multiple implementations and can connect you with contacts outside the standard reference check list.
The Reference Check Takeaway
Reference checks work. They surface real information — about how vendors operate, what successful projects require, where the risks are, and whether your organization is a genuine fit for the clients who have had the best experiences.
Don't treat them as a formality. Treat them as the final, most important research you'll do before making a multi-year technology commitment.
🎙 This post is a companion to Between the Dots Episode 15. Listen to the full conversation here.
→ Find all of our Between the Dots episodes here
And if you're navigating a selection process and want a second opinion on where you are, we'd love to talk it through.
