Uncovering the Drivers of Membership and Retention
[10 min read] For more than a decade, our customers asked us for member engagement scoring. They wanted a way to put a number on each member so they could tell, at a glance, who was thriving and who was drifting away. As a software company, we wanted to build it. But every time I pressed someone on how the scoring should actually work, I received a different answer. There was no shared definition, no agreed-upon method, no consensus on what counted or how much. Without a straight answer, it's hard to build a feature. So the request kept coming, and we kept struggling to fulfill it.
Then, at a New England Society of Association Executives conference a few years ago, I heard Amanda Lea Kaiser speak. Amanda began her career in brand marketing at Crayola before moving into the association world as a director of marketing, and she has spent years since as a member engagement researcher, speaker, and author. She interviewed hundreds of members, one by one, to understand why people join and stay. At the very end of her session, she said something that stuck with me:
“Member engagement is not what people do. It is how they feel.”
I found that deeply compelling, and I thought it pointed to a path forward. If engagement is a feeling, then we did not need to track activities at all. We needed to measure hearts and minds. The best tool I knew for measuring how people feel about an organization was Net Promoter Score.
We were so confident in this argument that we built an NPS tool directly into our software. I personally went on the speaker circuit making the case that this was the right way to understand member engagement.
Then, at Associations North’s Executive Edge, I shared the argument with the chief staff executive of a national association, and he pushed back hard. He told me he still needed member engagement scoring. When I asked why, his answer was simple and completely reasonable: he needed to know whether a member would renew before he sent the invoice. Before the invoice is sent, staff outreach is member services, he explained. After the invoice is sent, it’s a collection call. NPS was never going to give him that insight.
⚖️ The comfortable illusion of the engagement score
That exchange forced me to take traditional member engagement scoring seriously again, so let me describe what it actually is. You take various member activities, such as attending an event, serving on a committee, and making a purchase, and assign each one an arbitrary weight. Add up the weighted values to get a number. The goal is to identify who is on track and who is at risk. It feels rigorous. It feels data-driven. And it gives an overworked membership team something they badly want: a list.
The same executive director who pushed me on NPS also suggested I read a book. One book led to another, and eventually I found my way to Daniel Kahneman, the Nobel Prize-winning psychologist who wrote Thinking, Fast and Slow and spent his career studying how people actually make decisions.
Kahneman has a phrase for what a homemade scoring model gives us: the illusion of validity. We build something that produces confident-looking numbers and conclude we have captured a truth about member behavior. In reality, the weights in a typical engagement score did not come from data. Someone decided in a meeting that an event is worth ten points and an email open is worth two. When we encode our hunches into an algorithm, we do not remove our bias. We institutionalize and amplify it. The score becomes a mirror that tells us what we already believed.
Kahneman points to an even more uncomfortable truth for anyone who considers themselves an expert: expertise can make our predictions worse, not better.
The longer you have worked in association management, the more confident you become in your gut, and the less you question it.
His research also found that simple yes/no models often outperform complex ones built on weighted guesses. The sophistication of the engagement score is not a strength. It is the very thing that hides how little we actually know.
So there I was, right back at the beginning. How are we going to solve the problem?
⁉️ Asking a harder question
The answer came at a family dinner. I talk about associations everywhere, even at the dinner table, and one evening I described the problem to my nephew. His response was almost dismissive: “Uncle Pete, this is solved science. We have known how to solve this kind of problem since the 1970s. We just did not have the computing power to do it until recently.”
Why should we trust my nephew? He is a computer science PhD candidate at the University of Texas at Dallas who has also worked under Ivy League professors. He volunteered to build us something that would let us make statistically valid predictions about the one thing that the executive director in Minnesota actually wanted to know: whether a given member would renew.
What he built us was a neural network.
Instead of assigning points based on what we assumed mattered, the approach was to let the data tell us what actually predicts renewal. A traditional engagement score starts with human assumptions and forces the data to fit them. A neural network does the opposite. It learns directly from real member behavior, discovers patterns that are not obvious to us, and gets more accurate over time as it sees more examples. Think of how a child learns to recognize a dog: not from a rulebook, but from seeing many dogs, making mistakes, and adjusting. The model works the same way.
After chewing through more than a million member records, it didn't find a longer or smarter scoring formula. It found something far more useful: four clear priorities that any association can act on. But to trust the four, you first have to see how the model got there.
⚙️ What we actually built
Working with association customers, we started with roughly 100 hypotheses about what might predict whether a member renews. We fed the model 1.2 million anonymized member records spanning more than a decade, and training it took 100 trillion operations. This was not a chatbot or a repurposed version of ChatGPT. It was a custom-built model using statistically proven methods, running on our own infrastructure.
The first result was humbling.
Of the roughly 100 things our members and we were sure would predict renewal, only 79 turned out to be genuinely predictive.
A meaningful share of the "obvious" drivers, the things experienced professionals will confidently tell you matter, showed no statistical significance at all. They were red herrings that had been quietly steering decisions for years. This is exactly the trap Kahneman warned about, made visible.
Some of what did matter was almost mundane. Members without a valid email address on file renewed at meaningfully lower rates. So did members missing a billing email, a website, or with an email flagged as rejected. This is not engagement in any inspiring sense. It is data hygiene. But a member you cannot reliably reach is a member drifting toward the exit, and the model saw it plainly.
Some of what mattered was genuinely surprising. Members without autopay renewed more often than those with it. On its face, that sounds absurd until you look closer: several associations in the sample had leaned on forced autopay as a retention crutch, prioritizing enrollment over the harder work of proving value. Autopay was not causing anyone to leave. It was a marker for organizations that had swapped a billing tactic for a value proposition, and those organizations saw more churn. The model surfaces correlation, not causation, and the story behind a correlation is often the most instructive part.
We could fill this article with such findings. But this work matters not for the length of the list. Once you clear away the noise, the strongest patterns in 1.2 million records point to just four things. And every one of them is something an association can act on.
✔️ The four findings that matter
Strip away the red herrings and the model's strongest signals converge on four questions. None of them is about your scoring model. Each one is backed by the data, and each one points to work a small staff can start on Monday morning.
1. Are we creating opportunities for new members to feel welcome?
The single strongest signal in the entire model was the first renewal: one successful renewal dramatically increases the odds of the next.
Put plainly, new members are the most vulnerable members you have, and the first year matters more than any other.
So the question is not whether onboarding is nice to have. It is what, specifically, you are doing to own that first year. The associations that do this well treat onboarding as a deliberate program, not an autoresponder: a welcome sequence, a personal touch from staff or a peer, and structured moments to plug in. With intentionality like this, many organizations see retention climb by double digits.
2. Are we giving staff the resources to deliver on the brand promise?
This is where pricing comes in, and it reframes one of our strangest findings. Higher prices correlated with higher renewal, not lower. Part of that is perceived value: people protect what they pay real money for. But there is a second effect that is easy to miss. A healthy price point is what funds a great experience in the first place. Underprice the membership, and you starve the very staff and services that keep members around. Charging appropriately is not at odds with member value. It is often the precondition for it.
3. Are we building authentic connection?
Event attendance correlated with renewal, and in-person mattered more than virtual. Both help, but being in a room together is something a webinar cannot replicate. This is a case for protecting the gatherings that create real human connection, and for being intentional about the kind of connection they foster. Some of the most effective associations lean into it well beyond the ballroom, with volunteer-led ambassador programs, peer-to-peer outreach, and shared purpose through service, from Habitat builds to food-pantry volunteering. Connection is not a byproduct of membership. It can be designed.
4. Are we offering meaningful service?
Committee service was one of the most powerful signals we found, with renewal odds climbing meaningfully for each term served. And this is not just about formal committees. It extends to the board, a task force, a working group, any role where a member is trusted with something that matters. The lesson is not to manufacture busywork. It is to create real opportunities for members to contribute, because a member who is serving is a member who has decided to stay.
☀️ The truth was never in the algorithm
The value of the data isn't that it replaces judgment. It's that it tells us where to direct our judgment. It clears away the guesses so a small, stretched staff can stop scattering their energy and focus it on the things that actually move renewal. It turns "where do we spend our time?" from a matter of opinion into a matter of evidence.
That, in the end, is what digging for truth is really about. Not a smarter score, but a clearer set of priorities. Own the first year. Price with confidence. Protect real connection. Offer service that means something. The neural network took 100 trillion operations to point back at four questions that any association leader can start answering today.
Dig for the truth in your own data, and you may find it was pointing at these questions all along.
Learn more about how Member Health can help amplify your association.
About Pete Zimek - Pete Zimek, CAE, is a member of MSAE and the CEO of Novi AMS, the highest-rated AMS on G2, Capterra, and Review My AMS. Built by a group of associations more than a decade ago, Novi AMS applies that same deep domain expertise to its AI work, including its Amplify suite, helping lean teams spend less time on busywork and more on the mission. Pete can be reached at pete@noviams.com.