Last week somebody asked me a question via email that I've already answered lots of times: How many stories should a PNI project collect?
I was about to say "it's on page whatever in my book," like I usually do, but then I thought -- why don't I write something new this time, just to see what happens? I'm glad I did, because I think my answer is getting better as I keep doing more projects. Anyway, here's what I wrote. Maybe it will be helpful to you as well.
The "how many stories" question comes up often when people are planning
story projects. The answer is a bit complicated, but it depends on six
things: issues, ambitions, abstractions, experiences, engagement, and people.
Issues: One or many?If you want to talk about one big, simple issue, you need one set of
stories. However, if you want to talk about multiple issues, or one very
complex issue with a lot of other issues embedded within it, you need
more stories.
One way I like to use to figure out if an issue is complex
is to keep asking "And what issues lie within that?" and then stop when
the answer is "there aren't any issues within it."
For whatever number of stories you plan to
collect, you must multiply it
by the number of discrete issues you want to talk about. For example, if I wanted
to help people talk about jobs and homelessness, I would gather two sets
of stories (with some common questions to tie them together), so people
can explore each issue with the depth it requires.
Ambitions: Exploratory or in-depth?
If you want to:
- prove without a doubt that something is happening (in a
way that cannot be dismissed),
- represent the voices of
people who have not been heard (in a way that cannot be ignored),
- help people think through an issue deeply enough to arrive
at useful conclusions and plans (in a way that will not fall apart later
on), then
- you need more stories than if you just want to explore a topic and
see what happens.
Ambitious projects need 2-4 times as many stories as exploratory
projects. In an ambitious project, the patterns in the stories must be
obvious, plentiful, and complex enough to be explored in depth. In an
exploratory project, it's okay if the patterns are just interesting
hints at things people might want to explore more fully in the future.
Abstractions: Concrete or vague?
If you want to explore abstract issues that are difficult to explain in ordinary words, you will need more stories than if you want to explore simple, concrete issues.
For example, say you want to know how people feel about the new traffic lights in your neighborhood. You can just ask people how they feel about the new traffic lights in your neighborhood. But if you want to explore how your community is building resilience for a 21st century future, or some other string of jargon that means a lot to some people and nothing to others, you might have trouble gathering relevant stories. Most likely, you'll get a lot of "scattershot" stories based on people's guesses as to what you might be asking them to talk about.
A good test is to write down a question you would like to ask people, then translate it into simple, everyday language. Search for the "1000 most common words" in whatever language the question will be in, then remove all the words in the question that are not in that list. Then ask yourself: if you frame your question in common words, will the stories told in response adequately address the issue you want to address? If yes, just ask the question that way, and you're fine. You won't need extra stories.
But if rephrasing your question with common words will push it far away from the issue you want to address, then you will need to collect more stories, so that some of the scattershot stories you collect will fall onto your target.
Experiences: With stories, or with stories and patterns?
If you want people to meet in rooms, share stories, and do some
sensemaking exercises together, you can gather as few as twenty stories
per session. You might do that a few times within a project, but as long
as it's people talking, you can see and work with patterns in a few dozen to
several dozen stories.
On the other hand, if you want to do what I call catalysis (which is
just analysis without the definitive conclusions), you need at least 100
stories to start finding statistical patterns in your data (answers to
questions about stories). At 100 stories most such patterns tend to be
weak. At 150 or 200 stories patterns are stronger (and less likely to be
considered fake or irrelevant). I get pretty nervous when I have do catalysis with
only 100 stories to work with. At around 200 stories I start to feel
more comfortable, because the patterns I find are easy to see and talk
about (without worrying that people will say "there's nothing there").
This more-is-better trend continues until
about 600 stories, when you start running into diminishing returns. At
that point you are better off using your time to collect stories on a
different issue (unless, of course, some other aspect of this list means you need to push the number up for other reasons).
Catalysis is not important to, or even advisable for, every PNI project. Sometimes you do need to
generate a lot of graphs and statistics. But sometimes you can get the
same result with fewer stories by having people work with the stories
directly, in sensemaking exercises. It all depends on what sorts of
experiences you want people to have.
I always advise people to imagine the people they want to help or reach
(whoever they are) responding to patterns in the stories and other data
they plan to collect. If you can picture those people looking at
graphs and statistical patterns and saying, "Oh, wow, now I get it," then
you want those things to show those
people, so you need catalysis.
But if you can picture the same people saying the same things
because they are working with the stories directly (i.e., without any
graphs and statistics), you don't need catalysis. In fact, it might be a bad idea. It might waste time you can use for
other, more important things, like talking to more people, holding more
sessions, covering more issues, getting more stories to more people,
helping more people learn how to gather and work with stories, or
iterating over the project more times.
On my web site I have an excerpt from a catalysis report which a client
allowed me to share. If you look at it, you can see what the patterns that come out of
catalysis tend to look like. If that seems like it would not be useful
to your project, you don't need catalysis, and you don't need hundreds
of stories. On the other hand, if that sort of report seems like just the
kind of thing you need, then you can look at the numbers listed on the
second page of the report. Those are typical numbers for projects that
support catalysis well.
Engagement: Deep conversations, or messages in bottles?
A lot of "what works" in story work has to do with facilitation and
engagement. I once saw a project with 80 stories work far better (in the
sense of generating more useful insights) than a project with 1600
stories.
- The stories in the first project came from a group session with
20 people that was carried out by an expert facilitator who helped the
people in the room feel welcome, safe, and heard. As a result, the
people really spoke to the issues, and their stories and answers to
questions contained many striking insights.
- The second project used a
web form that had embedded in it some constraining expectations about
what respondents ought to say. Those 1600 people said more
surface-level things, so even with 20 times more stories, less useful
insight came out of the project. It was still a good project, but it did not explore its issues as deeply as the project with 80 stories.
So there is a quality-quantity balance. The more quality you can get in
your stories (in terms of how deeply and authentically people can
explore the issues at hand), the fewer stories will provide the same
result. Conversely, if for some reason you cannot gather quality
stories (maybe people are reluctant, or you can't talk to them in
person), a greater quantity of stories can make up for it, to some extent.
On some projects, quality is the primary constraint (so you need more stories), and on other projects, quantity is the primary constraint (so you need deeper engagement in the stories you can collect).
People: Small or large community? Small or large need?
The more people you want to listen to, and the more the people in that
group need to feel heard, the more stories you need to collect.
Participatory story work never results in statistical sampling (because
it's self-selecting), but you do need more stories to talk about issues
in a community of 10,000 than in a community of 100. And you need more
stories in a community with a strong need to be heard than in a
community where people have already had plenty of chances to speak up.
My general rule is that if at least 20% of the people in any community
have shared stories in a project, people tend to feel that the collected
stories are representative of the community. In cases where people in a
community feel especially unheard, that percent has to go up, maybe to
30% or 40%. The story collection also has to be balanced to represent
all relevant viewpoints, but that is the shape of the collection, not
its size.
Having said that, a rule of thumb based on percentage doesn't work as well if
the population is huge. If, say, there are 50,000 people in a community,
hearing from 20,000 of them might pose logistical problems. I have
seen story projects collect 10,000 stories, but it's not the norm. Most
projects have fewer than 1000 stories, just because the people doing the
projects have limited time to gather and work with the stories.
In the case of a larger community, it's reasonable to say that 20-40% of the community should be
invited to share stories. After all, it's more about who is
allowed
to speak than who actually speaks.
Web-based surveys tend to get a 5-10% response rate, so if 20,000 people
are invited to speak, you would get something like 1000-2000 stories, which is
doable logistically.
If stories are collected in person, in interviews or groups, it's hard to get 1000 stories, even if you invite 20,000 people. It takes more
time and energy to come to a session or interview than to fill out a web
form, so instead of a 5-10% response rate you will tend to get more like 1%.
On the other hand, stories gathered in interviews and sessions are so
much deeper and richer than web-collected stories that smaller numbers
of stories may not be a problem (see above).
Another thing is that, if a project contains multiple sub-projects that explore different issues (also see above), they
can together add up to hearing from 20-40% of the population, even when
the population is large. You can link sub-projects together
by using some common questions. If you do that, you can get to huge numbers of stories, spread across sub-projects within a larger, overarching project.
