“Are you a member or happy?”
Paraphrasing that other “Mo” — the one with an “e,” the State of Bar of Arizona is still asking. How happy are members with the Bar’s programs and services?
Although I already took the meaningless member survey, Monday morning I got another email reminding me to complete it. This is the third reminder — but who’s counting? The survey period ends this Friday.
Doubtless the Bar’s trying to gin up something close to a representative response rate so its leaders can afterward claim victory just like they did three years ago.
Never mind the biased questions with their implicit assumptions or the forced choices from preselected answers. The results are preordained. But then I already I discussed all this last week at “Another ‘poppycock’ survey from the State Bar of Arizona.” At least the Bar’s not planning to survey us so they can later sell us the results.
After last week’s post, I got an email from State Bar of Arizona Chief Communications Officer Rick DeBruhl. Understandably defensive, he wanted to explain a few things. With his consent, I’m posting his clarifications verbatim below.
“A couple thoughts on your blog:
“Our survey was designed with input by Bruce Merrill. Formerly with ASU, he is one of the premiere survey experts in the country. Surveys are inherently imperfect. Experts like Merrill design them in such a way as to work around our human idiosyncrasies. I’ve never taken a survey that I considered perfect, and I wouldn’t put ours in that category. Using consultants such as Merrill simply allow us to step beyond the amateur ranks to get reasonably valid information.
“You mentioned the fact that 80% of the bar’s membership did not respond in 2011. Survey experts will tell you that a 20% response rate is phenomenal. In addition, we made sure to check that the final numbers were demographically similar to our membership. That gives our survey validity.
“Questions such as the “printed directory or a more robust online member search tool” were designed to give us guidance. We know from previous surveys that members overwhelmingly use and value the directory. We’ve heard that the current online search tool doesn’t go far enough. Would people be satisfied if we dumped the print directory for our existing search tool? Possibly not. What we’re ultimately trying to determine is whether they like the printed version because they want a book, or because they don’t have a better option.
“As for the answers on the “most serious problems” question, we actually got them from another state’s survey. We’re trying to build not just Arizona data but national trends as well. We looked them over carefully before deciding to include them. Any time you create a list, it has the potential to create bias to those answers. And yet survey experts say that respondents need lists to be reminded of the options.
“As for why we use SurveyMonkey, the answer is simple. It is the highest quality for the lowest cost. Of course there are other ways to survey members with greater anonymity. However, they cost significantly more money. Merrill feels we can achieve the same results with lower cost using this method. One other option would be a random sampling. Experts say we could get statistically valid results with just hundreds of responses. Perhaps, but we felt that if we were going to talk about the results of the member survey that all members should have the ability to give answers.
“Incidentally, we had a technical problem on the first day of the survey that prevented some people from submitting. As a result, our consultant decided that our best option was to remove the block that prevents a second survey from the same computer. The consultant feels that the number of people submitting two will be significantly small so that it won’t likely affect overall trends. We do have the ability to run a check on the number of repeat IP addresses which will let us know whether that number was significant. I’m sure if we had unlimited resources, we could no doubt determine the identity of each IP address, but that’s simply not in our realm. Incidentally, we don’t use the SurveyMonkey invitation system.
“We’d love it to be shorter, but we’ve done our best to chop it down. Because of the skip logic built into the survey, no one actually answers every question.
“The bottom line is that we understand that surveys are imperfect. We look for trends and directions and feel that gives us guidance as an organization.
“As always, let me know if you have questions or thoughts.”
Dollars to donuts.
Survey design is as much art as science. So what’s a good response rate? Depends who’s asking. And it depends on how they ask the question. Also see AAPOR | Response Rate – An Overview.
But I’m glad at least that Rick DeBruhl conceded more than once that the Bar survey was “imperfect” — because it is. That was the crux of my post.
There’s a reason some 80% of lawyers don’t bother answering these surveys. They’ve figured out what a medical historian once said about something else, “The experience of the ignorant has routed the wisdom of the learned.”
Too bad my point about the Bar’s expensive printed directory got lost. The survey’s either/or question about the directory was a leading question. It was biased toward a choice preference for “a more robust online member search tool.”
Not long ago, the Arizona Bar spent well over six figures supposedly improving and updating its website and its online member search tool. And now it appears some Bar executive is itching to spend even more money on what’s become a bloated website and online member search tool. Meantime regardless of the Bar’s claims at being eco-friendly — it’s just not yet because the Bar continues to print member directories and kill trees.
As for the survey having been designed “with input by Bruce Merrill” — well, that was a point I already footnoted last week. Still with an expert “like Merrill” on board, you have to ask why the Bar needed to crib stuff from other state bar surveys?
Finally, as a learned colleague pointed out to me when I showed her Rick DeBruhl’s response, his email didn’t address the matter of his boss’s oversimplification of the Bar’s percentage of so-called ‘satisfied customers.’ This was last February 2013 when AZ Bar CEO John Phelps who’s also a lawyer addressed the state legislature’s house judiciary committee.
At about 27:33 on the tape and transcript, he omits the qualifier “somewhat” and asserts instead that “75% of the lawyers polled. . . were satisfied and 25% were not satisfied.”
Does such shorthand, she opined, potentially rise to an ethical rules violation under ER 7.1, i.e., that “a lawyer shall not make or knowingly permit to be made on the lawyer’s behalf a false or misleading communication”? Or as she also speculated, was it a possible violation of ER 8.4 (c) concerning “misleading” statements? The Bar’s communications chief doesn’t explain.