“Overall, how satisfied are you with your State Bar membership?”
That’s the first of 40 questions asked by the State Bar of Arizona in the 2014 version of its triennial Member Survey. A week ago Wednesday, the Bar sent a blast email asking“its members to evaluate our member services and your opinions on key issues facing the Arizona legal profession.” If ‘what’s past is prologue,’ Bar executives and their collaborators will again use the results to spin member satisfaction like they did in 2011.
In fact, as recently as February 2013, the Bar’s CEO was citing supposed member satisfaction to stymie a half-baked Arizona legislative effort to make Bar membership voluntary.1
Poppycock.
The Bar’s email went on to state “Your participation in this survey will help the State Bar to better provide the services and information that meet your needs and interests.” Well as Tricky Dicky used to say, “that’s just plain poppycock.”
Damning case in point, of the 40 questions posed — not a single one pertained to member interests in either the cost of bar membership or the Bar’s lack of transparency. These were two of the biggest criticisms lodged against the dues-raising Arizona bar this year. And they were hot-button concerns raised by candidates during the just concluded Bar board of governors elections. So much for professing to sincerely inquire about members’ “needs and interests.”
Indeed, contrary to the specious claim that the Bar will use the survey “to better provide the services and information that meet your needs,” the real intent is pretextual. As it has before, the Arizona Bar will use the data as cover — as both sword and shield to fend off critics who contend the association is bloated, out-of-touch, and high-cost.
Happy Campers.

Thomas Holding, Wikipedia/public domain
Three years ago, 51% of all respondents reported only being “Somewhat Satisfied” with the Bar. Somewhat satisfied? As in 2011, that’s again one of the preselected choices. But what does it mean? Try telling your kid he’s “somewhat” smart or your girlfriend she’s “somewhat” pretty or your spouse you’re “somewhat satisfied” with your relationship. Let me know how well that works.
And an additional 23% said they were either “Somewhat Dissatisfied” or “Very Dissatisfied” with the Bar. But forget all that. The Bar’s spinmeister magazine, “Arizona Attorney,” nonetheless headlined the 2011 survey results with the misleadingly titled, “High Satisfaction, Room for Growth” and bragged about what supposed ‘happy campers’ Arizona lawyers were. And never mind that fully 80% of the Bar’s members were too indifferent or too busy to respond to the survey or that the Bar failed to follow-up with those 17,165 nonrespondents.2
N.Q.R. Factor.
Now I don’t pretend to be an expert on surveys.3 But I do know this. Surveys should be concise. Questions are supposed to be clearly worded. And while it matters who’s paying for the survey, they’re also supposed to be neutral. What’s more, there’s as much art as science involved.
So I have my doubts about the Bar’s Member Survey. Besides the survey having too many questions requiring way too much work to fill out, there’s an N.Q.R.4 factor again emanating off this year’s survey.

Several questions appeared biased either by the implicit assumptions they make or by forcing respondents to make choices when they’d rather not. No wonder it feels like the deck is stacked.
When you ask, for example, “how satisfied” you are with your membership or “how valuable to you” bar services are, you know there’s something not quite right. Both questions are biased because of the implicit assumptions concerning satisfaction and valuable they make.5 The“words you use in the questions can affect respondents’ reaction and choices.”
Or take the leading question about whether or not there’s a preference for “a printed Member Directory or a more robust online member search tool?” [emphasis added]
Or how about the barely hidden Bar-agenda questions? For instance, there’s the forced choice made by Question 6, which in order to continue with the rest of the survey, requires respondents to pick at least one of 17 preselected positive choices under, “Which of the following are the features or uses for the [printed Membership] directory that you find to be the most valuable?” For those of us who think printed directories are a waste of money and of no value in a digital age, too bad. You can’t skip the question or choose ‘no opinion.’
And talk about agenda-driven responses like those in the survey category, “Professional Barriers.” Question 9 asks, “What do you believe are the three most serious problems faced by the legal profession today?” and Question 10 queries, “Please list the three most important issues that you would like to see the State Bar concentrate its efforts on in the next few years.”
For multiple-choice answers, the Bar provides its predetermined long list of alleged lawyer concerns like “lawyer advertising,” “diversity,” “lack of appropriate judicial system funding” and “threat to judicial independence” [even in a merit selection state where for the past 40 years 99.9% of Arizona judges are retained]. Who came up with those personal agenda-driven responses? But don’t look for choices about improving fiscal stewardship or treating members like clients or cutting costs or increasing bar transparency or heightening member due process.

And conveniently disingenuous about lawyer apprehension if not their outright paranoia when dealing with the almighty keeper of their meal-ticket-license — the survey asks intrusive demographic questions under the category, “About You and Your Work.” Like lawyers are going to trust privacy and confidentiality assurances about respondent anonymity when questions specifically ask for county of primary practice; year of admission generally, and in Arizona; age; gender; areas of practice; number of firm lawyers; and optionally, race and ethnicity. Why not just ask for names?
SurveyMonkey.
Not that respondents shouldn’t be concerned about survey integrity and anonymity. Where surveys contain “sensitive or potentially identifying information,” the U.S. Navy, for example, strongly recommends against commercial providers like SurveyMonkey that do not conform to its security regulations. “Since the data will be stored on commercial servers there is increased risk of harm or embarrassment if the data are somehow compromised.” As it happens, SurveyMonkey is the commercial survey provider used by the Bar.
On its website, SurveyMonkey explains that anonymity is up to the survey creator and not its job. While the survey creator has options to collect responses anonymously, SurveyMonkey explains, “All collection methods permit the tracking of respondent IP addresses. Anyone using the Email Invitation collector could potentially track an email address on the response.”
And according to a SurveyMonkey Review posted on the business software review site, TrustRadius, email links to its surveys allegedly allow individuals to “complete the survey more than once if they access the link through 2 different computers.” I don’t know about all that.
But I do know that after completing my survey, I could still access the same email link and begin completing another survey — not that I had any interest in wasting my time twice.
So is it one anonymous survey per ‘customer’? Or is that just more poppycock? In truth, I don’t care.
__________________________________________________________________________________________
[1] See Arizona House Judiciary Committee videotaped hearing “Arizona Supreme Court’s control over state bar debated, contested . . .” at http://www.youtube.com/watch?v=xotdkMf61Ic, February 14, 2013 and remarks by Arizona Bar CEO John Phelps at 27:14 conveniently omitting the faint praise qualifier “somewhat” and asserting instead that “75% of the lawyers polled. . . were satisfied and 25% were not satisfied.”
[2] See Ten Reasons Why Surveys Fail by Dr. David Futrell, Quality Progress Magazine, April 1994, noting, “Failure to follow up with the nonrespondents can yield grossly misleading data. In general, people who respond to a survey will be more extreme in terms of what is being measured than the nonrespondents.”
[3] To be fair, the Bar hired an expert, noted local researcher Bruce Merrill, Ph.D., to assist its 2011 survey. Dr. Merrill once ran a golf hole Ad-in-the-Hole Research Study to evaluate “Name/Brand Awareness,” “Ad Recall” and whether golf hole ads are “Bothersome to golfers.” Golf hole advertising’s a dumb idea — but no worse than urinal advertising.
[4] NQR means Not Quite Right. I was first introduced to the acronym by a friend and former F-18 fighter pilot.
[5] Compare the Bar’s “how satisfied” question with the example borrowed from Sterngold, Warland and Herrmann (1994) by Professors Hershey Friedman, Ph.D. and Taiwo Amoo, Ph.D. in Ranking the Rating Scales, published in the Journal of Marketing Management, Vol. 9:3, Winter 1999, 114-123. Referring to an earlier study, the professors write “that a question like “How concerned are you about…?” causes a bias in the direction of concern because it assumes that subjects should be concerned about an issue. Using a filter question first asking respondents whether or not they were concerned with an issue and then asking those that were concerned to rate their degree of concern resulted in significantly fewer people showing concern than the former approach.” Similarly, I posit that asking how satisfied members are with the Bar “causes a bias in the direction of [satisfaction] because it assumes that subjects should be [satisfied]” with the Bar.
Read Full Post »