Missing my chance — again!

Yesterday I got another mass email from the State Bar of Arizona (SBA). “Last Chance to Answer the State Bar of Arizona Member Survey” declared the subject line. “Please take a few minutes NOW to fill out the survey.”  And then to sweeten my interest, “As a token of our appreciation for your prompt response, we’ll enter your name into a raffle to win a $100 Visa gift card.” 

As usual, I ignore the Bar’s survey requests. This was the third mass email about the same topic. After the second email, I asked for a blank copy of the survey questions. After a couple of days, the Bar’s public relations chief emailed a copy. The survey runs 9 pages and 48 questions. By my stopwatch, it won’t “take a few minutes” to answer. Not like it matters. I won’t be taking the survey and humbly suggest no one else should either.

Long ago I worked for a guy whose favorite expression was the blindingly obvious, “Timing is everything.” So right in the middle of a big fight at the Arizona Legislature to reform the State Bar comes a self-serving survey replete with the usual leading questions. Talk about timing.

The questions are meant to lead survey-takers down the Bar’s primrose path. Tell us “how valuable” a member benefit is? “How well does the SBA deliver . . . .”

It’s been a while since I read Elizabeth Barrett Browning but in other words, “How Do I Love Thee?” The State Bar of Arizona wants us to “count the ways.”

And among the 48 questions is the one nearest to the unsated stomach of every bloated bureaucrat, “If the SBA could provide you with one additional resource or service that currently is not offered, what would it be?”

Just in case, there’s a second follow-up that fishes for “a second additional resource or service” to add to your expense line. Missing is the better question, “Does this program make me look fat?”

Buried in the questionnaire is the seemingly innocuous question No. 45, “Do you have a succession plan for continuation of your practice in the event that you are unable to continue in the practice of law due to death, disability, or bar discipline?” On that last point of “bar discipline,” Arizona lawyers may not all be aware of this but the Bar recently made a succession plan a mandatory ethical obligation. So if you answer in the negative and then disclose your identity to enter the raffle, beware you aren’t also entering an unintended second raffle for a bar complaint.

The truth is this survey like all the others isn’t intended to identify or to serve members’ interests. These surveys only serve the Bar’s interests. They are tools to drive the Bar’s mission-creeping agenda and to cover its analysis (CYA).

Moreover, the survey and whatever the results may be — either good or bad — serve as a sword and a shield the Bar employs to hide behind or to flagellate critics, especially those ‘pests’ at the Arizona Legislature.

No matter if the Bar again gets an underwhelming response. It usually does. By overwhelming numbers, members ignore these surveys knowing full well what the agenda is about — $100 Visa gift card or not.

But just the same, count on the Bar to loudly trumpet the fake news: ‘Look at the great job we’re doing! Members love us.’ Such pronouncements will continue to fly in the face of reality. How can you dare to assess the satisfaction of captive members forced to join and forced to finance an ever-expanding bureaucratic empire in order to earn a living as lawyers?

And finally leaving absolutely no doubt its intentions comes the money paragraph: “The survey will help us serve you better. It’s also critical to the long-term success of State Bar of Arizona.”

This from the same organization that as ordered by the state supreme court “exists to serve and protect the public with respect to the provision of legal services and access to justice.”

There again is the State Bar of Arizona’s two-headed conflict of interest. It rears its two heads once more. Serve and protect the public but at the same time serve and protect the interests of lawyers —“help us serve you better.”

Reminds me of Jerry Maguire — minus the humor. ‘Help me, help you.’

______________________________

Photos: Via Morguefile.com, no attribution required.

Read Full Post »

AZ Bar still asks ‘are you happy?’ and responds to ‘poppycock’ survey post.

“Are you a member or happy?”

Paraphrasing that other “Mo” — the one with an “e,” the State of Bar of Arizona is still asking. How happy are members with the Bar’s programs and services?

Although I already took the meaningless member survey, Monday morning I got another email reminding me to complete it. This is the third reminder — but who’s counting? The survey period ends this Friday.

 

Doubtless the Bar’s trying to gin up something close to a representative response rate so its leaders can afterward claim victory just like they did three years ago.

Never mind the biased questions with their implicit assumptions or the forced choices from preselected answers. The results are preordained. But then I already I discussed all this last week at “Another ‘poppycock’ survey from the State Bar of Arizona.” At least the Bar’s not planning to survey us so they can later sell us the results.

Defensively speaking.

After last week’s post, I got an email from State Bar of Arizona Chief Communications Officer Rick DeBruhl. Understandably defensive, he wanted to explain a few things. With his consent, I’m posting his clarifications verbatim below.

_____________________________________________________________

“A couple thoughts on your blog:

“Our survey was designed with input by Bruce Merrill. Formerly with ASU, he is one of the premiere survey experts in the country. Surveys are inherently imperfect. Experts like Merrill design them in such a way as to work around our human idiosyncrasies. I’ve never taken a survey that I considered perfect, and I wouldn’t put ours in that category. Using consultants such as Merrill simply allow us to step beyond the amateur ranks to get reasonably valid information.

“You mentioned the fact that 80% of the bar’s membership did not respond in 2011. Survey experts will tell you that a 20% response rate is phenomenal. In addition, we made sure to check that the final numbers were demographically similar to our membership. That gives our survey validity.

 “Questions such as the “printed directory or a more robust online member search tool” were designed to give us guidance. We know from previous surveys that members overwhelmingly use and value the directory. We’ve heard that the current online search tool doesn’t go far enough. Would people be satisfied if we dumped the print directory for our existing search tool? Possibly not. What we’re ultimately trying to determine is whether they like the printed version because they want a book, or because they don’t have a better option.

“As for the answers on the “most serious problems” question, we actually got them from another state’s survey. We’re trying to build not just Arizona data but national trends as well. We looked them over carefully before deciding to include them. Any time you create a list, it has the potential to create bias to those answers. And yet survey experts say that respondents need lists to be reminded of the options.

“As for why we use SurveyMonkey, the answer is simple. It is the highest quality for the lowest cost. Of course there are other ways to survey members with greater anonymity. However, they cost significantly more money. Merrill feels we can achieve the same results with lower cost using this method. One other option would be a random sampling. Experts say we could get statistically valid results with just hundreds of responses. Perhaps, but we felt that if we were going to talk about the results of the member survey that all members should have the ability to give answers.

“Incidentally, we had a technical problem on the first day of the survey that prevented some people from submitting. As a result, our consultant decided that our best option was to remove the block that prevents a second survey from the same computer. The consultant feels that the number of people submitting two will be significantly small so that it won’t likely affect overall trends. We do have the ability to run a check on the number of repeat IP addresses which will let us know whether that number was significant. I’m sure if we had unlimited resources, we could no doubt determine the identity of each IP address, but that’s simply not in our realm. Incidentally, we don’t use the SurveyMonkey invitation system.

“We’d love it to be shorter, but we’ve done our best to chop it down. Because of the skip logic built into the survey, no one actually answers every question.

“The bottom line is that we understand that surveys are imperfect. We look for trends and directions and feel that gives us guidance as an organization.

“As always, let me know if you have questions or thoughts.”

__________________________________________________________________________________

Dollars to donuts.

I said last week I wasn’t a survey expert. But dollars to donuts, it’s like asking 10 lawyers about anything. If you get 10 survey experts in one room — you’ll get 10 different opinions.

Survey design is as much art as science. So what’s a good response rate? Depends who’s asking. And it depends on how they ask the question. Also see AAPOR | Response Rate – An Overview.

But I’m glad at least that Rick DeBruhl conceded more than once that the Bar survey was “imperfect” — because it is. That was the crux of my post.

There’s a reason some 80% of lawyers don’t bother answering these surveys. They’ve figured out what a medical historian once said about something else, “The experience of the ignorant has routed the wisdom of the learned.”

Too bad my point about the Bar’s expensive printed directory got lost. The survey’s either/or question about the directory was a leading question. It was biased toward a choice preference for “a more robust online member search tool.”

Not long ago, the Arizona Bar spent well over six figures supposedly improving and updating its website and its online member search tool. And now it appears some Bar executive is itching to spend even more money on what’s become a bloated website and online member search tool. Meantime regardless of the Bar’s claims at being eco-friendly — it’s just not yet because the Bar continues to print member directories and kill trees.

As for the survey having been designed “with input by Bruce Merrill” — well, that was a point I already footnoted last week. Still with an expert “like Merrill” on board, you have to ask why the Bar needed to crib stuff from other state bar surveys?

Satisfied?

Finally, as a learned colleague pointed out to me when I showed her Rick DeBruhl’s response, his email didn’t address the matter of his boss’s oversimplification of the Bar’s percentage of so-called ‘satisfied customers.’ This was last February 2013 when AZ Bar CEO John Phelps who’s also a lawyer addressed the state legislature’s house judiciary committee.

At about 27:33 on the tape and transcript, he omits the qualifier “somewhat” and asserts instead that “75% of the lawyers polled. . . were satisfied and 25% were not satisfied.”

Does such shorthand, she opined, potentially rise to an ethical rules violation under ER 7.1, i.e., that “a lawyer shall not make or knowingly permit to be made on the lawyer’s behalf a false or misleading communication”? Or as she also speculated, was it a possible violation of ER 8.4 (c) concerning “misleading” statements? The Bar’s communications chief doesn’t explain.

 

Read Full Post »

Another ‘poppycock’ survey from the State Bar of Arizona.

“Overall, how satisfied are you with your State Bar membership?”

That’s the first of 40 questions asked by the State Bar of Arizona in the 2014 version of its triennial Member Survey. A week ago Wednesday, the Bar sent a blast email asking“its members to evaluate our member services and your opinions on key issues facing the Arizona legal profession.” If ‘what’s past is prologue,’ Bar executives and their collaborators will again use the results to spin member satisfaction like they did in 2011.

In fact, as recently as February 2013, the Bar’s CEO was citing supposed member satisfaction to stymie a half-baked Arizona legislative effort to make Bar membership voluntary.¹

Poppycock.

The Bar’s email went on to state “Your participation in this survey will help the State Bar to better provide the services and information that meet your needs and interests.” Well as Tricky Dicky used to say, “that’s just plain poppycock.”

Damning case in point, of the 40 questions posed — not a single one pertained to member interests in either the cost of bar membership or the Bar’s lack of transparency. These were two of the biggest criticisms lodged against the dues-raising Arizona bar this year. And they were hot-button concerns raised by candidates during the just concluded Bar board of governors elections. So much for professing to sincerely inquire about members’ “needs and interests.”

Indeed, contrary to the specious claim that the Bar will use the survey “to better provide the services and information that meet your needs,” the real intent is pretextual. As it has before, the Arizona Bar will use the data as cover — as both sword and shield to fend off critics who contend the association is bloated, out-of-touch, and high-cost.

Happy Campers.

Thomas Holding, Wikipedia/public domain

Three years ago, 51% of all respondents reported only being “Somewhat Satisfied” with the Bar. Somewhat satisfied? As in 2011, that’s again one of the preselected choices. But what does it mean? Try telling your kid he’s “somewhat” smart or your girlfriend she’s “somewhat” pretty or your spouse you’re “somewhat satisfied” with your relationship. Let me know how well that works.

And an additional 23% said they were either “Somewhat Dissatisfied” or “Very Dissatisfied” with the Bar. But forget all that. The Bar’s spinmeister magazine, “Arizona Attorney,” nonetheless headlined the 2011 survey results with the misleadingly titled, “High Satisfaction, Room for Growth” and bragged about what supposed ‘happy campers’ Arizona lawyers were. And never mind that fully 80% of the Bar’s members were too indifferent or too busy to respond to the survey or that the Bar failed to follow-up with those 17,165 nonrespondents.²

N.Q.R.  Factor.

Now I don’t pretend to be an expert on surveys.³ But I do know this. Surveys should be concise. Questions are supposed to be clearly worded. And while it matters who’s paying for the survey, they’re also supposed to be neutral. What’s more, there’s as much art as science involved.

So I have my doubts about the Bar’s Member Survey. Besides the survey having too many questions requiring way too much work to fill out, there’s an N.Q.R.⁴ factor again emanating off this year’s survey.

Several questions appeared biased either by the implicit assumptions they make or by forcing respondents to make choices when they’d rather not. No wonder it feels like the deck is stacked.

When you ask, for example, “how satisfied” you are with your membership or “how valuable to you” bar services are, you know there’s something not quite right. Both questions are biased because of the implicit assumptions concerning satisfaction and valuable they make.⁵ The“words you use in the questions can affect respondents’ reaction and choices.”

Or take the leading question about whether or not there’s a preference for “a printed Member Directory or a more robust online member search tool?” [emphasis added]

Or how about the barely hidden Bar-agenda questions? For instance, there’s the forced choice made by Question 6, which in order to continue with the rest of the survey, requires respondents to pick at least one of 17 preselected positive choices under, “Which of the following are the features or uses for the [printed Membership] directory that you find to be the most valuable?” For those of us who think printed directories are a waste of money and of no value in a digital age, too bad. You can’t skip the question or choose ‘no opinion.’

And talk about agenda-driven responses like those in the survey category, “Professional Barriers.” Question 9 asks, “What do you believe are the three most serious problems faced by the legal profession today?” and Question 10 queries, “Please list the three most important issues that you would like to see the State Bar concentrate its efforts on in the next few years.”

For multiple-choice answers, the Bar provides its predetermined long list of alleged lawyer concerns like “lawyer advertising,” “diversity,” “lack of appropriate judicial system funding” and “threat to judicial independence” [even in a merit selection state where for the past 40 years 99.9% of Arizona judges are retained]. Who came up with those personal agenda-driven responses? But don’t look for choices about improving fiscal stewardship or treating members like clients or cutting costs or increasing bar transparency or heightening member due process.

And conveniently disingenuous about lawyer apprehension if not their outright paranoia when dealing with the almighty keeper of their meal-ticket-license — the survey asks intrusive demographic questions under the category, “About You and Your Work.” Like lawyers are going to trust privacy and confidentiality assurances about respondent anonymity when questions specifically ask for county of primary practice; year of admission generally, and in Arizona; age; gender; areas of practice; number of firm lawyers; and optionally, race and ethnicity. Why not just ask for names?

SurveyMonkey.

 

Not that respondents shouldn’t be concerned about survey integrity and anonymity. Where surveys contain “sensitive or potentially identifying information,” the U.S. Navy, for example, strongly recommends against commercial providers like SurveyMonkey that do not conform to its security regulations. “Since the data will be stored on commercial servers there is increased risk of harm or embarrassment if the data are somehow compromised.” As it happens, SurveyMonkey is the commercial survey provider used by the Bar.

On its website, SurveyMonkey explains that anonymity is up to the survey creator and not its job. While the survey creator has options to collect responses anonymously, SurveyMonkey explains, “All collection methods permit the tracking of respondent IP addresses. Anyone using the Email Invitation collector could potentially track an email address on the response.”

And according to a SurveyMonkey Review posted on the business software review site, TrustRadius, email links to its surveys allegedly allow individuals to “complete the survey more than once if they access the link through 2 different computers.” I don’t know about all that.

But I do know that after completing my survey, I could still access the same email link and begin completing another survey — not that I had any interest in wasting my time twice.

So is it one anonymous survey per ‘customer’? Or is that just more poppycock? In truth, I don’t care.

__________________________________________________________________________________________

[1] See Arizona House Judiciary Committee videotaped hearing “Arizona Supreme Court’s control over state bar debated, contested . . .” at http://www.youtube.com/watch?v=xotdkMf61Ic, February 14, 2013 and remarks by Arizona Bar CEO John Phelps at 27:14 conveniently omitting the faint praise qualifier “somewhat” and asserting instead that “75% of the lawyers polled. . . were satisfied and 25% were not satisfied.”

[2] See Ten Reasons Why Surveys Fail by Dr. David Futrell, Quality Progress Magazine, April 1994, noting, “Failure to follow up with the nonrespondents can yield grossly misleading data. In general, people who respond to a survey will be more extreme in terms of what is being measured than the nonrespondents.”

[3] To be fair, the Bar hired an expert, noted local researcher Bruce Merrill, Ph.D., to assist its 2011 survey. Dr. Merrill once ran a golf hole Ad-in-the-Hole Research Study to evaluate “Name/Brand Awareness,” “Ad Recall” and whether golf hole ads are “Bothersome to golfers.” Golf hole advertising’s a dumb idea — but no worse than urinal advertising.

[4] NQR  means Not Quite Right. I was first introduced to the acronym by a friend and former F-18 fighter pilot.

[5] Compare the Bar’s “how satisfied” question with the example borrowed from Sterngold, Warland and Herrmann (1994) by Professors Hershey Friedman, Ph.D. and Taiwo Amoo, Ph.D. in Ranking the Rating Scales, published in the Journal of Marketing Management, Vol. 9:3, Winter 1999, 114-123. Referring to an earlier study, the professors write “that a question like “How concerned are you about…?” causes a bias in the direction of concern because it assumes that subjects should be concerned about an issue. Using a filter question first asking respondents whether or not they were concerned with an issue and then asking those that were concerned to rate their degree of concern resulted in significantly fewer people showing concern than the former approach.” Similarly, I posit that asking how satisfied members are with the Bar “causes a bias in the direction of [satisfaction] because it assumes that subjects should be [satisfied]” with the Bar.

Read Full Post »