An early RRForum study that examined the impact of mystery shops and the optimal frequency of inspections unexpectedly revealed how public health and safety agencies, and retailers, can significantly leverage their efforts to improve ID-checking through effective communications.

Context: Walgreens was the first national chain in 2001 to enter an Assurance of Voluntary Compliance with 43 state Attorneys General to reduce unlawful sales of tobacco, alcohol, and other age-restricted products. Walgreens, like other signatory chains, agreed to monitor staff ID-checking performance. RRForum, which was providing technical assistance to the AGs and the signatory chains, was concurrently developing a strategy to improve ID-checking that included real-time feedback from young, legal-age mystery shoppers. (Eventually this led to an NIH award that validated the effectiveness of mystery shopper programs, described in Research Brief #2.) Through the assistance of Alan Lieberman, Deputy AG of California overseeing the implementation of AVCs by Walgreens and other national chains, Walgreens allowed RRForum to embed a small study within the pharmacy chain’s larger monitoring program that would test the effectiveness of mystery shop inspections and examine the optimal frequency for inspections.

Research design. Forty-five Walgreens pharmacies – 15 each in St Louis, Houston, and Milwaukee – were identified and five stores from each of the three cities were randomly assigned to two treatment arms and a control arm.  All stores were inspected 20 times at intervals of approximately 2 weeks. The first 4 inspections, in which no reports were given to any store, served as a baseline indicator of staff ID-checking propensities. Thereafter, stores assigned to the first treatment arm received real-time feedback at the end of every attempted purchase, i.e., 2 Reports/Month.  Stores in the second treatment arm received real-time feedback at the conclusion of every second mystery shopper purchase attempt after the baseline i.e., 1 Report/Month. Stores in the control study arm did not receive feedback after any visits.

Results. ID-checking rates improved steadily and significantly for both treatment arms.  As shown below, 2 Reports/Month ID-checking rates increased from 66% at baseline to 78%, 90%, and 85% in the following 3 quarters. 1 Report/Month ID-checking rates increased from 56% at baseline to 83%, 89%, and 88% in the following 3 quarters. Clearly, mystery shop reports produced significant improvement, with no significant benefit accruing from the greater frequency.

The problem, however, was the third, control arm. If these 15 stores were receiving no feedback from the mystery shop inspections, why had their ID-checking rates risen from 55% at baseline to 85%, essentially the same level as the two treatment arms?

This aberration signaled possible corruption in the program implementation. To understand the anomalies, we contacted managers of the control stores. No, they had received no feedback from any of the 20 mystery shop visits. But they attended regional meetings and trainings whose managers included those from the two treatment arms. They learned of the inspections and wondered if they, too, might also be inspected. That concern translated into heightened communications with staff about ID-checking and heightened supervision. The lag in ID-checking improvements for the control arm is consistent with their managers learning of the inspections later in regional meetings and other communications, not in real time.

Discussion. Law enforcement compliance checks boost ID-checking performance. So, too, do inspections by legal-age mystery shoppers (described in Research Brief #2) and “reward and reminder” programs conducted by public health and prevention agencies.  Any of these inspection programs require a significant investment of time and resources. Getting as much “bang for the buck” for these programs is important, especially when law enforcement, public health / prevention, and industry face so many other challenges requiring their time and resources.

The Walgreens study shows that licensees who are made aware of inspections exhibit improvements in ID-checking conduct, opening an additional strategy to leverage inspections by communicating with licensees.

When the Walgreens study was presented at the 2008 RRForum national conference in Pittsburgh, PA, Lt. James Jones, Jr. of the Pennsylvania State Police Bureau of Liquor Control Enforcement privately shared the insight he had drawn: “It doesn’t really matter how many checks we do, what matters is how many the licensees think we do.” Now retired as a captain but still active in NLLEA, Jones developed anecdotal evidence that supported the strategy he then implemented.  “When I had the offices distribute news releases announcing checks in several different areas, but conducted checks in only one or two, we saw a notable increase in overall compliance rates.”

[RRForum had hoped to validate what we termed the “Walgreens effect” through a larger study in Oregon and Texas, funded by DOJ. However, very high ID checking rates were observed in Oregon during the baseline period, leaving little room for improvement and making it very difficult to detect intervention effects. Unfortunately, the study was not sufficiently powered in terms of numbers of communities to detect intervention effects within the two states separately. Future studies should include more communities across more states.]

Takeaway. Inspections of ID-checking conduct are effective in improving that conduct, but inspections require a significant investment of time and resources.  Communicating the intent to conduct inspections will, in and of itself, improve that conduct.  A small additional effort to publicize the intention to check age-verification conduct in the community, and to publicize the results of those inspections to all licensees, whether those licensees were inspected or not, significantly leverages that investment.

To read the journal article of the Walgreens project, select this link:

The effect of mystery shopper reports on age verification for tobacco purchases – PubMed (nih.gov)

If you would like to receive additional Research Briefs in this series, and other RRForum news, register here.