Evaluating a Mystery Shopping Provider
Content
Evaluating a Mystery Shopping Provider
White Paper June 16, 2008 By: Steven Di Pietro Managing Director Service Integrity Mystery Shopping How do you decide which Mystery Shopping company is best for you? The answer depends on more than the normal criteria you would use to evaluate service providers. The largest, oldest or best known provider may be the WRONG one for you. Unlike other forms of research, with Mystery Shopping each and every result is dissected and scrutinized by each site, so you want to be comfortable the results are valid. You will need to ask questions about Shopper Management, Quality Control, Operations, and Reporting before comparing costs. The following guide describes some of the more detailed issues you should address when selecting a provider.
Evaluating a Mystery Shopping Provider – White Paper
1
2
3
4
5
6
Shopper Management ........................................................................................ 3 1.1 Shopper Recruitment ................................................................................... 3 1.2 Data Collected............................................................................................. 3 1.3 Shopper status ............................................................................................. 3 1.3.1 Advantage of Employees...................................................................... 3 1.3.2 Advantage of Contractors ..................................................................... 3 1.4 Shopper vetting ........................................................................................... 3 1.5 Client screening .......................................................................................... 4 1.6 Pre-Shop screening...................................................................................... 4 1.7 Shopper Opt-in? .......................................................................................... 4 1.8 How do shoppers select sites? ..................................................................... 4 1.9 Are the shoppers locals? .............................................................................. 4 1.10 Shopper deletions .................................................................................... 5 1.11 Match shoppers to each of your locations................................................. 5 1.12 How many shoppers are active and fully registered? ................................ 5 1.13 Shopper rotation ...................................................................................... 6 1.14 Shopper frequency ................................................................................... 6 1.15 Shops per month ...................................................................................... 6 1.16 Demographics of current shoppers ........................................................... 7 Quality Control .................................................................................................. 7 2.1 Reading of each result ................................................................................. 7 2.2 Proportion of results entered on-line ............................................................ 7 2.3 Receipts verification .................................................................................... 7 2.4 IT in-house or outsourced ............................................................................ 8 Operations ......................................................................................................... 8 3.1 Ability to introduce off-cycle shops ............................................................. 8 3.2 Shopper briefing process ............................................................................. 8 3.3 Result timeliness ......................................................................................... 8 3.4 Live updates on shop status and reassigning ................................................ 8 Reporting........................................................................................................... 9 4.1 Web ............................................................................................................ 9 4.2 Speed of updates ......................................................................................... 9 4.3 Receipts on-line .......................................................................................... 9 4.4 Multiple access levels.................................................................................. 9 Pricing ............................................................................................................... 9 5.1 Changes to questionnaires ........................................................................... 9 5.2 Additional auditing...................................................................................... 9 5.3 Length of contract ....................................................................................... 9 5.4 Travel reimbursements ................................................................................ 9 Other ................................................................................................................10 6.1 International experience/peers ....................................................................10 6.2 Transparency of data ..................................................................................10 6.3 Staff queries – response mechanism ...........................................................10
Steven Di Pietro
Service Integrity Mystery Shopping ABN 53 107 407 607
Page 2 of 10
Evaluating a Mystery Shopping Provider – White Paper
1 Shopper Management
Shopper management is one of the more critical aspects of Mystery Shopping and often distinguishes different providers. 1.1 Shopper Recruitment A variety of methods may be used to employ shoppers including, cold calling, newspaper/magazine advertising, referrals, or even bureaus with lists of available names. At times the customers own database can be used to recruit shoppers. As the client it is important for you to understand these sources. Be careful of shoppers being used from Mystery Shopping bureaus because the detailed shopper information may not be available to the Mystery Shopping Company providing your service. If advertising is used it may compromise speedy delivery of new shoppers. 1.2 Data Collected Ask for a list of all data fields captured for the shoppers. In many cases information such as height, weight, income group and sex can be used to determine if the data is skewed to one demographic group. Alternately this information can be used to screen out inappropriate shoppers such as a man trying women’s clothing. 1.3 Shopper status Some Mystery Shopping companies use the Employees (whether full or part-time) while others use Contractors. Most companies are now moving towards the Contractor route. 1.3.1 Advantage of Employees • • • Stability Training Reasonably assured of identity Can be quickly recruited Can be easily terminated (because they can be more easily replaced) Fresher approach (not as jaded) More chance of random demographics of shoppers Usually opt-in for the jobs they ‘want’ to do. Shopper vetting
1.3.2 Advantage of Contractors • • • • • 1.4
The most basic requirement is to ensure that shoppers can read and write. Although many on-line edit programs can detect spelling errors, they cannot detect the incorrect use of a word. For example if a shopper uses ‘their’ instead of ‘there’.
Steven Di Pietro
Service Integrity Mystery Shopping ABN 53 107 407 607
Page 3 of 10
Evaluating a Mystery Shopping Provider – White Paper With strict auditing many of these errors are detected, but then some can slip through. To prevent problems for the exceptions a Mystery Shopping company needs to be sure the Shoppers have good literary skills. If a shopper cannot comprehend the shop guidelines or report back accurate results, the shop can be made invalid, which then puts pressure on the Mystery Shopping Company to re-shop the store, thus putting pressure on on-time delivery. One way to ensure the shoppers have basic literary skills is to ask them to complete a Test Shop to show they can a) follow instructions, and b) write clear responses. Ask your provider how they vet their shoppers on recruitment. 1.5 Client screening Ask whether the shopper demographics allow you to remove shoppers with compromised interests. For example, as a clothing retailer you may not want a competitor’s employee Mystery Shopping your store. Not only can your results be tainted but the questionnaire could fall into the possession of the competitor company itself. This screening also allows you to remove shoppers who would be unnatural in your store (usually a sign of a mystery shopper). 1.6 Pre-Shop screening At times it may be necessary to pre-screen shoppers before conducting an assignment. You may have some very specific requirements for the round of shops, or require filtering beyond the initial questions asked by your provider. For example, you may only want shoppers who have not been in your store in the past month, or who do not currently have your product. If your filters are too restrictive, they may have an impact on price. The more restrictive your requirements, the harder it will be for the provider to find shoppers and may increase cost, or compromise the on-time performance of shops. 1.7 Shopper Opt-in? You must decide whether you want shoppers who ‘want’ to be in your store or who are forced to go to your store. Your normal customers go to your store or call centre because they have a want or need, usually not because someone has forced them. Allowing the shoppers to opt-in to the shop places normal people in their normal environment. 1.8 How do shoppers select sites? Ask how shoppers are given the opportunity to shop your store. Are the shops assigned manually or automatically? This can impact on you as the client because efficiency in deployment of shops will increase the likelihood of you receiving ontime results. Most sophisticated Mystery Shopping companies will employ an electronic process of matching the shopper’s latitude and longitude with that of the store. 1.9 Are the shoppers locals? Shoppers who do not reside in the area of your stores can be easy to detect, especially in small country towns.
Steven Di Pietro
Service Integrity Mystery Shopping ABN 53 107 407 607
Page 4 of 10
Evaluating a Mystery Shopping Provider – White Paper Ask whether the shoppers are local, or travel a distance to the Shop assignment. 1.10 Shopper deletions Under what circumstances can your provider delete a shopper? The answer to this question usually depends on the status of shoppers, whether they are contractors or employees. Shoppers come in various shapes and sizes. A typical Mystery Shopping Company will have thousands of shoppers. In any cross section of the population you will find people who are unreliable and incapable of conducting a mystery shop. It is not easy to conduct a mystery shop, in fact it is much more difficult than it may otherwise seem. The easier to delete a shopper, the less likelihood you will have a rogue shopper conduct a shop on your premises. 1.11 Match shoppers to each of your locations Your provider will require at least three shoppers for each site and each evaluation. Whatever the site, some shoppers will not be available for a number of reasons such as illness, or other commitments. Therefore more than one shopper is required per site. The more frequently a location is shopped, the more shoppers are required because the same shopper should not shop the same store too frequently. When you ask the provider to match shoppers to your site locations, ask for the breakdown showing number of shoppers within a 30km radius of each site. There will be gaps, and the best company is not necessarily the one with the least gaps. Those gaps need to be explained by the recruiting processes. Often Mystery Shopping companies keep the number of shoppers as low as possible, pending the signing of new clients. The reason is that they do not want idle shoppers in locations where there are no assignments. 1.12 How many shoppers are active and fully registered? Many mystery shopping companies quote ridiculously large numbers of shoppers forming their shopper database. A common way to achieve large numbers is to simply ask people to provide a name and email address in a simple registration process. These are not shoppers, simply people interested in one day becoming a shopper. When you ask for shopper numbers, ask for the number of shoppers that are fully accredited and who can already see assignments. It is very difficult to cut through these statistics to find the truth. Some companies deem that any person who has provided even a fleeting interest is an available shopper. This may be true; however you may then wish to call into question what vetting and screening processes are adopted. Two pieces of information can help dissect the information by asking 1) how many of the shoppers have previously conducted a valid shop, and 2) looking at the proportion of accredited shoppers to total names on the database. In the case of the latter, it would be typical to only have approximately half the registered shoppers available for your assignments.
Steven Di Pietro
Service Integrity Mystery Shopping ABN 53 107 407 607
Page 5 of 10
Evaluating a Mystery Shopping Provider – White Paper This is important to you as the client because it will a) help you cut through the sales talk and b) help you understand the capability of your provider. It is then your decision as to whether you employ the smaller company with a definite recruiting plan, or a larger company who has tried to artificially inflate their position even greater. 1.13 Shopper rotation Depending on the industry, this is a critical factor in managing a program. Shopper rotation refers to how often a shopper can return to the same store in a cycle. If you have the same shopper visiting the same store for every shop, you will receive the input of the same person. If that person is barred from revisiting the same store too frequently, it will allow other shoppers (with another perspective) the opportunity to evaluate the service. Although Mystery Shopping questionnaires tend to be very binary Yes/No answers to remove subjectivity, the shoppers are often asked to provide free form comments which often assist the store in understanding more background to the results. Having the same result from the same shopper each month provides no additional insights. Having the same shopper visit also has the shopper internally comparing results between shops. This can be especially unfair if the shopper begins to form an affinity with a staff member or the store itself. In other words, they may start marking the store higher and higher as their affinity grows. In some cases such as the car industry, it would be unnatural for the same shopper to enter the store each month to ask about a new vehicle. In a large grocery store or fast food outlet, it would be less problematic, but nonetheless one dimensional to receive the same shoppers results each month. 1.14 Shopper frequency Shopper frequency refers to how many shops can be conducted for your Brand over all. This should be restricted to remove the opportunity to form an affiliation with the brand. The other reason to restrict the number of shops per client per month is to reduce the damage if the shopper is discovered. If the shopper is discovered, the stores may talk among themselves and compromise your shops in many locations. 1.15 Shops per month The final restriction on shoppers is the number of shops per month, across all clients. Professional shoppers tend to internally compare clients, and stores. Mystery Shopping is about the singular experience at a point in time, not comparisons. The most alarming aspect of allowing too many shops per month is inconsistency within a day or month. A shopper’s attitude on a fresh Monday morning will be very different to a rushed shop assignment at the end of the week on a rainy day. Restricting the number of shops per month keeps rotation in the shopper base. This restriction is not popular with shoppers but ensures you retain randomness of shopper demographics, shop times and even moods of the shoppers. From a client perspective most importantly, you can be more assured you have a fresh, keen shopper, with less likelihood of internal comparisons.
Steven Di Pietro
Service Integrity Mystery Shopping ABN 53 107 407 607
Page 6 of 10
Evaluating a Mystery Shopping Provider – White Paper 1.16 Demographics of current shoppers To keep the randomness of shoppers cycling through your stores, ask the provider to show the demographic break-up of your shoppers in terms of sex, age, income group and state.
2 Quality Control
2.1 Reading of each result Despite computer editing and auditing of results, there remains the opportunity for error where the wrong word is used in a questionnaire. As mentioned earlier, spell checkers will not detect the incorrect use of a word such as ‘their’ instead of ‘there’. Reading each result will also pick up inconsistencies in the questionnaire. At times the free form comments can contradict the shop score from yes/no questions. These inconsistencies can only be detected by the human eye. For example, the Mystery Shop result may show a low score but the shopper comments indicate they had a wonderful experience. These inconsistencies can sometimes be explained, but may also indicate a problem with the results. Another example is if the shopper indicated they waited 5 minutes to be served, and then in the comments makes a comment that they were impressed by the quick service. The most common form of verification is to simply read each result. Ask your provider to explain their verification process and whether there is any additional charge for the service. 2.2 Proportion of results entered on-line Ask how many shopper results are entered on-line as opposed to being called in over the phone, mailed, or faxed. Unless the shopper enters results directly there is a great opportunity for transcription error, regardless of the method. Phone shops are notorious for transcription especially where shoppers and/or transcribers take small shortcuts to speed the process, e.g. The shopper says something like “The first five are Yes, followed by a No, two Yes’s and a No”. Mailed questionnaires are problematic because in addition to the transcription issue the results can come late. This is not only important for the service standards, if there are questionable results, and the shopper needs to be contacted, there is less likelihood they can remember the intricate details of the shop. Faxed shops are subject to the additional transcription error where unclear transmissions may result in ‘guesses’. Faxes are also notorious for being lost. 2.3 Receipts verification Where possible the provider should retain a receipt of the shop to verify the shop took place. Receipt verification will also allow the provider to check the shop took place during the times (restrictions) required by the client and that the shopper went to the correct location.
Steven Di Pietro
Service Integrity Mystery Shopping ABN 53 107 407 607
Page 7 of 10
Evaluating a Mystery Shopping Provider – White Paper Typically, although the shopper will enter the shop time and date on the Shopping system, the receipt will verify whether this was in fact the time of the shop. As a client, if you do NOT want evening shops, this process will ensure the shopper followed the instructions. 2.4 IT in-house or outsourced Ask whether the IT resources are internal or outsourced. Ask questions about the IT structure to give you a feel for the robustness of systems. Outsourcing IT may at times provide more robustness but may also restrict flexibility to provide the service you require.
3 Operations
3.1 Ability to introduce off-cycle shops Ask whether the system can cater for off-cycle shops. Although your program may be monthly, occasionally you may want to add a wild card shop (off-cycle). You may decide to conduct additional shops for whatever reason but once in a program find there is an additional cost of implementation or that the off-cycle shops are not possible. 3.2 Shopper briefing process Understand the process of briefing the shoppers. Are they simply ‘dumped’ with a questionnaire or are they pre-briefed on the requirements and expectations? 3.3 Result timeliness Ask how quickly the shopper is required to submit their results. A critical aspect of ensuring validity of results is to have results input and returned quickly. Typically results should be input within 24 hours of the shop taking place. 3.4 Live updates on shop status and reassigning Ask how the provider monitors how the shops are being assigned, and taken up by the shoppers. You can be confident the shops will be completed on time if operationally there is a live update of each shops status. Typically the shop status includes classifications such as • • • • • • • • • Assigned, Confirmed, Audit, Pending (if information is missing) Approved Incomplete Rejected Invalid (go to wrong site or do not follow guidelines) Un-shoppable (if the site was closed)
Steven Di Pietro
Service Integrity Mystery Shopping ABN 53 107 407 607
Page 8 of 10
Evaluating a Mystery Shopping Provider – White Paper
4 Reporting
4.1 Web Best practice is to now provide web reporting. Web reporting will also give you, as the client the ability to store results and disseminate to a greater audience. 4.2 Speed of updates As the client, you should have the choice of asking for website updates as quickly or slowly as you require. In some cases, you may wish to have results updates every hour or two, whereas in other cases you may seek to have results updates only once a month. At times you may seek a staged release of results where the client principal has an opportunity to review results before they are released to the broader company population. Sometimes this additional check can pick up some inconsistencies that allow both client and provider to take corrective action if possible. 4.3 Receipts on-line Typically the individual receipts should be stored on-line but are not necessarily made available to the store on-line. If there is a query, you will want quick access to the receipt and be confident it can be found, as opposed to being lost in a filing cabinet. Providing receipts on-line can provide the opportunity for the store to undertake investigations to identify the shopper, thus compromising future shops. 4.4 Multiple access levels In larger organisations it is important to have different levels of access where each level can only see their own area of responsibility. You may not want individual stores or State Managers to see each other’s results, but still see their rankings.
5 Pricing
At times during a pitch, the costs are simply stated as a cost per shop. The following costs may not be made clear until the decision made to appoint the successful company. 5.1 Changes to questionnaires Changes to questionnaires are not abnormal, especially early in the program. Despite the best intentions by both parties to refine the questionnaires, there is no substitute for seeing real data flow from the program. Understand the pricing of minor changes. 5.2 5.3 Additional auditing Length of contract Understand if there are additional costs for manual reading of the results. Short term, even monthly contracts are more and more common and a reasonable request of your provider. 5.4 Travel reimbursements At times, especially with contractor shoppers, the provider may introduce mileage reimbursements for shoppers. If these are necessary, ensure there are minimum and maximum distances.
Page 9 of 10
Steven Di Pietro
Service Integrity Mystery Shopping ABN 53 107 407 607
Evaluating a Mystery Shopping Provider – White Paper It would be unreasonable to have travel payments for short distances 5 minutes from your site, and similarly unreasonable to pay mileage for over 1 hour travel times.
6 Other
6.1 International experience/peers Being part of affiliations may give the impression of being an industry leader, however more pertinent may be the ability for the provider to coopt with other worldwide offices where fluid communications and learning can provide immense leverage to the client. 6.2 Transparency of data As a client it is reasonable to request not only the raw data from completed shops, but also shopper statistics such as demographics. 6.3 Staff queries – response mechanism Regardless of the quality of questionnaires and provider, there will be questions asked from the recipients of questionnaires. Ask you provider about the mechanisms for feeding store questions. Current best practice is to allow on-line queries directly linked to the questionnaire results. This allows the feedback and responses to be perpetually linked to the store result. Steven Di Pietro Managing Director Service Integrity Mystery Shopping mailto:[email protected]
Steven Di Pietro
Service Integrity Mystery Shopping ABN 53 107 407 607
Page 10 of 10
White Paper June 16, 2008 By: Steven Di Pietro Managing Director Service Integrity Mystery Shopping How do you decide which Mystery Shopping company is best for you? The answer depends on more than the normal criteria you would use to evaluate service providers. The largest, oldest or best known provider may be the WRONG one for you. Unlike other forms of research, with Mystery Shopping each and every result is dissected and scrutinized by each site, so you want to be comfortable the results are valid. You will need to ask questions about Shopper Management, Quality Control, Operations, and Reporting before comparing costs. The following guide describes some of the more detailed issues you should address when selecting a provider.
Evaluating a Mystery Shopping Provider – White Paper
1
2
3
4
5
6
Shopper Management ........................................................................................ 3 1.1 Shopper Recruitment ................................................................................... 3 1.2 Data Collected............................................................................................. 3 1.3 Shopper status ............................................................................................. 3 1.3.1 Advantage of Employees...................................................................... 3 1.3.2 Advantage of Contractors ..................................................................... 3 1.4 Shopper vetting ........................................................................................... 3 1.5 Client screening .......................................................................................... 4 1.6 Pre-Shop screening...................................................................................... 4 1.7 Shopper Opt-in? .......................................................................................... 4 1.8 How do shoppers select sites? ..................................................................... 4 1.9 Are the shoppers locals? .............................................................................. 4 1.10 Shopper deletions .................................................................................... 5 1.11 Match shoppers to each of your locations................................................. 5 1.12 How many shoppers are active and fully registered? ................................ 5 1.13 Shopper rotation ...................................................................................... 6 1.14 Shopper frequency ................................................................................... 6 1.15 Shops per month ...................................................................................... 6 1.16 Demographics of current shoppers ........................................................... 7 Quality Control .................................................................................................. 7 2.1 Reading of each result ................................................................................. 7 2.2 Proportion of results entered on-line ............................................................ 7 2.3 Receipts verification .................................................................................... 7 2.4 IT in-house or outsourced ............................................................................ 8 Operations ......................................................................................................... 8 3.1 Ability to introduce off-cycle shops ............................................................. 8 3.2 Shopper briefing process ............................................................................. 8 3.3 Result timeliness ......................................................................................... 8 3.4 Live updates on shop status and reassigning ................................................ 8 Reporting........................................................................................................... 9 4.1 Web ............................................................................................................ 9 4.2 Speed of updates ......................................................................................... 9 4.3 Receipts on-line .......................................................................................... 9 4.4 Multiple access levels.................................................................................. 9 Pricing ............................................................................................................... 9 5.1 Changes to questionnaires ........................................................................... 9 5.2 Additional auditing...................................................................................... 9 5.3 Length of contract ....................................................................................... 9 5.4 Travel reimbursements ................................................................................ 9 Other ................................................................................................................10 6.1 International experience/peers ....................................................................10 6.2 Transparency of data ..................................................................................10 6.3 Staff queries – response mechanism ...........................................................10
Steven Di Pietro
Service Integrity Mystery Shopping ABN 53 107 407 607
Page 2 of 10
Evaluating a Mystery Shopping Provider – White Paper
1 Shopper Management
Shopper management is one of the more critical aspects of Mystery Shopping and often distinguishes different providers. 1.1 Shopper Recruitment A variety of methods may be used to employ shoppers including, cold calling, newspaper/magazine advertising, referrals, or even bureaus with lists of available names. At times the customers own database can be used to recruit shoppers. As the client it is important for you to understand these sources. Be careful of shoppers being used from Mystery Shopping bureaus because the detailed shopper information may not be available to the Mystery Shopping Company providing your service. If advertising is used it may compromise speedy delivery of new shoppers. 1.2 Data Collected Ask for a list of all data fields captured for the shoppers. In many cases information such as height, weight, income group and sex can be used to determine if the data is skewed to one demographic group. Alternately this information can be used to screen out inappropriate shoppers such as a man trying women’s clothing. 1.3 Shopper status Some Mystery Shopping companies use the Employees (whether full or part-time) while others use Contractors. Most companies are now moving towards the Contractor route. 1.3.1 Advantage of Employees • • • Stability Training Reasonably assured of identity Can be quickly recruited Can be easily terminated (because they can be more easily replaced) Fresher approach (not as jaded) More chance of random demographics of shoppers Usually opt-in for the jobs they ‘want’ to do. Shopper vetting
1.3.2 Advantage of Contractors • • • • • 1.4
The most basic requirement is to ensure that shoppers can read and write. Although many on-line edit programs can detect spelling errors, they cannot detect the incorrect use of a word. For example if a shopper uses ‘their’ instead of ‘there’.
Steven Di Pietro
Service Integrity Mystery Shopping ABN 53 107 407 607
Page 3 of 10
Evaluating a Mystery Shopping Provider – White Paper With strict auditing many of these errors are detected, but then some can slip through. To prevent problems for the exceptions a Mystery Shopping company needs to be sure the Shoppers have good literary skills. If a shopper cannot comprehend the shop guidelines or report back accurate results, the shop can be made invalid, which then puts pressure on the Mystery Shopping Company to re-shop the store, thus putting pressure on on-time delivery. One way to ensure the shoppers have basic literary skills is to ask them to complete a Test Shop to show they can a) follow instructions, and b) write clear responses. Ask your provider how they vet their shoppers on recruitment. 1.5 Client screening Ask whether the shopper demographics allow you to remove shoppers with compromised interests. For example, as a clothing retailer you may not want a competitor’s employee Mystery Shopping your store. Not only can your results be tainted but the questionnaire could fall into the possession of the competitor company itself. This screening also allows you to remove shoppers who would be unnatural in your store (usually a sign of a mystery shopper). 1.6 Pre-Shop screening At times it may be necessary to pre-screen shoppers before conducting an assignment. You may have some very specific requirements for the round of shops, or require filtering beyond the initial questions asked by your provider. For example, you may only want shoppers who have not been in your store in the past month, or who do not currently have your product. If your filters are too restrictive, they may have an impact on price. The more restrictive your requirements, the harder it will be for the provider to find shoppers and may increase cost, or compromise the on-time performance of shops. 1.7 Shopper Opt-in? You must decide whether you want shoppers who ‘want’ to be in your store or who are forced to go to your store. Your normal customers go to your store or call centre because they have a want or need, usually not because someone has forced them. Allowing the shoppers to opt-in to the shop places normal people in their normal environment. 1.8 How do shoppers select sites? Ask how shoppers are given the opportunity to shop your store. Are the shops assigned manually or automatically? This can impact on you as the client because efficiency in deployment of shops will increase the likelihood of you receiving ontime results. Most sophisticated Mystery Shopping companies will employ an electronic process of matching the shopper’s latitude and longitude with that of the store. 1.9 Are the shoppers locals? Shoppers who do not reside in the area of your stores can be easy to detect, especially in small country towns.
Steven Di Pietro
Service Integrity Mystery Shopping ABN 53 107 407 607
Page 4 of 10
Evaluating a Mystery Shopping Provider – White Paper Ask whether the shoppers are local, or travel a distance to the Shop assignment. 1.10 Shopper deletions Under what circumstances can your provider delete a shopper? The answer to this question usually depends on the status of shoppers, whether they are contractors or employees. Shoppers come in various shapes and sizes. A typical Mystery Shopping Company will have thousands of shoppers. In any cross section of the population you will find people who are unreliable and incapable of conducting a mystery shop. It is not easy to conduct a mystery shop, in fact it is much more difficult than it may otherwise seem. The easier to delete a shopper, the less likelihood you will have a rogue shopper conduct a shop on your premises. 1.11 Match shoppers to each of your locations Your provider will require at least three shoppers for each site and each evaluation. Whatever the site, some shoppers will not be available for a number of reasons such as illness, or other commitments. Therefore more than one shopper is required per site. The more frequently a location is shopped, the more shoppers are required because the same shopper should not shop the same store too frequently. When you ask the provider to match shoppers to your site locations, ask for the breakdown showing number of shoppers within a 30km radius of each site. There will be gaps, and the best company is not necessarily the one with the least gaps. Those gaps need to be explained by the recruiting processes. Often Mystery Shopping companies keep the number of shoppers as low as possible, pending the signing of new clients. The reason is that they do not want idle shoppers in locations where there are no assignments. 1.12 How many shoppers are active and fully registered? Many mystery shopping companies quote ridiculously large numbers of shoppers forming their shopper database. A common way to achieve large numbers is to simply ask people to provide a name and email address in a simple registration process. These are not shoppers, simply people interested in one day becoming a shopper. When you ask for shopper numbers, ask for the number of shoppers that are fully accredited and who can already see assignments. It is very difficult to cut through these statistics to find the truth. Some companies deem that any person who has provided even a fleeting interest is an available shopper. This may be true; however you may then wish to call into question what vetting and screening processes are adopted. Two pieces of information can help dissect the information by asking 1) how many of the shoppers have previously conducted a valid shop, and 2) looking at the proportion of accredited shoppers to total names on the database. In the case of the latter, it would be typical to only have approximately half the registered shoppers available for your assignments.
Steven Di Pietro
Service Integrity Mystery Shopping ABN 53 107 407 607
Page 5 of 10
Evaluating a Mystery Shopping Provider – White Paper This is important to you as the client because it will a) help you cut through the sales talk and b) help you understand the capability of your provider. It is then your decision as to whether you employ the smaller company with a definite recruiting plan, or a larger company who has tried to artificially inflate their position even greater. 1.13 Shopper rotation Depending on the industry, this is a critical factor in managing a program. Shopper rotation refers to how often a shopper can return to the same store in a cycle. If you have the same shopper visiting the same store for every shop, you will receive the input of the same person. If that person is barred from revisiting the same store too frequently, it will allow other shoppers (with another perspective) the opportunity to evaluate the service. Although Mystery Shopping questionnaires tend to be very binary Yes/No answers to remove subjectivity, the shoppers are often asked to provide free form comments which often assist the store in understanding more background to the results. Having the same result from the same shopper each month provides no additional insights. Having the same shopper visit also has the shopper internally comparing results between shops. This can be especially unfair if the shopper begins to form an affinity with a staff member or the store itself. In other words, they may start marking the store higher and higher as their affinity grows. In some cases such as the car industry, it would be unnatural for the same shopper to enter the store each month to ask about a new vehicle. In a large grocery store or fast food outlet, it would be less problematic, but nonetheless one dimensional to receive the same shoppers results each month. 1.14 Shopper frequency Shopper frequency refers to how many shops can be conducted for your Brand over all. This should be restricted to remove the opportunity to form an affiliation with the brand. The other reason to restrict the number of shops per client per month is to reduce the damage if the shopper is discovered. If the shopper is discovered, the stores may talk among themselves and compromise your shops in many locations. 1.15 Shops per month The final restriction on shoppers is the number of shops per month, across all clients. Professional shoppers tend to internally compare clients, and stores. Mystery Shopping is about the singular experience at a point in time, not comparisons. The most alarming aspect of allowing too many shops per month is inconsistency within a day or month. A shopper’s attitude on a fresh Monday morning will be very different to a rushed shop assignment at the end of the week on a rainy day. Restricting the number of shops per month keeps rotation in the shopper base. This restriction is not popular with shoppers but ensures you retain randomness of shopper demographics, shop times and even moods of the shoppers. From a client perspective most importantly, you can be more assured you have a fresh, keen shopper, with less likelihood of internal comparisons.
Steven Di Pietro
Service Integrity Mystery Shopping ABN 53 107 407 607
Page 6 of 10
Evaluating a Mystery Shopping Provider – White Paper 1.16 Demographics of current shoppers To keep the randomness of shoppers cycling through your stores, ask the provider to show the demographic break-up of your shoppers in terms of sex, age, income group and state.
2 Quality Control
2.1 Reading of each result Despite computer editing and auditing of results, there remains the opportunity for error where the wrong word is used in a questionnaire. As mentioned earlier, spell checkers will not detect the incorrect use of a word such as ‘their’ instead of ‘there’. Reading each result will also pick up inconsistencies in the questionnaire. At times the free form comments can contradict the shop score from yes/no questions. These inconsistencies can only be detected by the human eye. For example, the Mystery Shop result may show a low score but the shopper comments indicate they had a wonderful experience. These inconsistencies can sometimes be explained, but may also indicate a problem with the results. Another example is if the shopper indicated they waited 5 minutes to be served, and then in the comments makes a comment that they were impressed by the quick service. The most common form of verification is to simply read each result. Ask your provider to explain their verification process and whether there is any additional charge for the service. 2.2 Proportion of results entered on-line Ask how many shopper results are entered on-line as opposed to being called in over the phone, mailed, or faxed. Unless the shopper enters results directly there is a great opportunity for transcription error, regardless of the method. Phone shops are notorious for transcription especially where shoppers and/or transcribers take small shortcuts to speed the process, e.g. The shopper says something like “The first five are Yes, followed by a No, two Yes’s and a No”. Mailed questionnaires are problematic because in addition to the transcription issue the results can come late. This is not only important for the service standards, if there are questionable results, and the shopper needs to be contacted, there is less likelihood they can remember the intricate details of the shop. Faxed shops are subject to the additional transcription error where unclear transmissions may result in ‘guesses’. Faxes are also notorious for being lost. 2.3 Receipts verification Where possible the provider should retain a receipt of the shop to verify the shop took place. Receipt verification will also allow the provider to check the shop took place during the times (restrictions) required by the client and that the shopper went to the correct location.
Steven Di Pietro
Service Integrity Mystery Shopping ABN 53 107 407 607
Page 7 of 10
Evaluating a Mystery Shopping Provider – White Paper Typically, although the shopper will enter the shop time and date on the Shopping system, the receipt will verify whether this was in fact the time of the shop. As a client, if you do NOT want evening shops, this process will ensure the shopper followed the instructions. 2.4 IT in-house or outsourced Ask whether the IT resources are internal or outsourced. Ask questions about the IT structure to give you a feel for the robustness of systems. Outsourcing IT may at times provide more robustness but may also restrict flexibility to provide the service you require.
3 Operations
3.1 Ability to introduce off-cycle shops Ask whether the system can cater for off-cycle shops. Although your program may be monthly, occasionally you may want to add a wild card shop (off-cycle). You may decide to conduct additional shops for whatever reason but once in a program find there is an additional cost of implementation or that the off-cycle shops are not possible. 3.2 Shopper briefing process Understand the process of briefing the shoppers. Are they simply ‘dumped’ with a questionnaire or are they pre-briefed on the requirements and expectations? 3.3 Result timeliness Ask how quickly the shopper is required to submit their results. A critical aspect of ensuring validity of results is to have results input and returned quickly. Typically results should be input within 24 hours of the shop taking place. 3.4 Live updates on shop status and reassigning Ask how the provider monitors how the shops are being assigned, and taken up by the shoppers. You can be confident the shops will be completed on time if operationally there is a live update of each shops status. Typically the shop status includes classifications such as • • • • • • • • • Assigned, Confirmed, Audit, Pending (if information is missing) Approved Incomplete Rejected Invalid (go to wrong site or do not follow guidelines) Un-shoppable (if the site was closed)
Steven Di Pietro
Service Integrity Mystery Shopping ABN 53 107 407 607
Page 8 of 10
Evaluating a Mystery Shopping Provider – White Paper
4 Reporting
4.1 Web Best practice is to now provide web reporting. Web reporting will also give you, as the client the ability to store results and disseminate to a greater audience. 4.2 Speed of updates As the client, you should have the choice of asking for website updates as quickly or slowly as you require. In some cases, you may wish to have results updates every hour or two, whereas in other cases you may seek to have results updates only once a month. At times you may seek a staged release of results where the client principal has an opportunity to review results before they are released to the broader company population. Sometimes this additional check can pick up some inconsistencies that allow both client and provider to take corrective action if possible. 4.3 Receipts on-line Typically the individual receipts should be stored on-line but are not necessarily made available to the store on-line. If there is a query, you will want quick access to the receipt and be confident it can be found, as opposed to being lost in a filing cabinet. Providing receipts on-line can provide the opportunity for the store to undertake investigations to identify the shopper, thus compromising future shops. 4.4 Multiple access levels In larger organisations it is important to have different levels of access where each level can only see their own area of responsibility. You may not want individual stores or State Managers to see each other’s results, but still see their rankings.
5 Pricing
At times during a pitch, the costs are simply stated as a cost per shop. The following costs may not be made clear until the decision made to appoint the successful company. 5.1 Changes to questionnaires Changes to questionnaires are not abnormal, especially early in the program. Despite the best intentions by both parties to refine the questionnaires, there is no substitute for seeing real data flow from the program. Understand the pricing of minor changes. 5.2 5.3 Additional auditing Length of contract Understand if there are additional costs for manual reading of the results. Short term, even monthly contracts are more and more common and a reasonable request of your provider. 5.4 Travel reimbursements At times, especially with contractor shoppers, the provider may introduce mileage reimbursements for shoppers. If these are necessary, ensure there are minimum and maximum distances.
Page 9 of 10
Steven Di Pietro
Service Integrity Mystery Shopping ABN 53 107 407 607
Evaluating a Mystery Shopping Provider – White Paper It would be unreasonable to have travel payments for short distances 5 minutes from your site, and similarly unreasonable to pay mileage for over 1 hour travel times.
6 Other
6.1 International experience/peers Being part of affiliations may give the impression of being an industry leader, however more pertinent may be the ability for the provider to coopt with other worldwide offices where fluid communications and learning can provide immense leverage to the client. 6.2 Transparency of data As a client it is reasonable to request not only the raw data from completed shops, but also shopper statistics such as demographics. 6.3 Staff queries – response mechanism Regardless of the quality of questionnaires and provider, there will be questions asked from the recipients of questionnaires. Ask you provider about the mechanisms for feeding store questions. Current best practice is to allow on-line queries directly linked to the questionnaire results. This allows the feedback and responses to be perpetually linked to the store result. Steven Di Pietro Managing Director Service Integrity Mystery Shopping mailto:[email protected]
Steven Di Pietro
Service Integrity Mystery Shopping ABN 53 107 407 607
Page 10 of 10
