Loading...
OIG No. 23-14: Trolley Survey Methodology ReviewJoseph M. Centorino, Inspector General TO: FROM: DATE: PROJECT: Honorable Mayor and Members of the City Commission Joseph M. Centorino, Inspector General July 14, 2023 OIG No. 23-14 Trolley Survey Methodology Review On April 27, 2023, the Office of the Inspector General (OIG) received a request from the City of Miami Beach Transportation and Mobility Department (TMD) Director to review the draft scope of services for the Miami Beach Trolley Passenger Survey that was prepared by the ETC Institute, a company specializing in governmental surveys. The request for review was in accordance with a request from the Finance and Economic Resiliency Committee (FERG) that the OIG review and evaluate the survey's methodology and provide input to the TMD prior to its being conducted. During a Finance and Economic Resiliency Committee (FERG) meeting on July 22, 2022, the TMD presented a Pre-Pandemic Trolley Survey conducted in 2019 by Marlin Engineering Inc. The survey was part of a proposed trolley enhancement project worth $2.3 million. The TMD explained that the 2019 survey aimed to gain a comprehensive understanding of the trolley service, its users, and areas for improvement. The proposed enhancement for fiscal year 2023 aimed to reduce passenger wait times from 30 to at least 20 minutes. The FERG allocated up to $100,000, in addition to the proposed $2.3 million enhancement, for a new survey to ensure better-informed decision-making. During the Committee's discussion, Vice-Chair Ricky Arriola expressed concerns about the ridership numbers in the 2019 survey, which led to an OIG inspection of the trolley system's Automatic Passenger Counters (APCs) (see OIG No. 23-11) as well as this OIG review of the methodology used for the new survey to be conducted by a different private contractor to be engaged by TMD to gather input from trolley riders. During conversations with TMD staff while the new survey was being developed, the OIG was informed that the primary purpose of the survey, projected to be conducted by the ETC Institute, was to evaluate customer service satisfaction and the percentage of residents among those using trolley services. TMD also wanted the new survey in a format that would allow for a valid comparison with the 2019 passenger survey. After a draft of the survey was provided to the OIG, the OIG submitted its recommendations, intended to optimize the survey's value, to TMD. The proposed survey and OIG's Page 1 of 2 recommendations to TMD are attached to this report, as well as the responses of TMD to those recommendations, which indicated a careful consideration of the issues raised and an intent to implement a number of the recommendations. The OIG appreciates the TM D's consideration of its input and the steps taken to implement some of the specific recommendations made by the OIG. uty Chief Auditor er> Date /be» Date cc: Alina T.Hudak, City Manager Rickelle Williams Assistant City Manager Jose R. Gonzalez, Director, Transportation and Mobility Department OFFICE OF THE INSPECTOR GENERAL, City of Miami Beach 1130 Washington Avenue, 6 Floor, Miami Beach, FL 33139 Tel: 305.673.7020 • Hotline: 786.897.1111 Email: CityofMiamiBeachOIG@miamibeachfl.gov Website: www.mbinspectorgeneral .com Page 2 of 2 Attachment A: Passenger Survey Scope of Services City of Miami Beach Trolley Assessment Onboard Rider Survey Scope of Work Prepared by ETC Institute (4 Routes, 600 Samples) Task I: Project Start-up and Survey Design Upon the execution of a contract, Aaron Hekele (senior manager from ETC Institute) will meet with the project team and HNTB. Aaron Hekele will serve as the project manager. Topics that will be addressed during this meeting will include: • Questionnaire design data elements, review of past surveys, etc. The survey questionnaire will be developed in a manner that allows for a valid comparison between the current survey results and the 2019 passenger survey performed by another firm for Miami Beach. ETC Institute understands that the City of Miami Beach and HNTB will be an active partner in the design of the survey and the project as a whole. HNTB will provide ridership data for each route prior to surveys beginning in order for ETC to develop an appropriately weighted schedule. Required Data Elements and Survey Length Trip Related 1. Which route 2. Are you a resident or visitor 3. Frequency of riding (e.g., how many days per week 4. How did you get to stop 5. Where did you get on? Intersection sufficient 6. Where did you get of? Intersection sufficient 7. How do you get to destination after getting off trolley 8. Trip purpose 9. Satisfaction with service 10. Satisfaction with reliability l l. Satisfaction with frequency l2. Awareness of trolley app 13. Use of trolley app 14. Satisfaction with trolley app 15. Prior means of making trip 16. Primary mode of transportation 17. Feedback- open ended Demographics I. Age 2. Primary language 3. Employment status 4. Household income 5. Number of people in household 6. Number of vehicles in household The survey will have two scenarios, 1) current trolley users, and 2) individuals who are not trolley users. The non-users may be encountered in the vicinity of trolley stops, or may take the survey on line. The two scenarios will be similar to the 2019 survey to allow for valid comparison. ETC Institute will have bi-lingual (English and Spanish) interviewers available on routes to ensure non-English speaking riders are well-represented. The survey form will be available in Spanish, English and Haitian Creole. Rider Outreach. Although ETC Institute will provide guidance for the communications efforts and examples of outreach materials, we propose letting the City of Miami Beach do the bulk of the outreach effort, which could include but would not be limited to: agency website announcements, social media, e-mail blasts, car cards, station posters, etc. Task 1 Deliverables: Final Survey Instrument and Examples of Outreach Materials Task 2: Sample Design At a minimum, ETC Institute will develop a sampling plan that has specific goals for each route by direction and time of day (this will ensure peak vs non-peak analysis), and weekday vs weekend (if desirable). The time periods will be defined by HNTB per the City's model time periods. This ensures adequate representation of each route by direction and time of day. ETC Institute will develop a sampling plan to ensure 600 completed surveys and a maximum margin of error of +/-4% at the 95% level of confidence interval. Task 2 Deliverables: Final Sampling Plan for the 600 Completed Surveys Task 3: Survey Administration (600 Completed Surveys) The survey will be administered by the Field Manager (FM), who will be directly supervised by the Project Manager (PM), who reports directly to the City of Miami Beach/HNTB. The key individuals who will oversee data collection include: Project Manager (Aaron Hekele), Field Manager (Kevin Arnold), Data Manager (Nick Jones), and QAQC Manager (Jason Jones). The same team has been in place for all recent onboard projects including the upcoming MDT Regional Onboard OD Survey. ETC will hire local surveyors and train each of them to administer the survey. Training will involve the following. 1) survey purpose and goals, 2) random selection, 3) survey options, 4) management of refusals, 5) conducting debriefs, 6) safety issues, 7) surveyor professionalism. ►Process for Selecting Riders for the Survey. Interviewers will select riders at random to participate in the survey based on the sampling goals that will have been established for each route. ETC Tnstitute's procedures will randomly select the persons (and location in the boarding area if this mode is used) to be surveyed based on the number of people who board a bus or the sequence in which passengers are positioned at a stop / station. Survey Administration Details. The survey will be available in the following modes 1) paper and 2) QR code (QR code must be provided by interview to ensure access control/randomness). 2 Standing Passengers: If a passenger is riding the bus standing up it may be difficult for them to complete the paper survey, especially if the bus is very crowded. In that case, the interviewer will offer the passenger the option of scanning the QR code printed on the back of the COVID safety placard each interviewer wears around her neck, which will link them to a survey they can complete online. Ordinarily, this option will not be offered to seated passengers, but if a seated passenger asks to use the QR code scan method, the interviewer will allow them to do so. Riders between the ages of 16-18: Riders 16-18 will be surveyed in the same manner as all other passengers, however, if they appear to be with a chaperone, the interviewer will first ask the adult if it would be ok for the minor child to participate. ETC recommends having a minimum age for the following reasons: 1) riders under a certain age do not have a clear understanding of certain data elements asked on the survey i.e., frequency, timeliness, connections/transfers, etc. and need the assistance of a chaperone to accurately complete the survey, and 2) riders under a certain age do not have literacy skills to complete the survey accurately without the assistance of the chaperone. Management of Refusals. If a rider refuses to complete the survey, interviewers will observe and record the refusal. This will be closely monitored to ensure nonresponse bias is minimized. Management ofLEP Routes (Language English Proficiency). ETC will have the survey available in multiple languages and print English on the backside on all translations. ETC will place bilingual interviewers on all high % LEP lines prior to starting collection on that specific run. Blind Passengers. Interviewers will read the survey questions to blind passengers. PPE (Personal Protective Equipment). Pens will be individually wrapped, and all interviewers will wear PPE. Task 3 Deliverables: Ongoing Completion Reports Detailing ETC's progress towards cell level goals. Task 4: Data Processing and Error Checking QA/QC Plan. As part of data processing, ETC Institute will execute their internal quality assurance/quality control (QA/QC) plan to guide all data collection and handling activities. Nick Jones, Data Manager for ETC, and Jason Jones, QAQC Manager for ETC, will be responsible for processing the data and conducting QAQC analysis. The QA/QC plan will address: 3 • How to ensure the collection of sufficient individual rider samples to assure that the statistical sampling goals in the sampling plan are met. • The definition for a "complete and usable" survey. ETC Institute's fees are based on the number of completed surveys that meet this standard. Surveys that do not meet the standard will not count. • Data processing and error checks. Debriefs w/ Riders Significantly Increase Survey Usability After the Administration of the Survey. As the surveys are administered, the survey data will be reviewed by the Field Manager. If any of the required information is missing or incomplete, the survey will not be counted toward the goal. ETC will collect surveys until enough fully usable surveys are captured to meet the goals and objectives of the survey. Data Management and Quality Control. ETC Institute has an ongoing quality control and quality assurance program in place. This program has been developed and refined through our experience with hundreds of studies that involved the design and administration of surveys. Our quality assurance program is directly monitored by the company CEO, Christopher Tatham. The program is designed to give clients error-free results, and all employees at ETC Institute are directly involved in the program. Core Elements of ETC Institute's Quality Assurance Process: Comprehensive survey design and review process. All survey instruments will be reviewed by all senior members of ETC Institute's team to ensure that all issues are adequately addressed. • Internal Pre-test. A pre-test will be conducted prior to the administration of all surveys. This will ensure that the survey instruments are understood as designed. • Data entry fields will be limited to specific ranges to minimize the probability of error. The data processing system that will be used by our firm for the study alerts data entry personnel with an audible alarm if entries do not conform to these specifications. • ETC Institute will select at least 10% of the records at random for verification. A supervisor will match records in the databases against the corresponding survey to ensure that the data entry is accurate and complete. • Double data entry will be completed for all surveys. The data from all surveys will be entered into two independent databases by different people. The two databases will then be merged. The 4 process will identify all records that do not match. All discrepancies will be corrected. The double data entry method ensures that survey data is 99.99% accurate. Sampling Methodology. Demographic questions will be included on each of the survey instruments. The demographic data will be used to monitor the distribution of the respondents to ensure that the responding population for each survey is representative of the universe for each sample. Task 4 Deliverables: Final Cleaned and Formatted Database delivered in Excel or CSV format. Task 5: Online survey. In addition to the onboard survey described in tasks 1 through 4 an online survey will be executed using the City's website. HNTB will use the survey questionnaire developed by ETC and work with the City to provide an opportunity for users to provide input on line. HNTB will use Survey Monkey or similar survey software. City staff will post the survey and related materials on the website. HNTB will compile the online survey results and summarize for comparison with onboard survey results and for use in the Trolley Assessment. Task 5 Deliverable: Survey questionnaire suitable for posting on the City's website. Task 5: Reporting. ETC will summarize findings and provide a final report that includes the following sections; 1) Summary Methodology, 2) Overall Charts and Graphs, 3) Tabular Data Route Level Data Tables, and 4) Survey Questionnaire. ETC will make up to two full rounds of edits and revisions before finalizing the report. Task 6 Deliverables: • One-page summary Describing Survey Methodology (by ETC). • A survey final report with results evaluation, graphics and comparisons to the results of the 2019 trolley passenger survey. Comparisons will also made between the onboard and online survey results. • A summary of the survey findings in "brochure style" format. • APowerPoint presentation of the_survey findings. Timing/Schedule • Survey Design and Sampling Plan- 2 weeks after Notice to Proceed • Staffing Plan, Recruitment, Training 5 business days after approval of Survey Design • Survey Administration- 2 weeks after completion of Staff Training • QAQC/Data Processing- one week after completion of Survey Administration • Data Delivery one week after completion of QA QC/Data Processing • Documentation 4 weeks after Data Delivery (by HNTB) Pricing ETC will perform the tasks in this Statement of Work for a total of$45,290.00. 5 Attachment B: OIG Comments on Trolley Survey 1. The objective of the survey is not clear, and the trip related questions should be aligned with the stated objective? For example, if the objective is related to customer service then there should be such related questions as was the trolley driver courteous, helpful, and/or professional, etc. In addition, we believe there should be direct questions related to the cleanliness and the maintenance of the trolley and not such broad questions as "satisfaction with service", etc. To the OIG, there are also too many questions related to the trolley app that should be replaced with other questions. 2. The 2019 survey contained a sample size of 1,240 and the 2023 survey sample size is 600. As your sample size increases, your margin of error (listed as+ or - 4%) will decrease and your confidence interval (95%) will increase as your sample size increases. As such, the OIG recommends increasing the sample size. 3. We believe trip related question #2 should be "Are you a resident, visitor, or work in CMB? 4. Is the survey anonymous or do applicants record their names? If anonymous, how does HTNB avoid duplicate entries by the same individuals which could skew results? 5. Non-riders should be prompted to answer a different survey to determine why residents or CMB workers don't use the trolley to gain more information as to the corresponding reasons and to help eliminate road congestion. 6. HTNB should have interviewers on the trolleys who speak Creole to disseminate the surveys, not just English and Spanish speakers. 7. Concerns exist that the sample may not be representative of the population. The sample should be performed at different times of the year, different days and at different times, different routes, etc. For example, summer testing may result in more locals and less tourists respondents and so on. 8. Is the 600 completed surveys figure all-inclusive, does it include both online and in-person? 9. Who pays for Survey Monkey, and is it included in the $45,290 price? 10. The OIG believes there should be an incentive for people to complete the survey to help increase the response rate. 11. The City should retain the right to select the size of the survey team and approve which HTNB employees participate based on their qualifications, experiences, etc. as determined through submitted resumes. 12. Why don't we require the usage of tablets to process responses rather than use paper questionnaires? If all responses are received electronically rather than manually, we can eliminate the need to have dual entries by HTNB staff and lessen the need to have supervisors inspect at least 10% of the records at random for verification as addressed in the "Core Elements of ETC Institutes' Quality Assurance Process" thereby most likely reducing the $45,290 price. 13. The City Manager and City Commissioners stated on the dais that this survey should be performed after trolley service levels were enhanced. Has this happened or were different instructions received? 14. We noted the survey did not include any questions related to people with special needs, or the elderly, or those with disabilities to determine how their trolley experience could be improved. For example, the existence of low-floor buses, electric lifts, etc. 15. Surveys can be administrated by emails to residents, business owners, etc. or placed on the City website to increase the sample size at a nominal cost. 16. Surveys should include a question asking what other areas should be covered by the trolleys, this relates to possibly route changes and not just to adding new trolleys. 17. Is the City performing outreach related to the survey? If so, what does it include? Attachment C: Transportation Response to OIG Comments on Trolley Survey 1. The objective of the survey is not clear, and the trip related questions should be aligned with the stated objective? For example, if the objective is related to customer service then there should be such related questions as was the trolley driver courteous, helpful, and/or professional, etc. In addition, we believe there should be direct questions related to the cleanliness and the maintenance of the trolley and not such broad questions as "satisfaction with service", etc. To the OIG, there are also too many questions related to the trolley app that should be replaced with other questions. The objective of the survey is to "assess the current trolley and microtransit service to determine opportunities for improvement." We'll ensure this is included as part of the scope. The City will work with HNTB and its subconsultant, ETC, to revise the survey questions related to customer service and cleanliness. 2. The 2019 survey contained a sample size of 1,240 and the 2023 survey sample size is 600. As your sample size increases, your margin of error (listed as+ or - 4%) will decrease and your confidence interval (95%) will increase as your sample size increases. As such, the OIG recommends increasing the sample size. The 2023 sample size of 600 is statistically more accurate and based on the current 9,000 daily boardings with 95% confidence level and 4% margin of error. The 2019 survey sample size was 697 based on approximately 15,000 boardings per day with 95% confidence level and 7% confidence interval. Please note that the sample size of both surveys doesn't include responses from online survey. The actual number of completed surveys in 2019 was 733 onboard plus 507 online (1,240), and we will use the same approach for 2023 survey. 3. We believe trip related question #2 should be "Are you a resident, visitor, or work in CMB? We will revise the Q2 to include "work in CMB." 4. Is the survey anonymous or do applicants record their names? If anonymous, how does HTNB avoid duplicate entries by the same individuals which could skew results? Per ETC, names and contact info are purely optional but they will be required for the raffle drawing (which serves as an incentive to complete). The survey results will be completely anonymous. ETC randomly selects passengers as they board the vehicle, and we hand deliver each survey. Riders will not be able to complete multiple surveys. 5. Non-riders should be prompted to answer a different survey to determine why residents or CMB workers don't use the trolley to gain more information as to the corresponding reasons and to help eliminate road congestion. Yes, non-riders prompt a different set of questions. Please see the question flow from 2019 attached. 6. HTNB should have interviewers on the trolleys who speak Creole to disseminate the surveys, not just English and Spanish speakers. Survey will be available in English, Spanish, and Creole. 7. Concerns exist that the sample may not be representative of the population. The sample should be performed at different times of the year, different days and at different times, different routes, etc. For example, summer testing may result in more locals and less tourists respondents and so on. The City would like to complete the survey prior to issuing a new RFQ, which is expected to be competitively bid by the end of summer. The survey will be performed at different days, different times, and different routes. Going forward, we'll make sure that the next survey is conducted at different times of the year. 8. Is the 600 completed surveys figure all-inclusive, does it include both online and in- person? 600 sample size is only for in-person. Please see response to Q2 for more details. 9. Who pays for Survey Monkey, and is it included in the $45,290 price? There is no additional charge for Survey Monkey, cost is included in the $45,290 price. 10. The OIG believes there should be an incentive for people to complete the survey to help increase the response rate. ETC will achieve a very high response rates using the proposed methodology, however, ETC wants to ensure that surveys are properly/fully completed, which is the primary reason for offering a raffle drawing. Offering an incentive per completed survey is too costly. Offering an opportunity to win a drawing will achieve the goals and objectives of this study and is a more practical approach given the size and scope of the survey. 11. The City should retain the right to select the size of the survey team and approve which HTNB employees participate based on their qualifications, experiences, etc. as determined through submitted resumes. This will be included as an option on the Scope of Services. 12. Why don't we require the usage of tablets to process responses rather than use paper questionnaires? If all responses are received electronically rather than manually, we can eliminate the need to have dual entries by HTNB staff and lessen the need to have supervisors inspect at least 10% of the records at random for verification as addressed in the "Core Elements of ETC Institutes' Quality Assurance Process" thereby most likely reducing the $45,290 price. Per ETC, paper collection is much more cost effective. An average surveyor can produce 2 to 3 times more completed surveys compared to a surveyor using a tablet/interviewer method. Data processing fees may decrease with tablets, however, the cost to administer the survey may increase. ETC has the capability to administer the survey via tablet, though fees associated with data collection will increase. 13. The City Manager and City Commissioners stated on the dais that this survey should be performed after trolley service levels were enhanced. Has this happened or were different instructions received? The service was enhanced from 15 vehicles to 21 on October 1, 2022. However, before the pandemic, the City had 25 vehicles in operation. There is a proposed enhancement in the operational budget for FY24 to add an additional trolley to South Beach Loop A and South Beach Loop B. However, if approved by the City Commission it will go into effect after October 1, 2023. We would like to complete the survey prior to issuing a new RFQ, which we expected to do by the end of summer. 14. We noted the survey did not include any questions related to people with special needs, or the elderly, or those with disabilities to determine how their trolley experience could be improved. For example, the existence of low-floor buses, electric lifts, etc. We do have an open question where riders can provide feedback (018 in the attachment). 15. Surveys can be administrated by emails to residents, business owners, etc. or placed on the City website to increase the sample size at a nominal cost. In 2019 the City's Communication Department promoted the survey online service via social media and email blast. We are planning to do the same this year. 16. Surveys should include a question asking what other areas should be covered by the trolleys, this relates to possibly route changes and not just to adding new trolleys. We do have an open question where riders can provide feedback (018 in the attachment). 17. Is the City performing outreach related to the survey? If so, what does it include? Yes, please see the response to Q15.