Summary of Performance Measures Verification Concluded During FY2007-08MEMORANDUM
City of Miami Beach, 1700 Convention Center Drive, Miami Beach, Florida 33139, www.rniarnibeachfl.gov
OFFICE OF BUDGET AND PERFORMANCE IMPROVEMENT, Internal Audit
Tel: 305-673-7020, Fax: 305-673-7519
TO:
VIA:
FROM:
DATE:
SUBJECT:
Jorge M. Gonzalez, City Manager
Kathie G. Brooks, Budget and Performance Improvement Directafb
James J. sutter, Internal Auditor
February 23, 2009
SUMMARY OF PERFORMANCE MEASURES VERIFICATION CONCLUDED DURING
FY2007/08
During Fiscal Year 2007/08, Internal Audit concluded six performance measure verifications for
performance data reported by the City's Building Department and the Capital Improvements Projects
(CIP) Department. The objective of our verifications was to confirm the integrity, reliability,
completeness, and accuracy of processes and documentation related to the calculations and reporting
of performance data on the City's performance management software. Results from our verifications
were reported on two separate memorandums, one for each department, and have been summarized
later on this document.
Once the verification is complete, an assessment category is assigned to the performance measure
considering the following criteria for each corresponding category:
I. •.••·· > A.s~~s$1'rient cate,gqry\C ·· · I >? . ' .. •
.······
.· .. Gf'iteria .··. •·. .·.·. , .. ·.· . ;
If reported performance is 100% accurate and if it appears that
Certified controls are in place to ensure accuracy for collecting and reporting
performance data. Measurement data is supported by source
documents.
This category is assigned under either one of two conditions:
1. Reported performance is within +/-3%, but the controls over
data collection and reporting are not adequate to ensure
Certified with Qualifications continued accuracy.
2. The department's calculation of actual performance deviates
from the measure definition, but was still within an acceptable
range.
Documentation is unavailable or incomplete and controls are not
Factors Prevented Certification
adequate to ensure accuracy. This category is also assigned when
there is a deviation from the measure definition and the reviewer
cannot determine the correct measure result.
Inaccurate Actual performance is not within 3% of reported performance, or
there is a greater than 5% error in the sample of documents tested.
In addition, we provided the departments with recommendations in how to correct any shortcomings
observed and ensure that employees generating, collecting, analyzing and reporting the performance
Page 1 of 4
Internal Audit Memorandum
Summary of Performance Measures Verifications Performed FY2007/08 February 23, 2009
data understand the purpose and importance of the data, its objectives, and the relationship with the
City's Strategic Plan.
During Fiscal Year 2008 the following Key Performance Indicators (KPI) and Department Performance
Indicators were verified and assigned the accompanying Assessment Category:
Factors Prevented Certification
In general, the factors preventing the certification of the measure reported by the Building
Department included the following:
a) No supporting documentation was (;lvailable in order to reproduce and/or substantiate data
reported.
b) There was no measure definition established.
c) There was deviation from intended measure definition and the correct result of the measure
could not be determined.
d) No review process was currently in place to ensure accuracy and reliability of data reported.
Department's response:
In response to the memorandum dated January 6, 2009, the Building Department has reviewed the
analysis and report prepared by Internal Audit addressing the 'Percent of Buildings 40 years or older
complying with recertification.' The Department agreed with the assessment and recommendations
as provided and subsequently implemented the following tactics to address the factors preventing
certification of the measure:
The measure was not clearly defined (b) and therefore the result could not be clearly determined (c).
The performance measure tied to the KIO, "preserve historic building stock,' is now defined as:
Total Number Buildings complying with recertification requirements
Total number of buildings requiring recertification
The objective of the measure is to capture the City's effectiveness in properly notifying and enforcing
the recertification process in order to protect the City's historic buildings. The Department will
continue its efforts to ensure that all building owners are notified on a timely basis that their building is
due for recertification and that violations are issued within 30 days of the compliance date.
The lack of supporting documentation (a) made it unfeasible to review the process (d). The IT and
Engineering sections of the Building Department successfully designed a method and corresponding
report to accurately capture the required data. The data is saved on a secured server and accessible
to run future reports that will generate reliable, consistent and complete analysis supported by
documentation. In addition, quality control is being monitored at the level of data collection and in the
administration of the process. The Chief Engineer supervises the collection and submission of data
to ensure accuracy in the field. The Senior Building Manager assesses the validity and reliability of
the data and corresponding report before the results are entered into Active Strategy.
The Department strives to ensure a thorough and meaningful understanding of performance
evaluation methods to all staff.
Page 2 of 4
Internal Audit Memorandum
Summary of Performance Measures Verifications Performed FY2007 /08 February 23, 2009
Ca_Eitallmprovement Projects (CIP)
I< > ;····. < .. , •· Na.mc{()f.'Measuie \ >. · :> ·.. .<., ... · ·. . :.l\.sses$rnentca.tegory .··• · •\
% of change orders generated by contractor Factors Prevented Certification
% of change orders generated by unforeseen conditions Factors Prevented Certification
% of change orders generated by designer Factors Prevented Certification
% of change orders generated by owner Factors Prevented Certification
%of projects with substantially completed construction and in Factors Prevented Certification
beneficial use within 120 days of schedule.
Meanwhile, factors preventing the certification of the measures reported by the Capital Improvement
Projects department were a combination of the following:
a) No measure definition/description currently exists on the performance management software.
b) No supporting documentation was provided in order to verify and reproduce the data reported on
performance management software.
c) No review processes were currently in place to ensure the accuracy and reliability of
performance data reported.
d) Data was not timely reported on the City's performance management software.
e) No data source, calculation methodology, or limitations were referenced on the performance
management software.
Department's Summarized Response:
1. % of change orders generated by contractor, unforeseen conditions, designer and/or owner
Although CIP believes that the measures' names were self-explanatory and were approved by the
City Administration, they have since been changed in Performance management software to add "as
Percent of Contracted Amount" to each measure's name.
CIP concurs that there were some input errors in the Active Construction Project Change Orders
Report, which were primarily due to data that had not been updated by the Assistant Director in his
spreadsheet, and mainly due to some change order credits that had not been reflected. Another
contributing factor was the inclusion in our tracking estimated amounts for change orders that were in
the process of being negotiated, for which we didn't have final documentation. These estimated
amounts tended to be on the higher side than the eventual negotiated change order amount and
actually hurt the department in terms of the performance measure. Consequently, we informed the
auditors' going forward that we would only track finalized changed orders that had been negotiated to
a precise cost. Currently, the Project Manager categorizes the change order and CIP's Financial
staff enters it into the recently implemented database that categorizes change order tracking by the
four categories (contractor, unforeseen, designer, owner). The process is working well and the
database is current.
With over 100 active projects, it's not realistic to expect that CIP management can verify all
categorization figures for all change orders. Management does validate change order scopes and
total values as they are negotiated items, and project on-going expenses are closely monitored. The
implementation of the new tracking system will make it easy to report the Active Strategy measures
and should address data accessibility for validation purposes, which appeared to be the overriding
issue of the audit. CIP's grander objective is to manage overall project construction costs so that
they don't exceed the project budget, which typically includes a 10% contingency to cover change
orders. In this regard, the sum of the target measure change order percentages was set at less than
10%, and the individual change order target percentages were arbitrarily set, with the anticipation that
with the review of historical change orders we would be better able to set better targets by category-
Page 3 of 4
Internal Audit Memorandum
Summary of Performance Measures Verifications Performed FY2007/08 February 23, 2009
still within the overall total of 10%. To this end, we have achieved significant reductions in overall CO
% in terms of construction contracted amounts, since we implemented the Best Value selection
criteria for construction contracts.
2. % of projects with substantially completed construction and in beneficial use within 120 days of
schedule
Despite believing that the measure's title is self-explanatory, CIP agrees to create a description of the
performance measure to be incorporated into City's performance management software.
CIP uses a spreadsheet that tracks the on-schedule or late status of all projects against contracted
schedule and each project is tracked with respect to planning, design and construction by quarter.
This spreadsheet is then used to tabulate the total projects completed each quarter. The completed
dates of those projects completed contractually late are then compared to the date that is 120 days
beyond the contracted completion date. In the case, of this measure for 01-08, two of ten projects
were completed contractually late, but only one of ten projects was completed beyond the 120-day
target. The one in ten equates to the 90% figure that was reported in the City's performance
management software.
Additional details and recommendations were included on the individual memorandums issued to the
corresponding departments. In addition, an exit meeting was conducted with each department covering
the results of our verifications.
Page 4 of 4