Joseph IngemiExecutive Consultant| Price SystemsMount Laurel, Nj, United States
When we hear about IT Auditing, we think of SOX, HIPAA, etc. I am interested in Auditing the projects themselves, specifically, auditing Project Risk Controls. Can anyone talk to me about methodologies? Saving Changes...
I've worked at several companies where I helped implement project reviews. We did things like project scorecards. I've seen some very basic (and in my opion not very helpful) reviews where the audit team essentially just went down the checklist of documents that the team was supposed to create and checked off if the document existed or not. This was done without reviewing the document for content or validity. I've also been places where the project audits were very useful and in depth. Not only did they use checklists but they actually reviewed documents and processes for validity, usefulness and to see if the team understood why they were using a specific process or document (and of course to gather lessons learned on how to revise and make better the document or process). In addition, some companies I've been at have actually tied the scorecard to bonuses of division heads and project managers - nothing like a little incentive :) Saving Changes...
George JucanManaging Partner| Organizational Perfomance Enablers NetworkWoodbridge, Ontario, Canada
I am not aware of a methodology specialized for auditing projects, I map them to PMBOK framework or the organization's project management methodology and identify gaps and deviations. You may want to take a look at PMI's OPM3, there are many interesting things about assessing projects in there, including what to look for, process, tools, checklists etc. Saving Changes...
I have seen project controls audited as part of a SOX and internal process audit. The auditors came in, asked to see policies and procedures, and then checked project documentation for evidence of compliance with those policies and procedures.
I have not seen auditors use a "methodology" for their audit the way you have asked for one. Their "methodology" is their method for performing the audit (interview-based, evidence-based, self-audit, etc.). If you have a risk-management methodology, they would check your adherence to your own method.
There are some auditing standards, like the SAS 70 for IT security. These standards provide different tests that the auditors look for during their audit. They are not a methdology, though, because they do not tell you how to pass the tests in the standard, just the standards that the auditors should use.
I do not know of any auditing standards focused specifically on risk management. If you want some model issues to look for, talk to an auditor with experience in the insurance industry, or an auditor who has experience auditing a corporate risk management group. They may have tests to exercise that could be adapted to project-level risk management.
George is also 100% correct that OPM3 can be useful. An auditor could compare your current practices to a target maturity level within OPM3, and give you a report of whether your organization conforms to that level of activity. OPM3 has sections listing expected behaviors and evidence, which would be really valuable for any auditor. Saving Changes...
Joseph IngemiExecutive Consultant| Price SystemsMount Laurel, Nj, United States
I think that reviewing a project accordding to a score card is the direction I am leaning. I am interested in auditing a project in terms of items such as resource utilization, cost estimation, etc. Saving Changes...
Mark Price PerryBusiness Driven PMO Evangelist| BOT InternationalOrlando, Fl, United States
Joseph, great post and I think the replies by all are spot on. My only only comment to add with respect to "Project Auditing" and it is really a question is, "Is anyone out there doing post project audits well after the product of the project has been released to determine if the actual business case benefits as originally attested to by the sponsor have been achieved, or missed, or exceeded." And as a follow up, if anyone is doing such as post project audit, is this a one-off exercise or a planned/periodic management activity? Any thoughts or advice..?
Saving Changes...
Like the rest, I know of no audit methodologies apart from proprietary ones. When I worked in the medical device industry, project audits (called "quality audits") were as regulated as the project work. This mapped to federal regulations and were necessary to maintain mandated quality standards. Outside of such highly regulated environments I've seen the gamut described below. Though I agree that scorecards are rarely useful in and of themselves, they are better than absolutely nothing (which I've also seen). Sometimes it's a matter of maturity: start with an easy process like a scorecard and then mature to a more analytic audit approach.
In response to Mark Perry's question, a portfolio process I've helped to organize and rollout takes post-implementation issues into consideration. It hasn't had a chance to run through the full cycle, but it did have absolute management support. We'll see how well it measures outcomes and business requirement expectations. We've dubbed these phases "maintenance" and "post maintenance" and the process schedules regular reviews of all portfolio projects in these late phases. Saving Changes...
I think we need to distinguish project versus post-project. The latter is not the responsibility of the project manager but for line-management as compared to the first one which is a responsibility of projectmenager and steering committee.
Having said that the risk audit needs to be planned and prepared. There is an initial risk assessment from the planning phase available and during the project new risk might become visible. Both need to be part of the scope of the audit.
From my perspective you would need to do the folllowing (perhaps the start of a methodology):
1) assess the known risks from register by interviewing project stakeholders and participants
2) Assess if new risks ave become valid to be part of the risk register
3) Check actions against these risks especially if risks have become active
4) check against previous risk audits
5) report to the person that has assigned you the risk audit
From having worked closely with internal auditors, I would respectfully disagree about the responsibility of the project manager in audits. I recommend getting involved in BOTH the project audits and the post-project audits.
You make a great explanation of the risk-related audits, and I agree that the PM should be involved in these.
The PM should also be involved in process-related audits, and outcome-related audits. Any good audit system should be built in the principles of
1. Good policies and procedures
2. Clear outcomes and expectations
3. Outside verification (external audit)
4. Internal benchmarking and verification (internal audit)
The PM should be the one setting policies and procedures for the project, setting expected outcomes, and doing an internal audit to verify that the PM him or herself is following the procedures and delivering the outcomes expected.
In highly regulated areas like brokerage or insurance, each operational department goes through a process of self-audit before the internal audit department and before the external auditors arrive. The internal audits help to make sure that the department is running cleanly according to internal standards, and gives the opportunity for improvement with each successive audit.
The external audits then provide verification of the results of the internal audits, plus additional ideas for improvement.
My recommendation to all project managers is to embrace every phase of the audit. Audits are almost never pleasant, but if you tackle the issue aggressively, starting with a thorough internal audit, you will survive the external audits with minimal problems. You may even find the audits are helpful, because they give you new ideas to improve. Saving Changes...
Dear Alex,
Thanks for your view on the project audits.
I think we agree on most of the matter. The difference is that most benefits from projects occur post project. My opinion is that this period - post-project- is not the responsibility oft he Project Manager.
This also because in my work environment project managers are not seldom external people to the company and they might have left the organization.
I do like you comment on involvement. The Project Manager should be able to faciltate independent project audits done under the responsibility of the steering committee and should discuss with his internal client the execution of post-project audits.
The difference in opinion might be the process related stuff. I think the word responsibility might clarify the difference. During the project - project related process are in scope - , post-project I doubt. Therefore a clear handover process and acceptance by the internal client should also take care of handover of responsibility.
A project end-report should include a sheet of activities that have not been finished during the project. I guess a good project manager also has set up the way to measure project outcomes (i.e. benefits) that occur post-project. The way to measure is part of accepting the business case or measurement process.
As for embracing audit we are in full agreement. Saving Changes...
Rob MartinConsulting (Contract)| Microsoft (Thailand)Lam Luk Ka, Pathum Thani, Thailand
Actually several large companies have excellent review processes for reviewing their projects. The one I am most familiar with is similar to a risk assessment where the auditor makes a list of questions around the aspects of Project Manafgement they want to score. Then plotting this on a spider chart and seeing how it scores graphically. It also enables the viewer to see exactly what areas of the project need work. Many times the audit will show you're over coooking part of the project and neglecting another. Use it as an ongoing tool to take a "bearing" of where you are in the greater scheme of your projects lifecycle.
The eight points on the spide might be:-
1. Team (Leadership, PM Skills, Buy in)
2. Scope and expectations
3. Identification, documentation and tracking of requirements
4. Plans and planning processes (Effort estimation)
5. Cultural and ethical alignment
6. Alignment between the project team and the business
7. PM Methods
8. Communication, including progress tracking and reporting
Saving Changes...
