Joseph IngemiExecutive Consultant| Price SystemsMount Laurel, Nj, United States
When we hear about IT Auditing, we think of SOX, HIPAA, etc. I am interested in Auditing the projects themselves, specifically, auditing Project Risk Controls. Can anyone talk to me about methodologies? Saving Changes...
Thanks so much for the reply. I believe we are in complete agreement, but we just have different types of organizations.
In mine, the project managers are almost always employees, and project management is an on-going, permanent discipline. The people setting the PM policies and procedures are employees and will be around to work with the auditors. If you have consultants running many projects, then the lines of ownership must be much clearer, and I agree with your suggestions.
It is also critical, as you say, to have a clear point where the responsibility for benefits and compliance shift over to non-project staff. At some point the project ends, and responsibility cannot stay with the project manager any longer. It sounds like you are using the post-project reviews as a way to achieve that hand-off. This is an excellent practice.
Thanks so much for the lively debate.
--Alex Saving Changes...
Rob MartinConsulting (Contract)| Microsoft (Thailand)Lam Luk Ka, Pathum Thani, Thailand
I have attached a very basic example of what I am talking about.
Using a questionaire tailored to your needs can drive out issues and you can work to an acceptable score in advance. Each phase of the project should have a different mean score. Overcooking can be as disasterous as undercooking.
To level set the exercise you get the Project Team to do the questionaire and mark themselves. The review team then comes in and asks questions based on the already completed scorecard and then updates it with their collective view. This way the Project Team can calibrate against a set of expectations that they may not have been aware of.
These should be prepared for in advance by the Project Team, not sprung on them. I believe that a well prepared review is more usefull than a spot inspection which serves nothing but to annoy and uindermine.
If you're not well prepared, don;t do it!
Rob Saving Changes...
Rob MartinConsulting (Contract)| Microsoft (Thailand)Lam Luk Ka, Pathum Thani, Thailand
There is no way to edit, so I will apologise in advance for my Typos.
Rob Saving Changes...
SuiteGerald SmithFounder & President| MJS & Associates LLCCincinnati, Oh, United States
But what would be a good passing score for a project audit? 70%, 80%, 90% .. and who determines what that score is, the PMO? Saving Changes...
Gabrielle MaherPMO Consultant| IndependentLondon, London, United Kingdom
Hi Joseph
Audit and risk management / control - mean different things at different levels of an organisation – and from my experience I have identified a few categories:
- Corporate Governance and Internal Control – A major factor in Corporate Risk management is increased focus on Corporate Governance – both in the UK and US – where the standards required of the Board of Directors are set out in SOX (US) and The Combined Code (UK – 1992 revised 2006). There are also specific mandatory regulatory requirements that have been introduced by help in Corporate Risk Management such as BASEL II (by 2004) in banking.
- The Function of Corporate Governance and Risk Management is around the ongoing activity and accountability of Directors to ensure effective management systems and controls are in place incl. Financial monitoring and control which includes Financial, operational, compliance and risk management. Internal and external auditors, compliance officers and enterprise risk managers are involved in regularly scrutinising internal management systems and controls as are Fraud auditors. In the world of IT – The IT governance Institute and ISACA are recognised as the leading authority in promoting IT Governance and standards for computer systems audits and controls. ISACA’s members include CIMA (Certified Information Management Audit), CISA (Certified Information Systems Auditor) and there is also a certified auditor for Fraud. (ISACA.org – links to ITGI).
- Programme / Project – If the IT programme you are managing affects any of the Corporate management systems and controls – you can be sure you will be subjected to a regular Internal Audit, and may also have to meet Audit standards from internal Compliance and Corporate Risk . You should be made aware of these requirements at the outset of starting up a large programme of work by these functions. Auditors have substantial authority and the Auditing report issued post audit requires a project or programme manager / and the project / programme Board to meet the requirements set out in the audit report – and if serious these will be time bound. The Board of Directors will be notified of an audit failure – and worst case it could result in the programme being shut down (I think its 3 strikes and you are out). Equally a large IT programme with a substantial budget can be audited at any time by internal audit, risk management and compliance. In the UK – for Government projects and programmes – the OGC (Office of Government commerce) have guidelines for mandatory programme assessments (0-5 Gated Review Process – available on their website). These are mostly used for large Government programmes involving 3rd party relationship / vendor management and procurement. But I find them a very useful guideline if conducting a project healthcheck – quality gate review.
- Quality Management – quality management used to mean ‘Testing’ in IT - but today – for programme and project portfolio management it represents the output and quality standards as defined in the programme or project quality plan. This is the plan that is audited by a dedicated Quality Manager (standards include ISO in the UK – or if it’s a government programme an External Assessor will conduct the audit – the OGC had dedicated assessors). Equally a PMO bod can act as a quality reviewer if sufficiently capable. The quality plan of any project or programme should be agreed by the Board from the outset of the programme. Quality from a PMO perspective is a more ‘supportive’ health check – the aim is to ensure the project or programme is healthy and ready to move to the next stage so that the PMO Assurance duty to the Project / Programme Board is met. Failure to meet the quality stds can result in the project or programme being suspended/delayed.
- Risk Management and controls - there are many techniques available in the market to help surface and define risks including dedicated systems, methodologies, frameworks and techniques. For Programme management – in the UK we tend to follow the OGC’s M_O_R (Management of Risk). PRAM is also used and PRINCE2 also sets out standards. There are many methods available on the market – which discuss risk strategies, risk techniques and controls. The Management of Risk principles understand the Barriers to overcoming Risk, such as the Organisational culture and lack of support structure. The M_O_R framework incorporates risk identification, assessment, contingency planning / risk strategy, and implementation of the plan. You should have a risk register, issue log and the risk and issue management process (defined by a PMO) should incorporate the escalation process. You should also have a risk contingency budget agreed at the outset of the project. All risks and issues should be reviewed by a project review board and more regularly by a PMO if you have one.
Saving Changes...
Hans RobbersSenior Director| SalesforceVlissingen, Netherlands
Gerald
In reply to your question what is a good passing score and who determines thsi level it would say this is audit dependent.
Each audit has an audit owner. In most cases this is the project sponsor however if regualr audits are a part of the project plan or the offer made by a supplier this can be respectively the project manager (delegating to the PMO manager) or a delivery/contract manager at the supplier organisatation.
Before the audit starts there will be an interview to discuss the objectives and the specific areas to be investigated.
The audit is conducted and from there a report will be delivered describing the scoep, material used, findings, conclusions and recommendations.
This report is discussed with the audit owner and the project manager and in the end a list of actions and a timeline to implement is determined. Based on the findings the audit owner will determine the number of recommendations to be implemented and thus the passing grade.
Your explanation is very clear. I see some audit criteria that are absolutely pass/fail. If any of these items are missing, the group completely fails the audit.
Other items are recommendations, and the audit report will congratulate the group on achieving these goals and suggest a few more that they could work on.
I have not typically seen a "score" or "passing grade" assigned. Sometimes I have seen requirements that there be "no more than x" items failed in a certain category, but not a score.
Every audit group will behave differently. It depends on the goal of the audit sponsor and the practices of the people performing the audit.
Each item may score Low/No (1 point), Medium /NotSure (2points) or High/Yes (3 points).
Items are clustered and result in a subtotal, all clusters adding up to a grand total. Grand total vs. maximum score results in a percentage.
It's a start.
Weak point : more items per cluster automatically increase its significance Saving Changes...
Working in Project Risk and Advisory and with many different Internal Audit shops I have been able to see a variety of attempts and methods to audit / evaluate projects. Seeing that we have come up with some evolved capabilities, methods and process for auditing / evaluating projects.
Based on seeing other shops work and performing services using the clients methods (at times against my better judgement) I find that it is often difficult to move traditional Internal Auditors away from project auditing bad habits.
Some of the example faults and common mistakes that Internal Auditors make which can hurt the projects are as follows:
1. They "Tick and Bop" compliance of documentation for project management methodology that is often out of date or not reflective of current best practices.
2. Identify that documentation is in place BUT do not assess content. At times the content is weak, incomplete or lacking proper depth and analysis.
3. They attempt to leverage Operational Business Audit methodologies to assess projects.
4. Often do not properly scope the Audit
5. Create issues and walk away without assessing the best course of remediation (As Internal Auditors they could in fact help with the remediation themselves)
6. Often look to the past rather than the future. Only identifying issues rather than Risks to the future of the project.
7. Lack PM knowledge of methodologies (i.e. Scrum, Agile, Waterfall)
As for audit methodologies we have developed a variety. Our most advanced leverages analytics and, in simplest terms, benchmarks a significant number of project components with other projects of similar complexity and risk to evaluate the control levels utilized by the project being assess at the current time. (I would go into more detail but it would be quite a large replay)
A simple audit / assessment: At a high level we assess a variety of factors under the following seven domains:
Governance Approach
Ownership
Delivery Management
Business Unit
Resource Management
Risk Management
Vendor Management
We would often leverage a subject matter expert in the project area in which we operate to leverage lessons learned from prior project as well as gain deeper understanding into what the Project's end result may be.
Finally the underlining mindset we should be in when auditing and evaluating is... Try to look at the present and planning going forward. If during the life of the project we simply look at mistakes made or what we should have done in the past it can cause problems for the project. Saving Changes...
"If you have an apple and I have an apple and we exchange apples then you and I will still each have one apple. But if you have an idea and I have an idea and we exchange these ideas, then each of us will have two ideas."
