On the Value of What We Do… and How to Measure It

Ortiz_ James226x170This past week we conducted a workshop with IPCE and our stakeholders to explore approaches and proposed quantitative indicators (metrics) to measure the value of the agency’s independent review process. The proposition is simple, that is, measure the impact independent reviews are having on the success of the agency’s Programs and projects. The implementation on the other hand is anything but straight forward. After much discussion it became clear that that one of the factors that complicates direct measurement is the fact independent review is not a separate process but is embedded and synergistic with the overall programmatic decision and approval approach implemented by the agency. A way to illustrate this point is by examining the decision process of a project going to a Key Decision Point (KDP). The project’s initial integrated baseline undergoes many adjustments on the way to the KDP as a better understanding develops of the project’s technical, programmatic, and overall risk posture during the approval cycle. The approval cycle incorporates inputs and analyses from many sources including the project, the program, the host center, the mission directorates, and the agency of which the Standing Review Board (SRB) is one part. These activities are iterative and involve several in-process decisions to make changes and adjustments affecting the project technical and programmatic posture leading to the approval at the KDP. So it is difficult to ascertain whether the resulting decision and subsequent results in terms of the project’s subsequent performance are singularly due to the SRB. Thus, the consensus at the workshop was that the emphasis needs to be placed on supporting good approval decisions at the governing boards recognizing that the independent review process is an integral and essential part of informing these decisions.

So as we go forward we will be generating metrics to assess our contribution to the agency’s technical and programmatic decision and approval process. Many factors influence our contributions (and thus our value) such as the precision of our analyses, the thoroughness of our risk assessments, the timeliness in providing our results, our effectiveness in communicating our assessments, and our due diligence ensuring that complete and accurate information is heard at the management forums (which is also a measure of our independence).

You will be participating in this key effort as it unfolds through the year. I want to encourage you to embrace it and to contribute to its development and implementation. This is another step in our quest for excellence.

As always, I welcome your comments.
James N. Ortiz, PhD
IPAO Director

On Emphasizing the Programmatics

Ortiz_ James226x170 Our office has recently completed the second drop of forensics analysis where we look at SRB reports over the last two years looking for systemic trends in strengths, issues, and concerns reported by our review teams. SRB reports are a reflection of the performance of our Program and projects. The most significant area of issues and concerns reported deals with cost and schedule, or as we call it, programmatics. The continued challenge for our office is how to better support our SRBs with analyses that provide clear insight to the Agency to inform setting realistic cost of schedule commitments and on the viability of our Programs and projects to meet these commitments as they continue down the life-cycle. Over the last two years we have taken very deliberate steps to develop the capability to perform true integrated programmatic analyses. By that I mean, to assess the combined effects of risk on cost and schedule and the interplay between them. This has required significant improvements in our schedule and cost programmatic analytical capabilities (people, methods, and tools), the cross training of our analysts, and a very arduous effort to prepare our SRB members, our Program and project personnel, and our management staff at the Center, Missions Directorate and Agency levels to receive and to act on the results. Working with CAD we have made great progress but we still have a ways to go. We have stepped into an arena where programmatics are informed by statistical analyses with these analyses expected to be initiated by the Programs and projects themselves to justify the approval of their baselines. Our role has been expanded to not only model the project’s programmatics but to validate their statistical analyses. As we go forward we need to continue learning how to explain, at all levels, the underling risk posture that drives the results. We need to become extremely proficient at “story telling”, by that I mean, communicating in the most simple terms possible, not solely the mathematical information but the “story” the statistical curves and confidence levels are telling us in terms of the challenges the project has moving forward and the resources and timing for those resources (phasing of cost and schedule) that are needed to mitigate those challenges. The story telling I refer to is very hard to do but I believe it is not until we conquer that skill that we will significantly increase our impact to the Agency. The “we” I refer to is not only the programmatic analysts in PAG but RMs in EAG and IPAO leadership, including myself. The analogy of the elevator speech comes to mind. So if at the time we are done with our review, we cannot succinctly explain to any of our leaders as we ride the elevator together at NASA Headquarters the “story” our review is telling about the project or Program; then we are not there yet.

As I close this last blog of the year, the following thoughts come to mind. 2010 has been a very challenging year for the Agency and IPAO. I am very proud of the way you all have continued to move forward meeting the ambitious review manifest of the agency while improving the Agency’s review process as we go.

I wish you all and your love ones a very happy and safe holiday season and I anticipate a great new year.

James N. Ortiz, PhD

On the Role of the Principal Review Manager…

Ortiz_ James226x170 This blog is the first in a series of suggested topics you identified in your response to my call for IPAO blog topics. I thank you for your input and I will work trough the remaining seven topics on the list as we move forward.

The role of the Principal Review Manager (PRM) has been one of the most talked-about topics since I joined IPAO two years ago. It has been a source of multiple organizational discussions and we have tried to better define it and adjust the operation of this important position. The fundamental function of the PRM is very straight forward: the PRM is a senior member of EAG whose function is to facilitate the review activities of one or more of our mission directorate review portfolios. The challenge is the implementation; in other words, determining what is involved in “facilitating” the review activities of a mission directorate and defining the lines of authority, responsibility, and accountability between Review Managers (RMs), the EAG supervisor, and the PRM. Let me start by describing what is involved in facilitating the review activity. Facilitation involves proactively coordinating the manifest of reviews with the mission directorate, including addressing issues with significant changes to the manifest once it is approved. Facilitation also includes proactively addressing systemic (or cross cutting) issues in the planning and execution of review activities involving the mission directorate; as such, the PRM is the “the process improvement agent” as it relates to that mission directorate and is expected to lead the PALs and other improvement activities. The PRM also facilitates SRB activity by maintaining an awareness of periodic Program/project performance review activity conducted at the mission directorate level (and center when appropriate) and sharing that awareness with other IPAO personnel such as RMs and PAG analysts working independent reviews of these Programs or projects.

The PRMs also facilitates activities within IPAO. The PRM facilitates the transition of new RMs into the fold by providing expert advice and support as required to ease the transition and to transfer corporate knowledge. PRMs participate in the review of products and documentation such as TORs, briefing packages, and final reports for their assigned area.

In a nut shell, the PRM has authority, responsibility and accountability for the functions outlined above. Also, as you can see from the description provided above the PRM is a leadership position and PRMs lead by influence in their interfaces with RMs, other IPAO personnel, and mission directorate personnel. Supervisory authority responsibility and accountability for RMs and PRMs rests with the EAG supervisor. Authority, responsibility and accountability for planning and execution for a particular review rest with the assigned RM. PRM and RMs need to have a very close working relationship with RMs supporting the PRM in leading the planning and process improvement activities for the mission directorate and with RMs benefiting from the expert support and guidance provided by the PRM. I hope this description helps in our general understanding of the PRM role.

As always, please let me hear your thoughts on this subject.

James N. Ortiz, PhD

“Them Versus Us”…

Ortiz_ James226x170This version of the blog is dedicated to the topic of team cohesion. To no one’s surprise team cohesion is one of the trademarks of highly effective teams. For our IPAO team, team cohesion takes on a very challenging dimension as we operate on a highly inter-related set of activities with groups that are both diverse and dispersed. So it is very easy to fall in the trap of the “them versus us” syndrome. Here are a few examples: EAG versus PAG; SRB members versus IPAO members; IPAO business and administrative personnel versus IPCE business and administrative personnel; IPAO analyst versus CAD analyst versus SID analyst; IPAO RMs versus Mission Directorate PEs; IPAO RMs versus Center SRMs, …and this list could go on and on…. At the end of the day, we are all in the same team and we all have the same common goal: to make our programs and projects successful. We can’t do it without having constructive relationships and working together. The degree by which we attain our goal is closely tied to our mutual success. As a result, we are spending a significant share of our time and energy in building these relationships. It should be clear to everyone in IPAO that I measure our success by how we contribute to the success of our programs and projects as part of the larger team and not only by what we do within the confines of our organization’s boundaries or any boundaries within our organization. So I expect each of us to have this mind set as we continue to move forward. Be proactive in reaching out, in communicating, in identifying issues and constructive solutions, in asking and providing help and assistance, and in working patiently with others. I am convinced that operating under this mindset will improve our team cohesion overall and our ultimate success.

As always, please let me hear your thoughts on this subject.
James N. Ortiz, PhD

On Staff Meetings

Ortiz_ James226x170 I tremendously enjoy visiting our non-LaRC locations. It gives me the opportunity for more direct interaction with our extended IPAO staff. As I talk with them, it is clear that there is significant value in “catching up” with the latest happenings in the office, the most recent events in the Agency’s policies and with the Programs and projects with which we are engaged. I also get significant insight from their perspective in the operation of the office as highly distributed organization. Recognizing that one-third of the office is distributed, frequent and timely communication is essential to the smooth operation of IPAO. We have two main activities that provide us opportunity for interaction: the Director’s Status Review (DSR) and the weekly staff meeting.

I will address the staff meeting in this blog. The weekly staff meeting provides a frequent opportunity for interaction. Its main purpose is to exchange the most current information and announcements and to synchronize the office’s calendar of activities. Periodically, we extend by a half-hour to run the IPAO forum intended to raise the awareness of the staff on a Program or project being reviewed or a topic of interest in programmatic analysis. To make the meeting as effective as possible the full participation of everyone attending is a must. In order to take full advantage of the short time we spend together, and to fully embrace the participation of those at remote locations, we need to devote our full attention to the discussion and avoid “multiplexing” during the meeting by doing e-mail or other work, also we need to congregate around our communication equipment so our comments can be heard by all. An hour a week is a small commitment we all should make to maintain and strengthen the lines of communication and teamwork in the office. I appreciate your attention and support to this matter.

As always, your feedback is appreciated.
James N. Ortiz, PhD

On SRB Written Reports

Ortiz_ James226x170I know that running SRB teams and performing programmatic analyses is a laborious and stressful endeavor. You invest several months of hard work, nominating the chair, review team members, vetting each team member, negotiating the Terms of Reference (TOR) for your review, holding the site review, compiling findings and recommendations, performing the peer-reviews, and going through debriefing process including the quick look report, the project debrief and the briefing to the Center, Mission Directorate and Agency PMC… but what remains after all of that? The answer is the SRB written report. The written report is the narrative that describes all the hard work performed by the SRB that resulted in the assessment of the Program or project. It is the one record that is kept by IPAO to tell the story to all stakeholders, present and future, in enough detail so they get the complete picture of the who, how, when, and what resulted from the review. Some of you have questioned the value of such a product and have offered the briefing package as a replacement, but the briefing package is an inadequate product to tell the story. The briefing package presents a brief summary to an audience that is very familiar with the plans and rules of engagement for the review and that is mainly interested in the review outcome and recommendations. Another argument I heard recently is that the written reports are put on a shelf and just collect dust (electronic dust these days since we went to an all electronic archival system). Let me just say that over the last month SMD requested access to our reports to support a study of the effectiveness of their AO system and that a recently published report was immediately forwarded by the Mission Directorate to the IG at their request.

So this is a very important product and one in which we need to show the standard of excellence in IPAO. As you have seen, we have put the spotlight on our written reports recently to make sure that our reports have a standard appearance and feel that meets the needs outlined above. I appreciate your contribution to this effort. This is an important part of the legacy we leave behind!

As always, I welcome your thoughts.
James N. Ortiz, PhD

2010: The Year Ahead

Ortiz_ James226x170Welcome back to the IPAO Director’s blog after a two month hiatus while waiting for the new IPAO website to come on line.

This year promises to be more challenging than last year for IPAO. We, along with the rest of the Agency, are reacting to the changes in direction as a result of the President’s 2011 budget. In mid-January, we started discussing contingency plans as part of the planning and approval of our 2010 review manifest. Now, we are implementing that plan while remaining flexible and responsive to the needs of the Constellation Program while they refine their plan to move forward.

Although the year is still young, we already have several achievements under our belt: We published the SRB handbook; and as a result, we were able to close several outstanding actions we had open with the IG. We need to remain vigilant as we execute our independence processes to make sure we stay within the “straight and narrow” path outlined in the policy, while always learning new lessons on how to implement it. We also conducted the second round of Pause and Learn (PAL) sessions with SMD, and we have identified several efficiency improvements to the review processes that we are bringing forward for approval at the upcoming Convening Authorities PAL. We also identified several areas that we need to improve regarding our analyses and we are committed to address them. More will be communicated as we complete that effort.

After several months of intense work, we have rolled out Phase I of our new and improved IPAO website. Check it out! Phase I is a face lift, but the upcoming release of Phase II will have new functionality to help us do our work more effectively, such as automating the production of the weekly reports. Also, we just completed our support of the PM Challenge 2010 with the second year of the “Independent Review in Action” track – briefings, panel, and booth. I am very proud of how the IPAO rallied together to support this effort and by the professionalism demonstrated by all who participated. Thank you! I know, we have to “spiffy up” the booth, but we already have a volunteer signed up to lead that effort.

So, we had a good start this year and by working together, remaining flexible, maintaining our focus, and communicating amongst ourselves and with our stakeholders, we can look forward to many more achievements!

I am always interested in your thoughts!
James N. Ortiz, PhD

Working together: E-mail Protocols

Ortiz_ James226x170As we continue on our road to excellence, I am addressing the topic of e-mail protocol in this blog. All of us appreciate the great flexibility and expediency of this medium of communication, but we are all too familiar with its drawbacks such as the enormous quantity of messages, the constant intrusion into our personal time, the large number of irrelevant or just FYI email, and the long chains of e-mail “conversations”. Below, I provide some guidelines for the use of e-mail in our office with the intent to continue taking full advantage of the strengths of e-mail while mitigating the drawbacks mentioned above.

I expect people in the office to check e-mail regularly during working hours, for our office that window is from 07:00 AM to 06:00 PM. Please try to minimize the e-mail after these hours to provide ourselves with personal after-hour time. During the weekend, I will encourage e-mail communication be limited to items that require a quick response. I expect people to respond to these e-mails within 24 hours. This means that I expect people to scan e-mail once a day during these times. Any urgent items should be handled via a phone call and I believe should be rare.

Regarding the volume of e-mail, I would like to encourage everyone to pause before sending an e-mail to review the distribution and ask themselves if all the people listed need to be included. For example, don’t assume that because the names were included in a previous forwarded e-mail that everybody downstream needs to be included. Also consider alternatives to sending e-mail (specifically with large attachments) when the information can be posted in share drives or web pages.

Also, please be sensitive to the fact that e-mail as vehicle of communication has its limits. Long chains of e-mails “conversations” are an indication that a face-to-face or a teleconference is needed to resolve the issue.

I know that I am just scratching the surface of this topic but I feel that if we start here, we can harvest the “low hanging fruit” in this area and work more effectively together.

As always, please let me know your thoughts.
James N. Ortiz, PhD


This month I want to address IPAO staffing. We are going through significant changes within every group in our office. The review management staff, the programmatic analysis staff, the administrative staff and the Director’s suite are all experiencing changes in staff.

As a consequence, you are seeing review managers who were on detail assignments return to their home centers. On the EAG side, we have new detail RMs come into the organization from three different field Centers, as well as the addition of long-term rotational RMs within the next few weeks. PAG has recently added an additional in-situ cost analyst and will add two more in-situ schedule analyst soon. In the front office, our resource analyst is transitioning and we will be hiring a business manager for the office in the next few weeks. Also, our acting deputy director completed his developmental assignment, and we are hoping to have the new director in place soon.

Anticipating these changes, we are deploying a new training program called “boot camp” to help enable a quick integration of new personnel. Also, several of you have been tapped as formal mentors and as such you will perform a key role in showing our new team members the “ropes” to get them up to speed quickly with the least amount of pain. We all need to make ourselves available to help them in any way we can and to make they feel welcome into the organization. We also need to take advantage of the new perspectives these additions bring to us as they may point better ways to do and manage our work.

One of the key elements to success in our quest for excellence is to be able to depend on each other. This is a perfect opportunity to demonstrate our “esprit de corps”.

As always, please let me know your thoughts (any way you choose to do so).

Pursuit of Excellence

This fourth blog is dedicated to the topic of excellence. By now you have heard me stress at staff meetings and at our director’s management reviews that we, as a team, should strive to better communicate our plans and needs. We need to set clear expectations and relay these plans to our customers and stakeholders. We need to pay attention to detail in our planning products, reports, and other documentation to make sure they are error free and consistent with Agency policies and direction. Our goal should be to pro-actively address issues so that our working relationships, internally and externally, are developed and maintained in an environment of trust. Perhaps you remember our push for continuous improvement, in which we are dedicating our organization to improve our processes to provide value added to the Agency’s investment in independent programmatic assessment. Our goal is to help define and implement a holistic programmatic analysis capability that fully integrates elements of technical, cost, schedule and risk. IPAO is becoming further challenged to better adjust to the scope of work we are chartered to perform across the Agency and to provide enhanced recruitment opportunities to a broader spectrum of talent from the field centers.

All of these are part of our drive for excellence. In my view, our office is more than capable of achieving excellence, and as you can see from the examples that I mentioned above, we are well on our way. Why drive for excellence? The answer is simple – the Agency deserves no less for the important role we have been entrusted. We have a strong foundation to build upon. Remember excellence requires an extra effort. In my view, excellence is achieved by a vibrant organization whose products, at all levels, are known for their quality. It is an organization that is trusted as a willing partner in achieving the Agency’s goals; and in our case, an organization that is trusted to be a competent, objective, and fair assessor in all our engagements. This is a high bar but one that we can certainly achieve!

I know you all want to be part of such an organization. As we continue moving forward, your support and dedication will be a key enabler. As always, let me hear what you think.

James Ortiz
Director (acting) IPAO