Aerospace testing is, by its nature, an expensive activity. Employing technical specialists, making use of advanced test facilities and navigating the governance processes required to deliver safe and effective testing can incur vast expense. It is therefore common to seek efficiency savings through technical innovation or, more usually, through a reduction in the scope of testing activities; this will ultimately lead to a reduction in capability for the system under test. But what if there was a way to achieve efficiency savings without accepting a reduction in the scope of testing and operational capability? A concept displaying increasing potential in this area is the Combined Test Team (CTT).
To take a simple example, the typical stakeholders involved in the release to service assessment process of a new aircraft type can be defined as follows:
The customer: Responsible for defining and managing system requirements, and coordinating the activities of other stakeholders.
The Design Organization (DO): Responsible for delivery of product to satisfy the capability requirements defined by the customer. Additionally, the DO must also demonstrate compliance with these requirements and other applicable standards.
The Independent Technical Evaluator (ITE): Often mandated by the applicable aviation regulatory authority, the ITE provides the customer with an additional layer of assurance as to the safety in role of the equipment under test.
The end user: In a military context, usually referred to as Operational Test & Evaluation (OT&E). Once safety in role has been established, the end-user representative is responsible for assessing the operational effectiveness of the system under test. Additionally, the end-user representative is largely responsible for development of standard operating procedures and training syllabi.Effective management of the efforts of these stakeholders can be particularly challenging, and is central to the concept of the CTT.
Legacy Model
In a non-CTT scenario, each of the stakeholders conducts their own, sequential, test programs. There is sound logic in this approach; the DO must prove that the system is safe within a defined envelope, and this must be verified by the ITE. Once the safety argument is in place, the end-user representative assesses operational effectiveness within the bounds of this envelope. This is a logical, systematic approach that has endured for many years, but one in which significant inefficiency exists.
For example, the stakeholders are almost certainly not co-located, and may not operate within the same airworthiness regulatory framework, so the process of establishing a test base, defining a test plan and gaining permission from the necessary regulatory authority to operate the system must be undertaken three times. Also, as the stakeholders are operating independently, there is no cross-visibility of test planning or findings, resulting in repetition of potentially expensive and often hazardous trials activity.
The Combined Test Team
The underlying concept of the CTT is that testing is a source of evidence that can be drawn from by any interested party in order to fulfill evidence requirements. In practice, this means an amalgamation of a subset of key personnel from each of the stakeholder organizations into a single entity: the CTT. However, this amalgamation is not a trivial task, and can determine the success or failure of the test program. The following considerations are particularly critical:
Timely formation: A hastily assembled team that is handed a set of evidence requirements and a test plan is unlikely to succeed. For the priorities, responsibilities and methods to be properly established, it is imperative that the CTT is engaged early in the project lifecycle. This can be achieved through the formation of a cross-stakeholder test and evaluation working group.
Proportionality: A CTT is not a democracy, and stakeholders do not necessarily require equal representation in terms of staffing. Instead, the constitution of the CTT should be driven by the proportion of ownership of evidence requirements – further justification for early CTT formation.
Freedom from commercial pressure: The primary objective of the CTT is to safely deliver the required test program. While any testing professional should be mindful of where their activity sits within the overall program, excessive pressure to meet program deadlines is not appropriate within a CTT. The key message here is that commercial decisions and pressure should be detached as far as practicable from the CTT.
Leadership: Although the CTT is a collaborative effort, it will ultimately operate under a single regulatory framework and under the leadership of a single responsible person: the CTT lead. The CTT lead must balance the complementary and contrary requirements of the stakeholders, while also filtering program and commercial pressures from above. There are few roles as challenging and diverse in the field of aerospace testing; a person with experience of working closely with CTT stakeholders is essential.
Developing a Joint Test Plan
With the evidence requirement and CTT in place, it is time to develop a combined test plan. This is an area in which significant efficiency savings can be achieved through intelligent test planning. For example, military aircraft undergoing OT&E will often be required to transit out to appropriate ranges or exercise areas. In the legacy testing model, these transitions could be straight and level flight at altitude, during which no data is captured; in the context of testing, this is a waste of time. The CTT test plan can identify test points requiring straight and level flight, for example engine performance or navigation systems testing.
Similarly, human-machine interface testing often requires the aircraft to be flown in an operationally representative manner; this data can be captured during the OT&E phase of the sortie. Thus, what would have been three or more sorties in a legacy style test program has been combined into a single sortie. This concept can be developed even further with the addition of ‘hanging’ test points – a list of tasks that can be conducted on an ad hoc basis should the opportunity arise. A cautionary note, however; the team must be mindful of the sortie priorities and resist the temptation to compromise safety by attempting to overachieve.
Exploiting Niche Capabilities
A key advantage of the CTT approach is the ability to pool resources between stakeholders, often across organizational divides. This is particularly important for retaining niche capabilities such as test facilities for night-vision systems, electromagnetic compatibility and environmental extremes. Such facilities are few in number and require significant investment to sustain capabilities which, due to their very specific nature, are often under-utilized. The CTT approach opens up such facilities to all stakeholders, increasing utilization and justifying both sustainment and future investment in additional capabilities.
The Future
As the main benefit of adopting a CTT approach is increased efficiency, future developments are likely to be in this area. For example, where stakeholders are secure in the knowledge that they are to be working together for the foreseeable future, a strategic partnership makes eminent sense.
This allows for the establishment of common processes and test methods, reducing the time required to “spool up” CTT activities. Once an effective working model has been established between two stakeholders on a single program, there is no reason that it cannot be transferred to other platforms to spread the benefit of these efficiency savings. The stability assured by such partnerships also raises the possibility of physical co-location, a key ingredient for CTT success.Many years of shrinking budgets for aerospace products has instilled a culture of driving for efficiency. Although technical innovation can sometimes achieve this, novel test team construction can provide tangible benefits for a modest increase in program risk and little in the way of up-front cost. Careful planning is required to ensure that the CTT constitution is appropriate for the task, but the concept is sufficiently mature that it should be considered as a matter of course. Fundamentally, it is a way of empowering aerospace professionals from across organizational divides to collaborate in the delivery of safe and effective test activities – a concept that is extremely hard to argue against.
Acknowledgements
The author would like to acknowledge the efforts of CTT personnel worldwide, for delivering safe and effective capability in the spirit of collaboration.
This article is based upon the personal experience and opinions of the author, and does not represent the views of any company or organization.
Garnet Ridgway works for a UK-based aircraft test and evaluation organization; and writes regularly for Aerospace Testing International
