LT Thompson's research won the Distinguished Professor Kenneth J. Euske Dean’s Medal for Innovative Contribution to National Defense, and the Department of Defense Management. Her research was also presented at the 20th Annual Acquisition Research Symposium, co-hosted by the Acquisition Research Program and Naval Warfare Studies Institute. 

Abstract

Department of Defense (DOD) current source selection methods are at an increased risk of experiencing sustained bid protests. During source selections, the government frequently contradicts itself between its advertised stated order of importance for acquisition evaluation criteria (pre-award) and its actual choice behavior during source selections (Butler, 2014).  This paper provides a summation of research, conducted from 2021 to 2022, that explored the following research objectives: 1) Determine the degree of disconnect between stated preferences during pre-award acquisition phase and actual choice behavior in defense acquisition source selections, 2) develop a deep understanding of quality attributes in evaluating logistics-based service acquisitions, 3) provide a Choice-Based Conjoint (CBC) framework that the DOD could utilize to enhance source selection criteria development in both logistics and further categories of government spending. The research utilized methods such as interviews and spend analysis techniques to identify quality attributes of logistics-based acquisitions that would best discriminate as evaluation factors for award. Later, these attributes were used to develop a CBC exercise that enabled us to calculate attribute utilities and relative importance for each attribute. The summarized research in this paper provides a way forward to empirically deduce the relative importance for source selection evaluation factors, potentially reducing bid protest occurrences in future source selections.

Introduction

In its annual letter to congress, the U.S. Government Accountability Office (GAO) repeatedly reports that one its most common reasons for sustaining a bid protest: government agencies continuing to unreasonably evaluate technical, past performance, and cost or price evaluation factors during source selections (GAO, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021). Such unreasonable evaluations persist, and agencies cannot follow their own solicitation’s evaluation criteria, ensuring a flawed source selection decision and increased chance of a GAO bid protest.  

Additionally, the MITRE Corporation further substantiates the same issue in competitive source selections and evaluations, with its Contract Protest Diagnostic Tool (CPDT). The CPDT uses a heatmapping visual technique to show the exposure to protests within each phase of the federal acquisition phases (MITRE Corporation, 2022). Two of the most historically problematic ‘hot spots’ indicate that the U.S. federal government is regularly exposed to protests because its agencies do not (1) perform fair and consistent evaluations that are consistent with the evaluation procedures described within solicitations and they do not 2) solicit with evaluations factors in a properly weighted relative order of importance that matches how these same factors are evaluated during the source selection process.

Procurement agents throughout the DOD aim to deliver “quality and timely products and services to the Warfighter and the Nation at the best value to the taxpayer” (DOD, 2016). While source selections and their procedures offer a structured approach to these agents to obtain best value, the increased risk of bid protests created by these procedures mean implicit consequences for the DOD. When faced with bid protests, the DOD must utilize valuable, finite resources to resolve said protests. In a time of increasingly varied global change and threats, losing valuable resources to preventative consequences, places the DOD in a precarious predicament.

Past research conducted by one of this paper’s contributors, finds further fault in current source selection procedures. In the graduate essay on “Perceived Service Quality and Perceived Value in Business-to-Government Knowledge-Based Services” researchers argue that government agencies use Lowest Price Technically Acceptable (LPTA) procedures in an increased effort to avoid the risks of exposures such as those described in the above paragraph (Finkenstadt, 2020). The study goes on to provide some unique insights into the Business-to-Government (B2G) buyer and their choice behavior in simulated source selections by leveraging conjoint methods. For example, individuals often rely on theoretical deduction, or an a priori judgement, to predict the ordered importance of price and non-price factors instead of utilizing empirical reasoning (Finkenstadt, 2020). When presented evaluation factors in list form, individuals also have a difficult time in properly shaping the relative order of importance because a list does not provide them an opportunity to consider these nonprice and price factors when presented in a full set of offers or grouped together (Finkenstadt, 2020).

The following paper offers initial insights into how the DOD can address the illustrated disconnect between stated preferences during pre-award acquisition phase and actual choice behavior. These findings support the issues revealed in MITRE’s CPDT tool, GAO sustained protests, and past research. By quantifying the disconnect and better understanding how the DOD acquisition workforce and its customers evaluate products to meet their needs, there can be a subtle, yet significant shift in how the organization can better utilize its finite resources. Furthermore, the research summarized in this paper offers better understanding on how the DOD evaluates perceived attributes of logistics-based services. While these findings were supplemental in nature to the overall agenda of the research, such information has the potential to enhance future evaluation criteria in source selections for these logistics-based services. Finally, the insights offered from the research described in this paper, may reduce the risk of acquisition protests, as it provides knowledge of perceived preferences, subconscious or otherwise for these services. All ensure that acquisition professionals can better prioritize evaluation criteria during the contract pre-award phase ensuring the right solution, at the right time, and for the right customer.

Issues with Source Selection Methods

            As outlined in FAR part 15.3, source selections procedures enable acquisition professionals to determine which contractor proposal provides the best value, or “the expected outcome of an acquisition that, in the Government’s estimation, provides the greatest overall benefit in response to the requirement”, to the government (DOD, 2016). While the procedures provide a structured approach to obtain best value, the way source selections procedures stand now jeopardize the three goals of government procurement: transparency, value for money, and meeting requirements (Finkenstadt & Hawkins, 2016). This is because the procedures do not provide a way in which to state what really matters to the government and how best to quantify it. Instead, source selections teams are often left to define evaluation factors and an a priori ranking for those factors based on presumed importance. In short source selection guidance does not offer an empirical method to acquisition personnel that allows for both effective evaluation factor determination and their order of importance. Conjoint analysis, and more specifically, choice-based conjoint analysis can be that method.    

Conjoint Analysis

            Conjoint analysis is a tool that enables managers, companies, and acquisition personnel alike to “model the factors that underlie and drive consumer choice” (McQuarrie, 2016) through utilization of a product or service’s “separate (yet conjoined) parts” (Orme, 2020). Through conjoint analysis, a product or service’s attributes can be purposefully varied while respondents’ reaction to the variability can be statistically deduced and these scores, or utilities and part-worths, for each attribute can help to define the value of the service (Orme, 2020). For reference, part-worths are fully defined as, “the utility associated with a particular level of an attribute” and utility, in reference to conjoint analysis, “refers to a buyer’s liking for (or the desirability of) a product alternative” (Orme, 2020).The advantage of conjoint analysis over other standard marketing techniques, like surveys, is that it’s a ‘back door’ method to develop insight into subconscious choice behavior when respondents are presented full product or service profiles (Orme, 2020).

While conjoint analysis offers far superior methods in terms of discovering consumer choice behavior, the simulated, hypothetical environment built from its use imposes a lack of real-life consequences to the respondents, preventing analysts from collecting the most realistic preferences and data from respondents (Ding et al., 2005). To help combat the consequences of a simulated environment, the incentive-aligned consequence of ‘expert scrutiny’ was incorporated within this research’s conjoint analysis exercise. This meant that respondents who participated in the choice exercise are “told to answer realistically because an expert in public procurement will analyze their responses for reasonableness prior to including it in any decision to change public acquisition methods or policy. This mimics the formal source selection review process found in many public agencies” (Finkenstadt, 2020, p. 101).

Finkenstadt discovered that expert scrutiny worked as well as other typical incentive-aligned CBC prompts for government acquisition personnel during an exercise in which over 600 personnel were randomly assigned to various prompt conditions (2020). Though the outside option utility was much smaller in a Bayesian-truth serum condition, it was determined that it did not skew the relative importance ranking of factors (Finkenstadt, 2020). Finkenstadt recommends expert scrutiny due to its lower costs and time to employ. Therefore, this study settled on the use of expert scrutiny as the incentive-alignment prompt for respondents.

Software advancements have allowed conjoint analysis to expand in terms of its approaches and data that it gathers.  What started as a method utilizing handwritten cards for product profiles in 1971, is now conducted on advanced statistical software that offers a multitude of options to its users depending on their research and what outcomes they hope to measure (Orme, 2020). Each conjoint analysis approach can be divided amongst the tactics researchers use within their exercise. A ratings-based approach has respondents ranking full-profile products, while choice-based conjoint techniques allow respondents to choose or trade-off amongst different product profiles. Other approaches utilize some form or combination of both techniques. Figure 3 details the types of conjoint analysis, but for the purposes of this research, CBC analysis and Sawtooth© Choice Based Conjoint Software were utilized.

Figure 3.          Types of Conjoint Analysis in Marketing. Adapted from Orme (2020).

Choice-Based Conjoint

CBC presents a series of choice tasks, or questions, that ask respondents to choose from three to five product profiles (Orme, 2020). What sets CBC apart from other conjoint analysis techniques is that it provides respondents an option to choose none of the product profiles, as consumers realistically can choose none of the product alternatives when presented options in a real market environment. CBC utilizes several analytical methods to estimate respondent preference; however, for this research, Hierarchical Bayes (HB) analysis was utilized. HB estimation offers a model to estimate the part-worths at an individual-level, by iteratively collecting data from multiple respondents and finding a point of convergence (Orme, 2020). Figure 4 shows one of the twelve choice tasks presented in the CBC exercise for this research. The relative levels of each attribute displayed below are further explained within the methodology section of this paper. 

Figure 4.          CBC Choice Scenario Example.

Real Property Maintenance

For conjoint analysis to be successful, it must only include a limited number of attributes per product/service profile, otherwise it risks unnecessary difficulties for respondents and possibly jeopardizing the results of the CBC. Ensuring the proper number of attributes and attribute levels is one of the most critical aspects in designing a successful CBC (Orme, 2020). To secure proper design of the CBC then, no more than eight attributes and five or fewer levels of attributes should be used (Orme, 2020). With that, the research presented in this paper determined that only one Product Service Codes (PSC), under the Transportation and Logistics Service federal category of spend, would be used to determine the limited scope and attributes for the CBC scenario.

In order to select one specific PSC, spend and data analysis techniques were utilized in order to discover a PSC associated with the highest dollar spend under the Transportation and Logistics spending category level one. The U.S. Air Force Installation Contracting Center’s (AFICC) Business Intelligence tool, AFBIT Lite, provided the spend data required to find this specific PSC.  Within the category one level of spend, it was determined that amongst all 25 PSCs under that category, it was PSC R706 – Support Management: Logistics Support, that held the highest dollar spend at approximately $18.6 Billion (AFICC, 2022).

There are many services associated with PSC R706, and so it was then decided that for the CBC to have the proper number of attributes, further narrowing of the PSC was necessary. Researchers concluded that only one service under that PSC would be the focal service that helped to develop the CBC exercise employed. To narrow it down from one PSC to a particular service associated with that PSC, the DOD’s Project Management Resource Tools (PMRT) Enterprise Analytics (EA) application CON-IT application was used. Like the process utilized with AFBIT Lite, the service with the highest usage for the DOD was sought. This ensured the impact this research had was greater than if completed for a service not often utilized or contracted out for. Through a thorough examination of over 17,831 relevant Contract Line-Item Numbers (CLIN) data, Real Property Maintenance (RPM), was chosen as the selected service for the CBC exercise. The maintenance of real property is defined as, “the upkeep of property only to the extent necessary to offset serious deterioration; also, such operation of utilities, including water supply and sewerage systems, heating, plumbing, and air-conditioning equipment, as may be necessary for fire protection, the needs of interim tenants, and personnel employed at the site, and the requirements for preserving certain types of equipment” (Real Property Policies, 2022). Real property can include “any interest in land, together with the improvements, structures, and fixtures located, and appurtenances thereto, under the control of any Federal agency” (Real Property Policies, 2022).

Methodology

            Several steps were taken in advance to ensure the conjoint analysis techniques used in this research were properly conducted and represented a hyper-realistic situation that respondents could possibly encounter if participating in a DOD source selection. First, a literature review was conducted to educate, inform, and build a foundation for the study. Topics such as DOD source selection procedures, logistics, conjoint analysis and its use in the Business to Consumer (B2C), Business to Business (B2B), and Business to Government (B2G) markets were explored.

Second, once an initial backbone of knowledge was built, the researchers moved forward by interviewing logistics personnel and acquisition experts that aided in the determination of service quality attributes for logistics service. The six interviews conducted were with government personnel that had acquired logistics-based services and/or commodities, had a military logistics background, or participated in source selections for a logistics-based service. Questions proposed to interviewees focused on their organization’s acquisition of logistics-based services, factors considered important when evaluating a contractor’s proposal, and essentially, what was important to government customers, acquisition personnel, and logistics personnel when it came to a logistics-based service. Despite the diverse backgrounds of each interviewee, certain trends and patterns emerged amongst the responses provided to the researcher. Upon conclusion of the interviews, it was determined that the same evaluation considerations interviewees consistently mentioned were those indicators attributed to SERVQUAL, a popular model that aids in measuring the perceived quality of a service (Parasuraman et al., 1985). Five concise dimensions of service quality perception are highlighted through the SERVQUAL model:

• Tangibles:      Appearance of physical facilities, equipment, personnel, and communication materials
• Reliability:     Ability to perform the promised service dependably and accurately
• Responsiveness: Willingness to help customers and provide prompt service
• Assurance:     Knowledge and courtesy of employees and their ability to convey trust and confidence
• Empathy:       Caring, individualized attention the firm provides its customers (Parasuraman et al., 1985)

Interviewees consistently indicated that four of the five SERVQUAL service quality dimensions were important. These four dimensions were modified to represent the four attributes, besides price, that the CBC would include for the simulated logistics-based source selection.

Third, and as mentioned in previous portions of this paper, a specific service was chosen through a tailored spend and data analysis utilizing AFBIT Lite and PMRT. This analysis helped narrow the scope of the CBC from a category of spend to the selected service highlighted within the CBC, RPM. Along with providing a high-use, high-spend service this analysis also enabled the determination of realistic prices to be used in the CBC as the fifth and final quality attribute of the RPM service. Through PMRT CLIN data, the average price per month for RPM services equated to $74,885.62. Once an average price was determined, a pivot table utilizing RPM CLIN monthly prices was created, and the average price previously determined was taken to identify four other price points to utilize in the CBC. Figure 5 represents that process and Table 2 shows all five of these prices when they were increased to reflect the price of a firm-fixed price contract with a 12-month base period, four 12-month option periods, and a six-month extension of service clause if necessary. (See all figures and tables in the PDF version of this paper.)

Finally, after the above steps were completed, the CBC was designed to provide 12 random choice scenarios to respondents utilizing the five attributes and price points determined through this research’s additional methods of discovery. Along with the five attributes, four levels per attribute were created utilizing Table 3’s scale ratings that were adapted from current DOD source selection procedures.

CBC respondents included those government personnel that had held the role of contracting officer, contracting manager/administrator, contracting officer representative, quality assurance personnel, program manager, customer, and other positions directly involved with government acquisition. Respondents were guided to assume that all contract offers observed within each random choice task were technically acceptable, the prices provided were realistic, and the final evaluation determination they were selecting was based on the contractor’s past performance and generated through a tradeoff decision-making process between the price of the contract offer and the four service quality attributes (Finkenstadt, 2020).

The CBC was designed using Sawtooth© Choice Based Conjoint Software, tested for functionality at the National Contract Management Association (NCMA) Conference in July 2022, and finally released to collect data in August 2022. Along with the 12 random choice tasks presented to respondents (see Figure 4), respondents were also asked demographic questions regarding their experience in government acquisition and were also guided to rank order attributes of logistics-based services. These attributes represented the same as those presented in the CBC choice tasks; however, respondents were provided only the definition of these same attributes, and asked after the CBC choice tasks, to minimize the opportunity to ‘game’ the system and memorize their choices in the CBC and match their rank ordered items similarly. Figure 6 displays the rank order choice exercise presented.  (See all figures and tables in the PDF version of this paper.)

Overall Findings and Contributions

            The CBC choice exercise was open to respondents from 01 August 2022 – 15 September 2022. 30 respondents completed the choice exercise, meeting the standards “for investigational work and developing [a] hypotheses about a market” (Orme, 2020). The experience of those that completed the choice exercise varied in terms of the role held and their years of experience in said role. Each respondent was asked to select one or more of the acquisition-focused roles they had held and how many years they held that role. Table 4 and Table 5 show the experience demographics collected from the 30 respondents. The totals aggregate to greater than 30 as some respondents had multiple position experiences

            Respondent’s data was validated, in terms of response quality, utilizing several methods Sawtooth Software provides its researchers. Visual inspection of repeated choice patterns, review of completion times, and computation of the Root Likelihood measure were the three chosen methods used to validate quality. As a note, the Root Likelihood (RLH) is “an intuitive measure of how well the solution(s) fit the data…[it] is an intuitive probability expression of how successful the utility scores are in predicting which items respondents pick” (Sawtooth Software, 2022). All three validation methods indicated that none of the 30 respondent’s choices appeared randomly selected.

Research Objective I

In order to determine the degree of disconnect between stated preferences and actual choice behavior, the data collected from both the CBC and ranked preference exercise was compared for each of the 30 respondents. The CBC data offered individual importance scores for each attribute, while the ranked preference exercise allowed respondents to directly input what they believed to be most important to least when acquiring RPM services. Table 6 and Table 7 represent the collected data that was then compared against each other, while Figure 7 is a visual example of the comparison of stated and observed choices for one respondent.

Figure 7.           Stated Ranked Preferences vs. CBC Choice Behavior

Once all 30 respondents’ stated preferences and CBC behavior was reviewed and match compared, the inverse of their match rates (the disconnect rate) could then be determined on an individual and aggregate level. The average match rate accumulated (as seen in Table 8) through all respondent match rates was 23%, leaving the average disconnect rate at 77%. In summation, in this simulated source selection, the disconnect between the stated preferences of respondents and actual choice behavior could be confirmed and measured at over three times the rate at which respondents, and their stated level of attribute importance, matched their choice behaviors.

Table 8.           Match Rate Trends Amongst CBC Choice Behavior and Stated Preference

Overall Match Rate Trends: 

• 0 of 30 Respondents got 100% Match Rate (5 of 5 Matches)
• 0 of 30 Respondents got 80% Match Rate (4 of 5 Matches)
• 5 of 30 Respondents got 60% Match Rate (3 of 5 Matches)
• 5 of 30 Respondents got 40% Match Rate (2 of 5 Matches)
• 10 of 30 Respondents got 20% Match Rate (1 of 5 Matches)
• 10 of 30 Respondents got 0% Match Rate (0 of 5 Matches)
   

            In addition to analyzing exact match rates for respondents, the researchers also analyzed the collected data for the inclusive proximal match rate. This match rate reviewed choice behavior from both exercises and searched amongst all respondents as to if their CBC choices were off by one or two ranks in comparison to their stated ranked preferences. Simple ‘if/then’ formulas were utilized in Microsoft Excel to conduct this comparison process that not only checked for an exact match but also to examine whether the ranked attribute matched one ranking above or below that same attribute in the CBC choice behavior. Figure 8 displays the proximal match rate comparisons by attribute, with the green ‘Yes’ representing an exact match, the red ‘No-Yes’ representing the number of   inclusive proximal matches, and the blue ‘No-No’ indicating a no match whatsoever. While there was an increase in match rate when utilizing the proximal match process, the rate at which respondents still presented no match was approximately 40%.

Research Objective II

To develop a deep understanding of quality attributes that government buyers perceive when evaluating logistics-based service acquisitions, a series of semi-structured interviews were conducted with government personal that held various positions but had related experience in purchasing logistics-based services. These interviews highlighted several service quality indicators not associated with current measures and standards utilized in programs like the Contractor Performance Assessment Reports System (CPARS) or in guidance for Performance-based Logistics (PBL). Instead, the research offered some valuable insight into a potential issue regarding a dissonance between how the DOD is measuring the performance of logistics-based services versus how government personnel truly value the service itself and what they are looking for in terms of the contractors who provide it. With that, the research also provided four quality attributes (as seen in Table 1) that could offer a way forward in terms how the DOD measures quality for these services.

Research Objective III

            The foundational knowledge and research collected through this project allowed for a procedural framework to be built that can improve DOD source selection procedures. This framework offers a path of empirical reasoning, as opposed to theoretical deduction when determining evaluation factors. Figure 9 represents the CBC framework as it is incorporated into current source selection procedures.

            With the CBC framework incorporated into source selection procedures, Source Selection Teams (SSTs) can develop latent quality indicators through the very methodology this research utilized (semi-structured interviews), transition these indicators and perhaps some objective indicators to evaluation factors, and finally rank these evaluation factors utilizing a CBC or other conjoint analysis technique. In utilizing this framework, SSTs avoid ranking evaluation factors on theoretical deduction, ensure they are examining grouped evaluation factors (as opposed to strictly list form), and later portions of the source selection are no longer compromised as early use of CBC ensures empirically sufficient evaluation determination and criteria.

Conclusion

            As the DOD continues to operate with limited resources, witnesses rising tensions with geopolitical powers, and contends with an extremely accelerated technological shift, it’s important it finds ways in which to effectively function and rapidly adjust to the changes these three factors present. The research presented in this paper provides opportunities to manage resources more effectively, avoid acquiring ill-suited acquisitions to meet the evolving geopolitical threats, and bring a technological advantage to avoid the risky, ad hoc status quo in DoD source selections. In essence, applying the CBC framework to current DOD source selections offers a small, yet important, shift in how the DOD can deliver best value to rapidly protect and defend the United Stated of America (DOD).

See all figures, tables, and references in the PDF version of this paper.