Adjudicating Executive Privilege: Federal Administrative Agencies and Deliberative Process Privilege Claims in U.S. District Courts

Abstract

Government transparency is a key component of democratic accountability. The U.S. Congress and the president have created multiple legislative avenues to facilitate executive branch transparency with the public. However, when the executive branch withholds requested information from the public, the federal judiciary has the power to determine whether agencies must release documents and information to requestors. When enforcing standards of executive branch transparency, judges must balance concerns of executive autonomy and judicial intrusion into administrative decisionmaking. While much judicial scholarship focuses on the decisionmaking on high courts, in the U.S. context, federal district courts play a key role in adjudicating transparency disputes. In this article, I examine case outcomes in disputes involving agency claims of deliberative process privilege over internal agency documents litigated between 1994 and 2004. I find that U.S. federal district courts largely defer to administrative agencies in transparency disputes. However, factors such as agency structure and the congruence between judicial and administrative agency policy preferences influence whether federal judges require executive branch officials to release requested information.

In 2012, an interbranch clash over documents held by the United States Department of Justice (DOJ) led to a contempt charge against sitting Attorney General Eric Holder, and a complex legal dispute involving all three branches of U.S. federal government. The congressional investigation of the Department of Justice and the Bureau of Alcohol, Tobacco, and Firearms (ATF) lies at the center of the controversy. Specifically, in 2011, the House Oversight and Government Reform Committee, headed by Darrell Issa (R-CA), began its investigation of the ATF's “Operation Fast and Furious.” During the “Fast and Furious” operation, which traced back to 2009, Arizona ATF officials allowed guns purchased illegally to “walk,” or travel, across the border in hope of tracing the illegal weapons to cartels and gangs in Mexico.¹

However, the program came under increased scrutiny and criticism after the ATF lost track of hundreds of weapons. In addition, weapons involved in the “Fast and Furious” operation were linked to multiple crimes, including the murder of a U.S. Border Patrol Agent, near the United States–Mexico border (Horwitz 2011). As a part of its investigation of “Fast and Furious,” the House Oversight and Government Reform Committee requested a number of documents pertaining to the operation from the Department of Justice. The DOJ initially released some documents; however, at the behest of Attorney General Holder, President Obama asserted executive privilege over the withheld documents (Horwitz and O'Keefe 2012). In response to the DOJ's decision not to release all of the requested documents, the House of Representatives voted to hold Attorney General Holder in contempt, and the Oversight Committee filed a suit in federal court requesting that the court order the release of the disputed documents.² In transparency disputes, federal district courts must weigh legislative pronouncements, executive branch autonomy, and in some cases, the requester's need for information. What is the likelihood that a federal court will go against an agency's judgment and release internal documents to requesters?

Transparency and Executive-Judicial Authority

In separation of power systems, the judiciary can define and clarify the boundaries of executive branch action (Haynie and Dumas 2014; Howell 2003; Sanchez-Urribarri 2011). While direct legal challenges to presidential action garner more attention than administrative agency litigation, judicial intervention into agency activity can have dramatic consequences for public policy. This includes policy reversals or delays and added financial costs. In legal conflicts surrounding government transparency, an agency's ability to act as a gatekeeper of internal information is also at stake.

Transparency, or “the willingness of a government to release policy relevant information,” is a key component of democratic governance (Hollyer et al. 2011: 1193). Transparency facilitates greater public understanding of the factors underlying government policy and performance (Bellver and Kaufmann 2005: 4; Hollyer et al. 2011). The ability to challenge the legitimacy and legality of government decisionmaking necessitates transparency. Transparency is necessary both for electoral accountability and to seek remedies for injury through the court system.

In the U.S. context, the 1946 Administrative Procedure Act (APA) provides for transparency through public notification of proposed agency rulemaking (Harris 2009), and the 1966 Freedom of Information Act (FOIA) provides for direct access to internal government documents through a routinized request process (Miles 1988). In 2016 alone, requesters made approximately 800,000 FOIA requests to federal agencies.³ Although transparency is essential to democratic accountability, agencies may be understandably reluctant to release information pertaining to national security, ongoing criminal and civil investigations, and preliminary policy discussions. Individuals, organizations, and government entities, such as the House Oversight and Government Reform Committee in the “Fast and Furious” example, can challenge an agency's decision to withhold requested information in U.S. federal courts (Mastrogiacomo 2010).

Factors that contribute to government success in court include judicial attentiveness to executive preferences, agency litigation experience, and the presence of judges with previous executive branch experience (Galanter 1974, Kapiszewski 2011, Robinson 2012). These factors would presumably confer advantages to agencies in transparency disputes as well. However, the trend in the U.S. legislative policy environment in recent decades has been toward greater legally mandated transparency, with avenues for judicial remedies when information is withheld (Gerstein 2016; Halstuk and Chamberlin 2006). Given these potentially countervailing legal and political influences, to what extent does the institutional structure and policy preferences of administrative agencies affect judicial rulings in executive branch transparency litigation?

For example, the degree of partisan influence from presidential administrations can vary depending on whether an agency falls within the president's cabinet or whether the agency is an independent commission with greater insulation from direct presidential control. Given the variation in partisan influences across administrative agency structures, judges could exhibit different levels of scrutiny for independent and cabinet agencies that withhold documents. However, judges could also be more willing to defer to agencies when judicial preferences align with the agency's ideological character. Agency mission, policy jurisdiction, and personnel preferences are all factors that contribute to an agency's ideological identity.

In addition, transparency disputes can arise in different legal contexts (Kennedy 2005). The federal civil discovery process incorporates common law privileges that agencies can claim to withhold requested documents during on-going litigation between an agency and litigant (Tomlinson 1984). However, FOIA allows the public, and those not party to ongoing agency litigation, access to internal government documents (Halstuk and Chamberlin 2006). Although FOIA imposes broad transparency requirements on federal agencies, FOIA also promotes executive branch autonomy through the codification of common law privileges that allow agency officials to withhold documents (Kennedy 2005). Given the “presumption of disclosure”⁴ that accompanies the FOIA requests that land in federal court, judges may show less deference to agencies in the FOIA context as compared to the common law/civil discovery context.

The deliberative process privilege is the most common form of executive privilege raised in federal courts (Miles 1988; Narayan 2008).⁵ The deliberative process privilege, which was raised in the “Fast and Furious” dispute, also illustrates the tension between executive branch effectiveness and agency transparency. Specifically, the privilege protects internal conversations surrounding preliminary agency policy to preserve the quality of agency deliberations (Harris 2009); however, transparency requires knowing how and why agency officials made certain decisions (Weaver and Jones 1989). Examining federal district court decisionmaking in deliberative process privilege litigation provides important insight of when judges require greater transparency from the executive branch.

The article proceeds as follows: First, I explain the deliberative process privilege and deliberative process privilege litigation. Next, I argue that district judges are more likely to require executive branch (i.e., cabinet) agencies to release disputed documents in transparency litigation. However, when the policy preferences of the agency and the judge align, district justices will be less likely to require agencies officials to release requested information. Judges should also require greater transparency from agencies in FOIA disputes, given the “presumption of disclosure” that accompanies FOIA litigation. I test my argument using an original data set of approximately 200 federal district court cases involving challenges to agency claims of deliberative process privilege between 1994 and 2004. After reviewing the results, I conclude by discussing the implications of my findings in relation to broader concerns surrounding government transparency.

Federal Courts and Deliberative Process Privilege

The 1958 Kaiser Aluminum & Chemical Corp. v. United States dispute is the initial application of the privilege in U.S. federal courts (Harris 2009; Jensen 1999; Weaver and Jones 1989; Wetlaufer 1990). Retired Justice Stanley Reed, sitting by designation on the Court of Claims, upheld the decision of the War Assets Administration to withhold internal documents requested by a plaintiff. Justice Reed stated⁶:

Free and open comments of the advantages and disadvantages of a proposed course of governmental management would be adversely affected if the civil servant or executive assistant were compelled by publicity to bear the blame for errors or bad judgment properly chargeable to the responsible individual with power to decide and act. Government from its nature has necessarily been granted a certain freedom from control beyond that given the citizen. It is true that it now submits itself to suit but it must retain privileges for the good of all.

The above quote from Justice Reed illustrates judicial recognition of the need for some degree of protection for internal agency documents and discussion. This protection is not only important for the specific documents under dispute, but as Justice Reed suggests, withholding is warranted to reduce the negative consequences on agency deliberation that could accompany the release of internal documents (Jensen 1999). The deliberative process privilege specifically protects information utilized and exchanged prior to the finalization of agency policy (i.e., predecisional) and information that reflects opinions, recommendations, or a give-and-take exchange of information among officials (i.e., deliberative) (Harris 2009; Jensen 1999; Kennedy 2005; Weaver and Jones 1989).⁷ Importantly, deliberative statements included in the final agency policy adoption would not necessarily receive privilege protection (Weaver and Jones 1989). Factual information is generally not protected under the privilege (Jensen 1999; Miles 1988). Typically, in deliberative process privilege disputes, administrative agencies must provide a description of each withheld document, and a detailed explanation of why the asserted privilege allows the agency to withhold the document (Kennedy 2005; Narayan 2008).

Exemption 5 of the FOIA

Prior to the passage of the 1966 Freedom of Information Act, scholars, politicians, journalists, and lawyers expressed frustration and concern toward the perceived excessive secrecy of executive branch officials, which in turn produced a lack of access to information on internal government activity (Halstuk and Chamberlin 2006). Although the 1946 Administrative Procedure Act directed agencies to share records with interested parties, many argued that the APA legislation obstructed rather than facilitated access (S. Rep. No. 89-813 at 38, 1966). A key goal of FOIA was to enable greater government transparency through a “judicially enforceable public right of access” to executive branch agency documents and information (Halstuk and Chamberlin 512, 2006). Importantly, a number of exemptions recognize the need for agency discretion referenced by Justice Reed in Kaiser and allow agencies to withhold information requested through FOIA. For example, agencies could refuse to release documents on the grounds of national security or personal privacy (Halstuk and Chamberlin 2006; Mart and Ginsburg 2014). Exemption 5 of FOIA protects “interagency or intraagency memorandums or letters which would not be available by law to a party other than an agency in litigation with the agency” and incorporates the deliberative process privilege (Jensen 1999; Kennedy 2005; Miles 1988).⁸ In reference to Exemption 5, FOIA legislative history notes, “agency witnesses argued that a full and frank exchange of opinions would be impossible if all internal communications were made public” and the routine disclosure of internal communication would be akin to working within a “fishbowl” (H. Rep. No. 89-1497 at 31, 1966).

The privilege protects the same type of information in the FOIA context and the common law/civil discovery context. Also, in both contexts, courts must determine whether the withheld information properly reflects an exchange of information, opinions, and recommendations prior to the final adoption of agency policy. However, the identity of the requester, or the requester's specific need for the requested material, is not relevant in FOIA deliberative process privilege claims.⁹ If a court finds the agency has properly applied the privilege, requester need does not override the application of the privilege in the FOIA context (Jensen 1999). In the common law/civil discovery context, however, “courts engage in an ad hoc balancing of the evidentiary need [emphasis added] for allegedly privileged documents against the harm that may result from disclosure.”¹⁰ Therefore, “a demonstrated need for requested material” can override the deliberative process privilege in the common law/civil discovery context (Narayan 2008: 1196).

In either legal context, when ruling on deliberative process privilege claims, federal courts can make a variety of rulings. The court could rule that the privilege does not apply and that the agency must release all of their documents to the requester. Courts could also rule that the deliberative process privilege protects some of the requested material, but the agency must release some documents because of inaccurate application of the deliberative process privilege. In addition, judges can show complete deference to agency claims of deliberative process privilege and rule that all documents are privileged and that the agency litigant does not have to turn over any of the disputed documents.

Understanding Deliberative Process Privilege Litigation Outcomes

In FOIA disputes, courts exhibit substantial deference to agency withholdings (Verkuil 2002), particularly in disputes related to national security/foreign policy information (Mart and Ginsburg 2014). However, existing research on FOIA litigation does not examine how factors such as agency structure (i.e., independent compared to executive branch) and agency ideological character affect judicial deference in transparency disputes. Specifically, when weighing executive privilege claims, does an agency's closeness to presidential control affect case outcomes? In addition, similar to political actors (i.e., judges, members of Congress), federal agencies are also classified along a liberal-conservative ideological spectrum (Chen and Johnson 2015; Clinton and Lewis 2008). How does the ideology of agencies affect whether judges require agencies to release information to requesters? Finally, does the legal context (i.e., FOIA vs. Non-FOIA) affect judicial deference to agencies? For example, while the “Fast and Furious” litigation initiated by the House Oversight and Ethics Committee was ongoing, government watchdog organization Judicial Watch filed a FOIA request for similar documents sought by the Ethics Committee (Hattem 2016). Given that Exemption 5 is a “permissive” exemption, and not a mandatory exemption, agencies have substantial discretion in applying the privilege.¹¹ This discretion in application of the privilege confers judicial discretion in determining whether the privilege legally holds. Examination of privilege claims raised in various contexts across the federal executive branch helps further our current understanding of the ways in which federal judicial actors balance concerns of agency autonomy and government transparency.

Judicial Deference to Agency Claims of Deliberative Process Privilege

Deferring to an agency's decision to withhold documents affirms an agency's legal autonomy. However, requiring an agency to release documents the agency deems as privileged has a negative effect on the agency's ability to act as a gatekeeper for administrative information. Building from existing research on judicial decisionmaking in federal administrative agency disputes, I hypothesize that agency institutional structure, preference congruence between judges and agencies, and legal context will substantially affect whether courts defer to an agency's claim of deliberative process privilege.

Agency Institutional Structure

Judges will exhibit more deference to claims of deliberative process privilege raised by independent agencies and commissions, as compared to executive branch agencies.¹² Executive branch agencies typically include cabinet agencies and agencies located within the Executive Office of the President (EOP), whereas independent agencies are those that fall outside of the cabinet structure (Devins and Lewis 2008, Lewis 2003). Generally, the president has greater control over policy, hiring and firing, budgets, rulemaking, and reorganization in executive branch agencies (Lewis 2008; Moe 1982; Sheehan 1992; Wood and Waterman 1991). Independence, as it relates to federal agencies, can refer to independence of agency leadership and/or an independence of policy decisions (Selin 2015). Independent agencies are associated with party-balancing requirements, term limits, and other features that limit the president's ability to appoint and remove agency leadership (Aberbach and Rockman 2009; Lewis 2003). Exemption from Office of Management and Budget (OMB) review can also reduce presidential influence over administrative policy (Selin 2015). Although expertise and policy considerations influence agency/commission creation, scholars find that partisanship significantly affects whether agencies are insulated from executive influence. Specifically, when the president's party enjoys majority party control of Congress by larger margins, Congress is more willing to place administrative structures under greater presidential control (Lewis 2003: 127).

Courts may apply greater scrutiny to executive branch agencies because of concerns of decisionmaking driven by partisan politics (Sheehan 1992; Stephenson 2004). One may expect that judges would show increased deference to those administrative agencies closest to the president (Johnson 2014), given the potential for sanctions or retaliatory response from the executive (Howell 2003; Johnson 2003). However, federal judges will apply more scrutiny to executive branch agencies due to concern over the influence of partisan and political considerations in agency decisionmaking (Sheehan 1992), particularly in disputes involving the release of politically sensitive information. In the deliberative process privilege context, parties have accused officials working in agencies in close proximity to the president of withholding information that could lead to negative scrutiny of a presidential administration (Kamen and Itkowitz 2014). In the dispute over documents pertaining to the “Fast and Furious” operation, Representative Darrell Issa's (R-CA) investigatory panel argued, “the President and the Attorney General attempted to extend the scope of the Executive Privilege well beyond its historical boundaries to avoid disclosing documents that embarrass or otherwise implicate senior Obama Administration officials” (Kamen and Itkowitz 2014). Although the desire to prevent the release of information that could shine a negative light on administration officials and policy is understandable,¹³ it does not represent a valid legal basis for withholding documents under the deliberative process privilege. Given the possibility that political/partisan concerns of agency officials may motivate privilege claims, I expect federal judges to apply heightened scrutiny, and less deference, to executive branch agencies in deliberative process privilege claim disputes.

1. Hypothesis 1: Federal judges will be more likely to defer to claims of deliberative process privilege from independent agencies, and less likely to defer to privilege claims from executive branch agencies.

Judge-Agency Ideological Congruence

Ideological congruence between federal judges and agencies will affect whether judges defer to an agency's privilege claim. The ideological environment consistently influences outcomes in cases involving federal agency litigants (Crowley 1987; Fix 2014; Humphries and Songer 1999; Mart and Ginsburg 2014; Revesz 1997; Sheehan 1990). Similar or congruent policy preferences usually produce greater judicial deference to agencies in court. Crowley's (1987) analysis of U.S. Supreme Court review of administrative agencies finds that conservative justices show less support for social policy agencies (i.e., OSHA), which promulgate more liberal polices (as compared to economic policy agencies). In addition to the policy jurisdiction of the agency, the ideological direction of the agency decision affects judicial deference in federal courts (Crowley 1987; Humphries and Songer 1999; Kaheny and Rice 2010; Sheehan 1990).

Mart and Ginsburg's (2014) analysis of FOIA litigation provides important insight into the nuanced way partisan considerations emerge in transparency disputes. They find that majority-Republican panels in the D.C. Circuit were less likely to grant disclosure in Exemption 1 disputes involving national security or foreign affairs information (Mart and Ginsburg 2014). The finding of increased deference was not present outside of the D.C. Circuit. However, Mart and Ginsburg's (2014) analysis does not examine how the ideological tenor of the agency affects judicial deference to agency privilege claims.

Withholding documents differs from other policy outputs and decisions aligned with the mission of the agency. For example, an Environmental Protection Agency (EPA) regulation that requires lower carbon emissions from energy producers would be a “liberal” policy decision under most circumstances. However, the decision by an agency to withhold documents under the deliberative process privilege does not easily map upon the liberal-conservative ideological spectrum given the decreased connection to substantive agency policy outcomes. Instead, the ideology of the agency will influence judicial deference to deliberative process privilege claims. As noted by Clinton et al. (2012: 352), “government agencies are fundamentally political.” Agencies reflect factors such as organizational, personnel, policy, and mission-based preferences (Bertelli and Grose 2009; Cohen 1986; Nixon 2004), which many times align upon a liberal-conservative ideological spectrum (Clinton and Lewis 2008; Clinton et al. 2012). An agency's ideological orientation, in the context of agency's decision to withhold information, provides an important cue, or signal, which affects judicial assessment of the credibility of an agency's claim of privilege. Incongruent policy views between judges and the agency reduce the credibility of agency arguments to withhold documents and should result in judicial rulings requiring greater agency transparency. Specifically, liberal judges should apply more scrutiny to deliberative process privilege claims of conservative agencies; whereas conservative judges should be less deferential to liberal agencies that attempt to withhold documents under the deliberative process privilege.

1. Hypothesis 2: Federal judges will be more likely to defer to agency claims of deliberative process privilege when there is ideological congruence between the judge and the agency, and less likely to defer to privilege claims from ideologically incongruent agencies.

Legal Context

Finally, the legal context of the deliberative process privilege claim will also influence whether a federal judge will defer to a claim of privilege. In addition to the influence of judicial policy preferences on decisionmaking (Segal and Spaeth 2002), legal factors such as precedent, the standard of review, and the controlling statute have a significant impact on agency litigation outcomes (Humphries and Songer 1999; Raso 2015; Schuck and Donald Elliot 1990). FOIA requires courts to use de novo review when examining agency withholdings and “this more intense level of review means that individuals are more likely to be protected and agency actions are more likely to be reversed” (Verkuil 2002: 699). However, in his analysis of FOIA litigation outcomes between 1990 and 1999, Verkuil (2002) finds that only 10 percent of agency withholding decisions are reversed, which suggests that the expected correlation between scope of review and case outcome is not always present. Similarly, Mart and Ginsburg's (2014) examination of Exemption 1 litigation shows that district courts rule in favor of the federal government in about 74 percent of cases.

Questions remain on the effect of FOIA itself of judicial outcomes. One way to examine the effect of FOIA is to explore judicial outcomes on FOIA and non-FOIA privilege claims. Judges explain that agencies should construe all FOIA exemptions narrowly, and that the government bears the burden of proving that withheld documents are exempt (Miles 1988).¹⁴ Given that FOIA dictates de novo review (Kwoka 2013; Verkuil 2002) and FOIA's strong “presumption of disclosure,”¹⁵ I expect federal judges to exhibit less deference toward claims of deliberative process privilege in the FOIA context, and more deference in the common law/discovery context.

1. Hypothesis 3: Federal judges will be more likely to defer to agency claims of deliberative process privilege raised in the common law/discovery context, and less likely to defer to agency claims of privilege in the FOIA context.

Research Design: Data, Variables, and Method

To test my hypotheses, I used LexisNexis to collect an original data set of federal district court cases involving challenges to federal agency claims of deliberative process privilege. I collected district court cases adjudicated between 1994 and 2004, and cases that involved deliberative process privilege claims raised in the FOIA context and the common law/discovery context.¹⁶ The 11-year time-period allows for analysis across presidential administrations. The district court deliberative process privilege data set includes 172 court cases from published and unpublished court opinions. I code court cases according to the individual cabinet department or independent agency raising the deliberative process privilege claims.¹⁷

A small number of cases involved more than one cabinet department or more than one independent agency. In cases that involved more one than cabinet department or independent agency, the federal district judge assesses each agency's claim of deliberative process privilege individually. For example, if the Department of Treasury and the Department of Justice both claimed the deliberate process privilege over related documents in a single case, a federal district court could potentially uphold the Department of Treasury's claim of deliberative process privilege, while ruling against the Justice Department's claim of deliberative process privilege. Therefore, if a court case involved more than one cabinet department or independent agency claim of deliberative process privilege, I coded each court ruling on the agency privilege claim individually. This coding method produces 181 individual observations for cabinet department and independent agency deliberative process privilege claims.¹⁸

My empirical analysis focuses on whether the district court defers to the agency's claim of deliberative process privilege. I categorize the district court's decision on the agency's claim of privilege in three ways.¹⁹ If the court rules that none of the disputed documents are privileged, I categorize this case outcome as “No Privilege.” Eighteen percent of court outcomes are included in the No Privilege category. If the court rules some of the disputed documents are privileged and others are not, I categorize this outcome as “Partial Privilege.” Twenty-three percent of cases fall into the Partial Privilege category. Finally, if the court rules that the agency did not have to release any documents because all documents are privileged, I categorized this outcome as “Complete Privilege.” Approximately 60 percent of cases are included in the Complete Privilege category. The breakdown of case outcomes appears in Table 1.

Table 1. Judicial Deference toward Agency Claims of Deliberative Process Privilege

The dependent variable in my analysis is whether the district judge requires the agency to release none (No Privilege = 0), some (Partial Privilege = 1), or all (Complete Privilege = 2) of the requested documents where the agency claimed the deliberative privilege. Because my dependent variable is naturally unordered and has more than two discrete outcome categories, I estimate a multinomial logit model (Long 1997; Long and Freese 2005). The multinomial logit model used in this analysis will estimate the likelihood that an agency receives Partial Privilege or No Privilege from the presiding district judge, relative to the base category of Complete Privilege. The analysis will produce coefficient estimates for the Partial Privilege category and No Privilege category. The probabilities across all categories sum to one (Gordon 2012). Because a single court case can have more than one observation, I cluster observations according to case citation.²⁰ All tests are two-tailed.

Independent Variables

My three key independent variables are agency institutional structure, judge-agency ideological congruence, and the legal context of the deliberative process privilege claim. For agency structure, I create the indicator variable “Independent Agency”: agencies and commissions outside of the cabinet structure are coded as 1, and cabinet departments and bureaus are coded as 0.²¹ I use the Sourcebook of United States Executive Agencies to categorize independent agencies and cabinet agencies. I expect federal district courts to exhibit greater deference (i.e., Partial Privilege and Complete Privilege) to deliberative process privilege claims raised by independent agencies and commissions. Approximately 24 percent of the observations in the data set involve independent administrative agencies and commissions.

Developing an indicator of judge-agency ideological congruence is somewhat complex. I use Judicial Common Space (JCS) scores to measure district judge ideology (Boyd 2010). For the judges in my analysis, the JCS scores range from −0.502 to 0.578 (liberal to conservative).²² Although many previous analyses use the party of the president as a proxy for agency ideology, Clinton and Lewis (2008) use expert ratings and agency characteristics to develop a static measure of agency ideology for administrative agencies in existence between 1988 and 2005. The Clinton and Lewis (2008) scores are based upon expert assessment of whether the agency in question tends liberal, tends conservative, or neither consistently. Similar to JCS scores, negative values of the Clinton and Lewis (2008) scores indicate liberal agencies and positive values indicate conservative agencies. For the agencies in my analysis, the ideological scores range from −1.72 (Peace Corps) to 2.21 (Department of Defense).²³

I create an indicator variable, “Judge-Agency Ideological Congruence,” to capture the ideological congruence between agencies and federal district judges. Specifically, if the district judge's JCS score and the agency's Clinton and Lewis (2008) score both fall on the right of 0 of their respective ideological scales, this suggests that the agency and the judge are both somewhat conservative. When the judge and the agency both fall on the same side of their respective ideological scale, I code this as 1. If the agency and the district judges fall on opposite sides of their respective scales, the observation receives a coding of 0. Fifty-three percent of the judges and administrative agencies observations are ideologically congruent. Judges should be more likely to defer to claims of deliberative process privilege when judges are ideologically congruent with federal agencies.

Finally, I create an indicator variable, “FOIA” for my measure of legal context. I code the “FOIA” variable as 1 when deliberative process disputes occur in the context of a FOIA request for agency documents and 0 otherwise (i.e., common law/civil discovery). I expect federal district judges to show greater deference to deliberative process privilege claims raised in the non-FOIA context and less deference in the common law/civil discovery context. Sixty-five percent of cases involve FOIA disputes.

Control Variables

I control for a number of additional factors in my analysis that could influence case outcomes and are correlated with my key variables of interest. The status of the parties (i.e., government vs. nongovernment) can affect litigation outcomes (Black and Boyd 2012; Galanter 1974; McGuire 1995). I divide the litigants in the deliberative privilege cases into four categories and create four indicator variables: Business (23 percent), Individuals (45 percent), Public Interest (29 percent), and Government (4 percent). Corporations and companies comprise the business categories, whereas advocacy groups, unions, church organizations, interest groups, and so on comprise the public interest category. The government category includes government entities or officials (i.e., cities or legislators) suing for access to executive branch information, and the individual category includes individuals such as taxpayers, individuals in federal custody, and researchers seeking access to privileged documents. Because the public interest category involves many groups with repeat experience before the court (i.e., Judicial Watch and Tax Analysts), I expect administrative agencies to receive the least deference when those litigants appear in court, and the most deference when individuals petition for privilege documents. Because of advantages in resources, the likelihood of success of those in the business and government category should fall in between that of the individual and public interest categories.

Because agency type can affect judicial decisionmaking (Crowley 1987), I control for whether the agency's jurisdiction involves Social Regulatory policy (48 percent), Economic Regulatory policy (11 percent), or both Economic and Social Regulatory policy (23 percent). No regulatory policy jurisdiction serves as the base category (Clinton and Lewis 2008). I also control for whether the Department of Justice is the agency claiming privilege. A comparatively high percentage of cases (19 percent) involve the Department of Justice. Federal judges may be wary of releasing documents held by law enforcement agencies out of deference to ongoing agency investigations/litigation. Cases involving the Department of Justice as a litigant are coded as 1, and cases are coded as 0 otherwise.

Federal Magistrates decide the outcome in a number of cases in my analysis. Federal magistrate judges, who sometimes sit in district courts, are appointed by the federal judges in each district and serve 8-year renewable terms. Given that magistrates do not have life terms, and potentially have progressive ambition in terms of advancing upward in the federal court ladder, I expect magistrate judges to show increased deference to the executive branch in privilege cases and uphold agency withholding decisions. I create an indicator variable, Magistrate, that is coded as 1 for magistrate judges and 0 for non-magistrate judges.²⁴ Magistrate judges decide 20 percent of outcomes in the data set.

Presidents can have a significant influence over the transparency of the executive branch (Rosenberg 2008). Specifically, individual presidents can issue executives orders that can either increase or decrease access to government documents (Rosenberg 2008); therefore, I control for the identity of the presidential administration. The time-period under investigation involves the Clinton Presidency (1994–2000) and the Bush II Presidency (2001–2004). I include an indicator variable for the Clinton Presidency (60 percent) in all models.

I create an indicator variable to control for potential effects of the D.C. District. The D.C. District handles a substantial number of disputes involving federal administrative agencies, and 43 percent of the disputes in the deliberative process privilege dataset occurred in the D.C. District. Existing research suggests that D.C. Circuit judges may be more deferential to government litigants in transparency cases (Mart and Ginsburg 2014).

Finally, I also control for whether the opinion for the decision is Published. Although many studies of judicial decisionmaking understandably focus on published opinions, much of the work of district court judges results in unpublished opinions (Boyd 2015). Circuits develop varying norms and procedures for opinion publishing, and individual, institutional, and political factors influence whether district judges formally publish their decisions.²⁵ Examining both published and unpublished opinions provides a more representative analysis of judicial decisionmaking in executive privilege litigation. Forty-eight percent of the observations involve published decisions.²⁶ The descriptive statistics for the models used in the analysis appear in Table 2.

Table 2. Descriptive Statistics

Before presenting the results of the multinomial analysis, Table 3 provides a preliminary examination of deliberative process privilege outcomes through difference of means comparisons. To facilitate the analysis, I create a dichotomous variable by collapsing the No Privilege and Partial Privilege categories and comparing them against the Complete Privilege category. The results provide varying levels of support for my hypotheses. Judges are more likely to show independent administrative agencies complete deference in deliberative process privilege claims. In other words, judges are more likely to uphold an independent agency's decision to withhold documents under the deliberative process privilege. As previously stated, concern for partisan influence in transparency decisions may motivate increased judicial scrutiny, and less deference, for executive branch agencies. The results for Agency-Judge ideological congruence and FOIA are less precise and less conclusive. According to the estimates in Table 3, agencies that are ideologically congruent with federal judges and observations involving FOIA litigation are more likely to fall within the Complete Privilege category. However, these results are shy of the upper-boundary of traditionally accepted levels of statistical significance. Below, I discuss the results of the multinominal logit analysis using all three outcome categories.

Table 3. Difference of Means Analyses

^** p < 0.05.

Multivariate Analysis Results

Institutional and ideological factors substantially influence judicial decisionmaking in government transparency cases. In Table 4, the Complete Privilege category serves as the reference category for the Partial Privilege and No Privilege categories. According to the coefficient estimates in Table 4, district judges are less likely to show no deference (No Privilege) to independent agencies, relative to the reference category of Complete Privilege. Viewing the probability estimates in Table 5, district judges are less likely to require independent agencies to release all withheld documents as compared to executive branch agencies. The probability that an independent agency receives complete deference (Complete Privilege) is 0.866, whereas the probability that a district judge exhibits complete deference to an executive branch agency's claim of privilege is 0.633. In other words, an independent agency is +0.233 more likely to have their deliberative privilege claim upheld in full compared to an executive branch agency. For example, a claim of privilege made by an independent commission like the National Labor Relation Board has a greater likelihood of surviving judicial scrutiny when compared to a claim raised by the Department of Labor.²⁷ The change in probability for independent and executive branch agencies across privilege outcomes appears in Figure 1.²⁸

Table 4. Judicial Decisionmaking on Deliberative Process Privilege Claims: Multinomial Logit Estimates

Standard errors clustered by case citation. Tests are two-tailed. Complete Privilege is reference category.

Note: ^**p < 0.05; ^*p < 0.10.

Table 5. Predicted Probabilities: Probability Estimates for Deliberative Process Privilege Outcomes

Predicted probabilities calculated by holding categorical variables at their mode. ^**p < 0.05, ^*p < 0.1. “+” indicates choice with the highest probability.

Figure 1. Agency Structure and Deliberative Process Privilege Claim Federal District Court Rulings. [Color figure can be viewed at wileyonlinelibrary.com]

In their discussion of FOIA Exemption 1 usage and litigation, Mart and Ginsburg (2014: 726) decry not only what they describe as excessive secrecy from agencies, but also “excessive judicial deference.” If agencies assume that federal courts will be more likely to uphold agency withholdings and not disrupt agency decisionmaking, this could potentially lead to an even greater propensity of certain agencies to restrict access to internal information. Specifically, the expectation of greater judicial deference may promote less agency transparency.

The results in Table 4 also show that courts are also less likely to require ideologically congruent agencies to release all withheld documents (No Privilege), relative to the base category of Complete Privilege.²⁹ According to the estimates in Table 5, the probability that an incongruent agency receives a ruling allowing the agency to withhold all disputed documents is 0.457; whereas the probability that a court rules an ideologically congruent agency can withhold all disputed deliberative process privilege documents is 0.633. For example, compared to a liberal judge, a conservative judge reviewing a privilege claim from a conservative executive branch agency such as the Department of Defense (DOD) is more likely to defer to the DOD's deliberative process privilege claim. Figure 2 presents the predicted probability changes for agency ideological congruence across all three dependent variable categories.³⁰

Figure 2. Ideological Congruence and Deliberative Process Privilege Claim Federal District Court Rulings. [Color figure can be viewed at wileyonlinelibrary.com]

Contrary to my hypothesis, the results in Table 4 suggest that federal judges exhibit greater deference to agencies in the FOIA context; however, the estimates for the FOIA dispute variable are less precise than those for structural and ideological indicator variables. Specifically, according to Table 5, the probability that an agency falls within the Complete Privilege category is 0.663, whereas cases in the common law/civil discovery context have a probability of 0.443 of receiving complete deference. Figure 3 presents the predicted probability changes for the FOIA variable across all dependent variable categories. Although the burden of proof for withholding documents falls on the government in the FOIA context, these results seem to echo the findings by Mart and Ginsburg (2014) and Verkuil (2002) of substantial deference to the government in FOIA claims.³¹ Verkuil (2002: 716) speculates that judicial “skepticism” toward FOIA litigation, given the costs to agencies for compliance and the nature of the requesters, could contribute to the deference agencies receive in court. Also, the fact that the common law/civil discovery deliberative privilege can be overcome with a showing of need, unlike the FOIA deliberative privilege (Jensen 1999), could also contribute to this result. The estimates for the control variables are imprecise and do not reach the traditional levels of statistical significance.³² Specifically, according to the findings in Table 2, factors such as the identity of the requester and whether the case appeared in the D.C. District court do not affect the level of deference districts judges show agencies in deliberative process privilege litigation.

Figure 3. Legal Context and Deliberative Process Privilege Claim Federal District Court Rulings. [Color figure can be viewed at wileyonlinelibrary.com]

Discussion and Conclusion

In June 2016, President Obama signed the FOIA Improvement Act into law. In addition to new requirements concerning electronic access to previously requested material, and search fee charges, the law also codified the “presumption of disclosure” alluded to by jurists and promoted by the Obama Administration (Gerstein 2016). One key aspect of the new FOIA law attaches a sunset provision to Exemption 5 explicitly aimed at limiting agency use of the deliberative process privilege. According to the amended provision, “the deliberative process privilege shall not apply to records created 25 years or more before the date on which the records were requested.”³³ Interestingly, in the House congressional record for the FOIA Improvement Act, one House member stated (162 Cong. Rec. H3714, 2016):

we celebrate today the fact that we have made some milestones. Codifying in law the presumption of openness and, once and for all ending the deliberative process' unlimited length and reducing it to 25 years long, long after a President has left office, is a good start.

In the congressional record, the above statement is attributed to House member Darrell Issa (R-CA), who was chair of the Committee on Oversight and Government Reform during its legal battle with the Obama Administration in the “Fast and Furious” dispute. The influence of this recent FOIA amendment is still unfolding in the judicial arena; however, the limiting of agency discretion in the use of the deliberative process privilege would seemingly limit judicial discretion in deliberative process privilege litigation as well.

Individuals and organizations frequently request federal agencies to release internal agency documents for a multitude of reasons. Environmental organizations may request information related to changes to the Endangered Species List.³⁴ Researchers may request documents to assist in the completion of a book or research project.³⁵ Individuals involved in litigation with the government may request documents for discovery purposes (Tomlinson 1984). In addition, the Trump Administration has been subject to multiple FOIA requests from organizations such as the American Civil Liberties Union seeking access to White House visitor logs and information surrounding President Donald Trump's firing of FBI Director James Comey (Hensch 2017; Stempel 2017). If administrative agencies deny a request for documents, federal district courts play an essential role in determining whether requester access to the withheld information is warranted.

Although previous findings about the influence of agency structure on judicial decisionmaking have been mixed (Kaheny and Rice 2010; Sheehan 1992; Smith 2007; Unah 1997; Wendy et al. 1995), I find that federal district judges are more likely to defer to claims of deliberative process privilege raised by independent agencies and commissions. Elected officials designed independent agencies and commissions to have greater insulation from partisan politics and considerations. However, these agencies receive a measure of insulation during executive privilege litigation as well.

The results also suggest that the ideological character of the agency claiming privilege affects the degree of transparency judges require of administrative agencies. Shared or similar preferences, broadly, lend greater credibility to agency claims of privilege producing increased judicial deference.

I find modest evidence of the influence of FOIA on deliberative process privilege outcomes. Specifically, agencies were more likely to receive deference to their claims of privilege in the FOIA context. These results are contrary to my legal context hypothesis; however, they seem to support previous empirical results from FOIA litigation research (Mart and Ginsburg 2014; Verkuil 2002). Although not directly examined within this analysis, differences within the FOIA litigation environment (i.e., FOIA repeat player litigants, policy preferences of the requesters) could potentially influence aggregate case outcomes. Interestingly, the FOIA Improvement Act's amendment of the deliberative process privilege potentially increases the likelihood of differential outcomes between legal contexts in the future. The sunset provision applies to the deliberative process privilege in FOIA context; therefore, requesters in the non-FOIA context potentially have access to a greater range of internal agency content and information.

Concern over government transparency has been present since the founding period of the United States. According to James Madison, “a popular Government without popular information, or the means of acquiring it, is but a Prologue to a Farce or a Tragedy; or perhaps both” (Ginsberg et al. 2012: 1). When executive branch scandals arise, such as the “Fast and Furious” controversy, scholars, activists, and journalists usually issue a renewed called for government openness. Although the public's focus on government transparency waxes and wanes, consistent procedural access to government activity, mediated and monitored by federal judges, helps ensure that the public can accurately assess, and if necessary, respond to the actions of government officials. While federal judges, such as in the “Fast and Furious” case are willing to overrule agency judgment and require agencies to release withheld information (Johnson 2016), more often than not, federal judges defer to agency claims of deliberative process privilege.