Covert US strikes in Pakistan, Yemen and Somalia – our methodology

Over the past two years, the Bureau has made an extensive study of reports of US covert activities in Pakistan, Yemen and Somalia.

We present our findings for each country in two key ways: firstly datasets, which comprise narrative accounts of each reported action, a casualty count of the number of individuals reported killed or wounded in that action, and all sources used in researching each action. Secondly, we produce overall casualty counts, which add up the total reported deaths caused by drone strikes and other covert actions in each country.

Our Pakistan datasets cover CIA drone strikes in that country since 2004. For Yemen and Somalia, the datasets include other US covert actions including airstrikes, missile attacks and ground operations, and we also note significant events to provide context for US actions.

The data is collected and researched by a team of Bureau journalists. For each reported US attack, the Bureau seeks to identify the time, location and likely target, and to present as clear a description as possible of what took place during the event. We also seek to identify the numbers of those reportedly killed and injured, and to ascertain when possible whether they were alleged militants, or civilians. Wherever possible, we include other information on casualties, such as name, gender, age, tribal affiliation and other identifying aspects. All this information is included in the datasets.

We present our casualty counts – the total numbers of those reported killed and wounded – in the right-hand panel of all articles in the Bureau’s Covert Drone War section.

This article breaks down our approach into two sections: our sources and our methodology.

Our sources

Every strike or event covered in our datasets contains active links to all news reports, statements, documents and press releases which we have used as our sources. We also incorporate images and video clips relating to specific events. The Bureau’s dataset is active: both the timelines and casualty counts change according to our best present understanding. New information on a particular strike or action can emerge months or even years after an event.

The most comprehensive public information on casualties generally lies in the thousands of press reports filed by reputable national and international media outlets. The bulk of our sources are in English but in addition, when they are drawn to our attention by local contacts, we also sometimes incorporate reporting in Urdu (for Pakistan) or Arabic (for Yemen) as well as other languages when relevant.

In the case of Pakistan, the CIA does not officially acknowledge or comment on its drone campaign, and the Islamabad government does not publish a casualty count. In early 2013 senior administration figures – including the president and CIA director-designate John Brennan – indicated the government may be preparing to release civilian casualty figures for the conflict, but at of the time of writing no such information was available.

In Yemen, the government sometimes publishes information about ‘air strikes’ with no indication as to who carried them out. The government has also occasionally claimed responsibility for attacks that it is unlikely its rickety air force could have carried out, such as night-time strikes, or those on moving vehicles. In Somalia, very little is published at all. The US government has also sometimes obliquely referred to attacks: for example, in March 2012 attorney general Eric Holder made a speech at Northwestern University’s School of Law in which he defended the targeted killing of US citizens on foreign soil by the Obama administration.

The majority of our information stems from news reporting. Commonly cited international media sources include CNN, MSNBC, ABC News, Reuters, the BBC, Associated Press, the Guardian, the Telegraph, the Independent, TIME, the Wall Street Journal, the Washington Post, the New York Times, the Los Angeles Times, Fox News, the Nation, the Atlantic, Salon, Xinhua, Army Times, Bloomberg, AFP, NPR, Al Jazeera, and Al Arabiya.

Other international sources include the New America Foundation, Critical Threats, Long War Journal, Al Akhbar, Jamestown Foundation, Jihadology, Empty Wheel, Wired, WikiLeaks, the UN, Reprieve, Human Rights Watch, the American Civil Liberties Union, and Amnesty International.

Pakistani media sources include Dawn, Express Tribune, The Nation, Jang, Geo TV and The News International. The Bureau also includes reports from the Pakistan Observer but as we do not view casualty reports from that publication as credible at present, we do not incorporate their figures into our casualty counts.

Somali media sources include Somalia Report, Africa Confidential, AllAfrica and Bar Kulan. Press TV has also reported on a number of ‘US drone strikes’ in the country. However following a Bureau investigation we do not view these reports as credible. Their claims of US drone strikes in Somalia are recorded separately (in Press TV’s Somalia claims 2011-12) and are not included in our casualty counts.

Yemen media sources include Yemen Post, Yemen Times, Yemen Observer, Saba News Agency, Gulf News, Waq-al-Waq, Al-Shorfa and Akhbar al-Youm.

Other sources

In Pakistan, the Bureau has carried out field investigations into possible civilian deaths on three occasions. It has incorporated further sources, including the fieldwork of credible researchers (for example Stanford and New York universities) and evidence filed in legal cases brought in Pakistan and elsewhere on behalf of civilian drone victims.

Leaked US intelligence reports and WikiLeaks diplomatic cables deal directly with specific drone attacks or airstrikes in all three countries. These are cited as sources by the Bureau where relevant. We have also incorporated pertinent material from research papers, books and articles by journalists, academics, politicians and former intelligence officers.

On many occasions, there is a reasonable consensus between sources. Where contradictory accounts occur, we strive to speak with particular journalists and sources about their reports to clarify discrepancies. But where these discrepancies remain we include the contrasting accounts in the datasets’ narratives of each strike.

We have also endeavoured to publish material which has not previously been in the public domain. For instance, we reveal in English the full names of all 44 civilians, including 22 children, killed by a December 2009 US airstrike on al-Majala, Yemen. The Bureau translated the details from an original report carried out by a Yemen parliamentary commission.

We also cross-reference events with the work of other organisations who record drone strikes in Pakistan, particularly the New America Foundation and the Long War Journal.

Our methodology

How we list the strikes:

We have given each strike a unique code. This is a sequential number, with a letter prefix.

For Pakistan, the strikes are numbered with the prefix B for the Bush years, or Ob for Obama years. On occasion, a single source reports an incident that may – or may not – be a drone strike. When this occurs, we do not include them in our casualty counts, but we do include them in our timelines, adding the suffix ‘C’ to the strike’s code (see Ob28C, Ob39C and Ob130C, for example). Ob0 is a non-lethal Pakistani operation supported by a US drone, which we include in the timeline for reference purposes.

For the datasets in Somalia and Yemen, each strike is numbered with the prefix of SOM or YEM respectively.

By ‘strike’, we refer to a missile or set of missiles fired in at a single location in a short time window. Where missiles hit more than an hour apart, we count these as separate strikes. Where drones hit locations more than a couple of miles apart we also count these as separate strikes, even when they take place in quick succession.

Confirmed/possible US actions:

Monitoring events in Yemen presents a particular challenge. The CIA, US special forces, and Yemeni air force all carry out strikes, and a report in January 2013 claimed the Saudi air force had also bombed the country. It is often unclear who is responsible for attacks or even whether it is the work of a manned plane or a drone – although airstrikes are routinely described as ‘drone’ strikes in news reporting.

Where any US government source or named senior Yemeni source acknowledges a strike has been carried out by a US drone, we include it in our dataset as a ‘confirmed’ drone strike. Where three different types of local source – such as government officials, tribal sources, or eyewitnesses – all report a strike was the work of a drone, we will also report this as ‘confirmed’. We classify all other reports of drone strikes as ‘possible’ drone strikes.

How we reconcile the material

Even within a single report there can be contradictory information on how many individuals were killed or what the target was – for example the report might say it was ‘either a house or a vehicle’ that was hit. Reconciling accounts from multiple sources can be even more difficult.

Where credible sources differ over how many people were killed we provide a minimum and maximum count of the number of people reported killed. This is why our casualty counts are a range – and over time the cumulative difference between minimum and maximum reported casualties becomes quite pronounced.

However, we do not simply present a range from the lowest to highest reported casualties. Where early reports are updated, for example when the death toll rises as people die of their injuries, or where an early high estimate is lowered as rescue work progresses, we use the updated total in our casualty count.

It is quite common for only one source to report civilian casualties, or for ambiguous reporting to hint at civilian casualties, for example by referring to the dead only as ‘local tribesmen’ or ‘people’ rather than the more frequently used ‘militant’.

Where the reporting is vague but appears to indicate civilian casualties, we will include the line ‘Possible reported civilian casualties’ in that strike’s casualty figures in the Timeline.  This aims to act as a marker for future investigators though has no impact on our casualty counts. Where the reporting is more specific, but conflicts with other reports or is from a single source, we use the formula 0-X in our count of civilian deaths, with X referring to the highest reported number of civilian casualties.

This ensures that the minimum total number of reported civilian deaths is unchanged but the maximum total incorporates these possible civilian deaths. Our own charts and other strike visualisations always use only our minimum reported casualty counts.

What are our definitions of who has been killed and injured?

Of more than 3,000 people the Bureau has identified as being reported killed in US covert attacks since 2002, fewer than a third have so far been identified by name. We do not know who the majority of the dead are. However, field reports from journalists, government officials and militant sources often provide clear suggestions that they are allegedly militants.

A house or compound might be identified as being linked with a particular militant faction. A destroyed vehicle may be claimed to have contained militants. In such cases – and where the Taliban in Pakistan, al Shabaab in Somalia, al Qaeda in the Arabian Peninsula or Ansar al-Sharia in Yemen, have not confirmed an attack – we refer to those injured and killed as ‘alleged militants’. The bulk of those killed fall into this category.

We use the term ‘militant’ to describe all organised, named groups that bear arms and that are not part of Pakistani, Somali, or Yemeni military, police, paramilitary or militia forces. In the case of Somalia, UN and African Union forces are also on the ground, and are distinguished from militants.

However, as academics at Columbia Law School’s Human Rights Clinic noted in a 2012 report on counting drone strike casualties, the term ‘militant’ is politically and emotively charged, yet has no accepted legal definition. Although the Bureau records reports of alleged militant casualties in the narratives describing each strike, we do not keep a specific count of reported ‘militant’ deaths. We keep tallies only of reported civilian deaths and of total casualties.

We report all instances where civilians are credibly reported to have been killed or injured, using the 0-X formula described above where accounts vary as to whether civilians or militants were killed. It is fairly common for reporting to refer to the dead as ‘people’, ‘local tribesmen’ or ‘family members’ rather than specifically referring to civilians. At times the indication of civilian casualties is clearer – such as when reports refer to ‘militants and local tribesmen’ being killed. For all of these formulations, we will include possible civilian casualties in our casualty count using the 0-X formula.

This approach has been borne out by field researchers, including those commissioned by the Bureau, who have often identified civilian deaths in incidents where the initial reporting referred to the dead as ‘people’ or ‘tribesmen’, or indicated uncertainty over the identity of the dead.

When reporting on casualties among children we employ the United Nations-designated age range of 0-17 years inclusive. Where possible we report the child’s age.

All times and dates used are harmonised to Pakistan, Somalia or Yemen local time.

* Last modified February 2013. This was to clarify our casualty counting process and how we count civilians, as well as to include recent developments. Previously modified in March 2012 to reflect new data sets for Yemen and Somalia.