Our methodology
Since 2010, the Bureau has made an extensive study of reports of US covert activities in Pakistan, Yemen and Somalia.
We present our findings for each country in timelines comprised of narrative accounts of each reported action, a casualty count of the number of individuals reported killed or wounded in that action, and all sources used in researching each action. Secondly, we produce spreadsheets that are available online, that can be downloaded.
Our Pakistan datasets cover US drone strikes in that country since 2004. For Yemen, Afghanistan and Somalia, the datasets include other US covert actions including airstrikes, missile attacks and ground operations, and we also note significant events to provide context for US actions.
The data is collected and researched by a team of Bureau journalists. For each reported US attack, the Bureau seeks to identify the time, location and likely target, and to present as clear a description as possible of what took place during the event. We also seek to identify the numbers of those reportedly killed and injured, and to ascertain when possible whether they were alleged militants, or civilians. Wherever possible, we include other information on casualties, such as name, gender, age, tribal affiliation and other identifying aspects. All this information is included in the timelines.
This explainer breaks down our approach into two sections: our sources and our methodology.
Our sources
Every strike or event covered in our timelines contains active links to news reports, statements, documents and press releases, which we have used as our sources. On occasion information has come to the Bureau directly from sources who we present in the timelines as transparently as possible. We also occasionally have drawn information from terrorist propaganda, such as Voice of Jihad in Afghanistan or Inspire in Yemen. We do not link to these sources.
Images and video clips relating to specific events have been incorporated into our timelines, when such media area available. The Bureau’s dataset is active: the timelines and spreadsheets change according to our best present understanding. New information on a particular strike or action can emerge months or even years after an event.
If you have any information that we may have missed or think we have got something wrong please contact us at [email protected].
The most comprehensive public information on casualties generally lies in the thousands of press reports filed by reputable national and international media outlets. The bulk of our sources are in English but in addition we also sometimes incorporate reporting in Urdu (for Pakistan), Pashto and Dari (for Afghanistan) or Arabic (for Yemen) as well as other languages when relevant.
The US government started publishing its own estimates of how many people it has killed in counter-terrorism strikes “outside areas of active hostilities”. This phrase was presumed to refer to Pakistan, Yemen, Somalia, and Libya (not including the NATO bombing campaign in 2011 and the US strikes against ISIS around Sirte in 2016) though no US government official would confirm specifically where was outside active hostilities.
The US data release was part of an executive order that included a commitment that the US would continue to publish such figures annually. However it remains to be seen if this executive order will survive into the Trump administration.
Unfortunately the first data release, on July 1 2016 comprised of the total number US counter-terrorism strikes in all countries outside of areas of active hostilities for all years between January 2009 and December 2015, as well as the minimum and maximum estimates for the total number of combatants and non-combatants killed.
This degree of aggregation makes serious analysis against the Bureau’s figures impossible. The US did release figures of 2016 in the first weeks of January 2017 – read this analysis of the 2016 data release to see how it stacked up against the Bureau’s figures.
In the case of Pakistan, the CIA does not officially acknowledge or comment on its drone campaign, and the Islamabad government does not publish a casualty count. The US military has conducted one strike in Pakistan, on May 21 2016, which was publicly acknowledged and discussed openly by administration officials including President Barack Obama.
The Yemeni government in the past would occasionally published some information about “air strikes” with no indication as to who carried them out. Sanaa did occasionally claim responsibility for attacks that it is unlikely its rickety air force could have carried out, such as night-time strikes, or those on moving vehicles. The US government historically would not discuss its strikes on the record, though much detail has been leaked to journalists over the years. However in February 2016 the US geographical command in charge of operations in Yemen, Central Command (Centcom) began publishing press releases detailing its air actions in Yemen. It has continued to do so, confirming previously reported strikes and occasionally reporting an air attack for the first time.
Centcom’s reports do not coincide with all reported US strikes in Yemen. The extra attacks are either CIA strikes or misattributed attacks by the Saudi Arabia-led coalition of countries, which has been bombing Yemen since March 2015.
In Afghanistan, the US-led NATO headquarters in Kabul, Resolute Support, often provide the Bureau with details of its air attacks in the country. In the past this did include casualty data though this information has since stopped being provided.
The US Air Force releases a monthly data sheet, which summarises its operations in Afghanistan. It includes both strikes and other air sorties, such as intelligence, surveillance and reconnaissance flights. The Bureau records these figures alongside the Bureau’s own data in its Afghan spreadsheet. Reports in early 2017 however cast some doubt on how comprehensive the US Air Force data actually is.
The majority of our information stems from news reporting. Commonly cited international media sources include CNN, MSNBC, ABC News, Reuters, the BBC, Associated Press, the Guardian, the Telegraph, the Independent, TIME, the Wall Street Journal, the Washington Post, the New York Times, the Los Angeles Times, Fox News, the Nation, the Atlantic, Salon, Xinhua, Army Times, Bloomberg, AFP, NPR, Al Jazeera, and Al Arabiya.
Other international sources include the New America Foundation, Critical Threats, Long War Journal, Al Akhbar, Jamestown Foundation, Jihadology, Empty Wheel, Wired, WikiLeaks, the UN, Reprieve, Human Rights Watch, the American Civil Liberties Union, and Amnesty International.
Pakistani media sources include Dawn, Express Tribune, The Nation, Jang, Geo TV and The News International. The Bureau also includes reports from the Pakistan Observer but as we do not view casualty reports from that publication as credible at present, we do not incorporate their figures into our casualty counts.
Somali media sources include Somalia Report, Africa Confidential, AllAfrica and Bar Kulan. Press TV has also reported on a number of ‘US drone strikes’ in the country. However following a Bureau investigation we do not view these reports as credible. Their claims of US drone strikes in Somalia are recorded separately (in Press TV’s Somalia claims 2011-12) and are not included in our casualty counts.
Yemen media sources include Yemen Post, Yemen Times, Yemen Observer, Saba News Agency, Gulf News, Waq-al-Waq, Al-Shorfa and Akhbar al-Youm.
Afghan media sources include Pajhwok, Khaama Press, TOLO News and Afghan Islamic Press.
Other sources
In Pakistan, the Bureau has carried out field investigations into possible civilian deaths on three occasions. It has incorporated further sources, including the fieldwork of credible researchers (for example Stanford and New York universities) and evidence filed in legal cases brought in Pakistan and elsewhere on behalf of civilian drone victims.
Leaked US intelligence reports and WikiLeaks diplomatic cables deal directly with specific drone attacks or airstrikes in all three countries. These are cited as sources by the Bureau where relevant. We have also incorporated pertinent material from research papers, books and articles by journalists, academics, politicians and former intelligence officers.
On many occasions, there is a reasonable consensus between sources. Where contradictory accounts occur, we strive to speak with particular journalists and sources about their reports to clarify discrepancies. But where these discrepancies remain we include the contrasting accounts in the datasets’ narratives of each strike.
We have also endeavoured to publish material which has not previously been in the public domain. For instance, we reveal in English the full names of all 44 civilians, including 22 children, killed by a December 2009 US airstrike on al-Majala, Yemen. The Bureau translated the details from an original report carried out by a Yemen parliamentary commission.
We also cross-reference events with the work of other organisations which record drone strikes in Pakistan, particularly the New America Foundation and the Long War Journal.
Our methodology
How we list the strikes:
We have given each strike a unique code. This is a sequential number, with a letter prefix.
For Pakistan, the strikes are numbered with the prefix B for the Bush years, or Ob for Obama years. On occasion, a single source reports an incident that may – or may not – be a drone strike. When this occurs, we do not include them in our casualty counts, but we do include them in our timelines, adding the suffix ‘C’ to the strike’s code (see Ob28C, Ob39C and Ob130C, for example). Ob0 is a non-lethal Pakistani operation supported by a US drone, which we include in the timeline for reference purposes.
For the datasets in Somalia, Afghanistan and Yemen, each strike is numbered with the prefix of SOM, AFG or YEM respectively.
By ‘strike’, we refer to a missile or set of missiles fired in at a single location in a short time window. Where missiles hit more than an hour apart, we count these as separate strikes. Where drones hit locations more than a couple of miles apart we also count these as separate strikes, even when they take place in quick succession.
Confirmed/possible US actions:
Monitoring events in Yemen presents a particular challenge. The CIA, US special forces, the Yemeni air force and the UAE and Saudi-led anti-Houthi coalition have all carried out strikes in Yemen at one time or another. In 2016 Emirati aircraft reportedly hit al Qaeda targets near the southern Mukalla port city and a report in January 2013 claimed the Saudi air force had also bombed the country. It is often unclear who is responsible for attacks or even whether it is the work of a manned plane or a drone – although airstrikes are routinely described as ‘drone’ strikes in news reporting.
Where any US government source or named senior Yemeni source acknowledges a strike has been carried out by a US drone, we include it in our dataset as a ‘confirmed’ drone strike. Where three different types of local source – such as government officials, tribal sources, or eyewitnesses – all report a strike was the work of a drone, we will also report this as ‘confirmed’. We classify all other reports of drone strikes as ‘possible’ drone strikes.
The Bureau has notably undercounted US air strikes in Afghanistan because most air attacks go unreported in open sources including news media. This is because news of the strikes do not reach the journalists or sometimes a days’ news agenda is so full with other important stories, Afghan journalists do not have the time or inclination to cover yet another possible US air operation.
There are occasions however where incidents have been attributed to US air attacks by tenuous sources. The Bureau therefore includes only strikes reported by named Afghan sources, or named and unnamed US sources, in its spreadsheet. Attacks reported by unnamed or vague sources are listed in the timeline as C strikes, for further investigation.
How we reconcile the material:
Even within a single report there can be contradictory information on how many individuals were killed or what the target was – for example the report might say it was ‘either a house or a vehicle’ that was hit. Reconciling accounts from multiple sources can be even more difficult.
Where credible sources differ over how many people were killed we provide a minimum and maximum count of the number of people reported killed. This is why our casualty counts are a range – and over time the cumulative difference between minimum and maximum reported casualties becomes quite pronounced.
However, we do not simply present a range from the lowest to highest reported casualties. Where early reports are updated, for example when the death toll rises as people die of their injuries, or where an early high estimate is lowered as rescue work progresses, we use the updated total in our casualty count.
It is quite common for only one source to report civilian casualties, or for ambiguous reporting to hint at civilian casualties, for example by referring to the dead only as ‘local tribesmen’ or ‘people’ rather than the more frequently used ‘militant’.
Where the reporting is vague but appears to indicate civilian casualties, we will include the line ‘Possible reported civilian casualties’ in that strike’s casualty figures in the Timeline. This aims to act as a marker for future investigators though has no impact on our casualty counts. Where the reporting is more specific, but conflicts with other reports or is from a single source, we use the formula 0-X in our count of civilian deaths, with X referring to the highest reported number of civilian casualties.
This ensures that the minimum total number of reported civilian deaths is unchanged but the maximum total incorporates these possible civilian deaths. Our own charts and other strike visualisations always use only our minimum reported casualty counts.
What are our definitions of who has been killed and injured?
Of more than 3,000 people the Bureau has identified as being reported killed in US covert attacks since 2002, less than a third have so far been identified by name. We do not know who the majority of the dead are. However, field reports from journalists, government officials and militant sources often provide clear suggestions that they are allegedly militants.
A house or compound might be identified as being linked with a particular militant faction. A destroyed vehicle may be claimed to have contained militants. In such cases – and where the Taliban in Pakistan, Islamic State in Afghanistan, al Shabaab in Somalia, al Qaeda in the Arabian Peninsula or Ansar al Sharia in Yemen, have not confirmed an attack – we refer to those injured and killed as ‘alleged militants’ or ‘alleged terrorists’. The bulk of those killed fall into this category.
We use the term ‘militant’ to describe all organised, named groups that bear arms and that are not part of Pakistani, Somali, or Yemeni military, police, paramilitary or militia forces. In the case of Somalia, UN and African Union forces are also on the ground, and are distinguished from militants.
However, as academics at Columbia Law School’s Human Rights Clinic noted in a 2012 report on counting drone strike casualties, the term ‘militant’ is politically and emotively charged, yet has no accepted legal definition. Although the Bureau records reports of alleged militant casualties in the narratives describing each strike, we do not keep a specific count of reported ‘militant’ deaths. We keep tallies only of reported civilian deaths and of total casualties.
We report all instances where civilians are credibly reported to have been killed or injured, using the 0-X formula described above where accounts vary as to whether civilians or militants were killed. It is fairly common for reporting to refer to the dead as ‘people’, ‘local tribesmen’ or ‘family members’ rather than specifically referring to civilians. At times the indication of civilian casualties is clearer – such as when reports refer to ‘militants and local tribesmen’ being killed. For all of these formulations, we will include possible civilian casualties in our casualty count using the 0-X formula.
This approach has been borne out by field researchers, including those commissioned by the Bureau, who have often identified civilian deaths in incidents where the initial reporting referred to the dead as ‘people’ or ‘tribesmen’, or indicated uncertainty over the identity of the dead.
When reporting on casualties among children we employ the United Nations-designated age range of 0-17 years inclusive. Where possible we report the child’s age.
All times and dates used are harmonised to Pakistan, Somalia, Afghanistan or Yemen local time.
* Last modified in February 2017 to clarify how the changing situation in Yemen has affected our counting processes. It was also amended to include details of the US government’s publishing its own casualty estimates from counterterrorism strikes. And it was modified to reflect our new Afghanistan dataset. It was previously changed in February 2013 to clarify our casualty counting process and how we count civilians, as well as to include recent developments, and in March 2012 to reflect new data sets for Yemen and Somalia.
