Matthew Connelly's, professor and lead investigator for Columbia's History Lab, The Declassification Engine details and explains the work to create the largest database of declassified government documents and then the use of machine learning tools to analyze these documents to decipher what the government wants to keep us from learning and why. It is a troubling history with a clear need for rethinking our practices and demanding greater transparency from politicians in the digital age.

Across 10 chapters, Connelly explores different facets of the data, looking both historically and at specific issues. There are chapters focused on Pearl Harbor, the bomb, codes, the military industrial complex, surveillance and quasi-science. In each of these Connelly details important events, the timeline of information distribution and the almost continual legal re-interpretation or outright dismissal of laws intended to make information freely available. It is frequently noted that the more often secrecy is invoked, the more information is hidden. An example used several times in the book is that more information about UFOs has been declassified than details about nuclear bombs.

When America entered a war, traditional freedoms were restricted for the course of the war, Connely states this as the case, until World War II, when new technologies (the bomb) leading into the Cold War caused the US Government to continue a heightened sense of information security. To the point that the President wasn't always informed of the status of the nuclear arsenal. At least when those granted the permission to know the information did not leak it for their own purposes. That is one of the striking facts, that in many cases those reading classified information typically found details of the reports in the news media, or were the ones leaking it. One of Connelly's points is that too much is classified that does not need to be.

Most impactful to our future is chapter 10, Deleting the Archive. You can keep something secret if you destroy all evidence. As the National Archive is no longer accepting paper records, this means all materials they collect are now going to be digital with all the access issues and long term storage concerns. As in many of the less well-funded portions of the government, those managing the records have frequently and continue to need to do more with less.

A frightful look at the ways the dark state operates, but one that asks us to question some of the traditional narratives and emphasis that history is an additive process. The more information available, the more one is able to create a better, well rounded interpretation of the past.

<i>The Declassification Engine</i> is an interesting window into thinking about classified data, but I think it might be a book I find valuable for what it represents, rather than what it contains. The exponential proliferation of data, and how to process, store, maintain, and declutter it, is a rapidly growing problem in our digital age, and the opacity of <i>classified</i> data makes it even harder to manage, so it's wonderful that at least <i>someone</i> is thinking about that problem. The book is, at minimum, a signpost that Matthew Connelly's research and advocacy toward various government agencies exists, and we need voices speaking up to remind those agencies that over-classification goes against the best interest of the country (and of those very agencies!).

Unfortunately, the actual experience of reading the book was somewhat frustrating to me. Connelly has a historian's conviction that freedom of information is, if not the highest good, certainly a very high one, but often comes across as outright contemptuous of those government organizations that generate classified information. Perhaps he has seen so many examples of frivolous or sloppy classification that it's difficult to acknowledge that in some cases classifiers a) know what they're doing and b) may have a genuine need to classify a document, but he barely bothers to mention that some information is legitimately sensitive. It's easy to get up on a soapbox and say "look at these examples of things that are stupidly classified! Classification is bad!" while focusing on the most egregious examples, but it's harder to make a determination of, say, what percentage of classified documents are over-classified.

Connelly also has some moments that struck me as a little off. While decrying the money spent on classified research projects that turned out to be boondoggles (and which escaped public scrutiny that might have demonstrated their boondogglery sooner), he makes the argument that it's preferable to spend a smaller amount of money making it <i>look</i> like we're researching dumb stuff and trick adversaries into wasting a large amount of money trying to keep up with that research, such as an instance in which the U.S. "tricked" the Soviet Union into spending a ton of money weaponizing some disease (apologies, I don't have the book handy to look up which one). Um, are we treating it as a "win" that the Soviet Union ended up with a weaponized version of that disease? In other parts of the book, Connelly scoffs at much classified data (particularly from the State Department) as information that is not actually secret, as it's usually common knowledge or reported on by newspapers if one knows where to look, yet he extensively bemoans the inability of historians and scholars to get valuable information about government activities because it's been classified (and later "lost" or deleted). Which is it? Is classified data unimportant and not secret, or too-secret and very important? Yes, now I'm being uncharitable myself: it can be both, situationally; but it all seems an outgrowth from Connelly's general attitude that anybody who works for a classifying agency is a talentless hack and whatever they do is wrong, for one reason or another (even if those reasons contradict each other) -- Connelly essentially says exactly that, in putting forth the idea that intellectual lightweights gravitate to classified projects because their inadequacy can be easily hidden behind the wall of secrecy. It's a tiresome tone, and given that part of the purpose of this book must be to prod classifiers into a little more introspection when reviewing documents, you'd hope for a little more diplomacy on the author's part.

This timely and accessible book about the government's practices around secrecy and transparency when it comes to the classification and declassification of documents. Anyone paying even cursory attention to the news lately will realize how dysfunctional and disorganized the system is. Connelly shares how the current system came to be, while sharing implications for the present and future. While the topic may be heavy, his writing style is approachable. Highly recommend for anyone interested in US history and current events.

I find government secrets and classification of documents especially interesting, so knew this would be a perfect book for me.

It did not disappoint at all! It was thorough and well-researched. Highly recommended to anyone interested in intelligence and data!

The Declassification Engine was surprisingly engrossing for a book about data. No matter how much we think we know about the dark parts of our government, there is always more there. There are a lot of sayings about history and the present-history repeats itself or those who don’t know their history are doomed to repeat it. Everything here is not only a history of the loss of transperancy in our government, but a grim look at the present and the future. But, we do need to learn from our history. I’d recommend this book to anyone who is interested in how the goverment takes care of its secrets, history, or data (there are some of us out here) I’d recommend a slow paced read of this one.

The Declassification Engine aims to explain why the government needs to determine better standards when classifying confidential data while also creating a process for declassifying it in a timely manner. Connelly used data science to analyze declassified documents to find patterns in keywords that explain classifying trends. The project's goal is to potentially create an algorithm that could help the government tackle the overwhelming amount of collected data and automatically declassify information. Since WWII, the government has exponentially increased the amount of data they consider to be secret and allowed government agencies to access more private citizen's data than ever before, all at an astronomical financial cost. Every new administration promises full transparency and yet continues to classify even more information related to the presidency's goals, leading to the public being left in the dark as to what is being done for them, largely in the name of the US military or economy. Overall, Connelly makes the case for allowing researchers and the public access to declassified information in order to form a more complete historical picture and to help guide current and future policies. A good addition to any collection with a focus on data science, politics, government policies, or US history.

Many thanks to the publisher for access to this book. The author is uniquely qualified to speak on the concept of the "deep state", though not in a way that conspiracy theorists might think. This book goes in depth into government data, and how it is managed and classified and some of the implications of this classification. Highly thought provoking and hopefully a book that policy makers will take action on.

My thanks to both NetGalley and the publisher Knopf, Pantheon, Vintage and Anchor for and advanced copy of this new history book that looks at the untold history of the United States as shown through declassified records and documents.

Secrets are both a big business and a sign of power. Governments spend huge amounts of money trying to get secrets, hide their own secrets, and even more destroying the secrets they are supposed to share. People in power use their levels of clearance like keys to the executive washroom, lording it over fellow workers, and underlings, I would tell you but you aren't cleared for this. This access does them well in life, allowing people to enter the world of boardrooms and consulting with almost a preset salary. Governments keep secrets for many reasons, to protect the nation, would be cited a lot, but to protect it from what is the real question. Matthew Connelly, professor of international and global history writes about the business to classification, and even more about difficulty in declassifying in the book The Declassification Engine: What History Reveals About America's Top Secrets a reveling look at America its secrets, the whys and what are really being hidden.

The book begins with a project that seems to have a lot of interest, using data science to interpret declassified documents, and using those documents to get a better grasp of what is being classified, and why. There is much push back, threats, NDA's until finally funding is secured for History Lab project, which is better explained by the book than myself. Readers are then introduced to an America that before World War II prided itself on transparency. Yes during wartime, secrecy would be instilled, sure even the Constitution might be bent a tad, but at the end, things, and America would go back to normal. Except after World War II. Was it the situation, the global powers being up for grab, the money that was being tossed around, the fact that people in power liked controlling the power. Or was it the hubris that generals and governments liked more to be feared, than to be known as making mistakes. Even worse putting American lives in danger for what seems like now totally stupid ideas.

The book is extremely well written and sourced, with information on every page that makes the reader go what the heck. Stories of Churchill telling a dinner party about what he saw the Americans doing when Pearl Harbor was announced. The numerous trials of the effects of radiation on American citizens without their consent. My favorite was surgically altering a cat to be a transmitter to hear conversations in a park, but the cat didn't have any interest in going where it was told, and eventually was run over by a cab. Stories of generals going from the Pentagon to the boardroom, the Joint Chiefs not saving any of their notes for the national archive. The numerous atomic incidents and accidents that could have caused a world war. This stories are found in declassified documents, which also points out the problems and th industry of keeping secrets, different departments decided what should be classified, and what shouldn't be.

An extremely interesting book, especially for lovers of history and people who like to know what is really going on. The problem with something like this is that it almost makes conspiracy theory people seem right. However it seems the biggest secret in government is how many mistakes they make, and would rather be thought of as doing bad things for America and competent rather than doing stupid things constantly without anyone calling them on it. A book that covers many subjects, and does a great job in keeping the reader interested, and informed.