Old-style knowledge systems hamper spooks too

Intelligence agencies are not very bright in the way they process the information they gather, according to an insider view. The reason is that they employ old-style top-down internal communication systems rather than the search and knowledge-sharing Net tools many people use in their homes.
Matthew Burton wrote to me earlier this year, as I reported in Reducing the fear of being wrong - for spooks too. At the time I wasn't sure how serious he was about revealing the shortcomings of spook software systems. However, he has now produced an article for Studies in Intelligence, which you can preview here.
In that he writes about the high expectations he had on joining the Defense Intelligence Agency in January 2003, and excitement he felt at using their systems.

The reality was a colossal letdown. Intelink-the network that was designed to negate the physical distance that separates intelligence agencies and their customers-seems anachronistic in comparison to the Web we use at home. As a technology enthusiast with seven years of Web development experience, I was appalled that the rest of the world had access to better online tools than did the US national security structure-the very creator of "online." Our search engines return results reminiscent of the pre-Google Web. Our online personnel directories are useless. Agencies and combatant commands use a hodgepodge of incompatible discussion and chat tools, furthering our tendency to speak only with those in our own buildings.

The reasons for failure, cited by Matthew, will bring additional re-assurance to those who argue for bottom-up development of knowledge systems. It seems blogs have potential benefits for spooks as well as home or business users.

Why is the Web so much more user-friendly than Intelink? Did the late-1990s Silicon Valley boom propel private industry ahead of the government? Do our unique security requirements make great tools inaccessible to us?
The answer is much simpler. The Web is user-friendly because its users control its content. Intelink's pages are published by technicians who neither use the system for research nor understand its content. The Web's 900 million users can instantly say whatever they like in their own personal publishing space; on Intelink, content is restricted to what our agencies call "official products," and several layers of supervisors, systems administrators, and Web programmers stand between intelligence officers and their online world.

Matthew explains that this hierarchial system make it difficult for analysts in different agencies to develop trusting relationships and so share intelligence. Why not take a lesson from the wider community of Internet users, and make use of systems in which each user has a personal publishing capability, strong cross linking, together with powerful search engines? He believes this could be done on an internal intelligence system while protecting identities.

Analysts and collectors understand their information better than Web programmers and technical editors, so we know what links to place where. And because the quality of a personal home page would reflect upon its owner, we would have motivation to see that our pages provide good information for readers.
A web-like structure would take some time to realize, but the benefits would be enormous. Imagine having tools that could spot emerging patterns for you and guide you to documents that might be the missing piece of evidence you're looking for. Analytical puzzles, such as terror plots, are often too piecemeal for individual brains to put together. Having our documents aware of each other would be like hooking several brains up in a line, so that each one knows what the others know, making the puzzle much easier to solve. The moral is that logical dots are easier to connect if the virtual ones are already connected.

For those worried about privacy and the intrusive nature of intelligence gathering, the situation revealed by Matthew may be reassuring. For those who thought security measures were underpinned by the best technology available, it is rather worrying.
It seems to me the lesson yet again is that you get systems that reflect the culture of the organations in which they are installed, and it may take a lot to bring change, even in the direst circumstances.
According to Matthew experiments are now taking place with blogs, and he sets out some of the steps needed to promote their use and gain acceptance among intelligence officers.

Once blogs have been deployed, managers must encourage their employees to use the new technologies. They should not see blogging as a waste of time, as dilly-dallying, or as haphazard intelligence. Instead, they should view it as a venue for brainstorming and relationship building. Active offices will see the benefits. Their staffs will be in the vanguard of establishing strong working relationships with other agencies and offices, reaping the benefits of increased contacts and access to information. Their intelligence products will accommodate customers' desire for details. And their work areas will become more vibrant atmospheres that buzz with new ideas.

He concludes:

Over the past four years, policymakers and the press have endlessly underscored the need for Intelligence Community agencies to work more closely together. Few of us in the IC can say they are wrong. But even fewer of us can say we have the necessary tools for doing so. The Commission on the Intelligence Capabilities of the United States Regarding Weapons of Mass Destruction understood this problem and recommended the creation of new technologies to aid IC communication. What it did not understand is that such tools already exist on our home computers. If these tools are good enough to help a whole world of people communicate-everyone from hermitic techies to senior citizens-then they are good enough for us. We should see what everyone is raving about.

Note: I don't have any way of verifying publication until the article appears at the CIA's Studies in Intelligence. Matthew says the website there will update shortly...

Comments

[Matthew Burton]

Thanks, David, for taking me at my word and posting this before the CIA site has been updated.

A few weeks ago, I was at an Edward Tufte seminar. He mentioned something called "Conway's Law," and I wish I had learned it before publishing this article. "A network reflects the structure of the organization that runs it." I think it's a good summary of what I'm trying to say. David, you put it another way above: "..you get systems that reflect the culture of the organations in which they are installed."

Also, I don't want this to seem as an advertisement for blogs in particular; instead, I'm just calling for Intelink's liberation from sysadmins in order to improve search. The Web has always had this freedom, and Google was using link analysis long before blogs came around. The comments, trackbacks etc. will be nice to have on Intelink (if users ever adopt them), but what it needs immediately is decentralization. Just the RIGHT to self-publish, regardless of the technology driving it, would do the job. In my initial paper for the Director of Central Intelligence (I mention it near the end of the article), I did not mention blogs. After it was published within the Community, Intelink managers approached me and told me about the blog project. So it seemed like the most practical method to espouse.