Data Mining & Intelligence Agencies

Moderators: Elvis, DrVolin, Jeff

Data Mining & Intelligence Agencies

Postby Wombaticus Rex » Wed Apr 04, 2012 10:55 pm


To make this thread more useful I'll keep a running ToC on this first post. (Note that four relatively excellent introductory articles immediately follow in the same post.)

Note parallel threads: Surveillance and General Patton's stellar tutorial on the tech tools behind this, WE NSA NOW

Recorded Futures and "Temporal Analytics"
Kevin Slavin & Karl Schroeder
ECHELON: America's Secret Global Surveillance Network / James Bamford on NSA data centers
Public Intelligence posts: CATALYST, ICE datamining systems & contracts
WaPo: Monitoring America,
Tim Shorrock: Domestic Spying, Inc.
Hank Asher, NCMEC, data mining to find pedophiles
Critical notes on Palantir
EXCELLENT: Tim Shorrock on how privatized intelligence leads to a flood of classified leaks
Tim Shorrock on US telecom coordination with DHS/NSA/FBI etc
60's vintage SigInt: Dark Gene & IBEX
Lockheed Martin's big investment in monitoring America
Reuters on the NSA's core curriculum and need for trained workers
2012 Congressional inquiry into "Data Sellers"
General Patton on quantum computing and exponential tech effects on data mining / visualization
General Patton on the observable limits of data mining, AGI, machine learning
Overview on World Simulators - also note parallel thread: SEAS world simulators and real time global surveillance

Solid introduction to Data Mining, via today's Atlantic: ... sk/255388/

Everything You Wanted to Know About Data Mining but Were Afraid to Ask

Big data is everywhere we look these days. Businesses are falling all over themselves to hire 'data scientists,' privacy advocates are concerned about personal data and control, and technologists and entrepreneurs scramble to find new ways to collect, control and monetize data. We know that data is powerful and valuable. But how?

This article is an attempt to explain how data mining works and why you should care about it. Because when we think about how our data is being used, it is crucial to understand the power of this practice. Without data mining, when you give someone access to information about you, all they know is what you have told them. With data mining, they know what you have told them and can guess a great deal more. Put another way, data mining allows companies and governments to use the information you provide to reveal more than you think.

To most of us data mining goes something like this: tons of data is collected, then quant wizards work their arcane magic, and then they know all of this amazing stuff. But, how? And what types of things can they know? Here is the truth: despite the fact that the specific technical functioning of data mining algorithms is quite complex -- they are a black box unless you are a professional statistician or computer scientist -- the uses and capabilities of these approaches are, in fact, quite comprehensible and intuitive.

For the most part, data mining tells us about very large and complex data sets, the kinds of information that would be readily apparent about small and simple things. For example, it can tell us that "one of these things is not like the other" a la Sesame Street or it can show us categories and then sort things into pre-determined categories. But what's simple with 5 datapoints is not so simple with 5 billion datapoints.

And these days, there's always more data. We gather far more of it then we can digest. Nearly every transaction or interaction leaves a data signature that someone somewhere is capturing and storing. This is, of course, true on the internet; but, ubiquitous computing and digitization has made it increasingly true about our lives away from our computers (do we still have those?). The sheer scale of this data has far exceeded human sense-making capabilities. At these scales patterns are often too subtle and relationships too complex or multi-dimensional to observe by simply looking at the data. Data mining is a means of automating part this process to detect interpretable patterns; it helps us see the forest without getting lost in the trees.

Discovering information from data takes two major forms: description and prediction. At the scale we are talking about, it is hard to know what the data shows. Data mining is used to simplify and summarize the data in a manner that we can understand, and then allow us to infer things about specific cases based on the patterns we have observed. Of course, specific applications of data mining methods are limited by the data and computing power available, and are tailored for specific needs and goals. However, there are several main types of pattern detection that are commonly used. These general forms illustrate what data mining can do.

Anomaly detection: in a large data set it is possible to get a picture of what the data tends to look like in a typical case. Statistics can be used to determine if something is notably different from this pattern. For instance, the IRS could model typical tax returns and use anomaly detection to identify specific returns that differ from this for review and audit.

Association learning: This is the type of data mining that drives the Amazon recommendation system. For instance, this might reveal that customers who bought a cocktail shaker and a cocktail recipe book also often buy martini glasses. These types of findings are often used for targeting coupons/deals or advertising. Similarly, this form of data mining (albeit a quite complex version) is behind Netflix movie recommendations.

Cluster detection: one type of pattern recognition that is particularly useful is recognizing distinct clusters or sub-categories within the data. Without data mining, an analyst would have to look at the data and decide on a set of categories which they believe captures the relevant distinctions between apparent groups in the data. This would risk missing important categories. With data mining it is possible to let the data itself determine the groups. This is one of the black-box type of algorithms that are hard to understand. But in a simple example - again with purchasing behavior - we can imagine that the purchasing habits of different hobbyists would look quite different from each other: gardeners, fishermen and model airplane enthusiasts would all be quite distinct. Machine learning algorithms can detect all of the different subgroups within a dataset that differ significantly from each other.

Classification: If an existing structure is already known, data mining can be used to classify new cases into these pre-determined categories. Learning from a large set of pre-classified examples, algorithms can detect persistent systemic differences between items in each group and apply these rules to new classification problems. Spam filters are a great example of this - large sets of emails that have been identified as spam have enabled filters to notice differences in word usage between legitimate and spam messages, and classify incoming messages according to these rules with a high degree of accuracy.

Regression: Data mining can be used to construct predictive models based on many variables. Facebook, for example, might be interested in predicting future engagement for a user based on past behavior. Factors like the amount of personal information shared, number of photos tagged, friend requests initiated or accepted, comments, likes etc. could all be included in such a model. Over time, this model could be honed to include or weight things differently as Facebook compares how the predictions differ from observed behavior. Ultimately these findings could be used to guide design in order to encourage more of the behaviors that seem to lead to increased engagement over time.

The patterns detected and structures revealed by the descriptive data mining are then often applied to predict other aspects of the data. Amazon offers a useful example of how descriptive findings are used for prediction. The (hypothetical) association between cocktail shaker and martini glass purchases, for instance, could be used, along with many other similar associations, as part of a model predicting the likelihood that a particular user will make a particular purchase. This model could match all such associations with a user's purchasing history, and predict which products they are most likely to purchase. Amazon can then serve ads based on what that user is most likely to buy.

Data mining, in this way, can grant immense inferential power. If an algorithm can correctly classify a case into known category based on limited data, it is possible to estimate a wide-range of other information about that case based on the properties of all the other cases in that category. This may sound dry, but it is how most successful Internet companies make their money and from where they draw their power.


Via: NYT, "How Companies Learn Your Secrets"

Andrew Pole had just started working as a statistician for Target in 2002, when two colleagues from the marketing department stopped by his desk to ask an odd question: “If we wanted to figure out if a customer is pregnant, even if she didn’t want us to know, can you do that? ”

Pole has a master’s degree in statistics and another in economics, and has been obsessed with the intersection of data and human behavior most of his life. His parents were teachers in North Dakota, and while other kids were going to 4-H, Pole was doing algebra and writing computer programs. “The stereotype of a math nerd is true,” he told me when I spoke with him last year. “I kind of like going out and evangelizing analytics.”

As the marketers explained to Pole — and as Pole later explained to me, back when we were still speaking and before Target told him to stop — new parents are a retailer’s holy grail. Most shoppers don’t buy everything they need at one store. Instead, they buy groceries at the grocery store and toys at the toy store, and they visit Target only when they need certain items they associate with Target — cleaning supplies, say, or new socks or a six-month supply of toilet paper. But Target sells everything from milk to stuffed animals to lawn furniture to electronics, so one of the company’s primary goals is convincing customers that the only store they need is Target. But it’s a tough message to get across, even with the most ingenious ad campaigns, because once consumers’ shopping habits are ingrained, it’s incredibly difficult to change them.

There are, however, some brief periods in a person’s life when old routines fall apart and buying habits are suddenly in flux. One of those moments — the moment, really — is right around the birth of a child, when parents are exhausted and overwhelmed and their shopping patterns and brand loyalties are up for grabs. But as Target’s marketers explained to Pole, timing is everything. Because birth records are usually public, the moment a couple have a new baby, they are almost instantaneously barraged with offers and incentives and advertisements from all sorts of companies. Which means that the key is to reach them earlier, before any other retailers know a baby is on the way. Specifically, the marketers said they wanted to send specially designed ads to women in their second trimester, which is when most expectant mothers begin buying all sorts of new things, like prenatal vitamins and maternity clothing. “Can you give us a list?” the marketers asked.

“We knew that if we could identify them in their second trimester, there’s a good chance we could capture them for years,” Pole told me. “As soon as we get them buying diapers from us, they’re going to start buying everything else too. If you’re rushing through the store, looking for bottles, and you pass orange juice, you’ll grab a carton. Oh, and there’s that new DVD I want. Soon, you’ll be buying cereal and paper towels from us, and keep coming back.”

The desire to collect information on customers is not new for Target or any other large retailer, of course. For decades, Target has collected vast amounts of data on every person who regularly walks into one of its stores. Whenever possible, Target assigns each shopper a unique code — known internally as the Guest ID number — that keeps tabs on everything they buy. “If you use a credit card or a coupon, or fill out a survey, or mail in a refund, or call the customer help line, or open an e-mail we’ve sent you or visit our Web site, we’ll record it and link it to your Guest ID,” Pole said. “We want to know everything we can.”

Also linked to your Guest ID is demographic information like your age, whether you are married and have kids, which part of town you live in, how long it takes you to drive to the store, your estimated salary, whether you’ve moved recently, what credit cards you carry in your wallet and what Web sites you visit. Target can buy data about your ethnicity, job history, the magazines you read, if you’ve ever declared bankruptcy or got divorced, the year you bought (or lost) your house, where you went to college, what kinds of topics you talk about online, whether you prefer certain brands of coffee, paper towels, cereal or applesauce, your political leanings, reading habits, charitable giving and the number of cars you own. (In a statement, Target declined to identify what demographic information it collects or purchases.) All that information is meaningless, however, without someone to analyze and make sense of it. That’s where Andrew Pole and the dozens of other members of Target’s Guest Marketing Analytics department come in.

Almost every major retailer, from grocery chains to investment banks to the U.S. Postal Service, has a “predictive analytics” department devoted to understanding not just consumers’ shopping habits but also their personal habits, so as to more efficiently market to them. “But Target has always been one of the smartest at this,” says Eric Siegel, a consultant and the chairman of a conference called Predictive Analytics World. “We’re living through a golden age of behavioral research. It’s amazing how much we can figure out about how people think now.”

The reason Target can snoop on our shopping habits is that, over the past two decades, the science of habit formation has become a major field of research in neurology and psychology departments at hundreds of major medical centers and universities, as well as inside extremely well financed corporate labs. “It’s like an arms race to hire statisticians nowadays,” said Andreas Weigend, the former chief scientist at “Mathematicians are suddenly sexy.” As the ability to analyze data has grown more and more fine-grained, the push to understand how daily habits influence our decisions has become one of the most exciting topics in clinical research, even though most of us are hardly aware those patterns exist. One study from Duke University estimated that habits, rather than conscious decision-making, shape 45 percent of the choices we make every day, and recent discoveries have begun to change everything from the way we think about dieting to how doctors conceive treatments for anxiety, depression and addictions.

This research is also transforming our understanding of how habits function across organizations and societies. A football coach named Tony Dungy propelled one of the worst teams in the N.F.L. to the Super Bowl by focusing on how his players habitually reacted to on-field cues. Before he became Treasury secretary, Paul O’Neill overhauled a stumbling conglomerate, Alcoa, and turned it into a top performer in the Dow Jones by relentlessly attacking one habit — a specific approach to worker safety — which in turn caused a companywide transformation. The Obama campaign has hired a habit specialist as its “chief scientist” to figure out how to trigger new voting patterns among different constituencies.

Researchers have figured out how to stop people from habitually overeating and biting their nails. They can explain why some of us automatically go for a jog every morning and are more productive at work, while others oversleep and procrastinate. There is a calculus, it turns out, for mastering our subconscious urges. For companies like Target, the exhaustive rendering of our conscious and unconscious patterns into data sets and algorithms has revolutionized what they know about us and, therefore, how precisely they can sell.

Inside the brain-and-cognitive-sciences department of the Massachusetts Institute of Technology are what, to the casual observer, look like dollhouse versions of surgical theaters. There are rooms with tiny scalpels, small drills and miniature saws. Even the operating tables are petite, as if prepared for 7-year-old surgeons. Inside those shrunken O.R.’s, neurologists cut into the skulls of anesthetized rats, implanting tiny sensors that record the smallest changes in the activity of their brains.

An M.I.T. neuroscientist named Ann Graybiel told me that she and her colleagues began exploring habits more than a decade ago by putting their wired rats into a T-shaped maze with chocolate at one end. The maze was structured so that each animal was positioned behind a barrier that opened after a loud click. The first time a rat was placed in the maze, it would usually wander slowly up and down the center aisle after the barrier slid away, sniffing in corners and scratching at walls. It appeared to smell the chocolate but couldn’t figure out how to find it. There was no discernible pattern in the rat’s meanderings and no indication it was working hard to find the treat.

The probes in the rats’ heads, however, told a different story. While each animal wandered through the maze, its brain was working furiously. Every time a rat sniffed the air or scratched a wall, the neurosensors inside the animal’s head exploded with activity. As the scientists repeated the experiment, again and again, the rats eventually stopped sniffing corners and making wrong turns and began to zip through the maze with more and more speed. And within their brains, something unexpected occurred: as each rat learned how to complete the maze more quickly, its mental activity decreased. As the path became more and more automatic — as it became a habit — the rats started thinking less and less.

This process, in which the brain converts a sequence of actions into an automatic routine, is called “chunking.” There are dozens, if not hundreds, of behavioral chunks we rely on every day. Some are simple: you automatically put toothpaste on your toothbrush before sticking it in your mouth. Some, like making the kids’ lunch, are a little more complex. Still others are so complicated that it’s remarkable to realize that a habit could have emerged at all.

Take backing your car out of the driveway. When you first learned to drive, that act required a major dose of concentration, and for good reason: it involves peering into the rearview and side mirrors and checking for obstacles, putting your foot on the brake, moving the gearshift into reverse, removing your foot from the brake, estimating the distance between the garage and the street while keeping the wheels aligned, calculating how images in the mirrors translate into actual distances, all while applying differing amounts of pressure to the gas pedal and brake.

Now, you perform that series of actions every time you pull into the street without thinking very much. Your brain has chunked large parts of it. Left to its own devices, the brain will try to make almost any repeated behavior into a habit, because habits allow our minds to conserve effort. But conserving mental energy is tricky, because if our brains power down at the wrong moment, we might fail to notice something important, like a child riding her bike down the sidewalk or a speeding car coming down the street. So we’ve devised a clever system to determine when to let a habit take over. It’s something that happens whenever a chunk of behavior starts or ends — and it helps to explain why habits are so difficult to change once they’re formed, despite our best intentions.

To understand this a little more clearly, consider again the chocolate-seeking rats. What Graybiel and her colleagues found was that, as the ability to navigate the maze became habitual, there were two spikes in the rats’ brain activity — once at the beginning of the maze, when the rat heard the click right before the barrier slid away, and once at the end, when the rat found the chocolate. Those spikes show when the rats’ brains were fully engaged, and the dip in neural activity between the spikes showed when the habit took over. From behind the partition, the rat wasn’t sure what waited on the other side, until it heard the click, which it had come to associate with the maze. Once it heard that sound, it knew to use the “maze habit,” and its brain activity decreased. Then at the end of the routine, when the reward appeared, the brain shook itself awake again and the chocolate signaled to the rat that this particular habit was worth remembering, and the neurological pathway was carved that much deeper.

The process within our brains that creates habits is a three-step loop. First, there is a cue, a trigger that tells your brain to go into automatic mode and which habit to use. Then there is the routine, which can be physical or mental or emotional. Finally, there is a reward, which helps your brain figure out if this particular loop is worth remembering for the future. Over time, this loop — cue, routine, reward; cue, routine, reward — becomes more and more automatic. The cue and reward become neurologically intertwined until a sense of craving emerges. What’s unique about cues and rewards, however, is how subtle they can be. Neurological studies like the ones in Graybiel’s lab have revealed that some cues span just milliseconds. And rewards can range from the obvious (like the sugar rush that a morning doughnut habit provides) to the infinitesimal (like the barely noticeable — but measurable — sense of relief the brain experiences after successfully navigating the driveway). Most cues and rewards, in fact, happen so quickly and are so slight that we are hardly aware of them at all. But our neural systems notice and use them to build automatic behaviors.

Habits aren’t destiny — they can be ignored, changed or replaced. But it’s also true that once the loop is established and a habit emerges, your brain stops fully participating in decision-making. So unless you deliberately fight a habit — unless you find new cues and rewards — the old pattern will unfold automatically.

“We’ve done experiments where we trained rats to run down a maze until it was a habit, and then we extinguished the habit by changing the placement of the reward,” Graybiel told me. “Then one day, we’ll put the reward in the old place and put in the rat and, by golly, the old habit will re-emerge right away. Habits never really disappear.”

Luckily, simply understanding how habits work makes them easier to control. Take, for instance, a series of studies conducted a few years ago at Columbia University and the University of Alberta. Researchers wanted to understand how exercise habits emerge. In one project, 256 members of a health-insurance plan were invited to classes stressing the importance of exercise. Half the participants received an extra lesson on the theories of habit formation (the structure of the habit loop) and were asked to identify cues and rewards that might help them develop exercise routines.

The results were dramatic. Over the next four months, those participants who deliberately identified cues and rewards spent twice as much time exercising as their peers. Other studies have yielded similar results. According to another recent paper, if you want to start running in the morning, it’s essential that you choose a simple cue (like always putting on your sneakers before breakfast or leaving your running clothes next to your bed) and a clear reward (like a midday treat or even the sense of accomplishment that comes from ritually recording your miles in a log book). After a while, your brain will start anticipating that reward — craving the treat or the feeling of accomplishment — and there will be a measurable neurological impulse to lace up your jogging shoes each morning.

Our relationship to e-mail operates on the same principle. When a computer chimes or a smartphone vibrates with a new message, the brain starts anticipating the neurological “pleasure” (even if we don’t recognize it as such) that clicking on the e-mail and reading it provides. That expectation, if unsatisfied, can build until you find yourself moved to distraction by the thought of an e-mail sitting there unread — even if you know, rationally, it’s most likely not important. On the other hand, once you remove the cue by disabling the buzzing of your phone or the chiming of your computer, the craving is never triggered, and you’ll find, over time, that you’re able to work productively for long stretches without checking your in-box.

Some of the most ambitious habit experiments have been conducted by corporate America. To understand why executives are so entranced by this science, consider how one of the world’s largest companies, Procter & Gamble, used habit insights to turn a failing product into one of its biggest sellers. P.& G. is the corporate behemoth behind a whole range of products, from Downy fabric softener to Bounty paper towels to Duracell batteries and dozens of other household brands. In the mid-1990s, P.& G.’s executives began a secret project to create a new product that could eradicate bad smells. P.& G. spent millions developing a colorless, cheap-to-manufacture liquid that could be sprayed on a smoky blouse, stinky couch, old jacket or stained car interior and make it odorless. In order to market the product — Febreze — the company formed a team that included a former Wall Street mathematician named Drake Stimson and habit specialists, whose job was to make sure the television commercials, which they tested in Phoenix, Salt Lake City and Boise, Idaho, accentuated the product’s cues and rewards just right.

The first ad showed a woman complaining about the smoking section of a restaurant. Whenever she eats there, she says, her jacket smells like smoke. A friend tells her that if she uses Febreze, it will eliminate the odor. The cue in the ad is clear: the harsh smell of cigarette smoke. The reward: odor eliminated from clothes. The second ad featured a woman worrying about her dog, Sophie, who always sits on the couch. “Sophie will always smell like Sophie,” she says, but with Febreze, “now my furniture doesn’t have to.” The ads were put in heavy rotation. Then the marketers sat back, anticipating how they would spend their bonuses. A week passed. Then two. A month. Two months. Sales started small and got smaller. Febreze was a dud.

The panicked marketing team canvassed consumers and conducted in-depth interviews to figure out what was going wrong, Stimson recalled. Their first inkling came when they visited a woman’s home outside Phoenix. The house was clean and organized. She was something of a neat freak, the woman explained. But when P.& G.’s scientists walked into her living room, where her nine cats spent most of their time, the scent was so overpowering that one of them gagged.

According to Stimson, who led the Febreze team, a researcher asked the woman, “What do you do about the cat smell?”

“It’s usually not a problem,” she said.

“Do you smell it now?”

“No,” she said. “Isn’t it wonderful? They hardly smell at all!”

A similar scene played out in dozens of other smelly homes. The reason Febreze wasn’t selling, the marketers realized, was that people couldn’t detect most of the bad smells in their lives. If you live with nine cats, you become desensitized to their scents. If you smoke cigarettes, eventually you don’t smell smoke anymore. Even the strongest odors fade with constant exposure. That’s why Febreze was a failure. The product’s cue — the bad smells that were supposed to trigger daily use — was hidden from the people who needed it the most. And Febreze’s reward (an odorless home) was meaningless to someone who couldn’t smell offensive scents in the first place.

P.& G. employed a Harvard Business School professor to analyze Febreze’s ad campaigns. They collected hours of footage of people cleaning their homes and watched tape after tape, looking for clues that might help them connect Febreze to people’s daily habits. When that didn’t reveal anything, they went into the field and conducted more interviews. A breakthrough came when they visited a woman in a suburb near Scottsdale, Ariz., who was in her 40s with four children. Her house was clean, though not compulsively tidy, and didn’t appear to have any odor problems; there were no pets or smokers. To the surprise of everyone, she loved Febreze.

“I use it every day,” she said.

“What smells are you trying to get rid of?” a researcher asked.

“I don’t really use it for specific smells,” the woman said. “I use it for normal cleaning — a couple of sprays when I’m done in a room.”

The researchers followed her around as she tidied the house. In the bedroom, she made her bed, tightened the sheet’s corners, then sprayed the comforter with Febreze. In the living room, she vacuumed, picked up the children’s shoes, straightened the coffee table, then sprayed Febreze on the freshly cleaned carpet.

“It’s nice, you know?” she said. “Spraying feels like a little minicelebration when I’m done with a room.” At the rate she was going, the team estimated, she would empty a bottle of Febreze every two weeks.

When they got back to P.& G.’s headquarters, the researchers watched their videotapes again. Now they knew what to look for and saw their mistake in scene after scene. Cleaning has its own habit loops that already exist. In one video, when a woman walked into a dirty room (cue), she started sweeping and picking up toys (routine), then she examined the room and smiled when she was done (reward). In another, a woman scowled at her unmade bed (cue), proceeded to straighten the blankets and comforter (routine) and then sighed as she ran her hands over the freshly plumped pillows (reward). P.& G. had been trying to create a whole new habit with Febreze, but what they really needed to do was piggyback on habit loops that were already in place. The marketers needed to position Febreze as something that came at the end of the cleaning ritual, the reward, rather than as a whole new cleaning routine.

The company printed new ads showing open windows and gusts of fresh air. More perfume was added to the Febreze formula, so that instead of merely neutralizing odors, the spray had its own distinct scent. Television commercials were filmed of women, having finished their cleaning routine, using Febreze to spritz freshly made beds and just-laundered clothing. Each ad was designed to appeal to the habit loop: when you see a freshly cleaned room (cue), pull out Febreze (routine) and enjoy a smell that says you’ve done a great job (reward). When you finish making a bed (cue), spritz Febreze (routine) and breathe a sweet, contented sigh (reward). Febreze, the ads implied, was a pleasant treat, not a reminder that your home stinks.

And so Febreze, a product originally conceived as a revolutionary way to destroy odors, became an air freshener used once things are already clean. The Febreze revamp occurred in the summer of 1998. Within two months, sales doubled. A year later, the product brought in $230 million. Since then Febreze has spawned dozens of spinoffs — air fresheners, candles and laundry detergents — that now account for sales of more than $1 billion a year. Eventually, P.& G. began mentioning to customers that, in addition to smelling sweet, Febreze can actually kill bad odors. Today it’s one of the top-selling products in the world.

Andrew Pole was hired by Target to use the same kinds of insights into consumers’ habits to expand Target’s sales. His assignment was to analyze all the cue-routine-reward loops among shoppers and help the company figure out how to exploit them. Much of his department’s work was straightforward: find the customers who have children and send them catalogs that feature toys before Christmas. Look for shoppers who habitually purchase swimsuits in April and send them coupons for sunscreen in July and diet books in December. But Pole’s most important assignment was to identify those unique moments in consumers’ lives when their shopping habits become particularly flexible and the right advertisement or coupon would cause them to begin spending in new ways.

In the 1980s, a team of researchers led by a U.C.L.A. professor named Alan Andreasen undertook a study of peoples’ most mundane purchases, like soap, toothpaste, trash bags and toilet paper. They learned that most shoppers paid almost no attention to how they bought these products, that the purchases occurred habitually, without any complex decision-making. Which meant it was hard for marketers, despite their displays and coupons and product promotions, to persuade shoppers to change.

But when some customers were going through a major life event, like graduating from college or getting a new job or moving to a new town, their shopping habits became flexible in ways that were both predictable and potential gold mines for retailers. The study found that when someone marries, he or she is more likely to start buying a new type of coffee. When a couple move into a new house, they’re more apt to purchase a different kind of cereal. When they divorce, there’s an increased chance they’ll start buying different brands of beer.

Consumers going through major life events often don’t notice, or care, that their shopping habits have shifted, but retailers notice, and they care quite a bit. At those unique moments, Andreasen wrote, customers are “vulnerable to intervention by marketers.” In other words, a precisely timed advertisement, sent to a recent divorcee or new homebuyer, can change someone’s shopping patterns for years.

And among life events, none are more important than the arrival of a baby. At that moment, new parents’ habits are more flexible than at almost any other time in their adult lives. If companies can identify pregnant shoppers, they can earn millions.

The only problem is that identifying pregnant customers is harder than it sounds. Target has a baby-shower registry, and Pole started there, observing how shopping habits changed as a woman approached her due date, which women on the registry had willingly disclosed. He ran test after test, analyzing the data, and before long some useful patterns emerged. Lotions, for example. Lots of people buy lotion, but one of Pole’s colleagues noticed that women on the baby registry were buying larger quantities of unscented lotion around the beginning of their second trimester. Another analyst noted that sometime in the first 20 weeks, pregnant women loaded up on supplements like calcium, magnesium and zinc. Many shoppers purchase soap and cotton balls, but when someone suddenly starts buying lots of scent-free soap and extra-big bags of cotton balls, in addition to hand sanitizers and washcloths, it signals they could be getting close to their delivery date.

As Pole’s computers crawled through the data, he was able to identify about 25 products that, when analyzed together, allowed him to assign each shopper a “pregnancy prediction” score. More important, he could also estimate her due date to within a small window, so Target could send coupons timed to very specific stages of her pregnancy.

One Target employee I spoke to provided a hypothetical example. Take a fictional Target shopper named Jenny Ward, who is 23, lives in Atlanta and in March bought cocoa-butter lotion, a purse large enough to double as a diaper bag, zinc and magnesium supplements and a bright blue rug. There’s, say, an 87 percent chance that she’s pregnant and that her delivery date is sometime in late August. What’s more, because of the data attached to her Guest ID number, Target knows how to trigger Jenny’s habits. They know that if she receives a coupon via e-mail, it will most likely cue her to buy online. They know that if she receives an ad in the mail on Friday, she frequently uses it on a weekend trip to the store. And they know that if they reward her with a printed receipt that entitles her to a free cup of Starbucks coffee, she’ll use it when she comes back again.

In the past, that knowledge had limited value. After all, Jenny purchased only cleaning supplies at Target, and there were only so many psychological buttons the company could push. But now that she is pregnant, everything is up for grabs. In addition to triggering Jenny’s habits to buy more cleaning products, they can also start including offers for an array of products, some more obvious than others, that a woman at her stage of pregnancy might need.

Pole applied his program to every regular female shopper in Target’s national database and soon had a list of tens of thousands of women who were most likely pregnant. If they could entice those women or their husbands to visit Target and buy baby-related products, the company’s cue-routine-reward calculators could kick in and start pushing them to buy groceries, bathing suits, toys and clothing, as well. When Pole shared his list with the marketers, he said, they were ecstatic. Soon, Pole was getting invited to meetings above his paygrade. Eventually his paygrade went up.

At which point someone asked an important question: How are women going to react when they figure out how much Target knows?

“If we send someone a catalog and say, ‘Congratulations on your first child!’ and they’ve never told us they’re pregnant, that’s going to make some people uncomfortable,” Pole told me. “We are very conservative about compliance with all privacy laws. But even if you’re following the law, you can do things where people get queasy.”

About a year after Pole created his pregnancy-prediction model, a man walked into a Target outside Minneapolis and demanded to see the manager. He was clutching coupons that had been sent to his daughter, and he was angry, according to an employee who participated in the conversation.

“My daughter got this in the mail!” he said. “She’s still in high school, and you’re sending her coupons for baby clothes and cribs? Are you trying to encourage her to get pregnant?”

The manager didn’t have any idea what the man was talking about. He looked at the mailer. Sure enough, it was addressed to the man’s daughter and contained advertisements for maternity clothing, nursery furniture and pictures of smiling infants. The manager apologized and then called a few days later to apologize again.

On the phone, though, the father was somewhat abashed. “I had a talk with my daughter,” he said. “It turns out there’s been some activities in my house I haven’t been completely aware of. She’s due in August. I owe you an apology.”

When I approached Target to discuss Pole’s work, its representatives declined to speak with me. “Our mission is to make Target the preferred shopping destination for our guests by delivering outstanding value, continuous innovation and exceptional guest experience,” the company wrote in a statement. “We’ve developed a number of research tools that allow us to gain insights into trends and preferences within different demographic segments of our guest population.” When I sent Target a complete summary of my reporting, the reply was more terse: “Almost all of your statements contain inaccurate information and publishing them would be misleading to the public. We do not intend to address each statement point by point.” The company declined to identify what was inaccurate. They did add, however, that Target “is in compliance with all federal and state laws, including those related to protected health information.”

When I offered to fly to Target’s headquarters to discuss its concerns, a spokeswoman e-mailed that no one would meet me. When I flew out anyway, I was told I was on a list of prohibited visitors. “I’ve been instructed not to give you access and to ask you to leave,” said a very nice security guard named Alex.

Using data to predict a woman’s pregnancy, Target realized soon after Pole perfected his model, could be a public-relations disaster. So the question became: how could they get their advertisements into expectant mothers’ hands without making it appear they were spying on them? How do you take advantage of someone’s habits without letting them know you’re studying their lives?

Before I met Andrew Pole, before I even decided to write a book about the science of habit formation, I had another goal: I wanted to lose weight.

I had got into a bad habit of going to the cafeteria every afternoon and eating a chocolate-chip cookie, which contributed to my gaining a few pounds. Eight, to be precise. I put a Post-it note on my computer reading “NO MORE COOKIES.” But every afternoon, I managed to ignore that note, wander to the cafeteria, buy a cookie and eat it while chatting with colleagues. Tomorrow, I always promised myself, I’ll muster the willpower to resist.

Tomorrow, I ate another cookie.

When I started interviewing experts in habit formation, I concluded each interview by asking what I should do. The first step, they said, was to figure out my habit loop. The routine was simple: every afternoon, I walked to the cafeteria, bought a cookie and ate it while chatting with friends.

Next came some less obvious questions: What was the cue? Hunger? Boredom? Low blood sugar? And what was the reward? The taste of the cookie itself? The temporary distraction from my work? The chance to socialize with colleagues?

Rewards are powerful because they satisfy cravings, but we’re often not conscious of the urges driving our habits in the first place. So one day, when I felt a cookie impulse, I went outside and took a walk instead. The next day, I went to the cafeteria and bought a coffee. The next, I bought an apple and ate it while chatting with friends. You get the idea. I wanted to test different theories regarding what reward I was really craving. Was it hunger? (In which case the apple should have worked.) Was it the desire for a quick burst of energy? (If so, the coffee should suffice.) Or, as turned out to be the answer, was it that after several hours spent focused on work, I wanted to socialize, to make sure I was up to speed on office gossip, and the cookie was just a convenient excuse? When I walked to a colleague’s desk and chatted for a few minutes, it turned out, my cookie urge was gone.

All that was left was identifying the cue.

Deciphering cues is hard, however. Our lives often contain too much information to figure out what is triggering a particular behavior. Do you eat breakfast at a certain time because you’re hungry? Or because the morning news is on? Or because your kids have started eating? Experiments have shown that most cues fit into one of five categories: location, time, emotional state, other people or the immediately preceding action. So to figure out the cue for my cookie habit, I wrote down five things the moment the urge hit:

Where are you? (Sitting at my desk.)

What time is it? (3:36 p.m.)

What’s your emotional state? (Bored.)

Who else is around? (No one.)

What action preceded the urge? (Answered an e-mail.)

The next day I did the same thing. And the next. Pretty soon, the cue was clear: I always felt an urge to snack around 3:30.

Once I figured out all the parts of the loop, it seemed fairly easy to change my habit. But the psychologists and neuroscientists warned me that, for my new behavior to stick, I needed to abide by the same principle that guided Procter & Gamble in selling Febreze: To shift the routine — to socialize, rather than eat a cookie — I needed to piggyback on an existing habit. So now, every day around 3:30, I stand up, look around the newsroom for someone to talk to, spend 10 minutes gossiping, then go back to my desk. The cue and reward have stayed the same. Only the routine has shifted. It doesn’t feel like a decision, any more than the M.I.T. rats made a decision to run through the maze. It’s now a habit. I’ve lost 21 pounds since then (12 of them from changing my cookie ritual).

After Andrew Pole built his pregnancy-prediction model, after he identified thousands of female shoppers who were most likely pregnant, after someone pointed out that some of those women might be a little upset if they received an advertisement making it obvious Target was studying their reproductive status, everyone decided to slow things down.

The marketing department conducted a few tests by choosing a small, random sample of women from Pole’s list and mailing them combinations of advertisements to see how they reacted.

“We have the capacity to send every customer an ad booklet, specifically designed for them, that says, ‘Here’s everything you bought last week and a coupon for it,’ ” one Target executive told me. “We do that for grocery products all the time.” But for pregnant women, Target’s goal was selling them baby items they didn’t even know they needed yet.

“With the pregnancy products, though, we learned that some women react badly,” the executive said. “Then we started mixing in all these ads for things we knew pregnant women would never buy, so the baby ads looked random. We’d put an ad for a lawn mower next to diapers. We’d put a coupon for wineglasses next to infant clothes. That way, it looked like all the products were chosen by chance.

“And we found out that as long as a pregnant woman thinks she hasn’t been spied on, she’ll use the coupons. She just assumes that everyone else on her block got the same mailer for diapers and cribs. As long as we don’t spook her, it works.”

In other words, if Target piggybacked on existing habits — the same cues and rewards they already knew got customers to buy cleaning supplies or socks — then they could insert a new routine: buying baby products, as well. There’s a cue (“Oh, a coupon for something I need!”) a routine (“Buy! Buy! Buy!”) and a reward (“I can take that off my list”). And once the shopper is inside the store, Target will hit her with cues and rewards to entice her to purchase everything she normally buys somewhere else. As long as Target camouflaged how much it knew, as long as the habit felt familiar, the new behavior took hold.

Soon after the new ad campaign began, Target’s Mom and Baby sales exploded. The company doesn’t break out figures for specific divisions, but between 2002 — when Pole was hired — and 2010, Target’s revenues grew from $44 billion to $67 billion. In 2005, the company’s president, Gregg Steinhafel, boasted to a room of investors about the company’s “heightened focus on items and categories that appeal to specific guest segments such as mom and baby.”

Pole was promoted. He has been invited to speak at conferences. “I never expected this would become such a big deal,” he told me the last time we spoke.

A few weeks before this article went to press, I flew to Minneapolis to try and speak to Andrew Pole one last time. I hadn’t talked to him in more than a year. Back when we were still friendly, I mentioned that my wife was seven months pregnant. We shop at Target, I told him, and had given the company our address so we could start receiving coupons in the mail. As my wife’s pregnancy progressed, I noticed a subtle upswing in the number of advertisements for diapers and baby clothes arriving at our house.

Pole didn’t answer my e-mails or phone calls when I visited Minneapolis. I drove to his large home in a nice suburb, but no one answered the door. On my way back to the hotel, I stopped at a Target to pick up some deodorant, then also bought some T-shirts and a fancy hair gel. On a whim, I threw in some pacifiers, to see how the computers would react. Besides, our baby is now 9 months old. You can’t have too many pacifiers.

When I paid, I didn’t receive any sudden deals on diapers or formula, to my slight disappointment. It made sense, though: I was shopping in a city I never previously visited, at 9:45 p.m. on a weeknight, buying a random assortment of items. I was using a corporate credit card, and besides the pacifiers, hadn’t purchased any of the things that a parent needs. It was clear to Target’s computers that I was on a business trip. Pole’s prediction calculator took one look at me, ran the numbers and decided to bide its time. Back home, the offers would eventually come. As Pole told me the last time we spoke: “Just wait. We’ll be sending you coupons for things you want before you even know you want them.”



I am also reminded of the Joshua-Michéle Ross series on O'Rielly a few years back...let's review.

Source: ... mmons.html

In January 2002 DARPA launched the Information Awareness Office. The mission was to, “ imagine, develop, apply, integrate, demonstrate and transition information technologies, components and prototype, closed-loop, information systems that will counter asymmetric threats by achieving total information awareness (emphasis added)” The notion of a government agency achieving total information awareness was too Orwellian to ignore. Under criticism that this “awareness” could quickly migrate to a mass surveillance system the program was defunded.

Fast-forward to last week and my near-purchase of Libbey Duratuff Gibralter Glasses (the perfect bourbon glass one might speculate). Over the course of the next few days I was peppered with exact-match ads for Libbey Duratuff glassware on several other websites; A small example of information awareness at work.

Personal data is the currency of Web 2.0. Knowing what we watch, buy, click, own, what we think, intend and ultimately do confers competitive advantage. Facebook possesses your social graph, your personal interests and your full profile (age, location, relationship status etc.) not to mention your daily (or hourly) answer to their persistent question, “what’s on your mind?”. Reviewing the “25 Surprising Things Google Knows About You” should give anyone pause. And it’s not just the Web 2.0 set. Credit Card Companies, Telcos, Insurance , Pharma… all are collecting vast stores of personal data. If you watch the trendline it is moving toward more data and more analytic capability - not less.

So why is it that we seem to have more comfort when the capacity for total information awareness lies with corporations as opposed to government? Experience shows that there is a very thin barrier between the two. To wit, the release of thousands of phone records to the U.S. government - and, conveniently, government immunity for those same corporations after the breach. Google and Yahoo! and Microsoft have all been accused of cooperating with the Chinese government to aid censorship and repression of free speech. What happens if/when we encounter the next version of the Bush administration that sees no problem abrogating civil rights in pursuit of “evildoers”?

What's more, when we deliver our personal information over to corporations we are giving this data over to an institution that is amoral. Companies are not yet structured to deliver moral or ethical results - they are encouraged to grow and deliver “shareholder value” (read money) which is a numb and narrow measure of value. Do I want my data to be managed by an amoral institution?

To be clear - I want the convenience and miracles that modern technology brings. I love the Internet and I am willing to give over lots of data in the trade. But I want two fundamental protections:

First, change the corporation. The structure of the corporation continues to be driven by 20th century hard goals of efficiency and scale - not by more complex measures of environmental sustainability, value creation and the commonweal. These are simply not adequately factored into any structural, organizational, incentive or taxation systems of business today. Profit and profit motive are fine - but hiding social and environmental costs is no longer acceptable. I want to deal with institutions capable of morality. This is no small task - but if we can build the Internet….

Second. We need a right to privacy that matches the 21st century reality. As a friend of mine likes to say, “privacy is now a responsibility - not a right.” While it is pithy (and perhaps true), the reason we grant rights - and laws to enforce those rights in society is the simple fact that people do not generally have the wherewithal to protect themselves from large, institutional interests. In the same way that regulatory structures are needed to keep a financial system in balance (alas even the Ayn Rand acolyte Greenspan finally agrees with this truism), we need new rights and regulations governing the use of our personal data - and simple sets of controls over who has access to it.

The true work of the 21st century lies not in refining our technology - this we will achieve without any political will. The work lies in re-imagining our institutions.

Of course, his first "solution" is such an obvious category error you can immediately tell he wasn't going through an editor. The challenge of building the internet was electrical engineering and physical logistics -- changing the nature of the corporation is an institutional crusade requiring a completely different skillset and strategy. Strategies, really.

Today on twitter, all the Big Thinkers are abuzz about the idea of "repurposing" Federal bureaucracy and I couldn't help but ask for examples of that being done successfully -- so far all the responses have been token "corporate turnaround" stories from the private sector. I can't tell if I'm blinded by my cynicism, or they're really that naive to the tremendous logistical gap between fixing IBM's management structure and turning around the US Federal Government...

Anyways, the final installment in Ross's series gets more meaty...


Source: ... ticon.html

The Digital Panopticon


Bentham was left frustrated in his vision to build the Panopticon. But the concept endured - not just as a literal architecture for controlling physical subjects (there are many Panopticons that now bear Bentham’s stamp) - but as a metaphor for understanding the function of power in modern times. French philosopher Michel Foucault dedicated a whole section of his book Discipline and Punish to the significance of the Panopticon. His take was essentially this: The same mechanism at work in the Panopticon - making subjects totally visible to authority - leads to those subjects internalizing the norms of power. In Foucault’s words “…the major effect of the Panopticon; to induce in the inmate a state of conscious and permanent visibility that assures the automatic functioning of power. So to arrange things that the surveillance is permanent in its effects, even if it is discontinuous in its action; that the perfection of power should tend to render its actual exercise unnecessary” In short, under the possibility of total surveillance the inmate becomes self regulating.

The social technologies we see in use today are fundamentally panoptical - the architecture of participation is inherently an architecture of surveillance.

In the age of social networks we find ourselves coming under a vast grid of surveillance - of permanent visibility. The routine self-reporting of what we are doing, reading, thinking via status updates makes our every action and location visible to the crowd. This visibility has a normative effect on behavior (in other words we conform our behavior and/or our speech about that behavior when we know we are being observed).

In many cases we are opting into automated reporting structures (Google Lattitude, Loopt etc.) that detail our location at any given point in time. We are doing this in exchange for small conveniences (finding local sushi more quickly, gaining “ambient intimacy”) without ever considering the bargain that we are striking. In short, we are creating the ultimate Panopticon - with our data centrally housed in the cloud (see previous post on the Captivity of the Commons) - our every movement, and up-to-the-minute status is a matter of public record. In the same way that networked communications move us from a one to many broadcast model to a many to many - so we are seeing the move to a many-to-many surveillance model. A global community of voyeurs ceaselessly confessing to "What are you doing? (Twitter) or "What's on your mind? (Facebook)

Captivity of the Commons focused on the risks of corporate ownership of personal data. This post is concerned with how, as individuals, we have grown comfortable giving our information away; how our sense of privacy is changing under the small conveniences that disclosure brings. How our identity changes as an effect of constant self-disclosure. Many previous comments have rightly noted that privacy is often cultural -- if you don't expect it - there is no such thing as an infringement. Yet it is important to reckon with the changes we see occurring around us and argue what kind of a culture we wish to create (or contribute to).

Jacques Ellul’s book, Propaganda, had a thesis that was at once startling and obvious: Propaganda’s end goal is not to change your mind at any one point in time - but to create a changeable mind. Thus when invoked at the necessary time - humans could be manipulated into action. In the U.S. this language was expressed by catchphrases like, “communism in our backyard,” “enemies of freedom” or the current manufactured hysteria about Obama as a “socialist”.

Similarly the significance of status updates and location based services may not lie in the individual disclosure but in the significance of a culture that has become accustomed to constant disclosure.

Tech guys waking up to social conditioning implications of their own work is a beautiful thing, innit?
User avatar
Wombaticus Rex
Posts: 10614
Joined: Wed Nov 08, 2006 6:33 pm
Location: Vermontistan
Blog: View Blog (0)

Re: Data Mining & Intelligence Agencies

Postby Wombaticus Rex » Wed Apr 04, 2012 10:56 pm

Recorded Future is a formerly private tech company based out of Cambridge, MA who specialize in "Temporal Analytics." I did some searching and was surprised I couldn't find any specific thread about the recent(ish) news that both Google and the CIA's tech investment firm In-Q-Tel had bought, heavily, into their's a quick background primer first: ... google-cia

Exclusive: Google, CIA Invest in ‘Future’ of Web Monitoring

The investment arms of the CIA and Google are both backing a company that monitors the web in real time — and says it uses that information to predict the future.

The company is called Recorded Future, and it scours tens of thousands of websites, blogs and Twitter accounts to find the relationships between people, organizations, actions and incidents — both present and still-to-come. In a white paper, the company says its temporal analytics engine “goes beyond search” by “looking at the ‘invisible links’ between documents that talk about the same, or related, entities and events.”

The idea is to figure out for each incident who was involved, where it happened and when it might go down. Recorded Future then plots that chatter, showing online “momentum” for any given event.

“The cool thing is, you can actually predict the curve, in many cases,” says company CEO Christopher Ahlberg, a former Swedish Army Ranger with a PhD in computer science.

Which naturally makes the 16-person Cambridge, Massachusetts, firm attractive to Google Ventures, the search giant’s investment division, and to In-Q-Tel, which handles similar duties for the CIA and the wider intelligence community.

Needless to say I've had an abiding interest in these folks ever since and my handy Alerts notified me of this white paper from the company on "Temporal Analytics." I found it to be an engaging read with surprising depth of reference -- some quality brainfood here!

READ: ... analytics/

Source: ... uture.html

Recorded Future, a privately-held firm, has distinguished itself in several ways. First, the company received financial support from In-Q-Tel, the investment arm of the US intelligence community and from Google, a company known for its voracious interest in next-generation technology. Second, the company has ignited the blogosphere with its fact-filled and informative posts on the firm Web log. Topics have included scanning the horizon of automobile technology which included eye-popping visualizations of “big data”. Third, the company has been the subject of considerable discussion by analysts, competitors, and legal experts.

On April 2, 2011, I spoke with Christopher Ahlberg, the founder of Recorded Future. The company is competing in what for lack of a better term a market sector I call “predictive analytics.” Part business intelligence and part content processing, Recorded Future processes a wide range of inputs, analyzes entities and other identified elements, and uses the content and metadata to fuel numerical processes. The object of the computationally intensive operations is to provide insights about likely outcomes. The name of the company, Recorded Future, evokes both traditional content processing and the next-generation techniques of predictive analytics.

Like Palantir, a data analytics company which made headlines after landing $90 million in venture funding, Recorded Future uses easy-to-grasp, high-quality graphical outputs. The system can output tables and hot linked results lists. Recorded Future recognizes that users want to “see” information in a context as well as have the ability to dig into the underlying data or explore a particular set of outputs through time. But what makes Recorded Future different is that interest from both In-Q-Tel and Google makes Recorded Future like a company that nails a million dollar contract and wins the Fields Medal on the same day.

I was able to talk with Mr. Ahlberg at his home base of Boston, Massachusetts. The development group of Recorded Future is in Sweden. The full text of my interview with Mr. Ahlberg appears below.

Thanks for taking the time to talk with me. What’s the driver for Recorded Future?

The founders behind Recorded Future were part of the same team that built Spotfire.

That’s a TIBCO company now, right?

Yes. It sounds as if you are familiar with Spotfire, which is a tool for visualizing and analyzing large sets of structured data. We ended up selling that company after building it to US$60 million, which we thought was a healthy business. After a short break, my colleagues and I started to think about what to do next.

What really stood out for us was the incredible richness of the Internet as a data source. Google and others certainly has shown how powerful indexes can show you the path to interesting documents. But what if we could make "the Internet" available for analysis.

So we set out to organize unstructured information at very large scale by events and time.

Can you give me an example?

Of course, a query might return a link to a document that says something like "Hu Jintao will tomorrow land in Paris for talks with Sarkozy" or "Apple will next week hold a product launch event in San Francisco"). We wanted to take this information and make insights available through a stunning user experiences and application programming interfaces. Our idea was that an API would allow others to tap into the richness and potential of Internet content in a new way.

Hakia has made quite an impact with its SENSEnews service. Is your system applicable to the financial community’s interests as well?

Absolutely. Let me provide you with some color on what our system can do. In quantitative analysis (for example in finance) we can prove the predictive power of our data (stock returns, volatility). The method applies to other areas as well. For instance, we think about this as providing data through user experiences for end users to do analysis. The key differentiator for our system is that the query and results can be about the past or the future.

Applications range from law enforcement to financial analysis to health and medical challenges.

When did you become interested in text and content processing?

That’s a good question. I've always had a keen interest in data analysis and visualization. Even back in 1993 as part of my PhD I worked on what was called the FilmFinder which took large amounts of textually oriented data (what's now IMDB) and allowed you to explore that data in a visual manner.

Later, when we were working on Spotfire in finance and government we had lots of interest in visualizing and analyzing textual data. - Some of this work required our working with outputs that were generated by a range of text analysis tools.

It struck me that if we could turn textual information into temporal events (through clever linguistics) we could organize data for analysis. I saw that if we actually built the whole stack as a service for people, we could do this in a really attractive fashion and solve some significant and difficult information problems for people.

Recorded Future has been investing in search, content processing, and text analysis for a number of years. What's the R&D strategy going forward?

We started thinking about how to do this in early 2009 and have had engineering staff (by now a pretty good group) on this for about 18 months. We've deployed our product to some of the absolutely most sophisticated analytic organizations in both finance and government. Also, we gained a whole range of users around the world. The goals we've set out has been quite ambitious: To allow analysts work with the "Internet as a source" and to succeed in actual financial predictions.

We know our technology delivers results, so we knew we were on a very promising path that few had traveled successfully. Thus, support of In-Q-Tel and Google has been extremely positive. It is good to have highly regarded organizations validate one’s methods. Each organization gives us access to quite sophisticated people in the domains in which we work.

For the technical work, we do not have any magic. There is just hard work and our desire to deliver results to our users and customers. Some people think that In-Q-Tel and Google have magic. But there is no magic, just more work and lots of problem solving. Some people think there is magic, but there is none. We benefit from both organization’s interest in and constructive criticism of our system.

Many vendors argue that mash ups are and data fusion are "the way information retrieval will work going forward? I am referring to structured and unstructured information. How does Recorded Future perceive this blend of search and user-accessible outputs?

Yes, I think it's very true that we live in a time when data can and should come together better than ever before. Now, that doesn't mean that that's easy.

Architecturally we try to address this by building our user interfaces on the very same API that we provide to customers and partners so that we force ourselves to have a very standard, transparent way of accessing our data. We document this publicly.

Our customers are very interested in mashing up data from different sources. I think you call this “data fusion” in your blog. We want to mash up data and applications, not just data.

Would you give me an example?

Of course, this might be integrating our timeline visualizations with geospatial applications. Alternatively we integrate our data on corporations through identifiers such as stock tickers with external equity pricing/returns data. We try to prepare for these scenarios. We've published open examples; for instance, ... ytics.html, with our data loaded in Google Spreadsheets, R, etc. The Recorded Future data will gain in value when our system becomes more pervasive.

Without divulging your firm's methods, will you characterize a typical use case for your firm's content processing, tagging, and search and retrieval capabilities?

We have two primary use cases. One is end users doing analytic research. This use case can be as simple as "I'm looking to buy a large block of Apple shares. Find me upcoming events, scheduled and unscheduled/speculative for Apple over next 12 months so that I can weigh these external catalysts in to my analysis".

The other major use case is integrating our data into quantitative analysis. For instance, an an analyst may for example have an equity or commodity pricing model and would like to weigh in events and time as a factor.

What are the benefits to a commercial organization or a government agency when working with your firm?

We realize that what we're doing is something totally new. As you know, there are plenty of tools for information/entity extraction, etc. But the way we focus on higher end concepts such as events and time we'd like to think is fairly unique. We've packaged all of this up into a hosted service that users can access with out even having to think about "entity extraction" or the like.

Building such a service certainly takes some fine tuning. We try to be very humble about that. And we have been most fortunate to build a solid group of customers who're very successful in using our tools. Quite encouraging!

How does an information retrieval engagement move through its life cycle?

Because we deliver a hosted service, we can listen to the customer and respond to each customer’s requirements. As a cloud or hosted service, we essentially manage the whole cycle. We add new sources on a continuous basis, new concepts that are extracted, new UI improvement, new API calls, etc.

Our commitment to innovation and system enhancements is a big part of the ethos of the company.

One challenge to those involved with squeezing useful elements from large volumes of content is the volume of content AND the rate of change in existing content objects. What does your firm provide to customers to help them deal with the problems of “big data”?

This is a great question. Of course, there is delay in information. When a barge with cobalt get stuck somewhere in Congo and the information eventually hits a trading floor in Chicago that information doesn't travel in milliseconds.

What we do is to tag information very, very carefully. For example, we add metatags that make explicit when we locate an item of data. We tag when that datum was published. We tag when we analyzed that datum. We also tag when we find it, when it was published, when we analyzed it, and what actual time point (past, present, future) to which the datum refers. The time precision is quite important. Time makes it possible for end users and modelers to deal with this important attribute.

At this stage in our technology’s capabilities, we're not trying to claim that we can beat someone like Reuters or Bloomberg at delivering a piece of news the fastest. But if you're interested in monitoring, for example, the co-incidence of an insider trade with a product recall we can probably beat most at that.

Another challenge, particularly in professional intelligence operations, is moving data from point A to point B; that is, information enters a system but it must be made available to an individual who needs that information or at least must know about the information. What does your firm offer licensees to address this issue of content "push", report generation, and personalization within a work flow?

Okay, good points. We've built and provided integrations to environments such as Google Spreadsheets, R, Spotfire, etc. We have loads and loads of ideas for how our data can hit productivity environments. I can’t reveal the details of what is coming from Recorded Future. I can ask you to take a close look at the enhancements to our cloud service that will become available in the near future?

Is that a “recorded future?”

Yes, the enhancements I referenced are a 0.999999 probability of becoming available. Our customers are quite vocal in their needs, and we are responding as you and I are talking today.

There has been a surge in interest in putting "everything" in a repository and then manipulating the indexes to the information in the repository. On the surface, this seems to be gaining traction because network resident information can "disappear" or become unavailable. What's your view of the repository versus non repository approach to content processing? What are the "hooks" between content processed by Recorded Future and a more traditional type of analytics system?

To be honest, at this stage we have really not worked at this at all. We are focused on Internet content and will be tackling other types of content in the near future.

No problem. I appreciate your focus. Let me ask about visualization. Visualization has been a great addition to briefings. On the other hand, visualization and other graphic eye candy can be a problem to those in stressful operational situations? What's your firm's approach to presenting "outputs" that end users can easily ingest?

I think the challenge is not so much whether visualization is good or not. I know that you and I can discuss at length ways to help an end user get the fact or insight needed. I think the key is that in some use cases users would like to explore/visualize/analyze very actively, whereas in other use cases users would like to just be alerted for interesting patterns or see a report that someone else has done. For example in Recorded Future anyone can share an analysis on Twitter or Facebook. That's the level of ease we want to provide for sharing.

Thank you for providing such interesting insights into Recorded Future’s capabilities. However, I am on the fence about the merging of retrieval within other applications. What's your take on the "new" method which some people describe as "search enabled applications"?

Our plan is not really at all to compete with someone like Autonomy, Endeca, or Exalead. We want to build great service which indexes "the Internet" and makes it available for analysis. We believe that will be very valuable for people in finance, government, marketing, sales, etc. Think about it perhaps as the next generation of business intelligence.

There seems to be a popular perception that the world will be doing computing via iPad devices and mobile phones. My concern is that serious computing infrastructures are needed and that users are "cut off" from access to more robust systems? How does Recorded Future Recorded Future see the computing world over the next 12 to 18 months?

You are right. There certainly is a massive thrust of data and applications moving into the cloud. We'd like to ride that wave. Business intelligence and search has been behind there. But it's happening now and we'll ride that wave. But I think you're right to be concerned that this might, in fact, cut off users from internal data and applications. We have some interesting strategies in mind to address this.

Put on your wizard hat. What are the three most significant technologies that you see affecting your search business?

The three trends affecting what we're doing are, first, the shift from on premises systems to the cloud. Data and tools are moving into hosted data centers aooooooooooond the pace is, in my opinion, accelerating.

Second, I am interested in the rapid uptake of scalable database technologies. These systems allow us to index very large amounts of data. Recorded Future’s innovations thrive on large volumes of data.

Third, I think HTML5 is important. That technology allows us to build very compelling, Web-based user experiences.

Where does a reader get more information about Recorded Future ?

I think I would suggest that anyone wanting more information visit I also want to invite people to read our blogs such as and You can also email us at

ArnoldIT Comment

Recorded Future is going to be a disruptive company. The firm has a solid base of customers among governmental entities in the US and in Europe. With the support of Google, Recorded Future is going to find that interest among Google’s enterprise customers is a certainty. Like other companies offering next-generation technology, Recorded Future will have to continue to innovate and ward off competitive thrusts from giants like IBM, Oracle, and SAP. In addition, established players in analytics like i2 Ltd and the upstart Palantir will challenge Recorded Future in certain markets. Nevertheless, our view is that the future of Recorded Future is bright. This is a company to watch.
User avatar
Wombaticus Rex
Posts: 10614
Joined: Wed Nov 08, 2006 6:33 pm
Location: Vermontistan
Blog: View Blog (0)

Re: Data Mining & Intelligence Agencies

Postby Wombaticus Rex » Wed Apr 04, 2012 10:57 pm

Kevin Slavin argues that we're living in a world designed for -- and increasingly controlled by -- algorithms. In this riveting talk from TEDGlobal, he shows how these complex computer programs determine: espionage tactics, stock prices, movie scripts, and architecture. And he warns that we are writing code we can't understand, with implications we can't control.

Kevin Slavin is the Chairman and co-Founder of Area/Code. Founded in 2005, Area/Code creates cross-media games and entertainment for clients including Nokia, CBS, Disney Imagineering, MTV, Discovery Networks, A&E Networks, Nike, Puma, EA, the UK’s Department for Transport, and Busch Entertainment.

Area/Code builds on the landscape of pervasive technologies and overlapping media to create new kinds of entertainment. They have built mobile games with invisible characters that move through real-world spaces, online games synchronized to live television broadcasts, and videogames in which virtual sharks are controlled by real-world sharks with GPS receivers stapled to their fins. Their Facebook game “Parking Wars” served over 1 billion pages in 2008.

Before founding Area/Code, Slavin spent over 10 years in ad agencies including DDB, TBWA\Chiat\Day and SS+K, focused primarily on technology, networks, and community. His work has been recognized through many industry awards and press.

Area/Code’s work has received awards from the Clios, the One Club, Creativity, and many others, and the co-founders were recently named to the Creativity 50 and the Gamasutra 20. Slavin has spoken at the BBC, Ad Age, 5D, MoMA, the Van Alen Institute, the Guardian, DLD, the Cooper Union, the Storefront for Art and Architecture, and NBC, and together with Adam Greenfield he teaches Urban Computing at NYU’s Interactive Telecommunications Program. His work has been exhibited internationally, including the Design Museum of London and the Frankfurt Museum fuer Moderne Kunst.


source: ... ction.html

Beyond Prediction

By Karl Schroeder

I've just spent two years working toward a Master's degree in Strategic Foresight and Innovation.

Because most people look at me blankly when I tell them this, I've developed two ways to describe what what I'm doing, and foresight is. The first is to say that foresight used to be called futurism, but that futurism has increasingly become associated with the idea of predicting the future. Foresight is not about predicting the future, it's about minimizing surprise. The second way I usually put it is that foresight is not about predicting the future; it's about designing the future.

Actually, I'll say it's just about anything, as long as it's understood that foresight is not about predicting the future.

The reason is that, frankly, I'm pretty tired of all those, "Dude, where's my flying car!" digs. There's always been a certain brand of futurist who's obsessed with getting it right: with racking up successful predictions like some modern-day Nostradamus. I'm sure you know who I'm talking about; some futurists play the prediction game very well, but in the end it is a game, and closer to charlatanism than it is to science. There's actually no method for seeing the future, and nobody's predictions are more reliable than anybody else's.

If actual prediction were possible, the insurance companies would be all over it. They don't try to predict how you're going to die, though, do they? They look at trends and probabilities, and try to minimize surprise for their investments. That's exactly how strategic foresight works--as a kind of institutional insurance policy against disruptive surprise. There's a whole raft of methodologies for this, ranging from Delphi polls to trends analysis and scenarios. For me, this way of looking at the future is complementary to my other way of looking, which is the more fun and disreputable wild-eyed prophet--that is, as a science fiction writer.

There are no limits on me when I write SF. In contrast, doing foresight is a disciplined activity. I like this combination; I'm finding that each way of looking forward influences and improves the other--as long as I don't get the two confused.

I'm still coming to grips with how these two years will affect my writing. One result of undertaken the programme is that I've developed a different attitude toward writing near-future SF. Most writers I know avoid at all costs writing about the near future, because nothing goes out of date quicker than next year. I've always tended to agree with this assessment and--because SF writers aren't in the job of predicting the future either--have tended to set my novels and short stories very, very far in the future. Thousands of years, usually.

I'm no longer satisfied with doing that. There's the little matter of my second way of describing what foresight is: not as prediction, but design. If you're afraid of being a poor predictor of the near future, you'll avoid writing about it. But what if you were never out to predict in the first place? What if you don't care if a story you set in 2012 gets immediately overtaken by events? What if you set the action there not to predict some event or outcome, but to encourage some action on the part of your readers?

In other words I have a new ambition for my own SF: not as prediction, and not cautionary, either--but aspirational.

The fact is that if I've learned one thing in two years of studying how we think about the future, it's that the one thing that's sorely lacking in the public imagination is positive ideas about where we should be going. We seem to do everything about our future except try to design it. It's a funny thing: nobody ever questions your credentials if you predict doom and destruction. But provide a rosy picture of the future, and people demand that you justify yourself. Increasingly, though, I believe that while warning people of dire possibilities is responsible, providing them with something to aspire to is even more important. The foresight programme has given me a lot of tools to do that in a justifiable way, so I might as well use them.

Now all I have to do is put my money where my mouth is. By, say, writing an optimistic, aspirational novel set in the near future and unflinchingly accurate to the possibilities, both positive and negative, of the next few years?

Yeah, okay. --At least, I'm going to try.
User avatar
Wombaticus Rex
Posts: 10614
Joined: Wed Nov 08, 2006 6:33 pm
Location: Vermontistan
Blog: View Blog (0)

Re: Data Mining & Intelligence Agencies

Postby Wombaticus Rex » Wed Apr 04, 2012 11:03 pm

HUGE info-dump goldmine going up on public intelligence this week.

Meet Catalyst: IARPA's Entity and Relationship Extraction Program

The Office of the Director of National Intelligence (ODNI) is building a computer system capable of automatically analyzing the massive quantities of data gathered across the entire intelligence community and extracting information on specific entities and their relationships to one another. The system which is called Catalyst is part of a larger effort by ODNI to create software and computer systems capable of knowledge management, entity extraction and semantic integration, enabling greater analysis and understanding of complex, multi-source intelligence throughout the government.

The intelligence community has been working for years to develop software and analytical frameworks capable of large-scale data analysis and extraction. Technological advances have now made it possible for spy agencies to not just capture the incredible amount of data flowing through public and private networks around the world, but to parse, contextualize and understand the intelligence that is being gathered. Automated software programs are now capable of integrating data into semantic systems, providing context and meaning to names, dates, photographs and practically any kind of data you can imagine.

Many agencies within the intelligence community have already created systems to do this sort of semantic integration. The Office of Naval Intelligence uses a system called AETHER “to correlate seemingly disparate entities and relationships, to identify networks of interest, and to detect patterns.” The NSA runs a program called APSTARS that provides “semantic integration of data from multiple sources in support of intelligence processing.” The CIA has a program called Quantum Leap that is designed to “find non-obvious linkages, new connections, and new information” from within a dataset. Several similar programs were even initiated by ODNI including BLACKBOOK and the Large Scale Internet Exploitation Project (LSIE).

Catalyst is an attempt to create a unified system capable of automatically extracting complex information on entities as well as the relationships between them while contextualizing this information within semantic systems. According to its specifications, Catalyst will be capable of creating detailed histories of people, places and things while mapping the interrelations that detail those entities’ interactions with the world around them. A study conducted by IARPA states that Catalyst is designed to incorporate data from across the entire intelligence community, creating a centralized repository of available information gathered from all agencies:

Many IC organizations have recognized this problem and have programs to extract information from the resources, store it in an appropriate form, integrate the information on each person, organization, place, event, etc. in one data structure, and provide query and analysis tools that run over this data. Whereas this is a significant step forward for an organization, no organization is looking at integration across the entire IC. The DNI has the charter to integrate information from all organizations across the IC; this is what Catalyst is designed to do with entity data. The promise of Catalyst is to provide, within the security constraints on the data, access to “all that is known” within the IC on a person, organization, place, event, or other entity. Not what the CIA knows, then what DIA knows, and then what NSA knows, etc., and put the burden on the analyst to pull it all together, but have Catalyst pull it all together so that analysts can see what CIA, DIA, NSA, etc. all know at once. The value to the intelligence mission, should Catalyst succeed, is nothing less than a significant improvement in the analysis capability of the entire IC, to the benefit of the national security of the US.

To fully grasp the capabilities of such a system, it is important to understand the concepts of “semantic integration” and “entity extraction” that Catalyst will perform. Using an example described in the IARPA study, we will follow data through the stages of processing in a Catalyst system:

For example, some free text may include “… Joe Smith is a 6’11″ basketball player who plays for the Los Angeles Lakers…” from which the string “Joe Smith ” may be delineated as an entity of class Athlete (a subclass of People) having property Name with value JoeSmith and Height with value 6’11″ (more on this example below). Note that it is important to distinguish between an entity and the name of the entity, for an entity can have multiple names (JoeSmith, JosephSmith, JosephQSmith, etc.).

Once entities and their associated relationship values are determined, the information is then integrated into a knowledge base to produce a semantic graph:

To continue the example, one entry in the knowledge base is the entity of class Athlete with (datatype property) Name having value JoeSmith, another is the entity of class SportsFranchise with Name having value Lakers, and another is an entity of class City having value LosAngeles. If each of these is viewed as a node in a graph, then an edge connecting the node (entity) with Name JoeSmith to the node with Name Lakers is named MemberOf and the edge connecting the node with Name Lakers to the node with Name LosAngeles is named LocatedIn. Such edges, corresponding to relationships (object properties) and have a direction; for example, JoeSmith is a MemberOf the Lakers, but the Lakers are not a MemberOf JoeSmith (there may be an inverse relationship, such as HasMember, that is between the Lakers and JoeSmith.).

Data that has been extracted and integrated can then produce patterns that determine unknown relations between an entity and other entities that may be of concern to a particular intelligence agency:

Another simple pattern could be: JoeSmith Owns Automobile, or Person Owns an instance of the class Automobile with Manufacturer Lexus and LicensePlate VA-123456 or even JoeSmith has-unknown-relationship-with an instance of the class Automobile with Manufacturer Lexus and LicensePlate VA-123456. In these last three examples, one of the entities or the relationship is uninstantiated. Note that JoeSmith Owns an instance of the class Automobile with Manufacturer Lexus and LicensePlate VA-123456 is not a pattern, for it has no uninstantiated entities or relationships. A more complex pattern could be: Person Owns Automobile ParticipatedIn Crime HasUnknownRelationshipWith Organization HasAffiliationWith TerroristOrganization. Any one or more of the entities and the has-unknown-relationship-with relationship (but not all) can be instantiated and it would still be a pattern, such as JoeSmith Owns Automobile ParticipatedIn Crime PerpetratedBy Organization HasAffiliationWith HAMAS.

While this example only provides a limited view of Catalyst functionality, it nonetheless helps to demonstrate the potential capabilities of the system. Far more detailed explanations of the system, as well as a useful overview of similar government systems across the intelligence community, are provided in IARPA’s one-hundred and twenty-two page study. ... al-report/

ICE Pattern Analysis and Information Collection ... ic-system/


* Stores over 332,000,000 records on more than 254,000,000 entities
* Allows “disparate sources of information to be analyzed to find previously unknown relationship data about individuals that are the subject of active investigation”
* Usage of the system more than doubled from August 2010 to February 2011

The ICE Pattern Analysis and Information Collection (ICEPIC) system was established in 2008 to enable ICE law enforcement agents and analysts to look for non-obvious relationship patterns among individuals and organizations that are indicative of violations of the customs and immigration laws that are enforced by DHS agencies, as well as possible terrorist threats and plots. From these relationships, ICE agents develop specific leads and intelligence for active and new investigations. Identified relationships are also recorded for reuse in subsequent investigative analyses. The information processed by ICEPIC comes from existing ICE investigative and apprehension records systems, as well as immigration and alien admission records systems. ICEPIC includes capabilities that assist investigators in recording results of analyses performed in support of investigations and in capturing additional relevant information obtained from outside sources. The information collected by, on behalf of, in support of, or in cooperation with DHS and its components may contain personally identifiable information collected by other Federal, State, local, tribal, foreign government agencies, or international organizations.

ICEPIC is a set of information analysis tools which allow disparate sources of information to be analyzed to find previously unknown relationship data about individuals who are the subject of ongoing and valid investigations. Relationship data is made up of information about how a place, person, or thing (e.g., automobile or other piece of property) relates to other persons, places, or things. For example, ICEPIC can determine relationship data about how certain events occurred at a certain address, or certain individuals under investigation who have shared the same address in the past. ICEPIC also includes capabilities that assist investigators in recording results of analyses performed in support of investigations. All ICEPIC searches are conducted with the appropriate predicate for a search, i.e., ongoing investigation into a violation of law.

Use of the ICEPIC System

ICEPIC enables ICE investigators and analysts to recognize non-obvious relationships among persons, resolve addresses collected in varied formats, understand organizational relationships using information within existing DHS record systems, and develop actionable leads needed to accomplish ICE law enforcement objectives.1

According to the DHS Privacy Impact Assessment of the ICEPIC system, information is used to generate analytical reports to identify potential violations of customs or immigration law, confirm suspected violations, or investigate potential terrorist threats. The system automates five processes for investigators:

1. Analysis of leads, law enforcement and intelligence reports, and referrals, and processing of queries of ICE and DHS information to locate relevant records and produce reports.
2. Integration and resolution of information from multiple ICE and DHS databases to provide leads for law enforcement investigations and disruption of potential terrorist activities.
3. Initiation of analyses that support investigative cases in ICE headquarters and field offices and recording of the results of beneficial analyses.
4. Production and dissemination of target indicator profiles and other intelligence.
5. Management of analysis workflows and information resources.2

Reports generated through the use of information in ICEPIC are used by the Department of Justice (DOJ) or other federal agencies in the review, settlement, and prosecution of claims, complaints, and lawsuits involving matters over which ICE exercises jurisdiction or when conducting litigation or in proceedings before any court, adjudicative, or administrative body. This includes any litigation matters where ICE, DOJ, or an employee that is acting in his or her official capacity in support of ICE, the United States, or any agency thereof is involved. The Federal Bureau of Investigation (FBI) uses ICEPIC information when ICE becomes aware of information that may be related to an individual in terrorism-related activity.

Information in the ICEPIC System

The information in ICEPIC consists of the biographical and biometric information obtained from individuals during DHS enforcement encounters or provided by individuals when applying for U.S. immigration benefits or admission to the U.S. Biographical data includes name, aliases, date of birth, phone numbers, addresses, and nationality; biometric information includes fingerprints and photographs. Prior law enforcement encounter information consists of data related to an individual’s case, including immigration history, alien registration information, and other identification or record numbers.

Agents/analysts may consult commercial data providers in order to verify information within ICEPIC. As a specific example, an agent/analyst investigating a lead may encounter a gap in residential addresses for a particular individual. The agent/analyst may consult a commercial data provider to search for the missing address. How much weight to give the information from commercial data provider is left to the professional discretion of the agent/analyst.

The ICEPIC system retains records for ten (10) years from ICE’s last use of the individual’s data, and then archives the information for an additional five (5) years. After the five (5) year period, information is destroyed unless it has become relevant to a legal action, at which point the retention schedule would reset.

Based on the need to know, ICE may share analytical reports generated from ICEPIC information with other parts of DHS including both law enforcement and intelligence agencies. These may include: the DHS Operations Center, U.S. Secret Service, the U.S. Coast Guard, U.S. Customs and Border Protection, Transportation Security Administration, and the Office of Intelligence and Analysis. Organizations will only receive the information which they are authorized to receive.3

ICE shares analytical reports (not raw data) generated from ICEPIC information with law enforcement or intelligence agencies that demonstrate a need to know the information in the performance of their missions, including Federal, State, tribal, local and foreign law enforcement agencies, as well as relevant international organizations such as INTERPOL. As mentioned above, DOJ and the FBI receive analytical reports regularly, but external sharing is not limited to those agencies. According to the DHS Privacy Impact Assessment, all sharing is in accord with the ICEPIC System of Records Notice and the Privacy Act.4

Information is transmitted via hand-delivered reports and via e-mail over DHS sensitive but unclassified (SBU) networks, the DHS Homeland Security Data Network (HSDN), and the Department of Defense’s Secret Internet Protocol Router Network (SIPRNET). Reports are marked as “For Official Use Only/Law Enforcement Sensitive.” As a pre-condition to receiving such reports, ICE prohibits the recipient agency from further disseminating the information without prior approval from ICE.5 Contractors have access to all systems, including ICEPIC developers and information technology operations and maintenance staff. 6

Number of Records in ICEPIC

As of February 17, 2011 the ICEPIC Core contained 332,985,720 records on 254,105,241 entities. The number of entities contained in the database has risen by over 1,141,114 since November 2010. The number of records in the database has grown by over 3,946,245 in the same time.
User avatar
Wombaticus Rex
Posts: 10614
Joined: Wed Nov 08, 2006 6:33 pm
Location: Vermontistan
Blog: View Blog (0)

Re: Data Mining & Intelligence Agencies

Postby Wombaticus Rex » Wed Apr 04, 2012 11:16 pm

ECHELON: America's Secret Global Surveillance Network
by Patrick S. Poole

Read this previous Privacy Paper by Patrick Poole:
Inside America's Secret Court: The Foreign Intelligence Surveillance Court

Read some of the press clippings on this report:
NY Times (May 27, 1999): Lawmakers Raise Questions About International Spy Network
La Monde Diplomatique (Jan. 1999): How the United States Spies on us all
Federal Computer Week(Nov. 17, 1998): EU May Investigate US Global Spy Network
Inter@ctive Week (Nov. 16, 1998): ECHELON: Surveilling Surveillance
WorldNetDaily (Nov. 12, 1998): Push for Hearings on ECHELON
Wired (October 27, 1998): Spying on the Spies
Baltimore Sun (October 18, 1998): Putting NSA under scrutiny

Executive Summary

In the greatest surveillance effort ever established, the US National Security Agency (NSA) has created a global spy system, codename ECHELON, which captures and analyzes virtually every phone call, fax, email and telex message sent anywhere in the world. ECHELON is controlled by the NSA and is operated in conjunction with the General Communications Head Quarters (GCHQ) of England, the Communications Security Establishment (CSE) of Canada, the Australian Defense Security Directorate (DSD), and the General Communications Security Bureau (GCSB) of New Zealand. These organizations are bound together under a secret 1948 agreement, UKUSA, whose terms and text remain under wraps even today.

The ECHELON system is fairly simple in design: position intercept stations all over the world to capture all satellite, microwave, cellular and fiber-optic communications traffic, and then process this information through the massive computer capabilities of the NSA, including advanced voice recognition and optical character recognition (OCR) programs, and look for code words or phrases (known as the ECHELON “Dictionary”) that will prompt the computers to flag the message for recording and transcribing for future analysis. Intelligence analysts at each of the respective “listening stations” maintain separate keyword lists for them to analyze any conversation or document flagged by the system, which is then forwarded to the respective intelligence agency headquarters that requested the intercept.

But apart from directing their ears towards terrorists and rogue states, ECHELON is also being used for purposes well outside its original mission. The regular discovery of domestic surveillance targeted at American civilians for reasons of “unpopular” political affiliation or for no probable cause at all in violation of the First, Fourth and Fifth Amendments of the Constitution – are consistently impeded by very elaborate and complex legal arguments and privilege claims by the intelligence agencies and the US government. The guardians and caretakers of our liberties, our duly elected political representatives, give scarce attention to these activities, let alone the abuses that occur under their watch. Among the activities that the ECHELON targets are:

Political spying: Since the close of World War II, the US intelligence agencies have developed a consistent record of trampling the rights and liberties of the American people. Even after the investigations into the domestic and political surveillance activities of the agencies that followed in the wake of the Watergate fiasco, the NSA continues to target the political activity of “unpopular” political groups and our duly elected representatives. One whistleblower charged in a 1988 Cleveland Plain Dealer interview that, while she was stationed at the Menwith Hill facility in the 1980s, she heard real-time intercepts of South Carolina Senator Strom Thurmond. A former Maryland Congressman, Michael Barnes, claimed in a 1995 Baltimore Sun article that under the Reagan Administration his phone calls were regularly intercepted, which he discovered only after reporters had been passed transcripts of his conversations by the White House. One of the most shocking revelations came to light after several GCHQ officials became concerned about the targeting of peaceful political groups and told the London Observer in 1992 that the ECHELON dictionaries targeted Amnesty International, Greenpeace, and even Christian ministries.

Commercial espionage: Since the demise of Communism in Eastern Europe, the intelligence agencies have searched for a new justification for their surveillance capability in order to protect their prominence and their bloated budgets. Their solution was to redefine the notion of national security to include economic, commercial and corporate concerns. An office was created within the Department of Commerce, the Office of Intelligence Liaison, to forward intercepted materials to major US corporations. In many cases, the beneficiaries of this commercial espionage effort are the very companies that helped the NSA develop the systems that power the ECHELON network. This incestuous relationship is so strong that sometimes this intelligence information is used to push other American manufacturers out of deals in favor of these mammoth US defense and intelligence contractors, who frequently are the source of major cash contributions to both political parties.

While signals intelligence technology was helpful in containing and eventually defeating the Soviet Empire during the Cold War, what was once designed to target a select list of communist countries and terrorist states is now indiscriminately directed against virtually every citizen in the world. The European Parliament is now asking whether the ECHELON communications interceptions violate the sovereignty and privacy of citizens in other countries. In some cases, such as the NSA’s Menwith Hill station in England, surveillance is conducted against citizens on their own soil and with the full knowledge and cooperation of their government.

This report suggests that Congress pick up its long-neglected role as watchdog of the Constitutional rights and liberties of the American people, instead of its current role as lap dog to the US intelligence agencies. Congressional hearings ought to be held, similar to the Church and Rockefeller Committee hearings held in the mid-1970s, to find out to what extent the ECHELON system targets the personal, political, religious, and commercial communications of American citizens. The late Senator Frank Church warned that the technology and capability embodied in the ECHELON system represented a direct threat to the liberties of the American people. Left unchecked, ECHELON could be used by either the political elite or the intelligence agencies themselves as a tool to subvert the civil protections of Constitution and to destroy representative government in the United States.


The culmination of the Cold War conflict brought home hard realities for many military and intelligence agencies who were dependent upon the confrontation for massive budgets and little civilian oversight. World War II Allied political and military alliances had quickly become intelligence alliances in the shadow of the Iron Curtain that descended upon Eastern Europe after the war.

But for some intelligence agencies the end of the Cold War just meant a shift in mission and focus, not a loss of manpower or financial resources. One such US governmental organization is the National Security Agency (NSA). Despite the disintegration of Communism in the former Soviet Union and throughout Eastern Europe, the secretive NSA continues to grow at an exponential rate in terms of budget, manpower and spying abilities. Other countries have noticed the rapid growth of NSA resources and facilities around the world, and have decried the extensive spying upon their citizens by the US.

A preliminary report released by the European Parliament in January 1998 detailed research conducted by independent researchers that uncovered a massive US spy technology network that routinely monitors telephone, fax and email information on citizens all over the world, but particularly in the European Union (EU) and Japan. Titled “An Appraisal of Technologies of Political Control,”<1> this report, issued by the Scientific and Technological Options Assessment (STOA) committee of the European Parliament, caused a tremendous stir in the establishment press in Europe. At least one major US media outlet, The New York Times,<2> covered the issuance of the report as well.

The STOA report also exposed a festering sore spot between the US and our EU allies. The widespread surveillance of citizens in EU countries by the NSA has been known and discussed by European journalists since 1981. The name of the system in question is ECHELON, and it is one of the most secretive spy systems in existence.

ECHELON is actually a vast network of electronic spy stations located around the world and maintained by five countries: the US, England, Canada, Australia, and New Zealand. These countries, bound together in a still-secret agreement called UKUSA, spy on each other’s citizens by intercepting and gathering electronic signals of almost every telephone call, fax transmission and email message transmitted around the world daily. These signals are fed through the massive supercomputers of the NSA to look for certain keywords called the ECHELON “dictionaries.”

Most of the details of this mammoth spy system and the UKUSA agreement that supports it remain a mystery. What is known of ECHELON is the result of the efforts of journalists and researchers around the world who have labored for decades to uncover the operations of our government’s most secret systems. The 1996 publication of New Zealand journalist Nicky Hager’s book, Secret Power: New Zealand’s Role in the International Spy Network,<3> provided the most detailed look at the system and inflamed interest in ECHELON as well as the debate regarding its propriety.

This paper examines the expanse of the ECHELON system along with the intelligence agreements and exchanges that support it. The operation of ECHELON serves the NSA’s goal of spying on the citizens of other countries while also allowing them to circumvent the prohibition on spying on US citizens. ECHELON is not only a gross violation of our Constitution, but it violates the good will of our European allies and threatens the privacy of innocent civilians around the world. The existence and expansion of ECHELON is a foreboding omen regarding the future of our Constitutional liberties. If a government agency can willingly violate the most basic components of the Bill of Rights without so much as Congressional oversight and approval, we have reverted from a republican form of government to tyranny.

The Parties

The success of the Allied military effort in World War II was due in no small part to successes in gathering enemy intelligence information and cracking those military and diplomatic messages. In addition, the Allied forces were able to create codes and encryption devices that effectively concealed sensitive information from prying Axis Power eyes. These coordinated signal intelligence (SIGINT) programs kept Allied information secure and left the enemies vulnerable.

But at the close of the conflict, a new threatening power – the Soviet Union – was beginning to provoke the Cold War by enslaving Eastern Europe. These signal intelligence agencies now had a new enemy toward which to turn their electronic eyes and ears to ensure that the balance of power could be maintained. The volleys of electronic hardware and espionage that would follow for forty years would be the breeding ground of the ECHELON spy system.

The diplomatic foundation that was the genesis of ECHELON is the UKUSA agreement. The agreement has its roots in the BRUSA COMINT (communications intelligence) alliance formed in the early days of World War II and ratified on May 17, 1943 by the United Kingdom and the United States.<4> The Commonwealth SIGINT Organization formed in 1946-47 brought together the UK, Canada, Australia and New Zealand post-war intelligence agencies.<5> Forged in 1947 between the US and UK, the still-secret UKUSA agreement defined the relations between the SIGINT departments of those various governments. Direct agreements between the US and these agencies also define the intricate relationship that these organizations engage in.

Foremost among those agencies is the US National Security Agency (NSA), which represents the American interest. The NSA is designated as the “First Party to the Treaty.” The Government Communications Headquarters (GCHQ) signed the UKUSA agreement on behalf of the UK and its Commonwealth SIGINT partners. This brought Australia’s Defense Signals Directorate (DSD), the Canadian Communications Security Establishment (CSE) and New Zealand’s Government Communications Security Bureau (GCSB) into the arrangement. While these agencies are bound by additional direct agreements with the US and each other, these four countries are considered the “Second Parties to the (UKUSA) Treaty.” Third Party members include Germany, Japan, Norway, South Korea and Turkey. There are sources that indicate China may be included in this group on a limited basis as well.<6>

National Security Agency (US)

The prime mover in the UKUSA arrangement is undeniably the National Security Agency (NSA). The majority of funds for joint projects and facilities (discussed below) as well as the direction for intelligence gathering operations are issued primarily through the NSA. The participating agencies frequently exchange personnel, divide up intelligence collection tasks and establish common guidelines for classifying and protecting shared information. However, the NSA utilizes its role as the largest spy agency in the world to have its international intelligence partners do its bidding.

President Harry Truman established the NSA in 1952 with a presidential directive that remains classified to this day. The US government did not acknowledge the existence of the NSA until 1957. Its original mission was to conduct the signal intelligence (SIGINT) and communications security (COMSEC) for the US. President Ronald Reagan added the tasks of information systems security and operations security training in 1984 and 1988 respectively. A 1986 law charged the NSA with supporting combat operations for the Department of Defense.<7>

Headquartered at Fort George Meade, located between Washington D.C. and Baltimore, Maryland, the NSA boasts the most enviable array of intelligence equipment and personnel in the world. The NSA is the largest global employer of mathematicians, featuring the best teams of codemakers and codebreakers ever assembled. The latter's job is to crack the encryption codes of foreign and domestic electronic communications, forwarding the revealed messages to their enormous team of skilled linguists to review and analyze the messages in over 100 languages. The NSA is also responsible for creating the encryption codes that protect the US government’s communications.

In its role as gang leader for UKUSA, the NSA is primarily involved with creating new surveillance and codebreaking technology, directing the other cooperating agencies to their targets, and providing them with training and tools to intercept, process and analyze enormous amounts of signals intelligence. By possessing what is arguably the most technologically advanced communications, computer and codebreaking equipment of any government agency in the world, the NSA serves as a competent and capable taskmaster for UKUSA.

The ECHELON Network

The vast network created by the UKUSA community stretches across the globe and into the reaches of space. Land-based intercept stations, intelligence ships sailing the seven seas and top-secret satellites whirling twenty thousand miles overhead all combine to empower the NSA and its UKUSA allies with access to the entire global communications network. Very few signals escape its electronic grasp.

Having divided the world up among the UKUSA parties, each agency directs its electronic "vacuum-cleaner" equipment towards the heavens and the ground to search for the most minute communications signals that traverse the system’s immense path. The NSA facilities in the US cover the communications signals of both American continents; the GCHQ in Britain is responsible for Europe, Africa and Russia west of the Ural Mountains; the DSD in Australia assists in SIGINT collection in Southeastern Asia and the Southwest Pacific and Eastern Indian Ocean areas; the GSCB in New Zealand is responsible for Southern Pacific Ocean collections, particularly the South Pacific island nations group; and CSE in Canada handles interception of additional northern Russian, northern European and American communications.<8>

The Facilities

The backbone of the ECHELON network is the massive listening and reception stations directed at the Intelsat and Inmarsat satellites that are responsible for the vast majority of phone and fax communications traffic within and between countries and continents. The twenty Intelsat satellites follow a geo-stationary orbit locked onto a particular point on the Equator.<9> These satellites carry primarily civilian traffic, but they do additionally carry diplomatic and governmental communications that are of particular interest to the UKUSA parties.

Originally, only two stations were responsible for Intelsat intercepts: Morwenstow in England and Yakima in the state of Washington. However, when the Intelsat 5 series was replaced with the Intelsat 701 and 703 satellites, which had much more precise transmission beams that prohibited reception of Southern Hemisphere signals from the Yakima base in the Northern Hemisphere, additional facilities were constructed in Australia and New Zealand.<10>

Today, the Morwenstow station directs its ears towards the Intelsats traversing the atmosphere above the Atlantic and Indian Oceans and transmiting to Europe, Africa and western parts of Asia. The Yakima station, located on the grounds of the Yakima Firing Station, targets Pacific Ocean communications in the Northern Hemisphere as well as the Far East. Another NSA facility at Sugar Grove, West Virginia, covers traffic for the whole of North and South America. A DSD station at Geraldton, Australia, and the Waihopai, New Zealand GCSB facility cover Asia, the South Pacific countries and the Pacific Ocean. An additional station on Ascension Island in the Atlantic Ocean between Brazil and Angola is suspected of covering the Atlantic Intelsat’s Southern Hemisphere communications.<11>

Non-Intelsat satellites are monitored from these same stations, as well as from bases in Menwith Hill, England; Shoal Bay, Australia; Leitrim, Canada; Bad Aibling, Germany, and Misawa, Japan. These satellites typically carry Russian and regional communications.<12> It is known that the Shoal Bay facility targets a series of Indonesian satellites and that the Leitrim station intercepts communications from Latin American satellites, including the Mexican telephone company's Morelos satellite.<13>

Several dozen other radio listening posts operated by the UKUSA allies dot the globe as well, located at military bases on foreign soil and remote spy posts. These stations played a critical role in the time prior to the development of satellite communications because much of the world’s communications traffic was transmitted on radio frequency bands. Particularly in the high-frequency (HF) range, radio communications continue to serve an important purpose despite the widespread use of satellite technology because their signals can be transmitted to military ships and aircraft across the globe. Shorter range very high-frequencies (VHF) and ultra high-frequencies (UHF) are also used for tactical military communications within national borders. Major radio facilities in the UKUSA network include Tangimoana, New Zealand; Bamaga, Australia, and the joint NSA/GCHQ facility at the Indian Ocean atoll of Diego Garcia.<14>

A separate high frequency direction finding (HFDF) network intercepts communications signals for the unique purpose of locating the position of ships and aircraft. While these stations are not actually involved in the analysis of messages, they play a critical role in monitoring the movements of mobile military targets. The Canadian CSE figures prominently in the HFDF UKUSA network, codenamed CLASSIC BULLSEYE and hosting a major portion of the Atlantic and Pacific stations that monitored Soviet ship and submarine movements during the Cold War. Stations from Kingston and Leitrim (Ontario) to Gander (Newfoundland) on the Atlantic side, to Alert (Northwest Territories) located at the northernmost tip of Canada on the Arctic Ocean that listens to the Russian submarine bases at Petropavlovsk and Vladivostok, and finally to Masset (British Columbia) in the Pacific -- monitor shipping and flight lanes under the direction of the NSA.<15>. The CSE also maintains a small contingent at Lackland Air Force Base in San Antonio, Texas, which probably monitors Latin American communications targets.

Another major support for the ECHELON system is the US spy satellite network and its corresponding reception bases scattered about the UKUSA empire. These space-based electronic communications "vacuum cleaners" pick up radio, microwave and cell phone traffic on the ground. They were launched by the NSA in cooperation with its sister spy agencies, the National Reconnaissance Office (NRO) and the Central Intelligence Agency (CIA). The Ferret series of satellites in the 1960s; the Canyon, Rhyolite and Aquacade satellites in the 1970s; and the Chalet, Vortex, Magnum, Orion, and Jumpseat series of satellites in the 1980s, have given way to the new and improved Mercury, Mentor and Trumpet satellites during the 1990s.

Table I. US Spy Satellites in Current UseSatellite No. Orbit Manufacturer Purpose
Advanced KH-11 3 200 miles Lockheed Martin 5-inch resolution spy photographs
LaCrosse Radar Imaging 2 200-400 miles Lockheed Martin 3 to 10-foot resolution spy photographs
Orion/Vortex 3 22,300 miles TRW Telecom surveillance
Trumpet 2 200-22,300 miles Boeing Surveillance of cellular phones
Parsae 3 600 miles TRW Ocean surveillance
Satellite Data Systems 2 200-22,300 miles Hughes Data Relay
Defense Support Program 4+ 22,300 miles TRW/Aerojet Missile early warning
Defense Meteorological Support Program 2 500 miles Lockheed Martin Meteorology, nuclear blast detection

Source: MSNBC<16>

These surveillance satellites act as giant scoops picking up electronic communications, cell phone conversations and various radio transmissions. The downlink stations that control the operations and targeting of these satellites are under the exclusive control of the United States, despite their location on foreign military bases. The two primary downlink facilities are at Menwith Hill, England, and Pine Gap, Australia.

Inside Menwith Hill

The Menwith Hill facility is located in North Yorkshire near Harrogate, England. The important role that Menwith Hill plays in the ECHELON system was recognized by the recent European Parliament STOA report:
Within Europe, all email, telephone and fax communications are routinely intercepted by the United States National Security Agency, transferring all target information from the European mainland via the strategic hub of London then by satellite to Fort Meade in Maryland via the crucial hub at Menwith Hill in the North York Moors of the UK.<17>
The existence and importance of the facility was first brought to light by British journalist and researcher Duncan Campbell in 1980.<18> Today, it is the largest spy station in the world, with over twenty-five satellite receiving stations and 1,400 American NSA personnel working with 350 UK Ministry of Defense staff on site. After revelations that the facility was coordinating surveillance for the vast majority of the European continent, the base has become a target for regular protests organized by local peace activists. It has also become the target of intense criticism by European government officials who are concerned about the vast network of civilian surveillance and economic espionage conducted from the station by the US.<19>

The beginnings of Menwith Hill go back to December 1951, when the US Air Force and British War Office signed a lease for land that had been purchased by the British government. The NSA took over the lease of the base in 1966, and they have continued to build up the facility ever since. Up until the mid-1970s, Menwith Hill was used for intercepting International Leased Carrier (ILC) and Non-Diplomatic Communications (NDC). Having received one of the first sophisticated IBM computers in the early 1960s, Menwith Hill was also used to sort through the voluminous unenciphered telex communications, which consisted of international messages, telegrams and telephone calls from the government, business and civilian sectors looking for anything of political, military or economic value.<20>

The addition of the first satellite intercept station at Menwith Hill in 1974 raised the base’s prominence in intelligence gathering. Eight large satellite communications dishes were installed during that phase of construction. Several satellite-gathering systems now dot the facility:<21>

STEEPLEBUSH – Completed in 1984, this $160 million system expanded the satellite surveillance capability and mission of the spy station beyond the bounds of the installation that began in 1974.

RUNWAY – Running east and west across the facility, this system receives signals from the second-generation geosynchronous Vortex satellites, and gathers miscellaneous communications traffic from Europe, Asia and the former Soviet Union. The information is then forwarded to the Menwith Hill computer systems for processing. RUNWAY may have recently been replaced or complemented by another system, RUTLEY.

PUSHER – An HFDF system that covers the HF frequency range between 3 MHz and 30 MHz (radio transmissions from CB radios, walkie-talkies, and other radio devices). Military, embassy, maritime and air flight communications are the main target of PUSHER.

MOONPENNY – Uncovered by British journalist Duncan Campbell in the 1980s, this system is targeted at the communication relay satellites belonging to other countries, as well as the Atlantic and Indian Ocean Intelsat satellites.

KNOBSTICKS I and II – The purpose of these antennae arrays are unknown, but they probably target military and diplomatic traffic throughout Europe.

GT-6 – A new system installed at the end of 1996, GT-6 is believed to be the receiver for the third generation of geosynchronous satellites termed Advanced Orion or Advanced Vortex. A new polar orbit satellite called Advanced Jumpseat may be monitored from here as well.

STEEPLEBUSH II – An expansion of the 1984 STEEPLEBUSH system, this computer system processes information collected from the RUNWAY receivers gathering traffic from the Vortex satellites.

SILKWORTH – Constructed by Lockheed Corporation, the main computer system for Menwith Hill processes most of the information received by the various reception systems.

One shocking revelation about Menwith Hill came to light in 1997 during the trial of two women peace campaigners appealing their convictions for trespassing at the facility. In documents and testimony submitted by British Telecomm in the case, R.G. Morris, head of Emergency Planning for British Telecomm, revealed that at least three major domestic fiber-optic telephone trunk lines – each capable of carrying 100,000 calls simultaneously – were wired through Menwith Hill.<22> This allows the NSA to tap into the very heart of the British Telecomm network. Judge Jonathan Crabtree rebuked British Telecomm for his revelations and prohibited Mr. Morris from giving any further testimony in the case for “national security” reasons. According to Duncan Campbell, the secret spying alliance between Menwith Hill and British Telecomm began in 1975 with a coaxial connection to the British Telecomm microwave facility at Hunter’s Stone, four miles away from Menwith Hill – a connection maintained even today.<23>

Additional systems (TROUTMAN, ULTRAPURE, TOTALISER, SILVERWEED, RUCKUS, et. al.) complete the monumental SIGINT collection efforts at Menwith Hill. Directing its electronic vacuum cleaners towards unsuspecting communications satellites in the skies, receiving signals gathered by satellites that scoop up the most minute signals on the ground, listening in on the radio communications throughout the air, or plugging into the ground-based telecommunications network, Menwith Hill, alongside its sister stations at Pine Gap, Australia, and Bad Aibling, Germany, represents the comprehensive effort of the NSA and its UKUSA allies to make sure that no communications signal escapes its electronic net.

The ECHELON Dictionaries

The extraordinary ability of ECHELON to intercept most of the communications traffic in the world is breathtaking in its scope. And yet the power of ECHELON resides in its ability to decrypt, filter, examine and codify these messages into selective categories for further analysis by intelligence agents from the various UKUSA agencies. As the electronic signals are brought into the station, they are fed through the massive computer systems, such as Menwith Hill’s SILKWORTH, where voice recognition, optical character recognition (OCR) and data information engines get to work on the messages.

These programs and computers transcend state-of-the-art; in many cases, they are well into the future. MAGISTRAND is part of the Menwith Hill SILKWORTH super-computer system that drives the powerful keyword search programs.<24> One tool used to sort through the text of messages, PATHFINDER (manufactured by the UK company, Memex),<25> sifts through large databases of text-based documents and messages looking for keywords and phrases based on complex algorithmic criteria. Voice recognition programs convert conversations into text messages for further analysis. One highly advanced system, VOICECAST, can target an individual’s voice pattern, so that every call that person makes is transcribed for future analysis.

Processing millions of messages every hour, the ECHELON systems churn away 24 hours a day, 7 days a week, looking for targeted keyword series, phone and fax numbers, and specified voiceprints. It is important to note that very few messages and phone calls are actually transcribed and recorded by the system. The vast majority are filtered out after they are read or listened to by the system. Only those messages that produce keyword “hits” are tagged for future analysis. Again, it is not just the ability to collect the electronic signals that gives ECHELON its power; it is the tools and technology that are able to whittle down the messages to only those that are important to the intelligence agencies.

Each station maintains a list of keywords (the “Dictionary”) designated by each of the participating intelligence agencies. A Dictionary Manager from each of the respective agencies is responsible for adding, deleting or changing the keyword search criteria for their dictionaries at each of the stations.<26> Each of these station dictionaries are given codewords, such as COWBOY for the Yakima facility and FLINTLOCK for the Waihopai facility.<27> These codewords play a crucial identification role for the analysts who eventually look at the intercepted messages.

Each message flagged by the ECHELON dictionaries as meeting the specified criteria is sorted by a four-digit code representing the source or subject of the message (such as 5535 for Japanese diplomatic traffic, or 8182 for communications about distribution of encryption technology,)<28> as well as the date, time and station codeword. Also included in the message headers are the codenames for the intended agency: ALPHA-ALPHA (GCHQ), ECHO-ECHO (DSD), INDIA-INDIA (GCSB), UNIFORM-UNIFORM (CSE), and OSCAR-OSCAR (NSA). These messages are then transmitted to each agency’s headquarters via a global computer system, PLATFORM,<29> that acts as the information nervous system for the UKUSA stations and agencies.

Every day, analysts located at the various intelligence agencies review the previous day’s product. As it is analyzed, decrypted and translated, it can be compiled into the different types of analysis: reports, which are direct and complete translations of intercepted messages; “gists,” which give basic information on a series of messages within a given category; and summaries, which are compilations from both reports and gists.<30> These are then given classifications: MORAY (secret), SPOKE (more secret than MORAY), UMBRA (top secret), GAMMA (Russian intercepts) and DRUID (intelligence forwarded to non-UKUSA parties). This analysis product is the raison d’être of the entire ECHELON system. It is also the lifeblood of the UKUSA alliance.

The Problem

The ECHELON system is the product of the Cold War conflict, an extended battle replete with heightened tensions that teetered on the brink of annihilation and the diminished hostilities of détente and glasnost. Vicious cycles of mistrust and paranoia between the United States and the Soviet Empire fed the intelligence agencies to the point that, with the fall of communism throughout Eastern Europe, the intelligence establishment began to grasp for a mission that justified its bloated existence.

But the rise of post-modern warfare – terrorism – gave the establishment all the justification it needed to develop even greater ability to spy on our enemies, our allies and our own citizens. ECHELON is the result of those efforts. The satellites that fly thousands of miles overhead and yet can spy out the most minute details on the ground; the secret submarines that troll the ocean floors that are able to tap into undersea communications cables;<31> and all power the efficient UKUSA signals intelligence machine.

There is a concerted effort by the heads of intelligence agencies, federal law enforcement officials and congressional representatives to defend the capabilities of ECHELON. Their persuasive arguments point to the tragedies seen in the bombings in Oklahoma City and the World Trade Center in New York City. The vulnerability of Americans abroad, as recently seen in the bombing of the American embassies in Dar es Salaam, Tanzania, and Nairobi, Kenya, emphasizes the necessity of monitoring those forces around the world that would use senseless violence and terror as political weapons against the US and its allies.

Intelligence victories add credibility to the arguments that defend such a pervasive surveillance system. The discovery of missile sites in Cuba in 1962, the capture of the Achille Lauro terrorists in 1995, the discovery of Libyan involvement in the bombing of a Berlin discotheque that killed one American (resulting in the 1996 bombing of Tripoli) and countless other incidents that have been averted (which are now covered by the silence of indoctrination vows and top-secret classifications) all point to the need for comprehensive signals intelligence gathering for the national security of the United States.

But despite the real threats and dangers to the peace and protection of American citizens at home and abroad, our Constitution is quite explicit in limiting the scope and powers of government. A fundamental foundation of free societies is that when controversies arise over the assumption of power by the state, power never defaults to the government, nor are powers granted without an extraordinary, explicit and compelling public interest. As the late Supreme Court Justice William Brennan pointed out:
The concept of military necessity is seductively broad, and has a dangerous plasticity. Because they invariably have the visage of overriding importance, there is always a temptation to invoke security “necessities” to justify an encroachment upon civil liberties. For that reason, the military-security argument must be approached with a healthy skepticism: Its very gravity counsels that courts be cautious when military necessity is invoked by the Government to justify a trespass on [Constitutional] rights.<32>
Despite the necessity of confronting terrorism and the many benefits that are provided by the massive surveillance efforts embodied by ECHELON, there is a dark and dangerous side of these activities that is concealed by the cloak of secrecy surrounding the intelligence operations of the United States.

The discovery of domestic surveillance targetting American civilians for reasons of “unpopular” political affiliation – or for no probable cause at all – in violation of the First, Fourth and Fifth Amendments of the Constitution is regularly impeded by very elaborate and complex legal arguments and privilege claims by the intelligence agencies and the US government. The guardians and caretakers of our liberties – our duly elected political representatives – give scarce attention to the activities, let alone the abuses, that occur under their watch. As pointed out below, our elected officials frequently become targets of ECHELON themselves, chilling any effort to check this unbridled power.

In addition, the shift in priorities resulting from the demise of the Soviet Empire and the necessity to justify intelligence capabilities resulted in a redefinition of “national security interests” to include espionage committed on behalf of powerful American companies. This quiet collusion between political and private interests typically involves the very same companies that are involved in developing the technology that empowers ECHELON and the intelligence agencies.

Domestic and Political Spying

When considering the use of ECHELON on American soil, the pathetic historical record of NSA and CIA domestic activities in regards to the Constitutional liberties and privacy rights of American citizens provides an excellent guidepost for what may occur now with the ECHELON system. Since the creation of the NSA by President Truman, its spying capability has frequently been used to monitor the activities of an unsuspecting public.


In 1945 Project SHAMROCK was initiated to obtain copies of all telegraphic information exiting or entering the United States. With the full cooperation of RCA, ITT and Western Union (representing almost all of the telegraphic traffic in the US at the time), the NSA's predecessor and later the NSA itself wereprovided with daily microfilm copies of all incoming, outgoing and transiting telegraphs. This system changed dramatically when the cable companies began providing magnetic computer tapes to the agency that enabled the agency to run all the messages through its HARVEST computer to look for particular keywords, locations, senders or addressees.

Project SHAMROCK became so successful that the in 1966 NSA and CIA set up a front company in lower Manhattan (where the offices of the telegraph companies were located) under the codename LPMEDLEY. At the height of Project SHAMROCK, 150,000 messages a month were printed and analyzed by NSA agents.<33>

NSA Director Lew Allen brought Project SHAMROCK to a crashing halt in May 1975 as congressional critics began to rip open the program’s shroud of secrecy. The testimony of both the representatives from the cable companies and of Director Allen at the hearings prompted Senate Intelligence Committee chairman Sen. Frank Church to conclude that Project SHAMROCK was “probably the largest government interception program affecting Americans ever undertaken.”<34>


A sister project to Project SHAMROCK, Project MINARET involved the creation of “watch lists” by each of the intelligence agencies and the FBI of those accused of “subversive” domestic activities. The watch lists included such notables as Martin Luther King, Malcolm X, Jane Fonda, Joan Baez and Dr. Benjamin Spock.

After the Supreme Court handed down its 1972 Keith decision,<35> which held that -- while the President could act to protect the country from unlawful and subversive activity designed to overthrow the government -- that same power did not extend to include warrantless electronic surveillance of domestic organizations, pressure came to bear on Project MINARET.<36> Attorney General Elliot Petersen shut down Project MINARET as soon as its activities were revealed to the Justice Department, despite the fact that the FBI (an agency under the Justice Department’s authority) was actively involved with the NSA and other intelligence agencies in creating the watch lists.

Operating between 1967 and 1973, over 5,925 foreigners and 1,690 organizations and US citizens were included on the Project MINARET watch lists. Despite extensive efforts to conceal the NSA’s involvement in Project MINARET, NSA Director Lew Allen testified before the Senate Intelligence Committee in 1975 that the NSA had issued over 3,900 reports on the watch-listed Americans.<37> Additionally, the NSA Office of Security Services maintained reports on at least 75,000 Americans between 1952 and 1974. This list included the names of anyone that was mentioned in a NSA message intercept.

Operation CHAOS

While the NSA was busy snooping on US citizens through Projects SHAMROCK and MINARET, the CIA got into the domestic spying act by initiating Operation CHAOS. President Lyndon Johnson authorized the creation of the CIA’s Domestic Operations Division (DOD), whose purpose was to “exercise centralized responsibility for direction, support, and coordination of clandestine operations activities within the United States….”

When Johnson ordered CIA Director John McCone to use the DOD to analyze the growing college student protests of the Administration’s policy towards Vietnam, two new units were set up to target anti-war protestors and organizations: Project RESISTANCE, which worked with college administrators, campus security and local police to identify anti-war activists and political dissidents; and Project MERRIMAC, which monitored any demonstrations being conducted in the Washington D.C. area. The CIA then began monitoring student activists and infiltrating anti-war organizations by working with local police departments to pull off burglaries, illegal entries (black bag jobs), interrogations and electronic surveillance.<38>

After President Nixon came to office in 1969, all of these domestic surveillance activities were consolidated into Operation CHAOS. After the revelation of two former CIA agents’ involvement in the Watergate break-in, the publication of an article about CHAOS in the New York Times<39> and the growing concern about distancing itself from illegal domestic spying activities, the CIA shut down Operation CHAOS. But during the life of the project, the Church Committee and the Commission on CIA Activities Within the United States (the Rockefeller Commission) revealed that the CIA had compiled files on over 13,000 individuals, including 7,000 US citizens and 1,000 domestic organizations.<40>

The Foreign Intelligence Surveillance Court (FISC)

In response to the discovery of such a comprehensive effort by previous administrations and the intelligence agencies, Congress passed legislation (the Foreign Intelligence Surveillance Act of 1978)<41> that created a top-secret court to hear applications for electronic surveillance from the FBI and NSA to provide some check on the domestic activities of the agencies. In 1995, Congress granted the court additional power to authorize surreptitious entries. In all of these actions, Congressional intent was to provide a check on the domestic surveillance abuses mentioned above.

The seven-member court, comprised of federal District Court judges appointed by the Supreme Court Chief Justice, sits in secret in a sealed room on the top floor of the Department of Justice building. Public information about the court’s hearings is scarce; each year the Attorney General is required by law to transmit to Congress a report detailing the number of applications each year and the number granted. With over 10,000 applications submitted to the FISC during the past twenty years, the court has only rejected one application (and that rejection was at the request of the Reagan Administration, which had submitted the application).

While the FISC was established to be the watchdog for the Constitutional rights of the American people against domestic surveillance, it quickly became the lap dog of the intelligence agencies. Surveillance requests that would never receive a hearing in a state or federal court are routinely approved by the FISC. This has allowed the FBI to use the process to conduct surveillance to obtain evidence in circumvention of the US Constitution, and the evidence is then used in subsequent criminal trials. But the process established by Congress and the courts ensures that information regarding the cause or extent of the surveillance order is withheld from defense attorneys because of the classified nature of the court.<42> Despite Congress’s initial intent for the FISC, it is doubtful that domestic surveillance by means of ECHELON comes under any scrutiny by the court.

Political Uses of ECHELON and UKUSA

Several incidents of domestic spying involving ECHELON have emerged from the secrecy of the UKUSA relationship. What these brief glimpses inside the intelligence world reveal is that, despite the best of intentions by elected representatives, presidents and prime ministers, the temptation to use ECHELON as a tool of political advancement and repression proves too strong.

Former Canadian spy Mike Frost recounts how former British Prime Minister Margaret Thatcher made a request in February 1983 to have two ministers from her own government monitored when she suspected them of disloyalty. In an effort to avoid the legal difficulties involved with domestic spying on high governmental officials, the GCHQ liaison in Ottawa made a request to CSE for them to conduct the three-week-long surveillance mission at British taxpayer expense. Frost’s CSE boss, Frank Bowman, traveled to London to do the job himself. After the mission was over, Bowman was instructed to hand over the tapes to a GCHQ official at their headquarters.<43>

Using the UKUSA alliance as legal cover is seductively easy. As Spyworld co-author Michel Gratton puts it,
The Thatcher episode certainly shows that GCHQ, like NSA, found ways to put itself above the law and did not hesitate to get directly involved in helping a specific politician for her personal political benefit…. [T]he decision to proceed with the London caper was probably not put forward for approval to many people up the bureaucratic ladder. It was something CSE figured they would get away with easily, so checking with the higher-ups would only complicate things unnecessarily.<44>
Frost also told of how he was asked in 1975 to spy on an unlikely target – Prime Minster Pierre Trudeau’s wife, Margaret Trudeau. The Royal Canadian Mounted Police’s (RCMP) Security Service division was concerned that the Prime Minister’s wife was buying and using marijuana, so they contacted the CSE to do the dirty work. Months of surveillance in cooperation with the Security Service turned up nothing of note. Frost was concerned that there were political motivations behind the RCMP’s request: “She was in no way suspected of espionage. Why was the RCMP so adamant about this? Were they trying to get at Pierre Trudeau for some reason or just protect him? Or were they working under orders from their political masters?”<45>

The NSA frequently gets into the political spying act as well. Nixon presidential aide John Ehrlichman revealed in his published memoirs, Witness to Power: The Nixon Years, that Henry Kissinger used the NSA to intercept the messages of then-Secretary of State William P. Rogers, which Kissinger used to convince President Nixon of Rogers’ incompetence. Kissinger also found himself on the receiving end of the NSA’s global net. Word of Kissinger’s secret diplomatic dealings with foreign governments would reach the ears of other Nixon administration officials, incensing Kissinger. As former NSA Deputy Director William Colby pointed out, “Kissinger would get sore as hell…because he wanted to keep it politically secret until it was ready to launch.”<46>

However, elected representatives have also become targets of spying by the intelligence agencies. In 1988, a former Lockheed software manager who was responsible for a dozen VAX computers that powered the ECHELON computers at Menwith Hill, Margaret Newsham, came forth with the stunning revelation that she had actually heard the NSA’s real time interception of phone conversations involving South Carolina Senator Strom Thurmond. Newsham was fired from Lockheed after she filed a whistleblower lawsuit alleging that the company was engaged in flagrant waste and abuse. After a top secret meeting in April 1988 with then-chairman of the House Permanent Select Committee on Intelligence, Rep. Louis Stokes, Capitol Hill staffers familiar with the meeting leaked the story to the Cleveland Plain Dealer.<47> While Sen. Thurmond was reluctant to pressure for a thorough investigation into the matter, his office revealed at the time that the office had previously received reports that the Senator was a target of the NSA.<48> After the news reports an investigation into the matter discovered that there were no controls or questioning over who could enter target names into the Menwith Hill system.<49>

The NSA, under orders from the Reagan administration, also targeted Maryland Congressman Michael Barnes. Phone calls he placed to Nicaraguan officials were intercepted and recorded, including a conversation he had with the Foreign Minister of Nicaragua protesting the implementation of martial law in that country. Barnes found out about the NSA’s spying after White House officials leaked transcripts of his conversations to reporters. CIA Director William Casey, later implicated in the Iran-Contra affair, showed Barnes a Nicaraguan embassy cable that reported a meeting between embassy staff and one of Barnes’ aides. The aide had been there on a professional call regarding an international affairs issue, and Casey asked for Barnes to fire the aide. Barnes replied that it was perfectly legal and legitimate for his staff to meet with foreign diplomats.

Says Barnes, “I was aware that NSA monitored international calls, that it was a standard part of intelligence gathering. But to use it for domestic political purposes is absolutely outrageous and probably illegal.”<50> Another former chairman of the Senate Intelligence Committee has also expressed his concerns about the NSA’s domestic targeting. “It has always worried me. What if that is used on American citizens?” queried former Arizona Senator Dennis DeConcini. “It is chilling. Are they listening to my private conversations on my telephone?”<51>

Seemingly non-controversial organizations have ended up in the fixed gaze of ECHELON, as several former GCHQ officials confidentially told the London Observer in June 1992. Among the targeted organizations they named were Amnesty International, Greenpeace and Christian Aid, an American missions organization that works with indigenous pastors engaged in ministry work in countries closed to Western, Christian workers.<52>

In another story published by the London Observer, a former employee of the British Joint Intelligence Committee, Robin Robison, admitted that Margaret Thatcher had personally ordered the communications interception of the parent company of the Observer, Lonrho, after the Observer had published a 1989 expose charging bribes had been paid to Thatcher’s son, Mark, in a multi-billion dollar British arms deal with Saudi Arabia. Despite facing severe penalties for violating his indoctrination vows, Robison admitted that he had personally delivered intercepted Lonrho messages to Mrs. Thatcher’s office.<53>

It should hardly be surprising that ECHELON ends up being used by elected and bureaucratic officials to their political advantage or by the intelligence agencies themselves for the purpose of sustaining their privileged surveillance powers and bloated budgets. The availability of such invasive technology practically begs for abuse, although it does not justify its use to those ends. But what is most frightening is the targeting of such “subversives” as those who expose corrupt government activity, protect human rights from government encroachments, challenge corporate polluters, or promote the gospel of Christ. That the vast intelligence powers of the United States should be arrayed against legitimate and peaceful organizations is demonstrative not of the desire to monitor, but of the desire to control.

Commercial spying

With the rapid erosion of the Soviet Empire in the early 1990s, Western intelligence agencies were anxious to redefine their mission to justify the scope of their global surveillance system. Some of the agencies’ closest corporate friends quickly gave them an option – commercial espionage. By redefining the term “national security” to include spying on foreign competitors of prominent US corporations, the signals intelligence game has gotten ugly. And it very well may have prompted the recent scrutiny by the European Union that ECHELON has endured.

While UKUSA agencies have pursued economic and commercial information on behalf of their countries with renewed vigor after the passing of communism in Eastern Europe, the NSA practice of spying on behalf of US companies has a long history. Gerald Burke, who served as Executive Director of President Nixon’s Foreign Intelligence Advisory Board, notes commercial espionage was endorsed by the US government as early as 1970: “By and large, we recommended that henceforth economic intelligence be considered a function of the national security, enjoying a priority equivalent to diplomatic, military, and technological intelligence.”<54>

To accommodate the need for information regarding international commercial deals, the intelligence agencies set up a small, unpublicized department within the Department of Commerce, the Office of Intelligence Liaison. This office receives intelligence reports from the US intelligence agencies about pending international deals that it discreetly forwards to companies that request it or may have an interest in the information. Immediately after coming to office in January 1993, President Clinton added to the corporate espionage machine by creating the National Economic Council, which feeds intelligence to “select” companies to enhance US competitiveness. The capabilities of ECHELON to spy on foreign companies is nothing new, but the Clinton administration has raised its use to an art:
In 1990 the German magazine Der Speigel revealed that the NSA had intercepted messages about an impending $200 million deal between Indonesia and the Japanese satellite manufacturer NEC Corp. After President Bush intervened in the negotiations on behalf of American manufacturers, the contract was split between NEC and AT&T.

In 1994, the CIA and NSA intercepted phone calls between Brazilian officials and the French firm Thomson-CSF about a radar system that the Brazilians wanted to purchase. A US firm, Raytheon, was a competitor as well, and reports prepared from intercepts were forwarded to Raytheon.<55>

In September 1993, President Clinton asked the CIA to spy on Japanese auto manufacturers that were designing zero-emission cars and to forward that information to the Big Three US car manufacturers: Ford, General Motors and Chrysler.<56> In 1995, the New York Times reported that the NSA and the CIA’s Tokyo station were involved in providing detailed information to US Trade Representative Mickey Kantor’s team of negotiators in Geneva facing Japanese car companies in a trade dispute.<57> Recently, a Japanese newspaper, Mainichi, accused the NSA of continuing to monitor the communications of Japanese companies on behalf of American companies.<58>
Insight Magazine reported in a series of articles in 1997 that President Clinton ordered the NSA and FBI to mount a massive surveillance operation at the 1993 Asian/Pacific Economic Conference (APEC) hosted in Seattle. One intelligence source for the story related that over 300 hotel rooms had been bugged for the event, which was designed to obtain information regarding oil and hydro-electric deals pending in Vietnam that were passed on to high level Democratic Party contributors competing for the contracts.<59> But foreign companies were not the only losers: when Vietnam expressed interest in purchasing two used 737 freighter aircraft from an American businessman, the deal was scuttled after Commerce Secretary Ron Brown arranged favorable financing for two new 737s from Boeing.<60>
But the US is not the only partner of the UKUSA relationship that engages in such activity. British Prime Minister Margaret Thatcher ordered the GCHQ to monitor the activities of international media mogul Robert Maxwell on behalf of the Bank of England.<61> Former CSE linguist and analyst Jane Shorten claimed that she had seen intercepts from Mexican trade representatives during the 1992-1993 NAFTA trade negotiations, as well as 1991 South Korean Foreign Ministry intercepts dealing with the construction of three Canadian CANDU nuclear reactors for the Koreans in a $6 billion deal.<62> Shorten’s revelation prompted Canadian Deputy Prime Minister Sheila Copps to launch a probe into the allegations after the Mexicans lodged a protest.

But every spy agency eventually gets beat at their own game. Mike Frost relates in Spyworld how an accidental cell phone intercept in 1981 of the American Ambassador to Canada discussing a pending grain deal that the US was about to sign with China provided Canada with the American negotiating strategy for the deal. The information was used to outbid the US, resulting in a three year, $2.5 billion contract for the Canadian Wheat Board. CSE out-spooked the NSA again a year later when Canada snagged a $50 million wheat sale to Mexico.<63>

Another disturbing trend regarding the present commercial use of ECHELON is the incestuous relationship that exists between the intelligence agencies and the US corporations that develop the technology that fuels their spy systems. Many of the companies that receive the most important commercial intercepts – Lockheed, Boeing, Loral, TRW and Raytheon – are actively involved in the manufacturing and operation of many of the spy systems that comprise ECHELON. The collusion between intelligence agencies and their contractors is frightening in the chilling effect it has on creating any foreign or even domestic competition. But just as important is that it is a gross misuse of taxpayer-financed resources and an abuse of the intelligence agencies’ capabilities.

The Warning

While the UKUSA relationship is a product of Cold War political and military tensions, ECHELON is purely a product of the 20th Century – the century of statism. The modern drive toward the assumption of state power has turned legitimate national security agencies and apparati into pawns in a manipulative game where the stakes are no less than the survival of the Constitution. The systems developed prior to ECHELON were designed to confront the expansionist goals of the Soviet Empire – something the West was forced out of necessity to do. But as Glyn Ford, European Parliament representative for Manchester, England, and the driving force behind the European investigation of ECHELON, has pointed out: “The difficulty is that the technology has now become so elaborate that what was originally a small client list has become the whole world.”<64>

What began as a noble alliance to contain and defeat the forces of communism has turned into a carte blanche to disregard the rights and liberties of the American people and the population of the free world. As has been demonstrated time and again, the NSA has been persistent in subverting not just the intent of the law in regards to the prohibition of domestic spying, but the letter as well. The laws that were created to constrain the intelligence agencies from infringing on our liberties are frequently flaunted, re-interpreted and revised according to the bidding and wishes of political spymasters in Washington D.C. Old habits die hard, it seems.

As stated above, there is a need for such sophisticated surveillance technology. Unfortunately, the world is filled with criminals, drug lords, terrorists and dictators that threaten the peace and security of many nations. The thought that ECHELON can be used to eliminate or control these international thugs is heartening. But defenders of ECHELON argue that the rare intelligence victories over these forces of darkness and death give wholesale justification to indiscriminate surveillance of the entire world and every member of it. But more complicated issues than that remain.

The shameless and illegal targeting of political opponents, business competitors, dissidents and even Christian ministries stands as a testament that if America is to remain free, we must bind these intelligence systems and those that operate them with the heavy chains of transparency and accountability to our elected officials. But the fact that the ECHELON apparatus can be quickly turned around on those same officials in order to maintain some advantage for the intelligence agencies indicates that these agencies are not presently under the control of our elected representatives.

That Congress is not aware of or able to curtail these abuses of power is a frightening harbinger of what may come here in the United States. The European Parliament has begun the debate over what ECHELON is, how it is being used and how free countries should use such a system. Congress should join that same debate with the understanding that consequences of ignoring or failing to address these issues could foster the demise of our republican form of government. Such is the threat, as Senator Frank Church warned the American people over twenty years ago.

At the same time, that capability at any time could be turned around on the American people and no American would have any privacy left, such [is] the capability to monitor everything: telephone conversations, telegrams, it doesn’t matter. There would be no place to hide. If this government ever became a tyranny, if a dictator ever took charge in this country, the technological capacity that the intelligence community has given the government could enable it to impose total tyranny, and there would be no way to fight back, because the most careful effort to combine together in resistance to the government, no matter how privately it was done, is within the reach of the government to know. Such is the capability of this technology…

I don’t want to see this country ever go across the bridge. I know the capacity that is there to make tyranny total in America, and we must see to it that this agency and all agencies that possess this technology operate within the law and under proper supervision, so that we never cross over that abyss. That is the abyss from which there is no return.<65>


1. Steve Wright, An Appraisal of Technologies of Political Control, European Parliament: Scientific and Technologies Options Assessment, Luxembourg, January 6, 1998.
2. Bruno Giussani, “European Study Paints a Chilling Portrait of Technology’s Uses," The New York Times, February 24, 1998.
3. Nelson, New Zealand: Craig Potton Publishing, 1996.
4. Desmond Ball and Jeffrey Richelson, The Ties That Bind: Intelligence Cooperation Between the UKUSA Countries, (Boston: Allen & Unwin, 1985) pp. 137-8.
5. Ibid., 142-143.
6. Secret Power, p. 40. See note 3.
7. National Security Agency, About the NSA.
8. The Ties that Bind, p. 143.
9. The coverage area of the various Intelsat satellites can be found at the Intelsat website at:
10. Secret Power, p. 28.
11. Ibid., p. 35.
12. Ibid.
13. Marco Campagna, Un Systeme De Surveillance Mondial, Cahiers de Television (CTV-France), June 1998; Peter Hum, "I Spy," The Ottawa Citizen, May 10, 1997.
14. Secret Power, pp. 35-36, 150; Ties That Bind, pp. 204-207.
15. Mike Frost and Michel Graton, Spyworld: How C.S.E. Spies on Canadians and the World (Toronto: Seal/McClelland-Bantam, 1995), p. 35
16. Robert Windrem, "Spy Satellites Enter New Dimension," MSNBC and NBC News, August 8, 1998.
17. An Appraisal of Technology of Political Control, p. 19.
18. Duncan Campbell and Linda Melvern, “America’s Big Ear on Europe,” New Statesman, July 18, 1980, pp. 10-14.
19. Simon Davies, “EU Simmers Over Menwith Listening Post,” London Telegraph, July 16, 1998.
20. Nicholas Rufford, “Spy Station F83,” The Sunday (London) Times, May 31, 1998.
21. Duncan Campell, "Somebody’s Listening," The New Statesman, August 12, 1988, pp. 10-12; “The Hill,” Dispatches, BBC Channel 4, October 6, 1993 (transcript provided by Duncan Campbell); Loring Wirbel, “Space – Intelligence Technology’s Embattled Frontier,” Electronic Engineering Times, April 22, 1997; Nicholas Rufford, “Cracking the Menwith Codes,” The Sunday (London) Times, May 31, 1998.
22. Duncan Campbell, BT Condemned for Listing Cables to US SIGINT Station, September 4, 1997.
23. Ibid.; Spy Station F83.
24. Mentioned in Dispatches: The Hill.
25. An Appraisal of Technologies of Political Control, p. 19. Memex maintains a website describing their defense and intelligence products and contracts:
26. Secret Power, p. 49.
27. Ibid., pp. 165-166.
28. Ibid., p. 44
29. James Bamford, The Puzzle Palace: Inside the National Security Agency, America's Most Secret Intelligence Organization, (New York: Penguin Books, 1983), pp. 138-139.
30. Secret Power, p. 45.
31. Ties That Bind, pp. 223-224.
32. Brown v. Glines, 444 U.S. 348 (1980).
33. Puzzle Palace, p. 314, 459.
34. External Collection Program: U.S. Senate, Select Committee on Intelligence, Supplementary Detailed Staff Reports on Intelligence and the Rights of Americans, Final Report, Book III, April 23, 1976, p. 765.
35. United States v. United States District Court, 407 U.S. 297 (1972)
36. Puzzle Palace, pp. 370-373.
37. Puzzle Palace, p. 381.
38. Morton Halperin, Jerry Berman, et. al., The Lawless State (Penguin: New York, 1976) p. 146..
39. Seymour Hersh, “Huge CIA Operation Reported in U.S. Against Antiwar Forces,” New York Times (December 22, 1974), p. 1.
40. The Lawless State, p. 153; US Commission on CIA Activites within the United States, Report to the President (US Government Printing Office: Washington DC, 1975), p. 144n3.
41. 50 USC Sec. 1801, et. seq.
42. For more information on the FISC, see this author’s essay “Inside America’s Secret Court: The Foreign Intelligence Surveillance Court,” The Privacy Papers, No. 2 (Washington D.C.: Free Congress Foundation, 1998).
43. Spyworld, pp. 234-238.
44. Ibid., p. 238.
45. Ibid., pp. 93-97.
46. Scott Shane and Tom Bowman, “Catching Americans in NSA’s Net,” Baltimore Sun, December 12, 1995.
47. Keith C. Epstein and John S. Long, “Security Agency Accused of Monitoring U.S. Calls,” Cleveland Plain Dealer, July 1, 1988, pp. 1A, 10A.
48. Pete Carey, “NSA Accused of Forbidden Phone Taps,” San Jose Mercury News, July 2, 1988, p. 1A.
49. Somebody's Listening, p. 11.
50. Catching Americans in NSA’s Net.
51. Ibid.
52. John Merritt, “UK: GCHQ Spies on Charities and Companies – Fearful Whistleblowers Tell of Massive Routine Abuse,” Observer (London), June 18, 1992.
53. Hugh O’Shaughnessy, “Thatcher Ordered Lonrho Phone-Tap Over Harrods Affairs,” Observer (London), June 28, 1992; cited in Secret Power, p. 54.
54. Dispatches: The Hill, op. cit.
55. Tom Bowman and Scott Shane, “Battling High-Tech Warriors,” Baltimore Sun, December 15, 1995.
56. Robert Dreyfuss, “Company Spies,” Mother Jones, May/June 1994.
57. Cited in Bruce Livesey, “Trolling for Secrets: Economic Espionage is the New Niche for Government Spies,” Financial Post (Canada), February 28, 1998.
58. U.S. Spy Agency Helped U.S. Companies Win Business Overseas, Nikkei English News, September 21, 1998.
59. Timothy W. Maier, “Did Clinton Bug Conclave for Cash,” Insight, September 15, 1997. The three article series is online at:
60. Timothy W. Maier, “Snoops, Sex and Videotape,” Insight, September 29, 1997.
61. Matthew Fletcher, “Cook Faces Quiz on Big Brother Spy Net,” Financial Mail (England), March 1, 1998.
62. Trolling for Secrets, op. cit.
63. Spyworld, pp. 224-227.
64. Lucille Redmond, “Suddenly There Came a Tapping…”, The Sunday Business Post (Ireland), March 9, 1998.
65. National Broadcasting Company, “Meet the Press” (Washington D.C.: Merkle Press, 1975), transcript of August 17, 1975, p. 6; quoted in Puzzle Palace, p. 477.

Original Document Source:
ECHELON: America's Secret Global Surveillance Network by Patrick S. Poole
Return to Top of Page
Return to ECHELON Reports Page
Return to Home Page
Civil Intelligence Association, Defense Oversight Group

The NSA Is Building the Country’s Biggest Spy Center (Watch What You Say)
By James Bamford Email Author March 15, 2012 | 7:24 pm , Surveillance

Photo: Name Withheld; Digital Manipulation: Jesse Lenz
The spring air in the small, sand-dusted town has a soft haze to it, and clumps of green-gray sagebrush rustle in the breeze. Bluffdale sits in a bowl-shaped valley in the shadow of Utah’s Wasatch Range to the east and the Oquirrh Mountains to the west. It’s the heart of Mormon country, where religious pioneers first arrived more than 160 years ago. They came to escape the rest of the world, to understand the mysterious words sent down from their god as revealed on buried golden plates, and to practice what has become known as “the principle,” marriage to multiple wives.

Today Bluffdale is home to one of the nation’s largest sects of polygamists, the Apostolic United Brethren, with upwards of 9,000 members. The brethren’s complex includes a chapel, a school, a sports field, and an archive. Membership has doubled since 1978—and the number of plural marriages has tripled—so the sect has recently been looking for ways to purchase more land and expand throughout the town.

But new pioneers have quietly begun moving into the area, secretive outsiders who say little and keep to themselves. Like the pious polygamists, they are focused on deciphering cryptic messages that only they have the power to understand. Just off Beef Hollow Road, less than a mile from brethren headquarters, thousands of hard-hatted construction workers in sweat-soaked T-shirts are laying the groundwork for the newcomers’ own temple and archive, a massive complex so large that it necessitated expanding the town’s boundaries. Once built, it will be more than five times the size of the US Capitol.

Rather than Bibles, prophets, and worshippers, this temple will be filled with servers, computer intelligence experts, and armed guards. And instead of listening for words flowing down from heaven, these newcomers will be secretly capturing, storing, and analyzing vast quantities of words and images hurtling through the world’s telecommunications networks. In the little town of Bluffdale, Big Love and Big Brother have become uneasy neighbors.

The NSA has become the largest, most covert, and potentially most intrusive intelligence agency ever.
Under construction by contractors with top-secret clearances, the blandly named Utah Data Center is being built for the National Security Agency. A project of immense secrecy, it is the final piece in a complex puzzle assembled over the past decade. Its purpose: to intercept, decipher, analyze, and store vast swaths of the world’s communications as they zap down from satellites and zip through the underground and undersea cables of international, foreign, and domestic networks. The heavily fortified $2 billion center should be up and running in September 2013. Flowing through its servers and routers and stored in near-bottomless databases will be all forms of communication, including the complete contents of private emails, cell phone calls, and Google searches, as well as all sorts of personal data trails—parking receipts, travel itineraries, bookstore purchases, and other digital “pocket litter.” It is, in some measure, the realization of the “total information awareness” program created during the first term of the Bush administration—an effort that was killed by Congress in 2003 after it caused an outcry over its potential for invading Americans’ privacy.

But “this is more than just a data center,” says one senior intelligence official who until recently was involved with the program. The mammoth Bluffdale center will have another important and far more secret role that until now has gone unrevealed. It is also critical, he says, for breaking codes. And code-breaking is crucial, because much of the data that the center will handle—financial information, stock transactions, business deals, foreign military and diplomatic secrets, legal documents, confidential personal communications—will be heavily encrypted. According to another top official also involved with the program, the NSA made an enormous breakthrough several years ago in its ability to cryptanalyze, or break, unfathomably complex encryption systems employed by not only governments around the world but also many average computer users in the US. The upshot, according to this official: “Everybody’s a target; everybody with communication is a target.”

For the NSA, overflowing with tens of billions of dollars in post-9/11 budget awards, the cryptanalysis breakthrough came at a time of explosive growth, in size as well as in power. Established as an arm of the Department of Defense following Pearl Harbor, with the primary purpose of preventing another surprise assault, the NSA suffered a series of humiliations in the post-Cold War years. Caught offguard by an escalating series of terrorist attacks—the first World Trade Center bombing, the blowing up of US embassies in East Africa, the attack on the USS Cole in Yemen, and finally the devastation of 9/11—some began questioning the agency’s very reason for being. In response, the NSA has quietly been reborn. And while there is little indication that its actual effectiveness has improved—after all, despite numerous pieces of evidence and intelligence-gathering opportunities, it missed the near-disastrous attempted attacks by the underwear bomber on a flight to Detroit in 2009 and by the car bomber in Times Square in 2010—there is no doubt that it has transformed itself into the largest, most covert, and potentially most intrusive intelligence agency ever created.

In the process—and for the first time since Watergate and the other scandals of the Nixon administration—the NSA has turned its surveillance apparatus on the US and its citizens. It has established listening posts throughout the nation to collect and sift through billions of email messages and phone calls, whether they originate within the country or overseas. It has created a supercomputer of almost unimaginable speed to look for patterns and unscramble codes. Finally, the agency has begun building a place to store all the trillions of words and thoughts and whispers captured in its electronic net. And, of course, it’s all being done in secret. To those on the inside, the old adage that NSA stands for Never Say Anything applies more than ever.

When construction is completed in 2013, the heavily fortified $2 billion facility in Bluffdale will encompass 1 million square feet.

1 Visitor control center
A $9.7 million facility for ensuring that only cleared personnel gain access.
2 Administration
Designated space for technical support and administrative personnel.
3 Data halls
Four 25,000-square-foot facilities house rows and rows of servers.
4 Backup generators and fuel tanks
Can power the center for at least three days.
5 Water storage and pumping
Able to pump 1.7 million gallons of liquid per day.
6 Chiller plant
About 60,000 tons of cooling equipment to keep servers from overheating.
7 Power substation
An electrical substation to meet the center’s estimated 65-megawatt demand.
8 Security
Video surveillance, intrusion detection, and other protection will cost more than $10 million.
Source: U.S. Army Corps of Engineers Conceptual Site plan
A swath of freezing fog blanketed Salt Lake City on the morning of January 6, 2011, mixing with a weeklong coating of heavy gray smog. Red air alerts, warning people to stay indoors unless absolutely necessary, had become almost daily occurrences, and the temperature was in the bone-chilling twenties. “What I smell and taste is like coal smoke,” complained one local blogger that day. At the city’s international airport, many inbound flights were delayed or diverted while outbound regional jets were grounded. But among those making it through the icy mist was a figure whose gray suit and tie made him almost disappear into the background. He was tall and thin, with the physique of an aging basketball player and dark caterpillar eyebrows beneath a shock of matching hair. Accompanied by a retinue of bodyguards, the man was NSA deputy director Chris Inglis, the agency’s highest-ranking civilian and the person who ran its worldwide day-to-day operations.

A short time later, Inglis arrived in Bluffdale at the site of the future data center, a flat, unpaved runway on a little-used part of Camp Williams, a National Guard training site. There, in a white tent set up for the occasion, Inglis joined Harvey Davis, the agency’s associate director for installations and logistics, and Utah senator Orrin Hatch, along with a few generals and politicians in a surreal ceremony. Standing in an odd wooden sandbox and holding gold-painted shovels, they made awkward jabs at the sand and thus officially broke ground on what the local media had simply dubbed “the spy center.” Hoping for some details on what was about to be built, reporters turned to one of the invited guests, Lane Beattie of the Salt Lake Chamber of Commerce. Did he have any idea of the purpose behind the new facility in his backyard? “Absolutely not,” he said with a self-conscious half laugh. “Nor do I want them spying on me.”

For his part, Inglis simply engaged in a bit of double-talk, emphasizing the least threatening aspect of the center: “It’s a state-of-the-art facility designed to support the intelligence community in its mission to, in turn, enable and protect the nation’s cybersecurity.” While cybersecurity will certainly be among the areas focused on in Bluffdale, what is collected, how it’s collected, and what is done with the material are far more important issues. Battling hackers makes for a nice cover—it’s easy to explain, and who could be against it? Then the reporters turned to Hatch, who proudly described the center as “a great tribute to Utah,” then added, “I can’t tell you a lot about what they’re going to be doing, because it’s highly classified.”

And then there was this anomaly: Although this was supposedly the official ground-breaking for the nation’s largest and most expensive cybersecurity project, no one from the Department of Homeland Security, the agency responsible for protecting civilian networks from cyberattack, spoke from the lectern. In fact, the official who’d originally introduced the data center, at a press conference in Salt Lake City in October 2009, had nothing to do with cybersecurity. It was Glenn A. Gaffney, deputy director of national intelligence for collection, a man who had spent almost his entire career at the CIA. As head of collection for the intelligence community, he managed the country’s human and electronic spies.

Within days, the tent and sandbox and gold shovels would be gone and Inglis and the generals would be replaced by some 10,000 construction workers. “We’ve been asked not to talk about the project,” Rob Moore, president of Big-D Construction, one of the three major contractors working on the project, told a local reporter. The plans for the center show an extensive security system: an elaborate $10 million antiterrorism protection program, including a fence designed to stop a 15,000-pound vehicle traveling 50 miles per hour, closed-circuit cameras, a biometric identification system, a vehicle inspection facility, and a visitor-control center.

Inside, the facility will consist of four 25,000-square-foot halls filled with servers, complete with raised floor space for cables and storage. In addition, there will be more than 900,000 square feet for technical support and administration. The entire site will be self-sustaining, with fuel tanks large enough to power the backup generators for three days in an emergency, water storage with the capability of pumping 1.7 million gallons of liquid per day, as well as a sewage system and massive air-conditioning system to keep all those servers cool. Electricity will come from the center’s own substation built by Rocky Mountain Power to satisfy the 65-megawatt power demand. Such a mammoth amount of energy comes with a mammoth price tag—about $40 million a year, according to one estimate.

Given the facility’s scale and the fact that a terabyte of data can now be stored on a flash drive the size of a man’s pinky, the potential amount of information that could be housed in Bluffdale is truly staggering. But so is the exponential growth in the amount of intelligence data being produced every day by the eavesdropping sensors of the NSA and other intelligence agencies. As a result of this “expanding array of theater airborne and other sensor networks,” as a 2007 Department of Defense report puts it, the Pentagon is attempting to expand its worldwide communications network, known as the Global Information Grid, to handle yottabytes (1024 bytes) of data. (A yottabyte is a septillion bytes—so large that no one has yet coined a term for the next higher magnitude.)

It needs that capacity because, according to a recent report by Cisco, global Internet traffic will quadruple from 2010 to 2015, reaching 966 exabytes per year. (A million exabytes equal a yottabyte.) In terms of scale, Eric Schmidt, Google’s former CEO, once estimated that the total of all human knowledge created from the dawn of man to 2003 totaled 5 exabytes. And the data flow shows no sign of slowing. In 2011 more than 2 billion of the world’s 6.9 billion people were connected to the Internet. By 2015, market research firm IDC estimates, there will be 2.7 billion users. Thus, the NSA’s need for a 1-million-square-foot data storehouse. Should the agency ever fill the Utah center with a yottabyte of information, it would be equal to about 500 quintillion (500,000,000,000,000,000,000) pages of text.

The data stored in Bluffdale will naturally go far beyond the world’s billions of public web pages. The NSA is more interested in the so-called invisible web, also known as the deep web or deepnet—data beyond the reach of the public. This includes password-protected data, US and foreign government communications, and noncommercial file-sharing between trusted peers. “The deep web contains government reports, databases, and other sources of information of high value to DOD and the intelligence community,” according to a 2010 Defense Science Board report. “Alternative tools are needed to find and index data in the deep web … Stealing the classified secrets of a potential adversary is where the [intelligence] community is most comfortable.” With its new Utah Data Center, the NSA will at last have the technical capability to store, and rummage through, all those stolen secrets. The question, of course, is how the agency defines who is, and who is not, “a potential adversary.”
Once it’s operational, the Utah Data Center will become, in effect, the NSA’s cloud. The center will be fed data collected by the agency’s eavesdropping satellites, overseas listening posts, and secret monitoring rooms in telecom facilities throughout the US. All that data will then be accessible to the NSA’s code breakers, data-miners, China analysts, counterterrorism specialists, and others working at its Fort Meade headquarters and around the world. Here’s how the data center appears to fit into the NSA’s global puzzle.—J.B.

1 Geostationary satellites
Four satellites positioned around the globe monitor frequencies carrying everything from walkie-talkies and cell phones in Libya to radar systems in North Korea. Onboard software acts as the first filter in the collection process, targeting only key regions, countries, cities, and phone numbers or email.
2 Aerospace Data Facility, Buckley Air Force Base, Colorado
Intelligence collected from the geostationary satellites, as well as signals from other spacecraft and overseas listening posts, is relayed to this facility outside Denver. About 850 NSA employees track the satellites, transmit target information, and download the intelligence haul.
3 NSA Georgia, Fort Gordon, Augusta, Georgia
Focuses on intercepts from Europe, the Middle East, and North Africa. Codenamed Sweet Tea, the facility has been massively expanded and now consists of a 604,000-square-foot operations building for up to 4,000 intercept operators, analysts, and other specialists.
4 NSA Texas, Lackland Air Force Base, San Antonio
Focuses on intercepts from Latin America and, since 9/11, the Middle East and Europe. Some 2,000 workers staff the operation. The NSA recently completed a $100 million renovation on a mega-data center here—a backup storage facility for the Utah Data Center.
5 NSA Hawaii, Oahu
Focuses on intercepts from Asia. Built to house an aircraft assembly plant during World War II, the 250,000-square-foot bunker is nicknamed the Hole. Like the other NSA operations centers, it has since been expanded: Its 2,700 employees now do their work aboveground from a new 234,000-square-foot facility.
6 Domestic listening posts
The NSA has long been free to eavesdrop on international satellite communications. But after 9/11, it installed taps in US telecom “switches,” gaining access to domestic traffic. An ex-NSA official says there are 10 to 20 such installations.
7 Overseas listening posts
According to a knowledgeable intelligence source, the NSA has installed taps on at least a dozen of the major overseas communications links, each capable of eavesdropping on information passing by at a high data rate.
8 Utah Data Center, Bluffdale, Utah
At a million square feet, this $2 billion digital storage facility outside Salt Lake City will be the centerpiece of the NSA’s cloud-based data strategy and essential in its plans for decrypting previously uncrackable documents.
9 Multiprogram Research Facility, Oak Ridge, Tennessee
Some 300 scientists and computer engineers with top security clearance toil away here, building the world’s fastest supercomputers and working on cryptanalytic applications and other secret projects.
10 NSA headquarters, Fort Meade, Maryland
Analysts here will access material stored at Bluffdale to prepare reports and recommendations that are sent to policymakers. To handle the increased data load, the NSA is also building an $896 million supercomputer center here.
Before yottabytes of data from the deep web and elsewhere can begin piling up inside the servers of the NSA’s new center, they must be collected. To better accomplish that, the agency has undergone the largest building boom in its history, including installing secret electronic monitoring rooms in major US telecom facilities. Controlled by the NSA, these highly secured spaces are where the agency taps into the US communications networks, a practice that came to light during the Bush years but was never acknowledged by the agency. The broad outlines of the so-called warrantless-wiretapping program have long been exposed—how the NSA secretly and illegally bypassed the Foreign Intelligence Surveillance Court, which was supposed to oversee and authorize highly targeted domestic eavesdropping; how the program allowed wholesale monitoring of millions of American phone calls and email. In the wake of the program’s exposure, Congress passed the FISA Amendments Act of 2008, which largely made the practices legal. Telecoms that had agreed to participate in the illegal activity were granted immunity from prosecution and lawsuits. What wasn’t revealed until now, however, was the enormity of this ongoing domestic spying program.

For the first time, a former NSA official has gone on the record to describe the program, codenamed Stellar Wind, in detail. William Binney was a senior NSA crypto-mathematician largely responsible for automating the agency’s worldwide eavesdropping network. A tall man with strands of black hair across the front of his scalp and dark, determined eyes behind thick-rimmed glasses, the 68-year-old spent nearly four decades breaking codes and finding new ways to channel billions of private phone calls and email messages from around the world into the NSA’s bulging databases. As chief and one of the two cofounders of the agency’s Signals Intelligence Automation Research Center, Binney and his team designed much of the infrastructure that’s still likely used to intercept international and foreign communications.

He explains that the agency could have installed its tapping gear at the nation’s cable landing stations—the more than two dozen sites on the periphery of the US where fiber-optic cables come ashore. If it had taken that route, the NSA would have been able to limit its eavesdropping to just international communications, which at the time was all that was allowed under US law. Instead it chose to put the wiretapping rooms at key junction points throughout the country—large, windowless buildings known as switches—thus gaining access to not just international communications but also to most of the domestic traffic flowing through the US. The network of intercept stations goes far beyond the single room in an AT&T building in San Francisco exposed by a whistle-blower in 2006. “I think there’s 10 to 20 of them,” Binney says. “That’s not just San Francisco; they have them in the middle of the country and also on the East Coast.”

The eavesdropping on Americans doesn’t stop at the telecom switches. To capture satellite communications in and out of the US, the agency also monitors AT&T’s powerful earth stations, satellite receivers in locations that include Roaring Creek and Salt Creek. Tucked away on a back road in rural Catawissa, Pennsylvania, Roaring Creek’s three 105-foot dishes handle much of the country’s communications to and from Europe and the Middle East. And on an isolated stretch of land in remote Arbuckle, California, three similar dishes at the company’s Salt Creek station service the Pacific Rim and Asia.

The former NSA official held his thumb and forefinger close together: “We are that far from a turnkey totalitarian state.”
Binney left the NSA in late 2001, shortly after the agency launched its warrantless-wiretapping program. “They violated the Constitution setting it up,” he says bluntly. “But they didn’t care. They were going to do it anyway, and they were going to crucify anyone who stood in the way. When they started violating the Constitution, I couldn’t stay.” Binney says Stellar Wind was far larger than has been publicly disclosed and included not just eavesdropping on domestic phone calls but the inspection of domestic email. At the outset the program recorded 320 million calls a day, he says, which represented about 73 to 80 percent of the total volume of the agency’s worldwide intercepts. The haul only grew from there. According to Binney—who has maintained close contact with agency employees until a few years ago—the taps in the secret rooms dotting the country are actually powered by highly sophisticated software programs that conduct “deep packet inspection,” examining Internet traffic as it passes through the 10-gigabit-per-second cables at the speed of light.

The software, created by a company called Narus that’s now part of Boeing, is controlled remotely from NSA headquarters at Fort Meade in Maryland and searches US sources for target addresses, locations, countries, and phone numbers, as well as watch-listed names, keywords, and phrases in email. Any communication that arouses suspicion, especially those to or from the million or so people on agency watch lists, are automatically copied or recorded and then transmitted to the NSA.

The scope of surveillance expands from there, Binney says. Once a name is entered into the Narus database, all phone calls and other communications to and from that person are automatically routed to the NSA’s recorders. “Anybody you want, route to a recorder,” Binney says. “If your number’s in there? Routed and gets recorded.” He adds, “The Narus device allows you to take it all.” And when Bluffdale is completed, whatever is collected will be routed there for storage and analysis.

According to Binney, one of the deepest secrets of the Stellar Wind program—again, never confirmed until now—was that the NSA gained warrantless access to AT&T’s vast trove of domestic and international billing records, detailed information about who called whom in the US and around the world. As of 2007, AT&T had more than 2.8 trillion records housed in a database at its Florham Park, New Jersey, complex.

Verizon was also part of the program, Binney says, and that greatly expanded the volume of calls subject to the agency’s domestic eavesdropping. “That multiplies the call rate by at least a factor of five,” he says. “So you’re over a billion and a half calls a day.” (Spokespeople for Verizon and AT&T said their companies would not comment on matters of national security.)

After he left the NSA, Binney suggested a system for monitoring people’s communications according to how closely they are connected to an initial target. The further away from the target—say you’re just an acquaintance of a friend of the target—the less the surveillance. But the agency rejected the idea, and, given the massive new storage facility in Utah, Binney suspects that it now simply collects everything. “The whole idea was, how do you manage 20 terabytes of intercept a minute?” he says. “The way we proposed was to distinguish between things you want and things you don’t want.” Instead, he adds, “they’re storing everything they gather.” And the agency is gathering as much as it can.

Once the communications are intercepted and stored, the data-mining begins. “You can watch everybody all the time with data- mining,” Binney says. Everything a person does becomes charted on a graph, “financial transactions or travel or anything,” he says. Thus, as data like bookstore receipts, bank statements, and commuter toll records flow in, the NSA is able to paint a more and more detailed picture of someone’s life.

The NSA also has the ability to eavesdrop on phone calls directly and in real time. According to Adrienne J. Kinne, who worked both before and after 9/11 as a voice interceptor at the NSA facility in Georgia, in the wake of the World Trade Center attacks “basically all rules were thrown out the window, and they would use any excuse to justify a waiver to spy on Americans.” Even journalists calling home from overseas were included. “A lot of time you could tell they were calling their families,” she says, “incredibly intimate, personal conversations.” Kinne found the act of eavesdropping on innocent fellow citizens personally distressing. “It’s almost like going through and finding somebody’s diary,” she says.

In secret listening rooms nationwide, NSA software examines every email, phone call, and tweet as they zip by.
But there is, of course, reason for anyone to be distressed about the practice. Once the door is open for the government to spy on US citizens, there are often great temptations to abuse that power for political purposes, as when Richard Nixon eavesdropped on his political enemies during Watergate and ordered the NSA to spy on antiwar protesters. Those and other abuses prompted Congress to enact prohibitions in the mid-1970s against domestic spying.

Before he gave up and left the NSA, Binney tried to persuade officials to create a more targeted system that could be authorized by a court. At the time, the agency had 72 hours to obtain a legal warrant, and Binney devised a method to computerize the system. “I had proposed that we automate the process of requesting a warrant and automate approval so we could manage a couple of million intercepts a day, rather than subvert the whole process.” But such a system would have required close coordination with the courts, and NSA officials weren’t interested in that, Binney says. Instead they continued to haul in data on a grand scale. Asked how many communications—”transactions,” in NSA’s lingo—the agency has intercepted since 9/11, Binney estimates the number at “between 15 and 20 trillion, the aggregate over 11 years.”

When Barack Obama took office, Binney hoped the new administration might be open to reforming the program to address his constitutional concerns. He and another former senior NSA analyst, J. Kirk Wiebe, tried to bring the idea of an automated warrant-approval system to the attention of the Department of Justice’s inspector general. They were given the brush-off. “They said, oh, OK, we can’t comment,” Binney says.

Sitting in a restaurant not far from NSA headquarters, the place where he spent nearly 40 years of his life, Binney held his thumb and forefinger close together. “We are, like, that far from a turnkey totalitarian state,” he says.

There is still one technology preventing untrammeled government access to private digital data: strong encryption. Anyone—from terrorists and weapons dealers to corporations, financial institutions, and ordinary email senders—can use it to seal their messages, plans, photos, and documents in hardened data shells. For years, one of the hardest shells has been the Advanced Encryption Standard, one of several algorithms used by much of the world to encrypt data. Available in three different strengths—128 bits, 192 bits, and 256 bits—it’s incorporated in most commercial email programs and web browsers and is considered so strong that the NSA has even approved its use for top-secret US government communications. Most experts say that a so-called brute-force computer attack on the algorithm—trying one combination after another to unlock the encryption—would likely take longer than the age of the universe. For a 128-bit cipher, the number of trial-and-error attempts would be 340 undecillion (1036).

Breaking into those complex mathematical shells like the AES is one of the key reasons for the construction going on in Bluffdale. That kind of cryptanalysis requires two major ingredients: super-fast computers to conduct brute-force attacks on encrypted messages and a massive number of those messages for the computers to analyze. The more messages from a given target, the more likely it is for the computers to detect telltale patterns, and Bluffdale will be able to hold a great many messages. “We questioned it one time,” says another source, a senior intelligence manager who was also involved with the planning. “Why were we building this NSA facility? And, boy, they rolled out all the old guys—the crypto guys.” According to the official, these experts told then-director of national intelligence Dennis Blair, “You’ve got to build this thing because we just don’t have the capability of doing the code-breaking.” It was a candid admission. In the long war between the code breakers and the code makers—the tens of thousands of cryptographers in the worldwide computer security industry—the code breakers were admitting defeat.

So the agency had one major ingredient—a massive data storage facility—under way. Meanwhile, across the country in Tennessee, the government was working in utmost secrecy on the other vital element: the most powerful computer the world has ever known.

The plan was launched in 2004 as a modern-day Manhattan Project. Dubbed the High Productivity Computing Systems program, its goal was to advance computer speed a thousandfold, creating a machine that could execute a quadrillion (1015) operations a second, known as a petaflop—the computer equivalent of breaking the land speed record. And as with the Manhattan Project, the venue chosen for the supercomputing program was the town of Oak Ridge in eastern Tennessee, a rural area where sharp ridges give way to low, scattered hills, and the southwestward-flowing Clinch River bends sharply to the southeast. About 25 miles from Knoxville, it is the “secret city” where uranium- 235 was extracted for the first atomic bomb. A sign near the exit read: what you see here, what you do here, what you hear here, when you leave here, let it stay here. Today, not far from where that sign stood, Oak Ridge is home to the Department of Energy’s Oak Ridge National Laboratory, and it’s engaged in a new secret war. But this time, instead of a bomb of almost unimaginable power, the weapon is a computer of almost unimaginable speed.

In 2004, as part of the supercomputing program, the Department of Energy established its Oak Ridge Leadership Computing Facility for multiple agencies to join forces on the project. But in reality there would be two tracks, one unclassified, in which all of the scientific work would be public, and another top-secret, in which the NSA could pursue its own computer covertly. “For our purposes, they had to create a separate facility,” says a former senior NSA computer expert who worked on the project and is still associated with the agency. (He is one of three sources who described the program.) It was an expensive undertaking, but one the NSA was desperate to launch.

Known as the Multiprogram Research Facility, or Building 5300, the $41 million, five-story, 214,000-square-foot structure was built on a plot of land on the lab’s East Campus and completed in 2006. Behind the brick walls and green-tinted windows, 318 scientists, computer engineers, and other staff work in secret on the cryptanalytic applications of high-speed computing and other classified projects. The supercomputer center was named in honor of George R. Cotter, the NSA’s now-retired chief scientist and head of its information technology program. Not that you’d know it. “There’s no sign on the door,” says the ex-NSA computer expert.

At the DOE’s unclassified center at Oak Ridge, work progressed at a furious pace, although it was a one-way street when it came to cooperation with the closemouthed people in Building 5300. Nevertheless, the unclassified team had its Cray XT4 supercomputer upgraded to a warehouse-sized XT5. Named Jaguar for its speed, it clocked in at 1.75 petaflops, officially becoming the world’s fastest computer in 2009.

Meanwhile, over in Building 5300, the NSA succeeded in building an even faster supercomputer. “They made a big breakthrough,” says another former senior intelligence official, who helped oversee the program. The NSA’s machine was likely similar to the unclassified Jaguar, but it was much faster out of the gate, modified specifically for cryptanalysis and targeted against one or more specific algorithms, like the AES. In other words, they were moving from the research and development phase to actually attacking extremely difficult encryption systems. The code-breaking effort was up and running.

The breakthrough was enormous, says the former official, and soon afterward the agency pulled the shade down tight on the project, even within the intelligence community and Congress. “Only the chairman and vice chairman and the two staff directors of each intelligence committee were told about it,” he says. The reason? “They were thinking that this computing breakthrough was going to give them the ability to crack current public encryption.”

In addition to giving the NSA access to a tremendous amount of Americans’ personal data, such an advance would also open a window on a trove of foreign secrets. While today most sensitive communications use the strongest encryption, much of the older data stored by the NSA, including a great deal of what will be transferred to Bluffdale once the center is complete, is encrypted with more vulnerable ciphers. “Remember,” says the former intelligence official, “a lot of foreign government stuff we’ve never been able to break is 128 or less. Break all that and you’ll find out a lot more of what you didn’t know—stuff we’ve already stored—so there’s an enormous amount of information still in there.”

The NSA believes it’s on the verge of breaking a key encryption algorithm—opening up hoards of data.
That, he notes, is where the value of Bluffdale, and its mountains of long-stored data, will come in. What can’t be broken today may be broken tomorrow. “Then you can see what they were saying in the past,” he says. “By extrapolating the way they did business, it gives us an indication of how they may do things now.” The danger, the former official says, is that it’s not only foreign government information that is locked in weaker algorithms, it’s also a great deal of personal domestic communications, such as Americans’ email intercepted by the NSA in the past decade.

But first the supercomputer must break the encryption, and to do that, speed is everything. The faster the computer, the faster it can break codes. The Data Encryption Standard, the 56-bit predecessor to the AES, debuted in 1976 and lasted about 25 years. The AES made its first appearance in 2001 and is expected to remain strong and durable for at least a decade. But if the NSA has secretly built a computer that is considerably faster than machines in the unclassified arena, then the agency has a chance of breaking the AES in a much shorter time. And with Bluffdale in operation, the NSA will have the luxury of storing an ever-expanding archive of intercepts until that breakthrough comes along.

But despite its progress, the agency has not finished building at Oak Ridge, nor is it satisfied with breaking the petaflop barrier. Its next goal is to reach exaflop speed, one quintillion (1018) operations a second, and eventually zettaflop (1021) and yottaflop.

These goals have considerable support in Congress. Last November a bipartisan group of 24 senators sent a letter to President Obama urging him to approve continued funding through 2013 for the Department of Energy’s exascale computing initiative (the NSA’s budget requests are classified). They cited the necessity to keep up with and surpass China and Japan. “The race is on to develop exascale computing capabilities,” the senators noted. The reason was clear: By late 2011 the Jaguar (now with a peak speed of 2.33 petaflops) ranked third behind Japan’s “K Computer,” with an impressive 10.51 petaflops, and the Chinese Tianhe-1A system, with 2.57 petaflops.

But the real competition will take place in the classified realm. To secretly develop the new exaflop (or higher) machine by 2018, the NSA has proposed constructing two connecting buildings, totaling 260,000 square feet, near its current facility on the East Campus of Oak Ridge. Called the Multiprogram Computational Data Center, the buildings will be low and wide like giant warehouses, a design necessary for the dozens of computer cabinets that will compose an exaflop-scale machine, possibly arranged in a cluster to minimize the distance between circuits. According to a presentation delivered to DOE employees in 2009, it will be an “unassuming facility with limited view from roads,” in keeping with the NSA’s desire for secrecy. And it will have an extraordinary appetite for electricity, eventually using about 200 megawatts, enough to power 200,000 homes. The computer will also produce a gargantuan amount of heat, requiring 60,000 tons of cooling equipment, the same amount that was needed to serve both of the World Trade Center towers.

In the meantime Cray is working on the next step for the NSA, funded in part by a $250 million contract with the Defense Advanced Research Projects Agency. It’s a massively parallel supercomputer called Cascade, a prototype of which is due at the end of 2012. Its development will run largely in parallel with the unclassified effort for the DOE and other partner agencies. That project, due in 2013, will upgrade the Jaguar XT5 into an XK6, codenamed Titan, upping its speed to 10 to 20 petaflops.

Yottabytes and exaflops, septillions and undecillions—the race for computing speed and data storage goes on. In his 1941 story “The Library of Babel,” Jorge Luis Borges imagined a collection of information where the entire world’s knowledge is stored but barely a single word is understood. In Bluffdale the NSA is constructing a library on a scale that even Borges might not have contemplated. And to hear the masters of the agency tell it, it’s only a matter of time until every word is illuminated.

James Bamford ( is the author of The Shadow Factory: The Ultra-Secret NSA from 9/11 to the Eavesdropping on America.
User avatar
Wombaticus Rex
Posts: 10614
Joined: Wed Nov 08, 2006 6:33 pm
Location: Vermontistan
Blog: View Blog (0)

Re: Data Mining & Intelligence Agencies

Postby Wombaticus Rex » Wed Apr 04, 2012 11:36 pm

Monitoring America


Monday, December 20, 2010; 1:40 AM

Correction to this article: An earlier version of this article contained several incorrect numbers that have since been updated. The errors occurred because of the accidental duplication of 74 records in a database of over 4,000 counterterrorism organizations that The Post assembled. While not affecting the overall conclusions of the article, the 74 duplications mean that there are 3,984 federal, state and local organizations working on domestic counterterrorism, not 4,058. Of the total, the number created since the 2001 attacks is 934, not 935.

Nine years after the terrorist attacks of 2001, the United States is assembling a vast domestic intelligence apparatus to collect information about Americans, using the FBI, local police, state homeland security offices and military criminal investigators.

The system, by far the largest and most technologically sophisticated in the nation's history, collects, stores and analyzes information about thousands of U.S. citizens and residents, many of whom have not been accused of any wrongdoing.

The government's goal is to have every state and local law enforcement agency in the country feed information to Washington to buttress the work of the FBI, which is in charge of terrorism investigations in the United States.

Other democracies - Britain and Israel, to name two - are well acquainted with such domestic security measures. But for the United States, the sum of these new activities represents a new level of governmental scrutiny.

This localized intelligence apparatus is part of a larger Top Secret America created since the attacks. In July, The Washington Post described an alternative geography of the United States, one that has grown so large, unwieldy and secretive that no one knows how much money it costs, how many people it employs or how many programs exist within it.

Today's story, along with related material on The Post's Web site, examines how Top Secret America plays out at the local level. It describes a web of 3,984 federal, state and local organizations, each with its own counterterrorism responsibilities and jurisdictions. At least 934 of these organizations have been created since the 2001 attacks or became involved in counterterrorism for the first time after 9/11.

(Search our database for your state to find a detailed profile of counterterrorism efforts in your community.) ... ca/states/

The months-long investigation, based on nearly 100 interviews and 1,000 documents, found that:

* Technologies and techniques honed for use on the battlefields of Iraq and Afghanistan have migrated into the hands of law enforcement agencies in America.

* The FBI is building a database with the names and certain personal information, such as employment history, of thousands of U.S. citizens and residents whom a local police officer or a fellow citizen believed to be acting suspiciously. It is accessible to an increasing number of local law enforcement and military criminal investigators, increasing concerns that it could somehow end up in the public domain.

* Seeking to learn more about Islam and terrorism, some law enforcement agencies have hired as trainers self-described experts whose extremist views on Islam and terrorism are considered inaccurate and counterproductive by the FBI and U.S. intelligence agencies.

* The Department of Homeland Security sends its state and local partners intelligence reports with little meaningful guidance, and state reports have sometimes inappropriately reported on lawful meetings.

Counterterrorism on Main Street
In cities across Tennessee and across the nation local agencies are using sophisticated equipment and techniques to keep an eye out for terrorist threats -- and to watch Americans in the process. Launch Gallery »

The need to identify U.S.-born or naturalized citizens who are planning violent attacks is more urgent than ever, U.S. intelligence officials say. This month's FBI sting operation involving a Baltimore construction worker who allegedly planned to bomb a Maryland military recruiting station is the latest example. It followed a similar arrest of a Somali-born naturalized U.S. citizen allegedly seeking to detonate a bomb near a Christmas tree lighting ceremony in Portland, Ore. There have been nearly two dozen other cases just this year.

"The old view that 'if we fight the terrorists abroad, we won't have to fight them here' is just that - the old view," Homeland Security Secretary Janet Napolitano told police and firefighters recently.

The Obama administration heralds this local approach as a much-needed evolution in the way the country confronts terrorism.

Top Secret America is a project two years in the making that describes the huge security buildup in the United States after the Sept. 11, 2001, attacks. Today’s story is about those efforts at the local level, including law enforcement and homeland security agencies in every state and thousands of communities. View previous stories, explore relationships between government organizations and the types of work being done, and view top-secret geography on an interactive map.

However, just as at the federal level, the effectiveness of these programs, as well as their cost, is difficult to determine. The Department of Homeland Security, for example, does not know how much money it spends each year on what are known as state fusion centers, which bring together and analyze information from various agencies within a state.

The total cost of the localized system is also hard to gauge. The DHS has given $31 billion in grants since 2003 to state and local governments for homeland security and to improve their ability to find and protect against terrorists, including $3.8 billion in 2010. At least four other federal departments also contribute to local efforts. But the bulk of the spending every year comes from state and local budgets that are too disparately recorded to aggregate into an overall total.

The Post findings paint a picture of a country at a crossroads, where long-standing privacy principles are under challenge by these new efforts to keep the nation safe.

The public face of this pivotal effort is Napolitano, the former governor of Arizona, which years ago built one of the strongest state intelligence organizations outside of New York to try to stop illegal immigration and drug importation.

Napolitano has taken her "See Something, Say Something" campaign far beyond the traffic signs that ask drivers coming into the nation's capital for "Terror Tips" and to "Report Suspicious Activity."

She recently enlisted the help of Wal-Mart, Amtrak, major sports leagues, hotel chains and metro riders. In her speeches, she compares the undertaking to the Cold War fight against communists.

"This represents a shift for our country," she told New York City police officers and firefighters on the eve of the 9/11 anniversary this fall. "In a sense, this harkens back to when we drew on the tradition of civil defense and preparedness that predated today's concerns."


From Afghanistan to Tennessee

On a recent night in Memphis, a patrol car rolled slowly through a parking lot in a run-down section of town. The military-grade infrared camera on its hood moved robotically from left to right, snapping digital images of one license plate after another and analyzing each almost instantly.

Suddenly, a red light flashed on the car's screen along with the word "warrant."

"Got a live one! Let's do it," an officer called out.

The streets of Memphis are a world away from the streets of Kabul, yet these days, the same types of technologies and techniques are being used in both places to identify and collect information about suspected criminals and terrorists.

The examples go far beyond Memphis.

* Hand-held, wireless fingerprint scanners were carried by U.S. troops during the insurgency in Iraq to register residents of entire neighborhoods. L-1 Identity Solutions is selling the same type of equipment to police departments to check motorists' identities.

* In Arizona, the Maricopa County Sheriff's Facial Recognition Unit, using a type of equipment prevalent in war zones, records 9,000 biometric digital mug shots a month.

* U.S. Customs and Border Protection flies General Atomics' Predator drones along the Mexican and Canadian borders - the same kind of aircraft, equipped with real-time, full-motion video cameras, that has been used in wars in Kosovo, Iraq and Afghanistan to track the enemy.

The special operations units deployed overseas to kill the al-Qaeda leadership drove technological advances that are now expanding in use across the United States. On the front lines, those advances allowed the rapid fusing of biometric identification, captured computer records and cellphone numbers so troops could launch the next surprise raid.

Here at home, it's the DHS that is enamored with collecting photos, video images and other personal information about U.S. residents in the hopes of teasing out terrorists.

The DHS helped Memphis buy surveillance cameras that monitor residents near high-crime housing projects, problematic street corners, and bridges and other critical infrastructure. It helped pay for license plate readers and defrayed some of the cost of setting up Memphis's crime-analysis center. All together it has given Memphis $11 million since 2003 in homeland security grants, most of which the city has used to fight crime.

"We have got things now we didn't have before," said Memphis Police Department Director Larry Godwin, who has produced record numbers of arrests using all this new analysis and technology. "Some of them we can talk about. Some of them we can't."

One of the biggest advocates of Memphis's data revolution is John Harvey, the police department's technology specialist, whose computer systems are the civilian equivalent of the fancier special ops equipment used by the military.

Harvey collects any information he can pry out of government and industry. When officers were wasting time knocking on the wrong doors to serve warrants, he persuaded the local utility company to give him a daily update of the names and addresses of customers.

When he wanted more information about phones captured at crime scenes, he programmed a way to store all emergency 911 calls, which often include names and addresses to associate with phone numbers. He created another program to upload new crime reports every five minutes and mine them for the phone numbers of victims, suspects, witnesses and anyone else listed on them.

Now, instead of having to decide which license plate numbers to type into a computer console in the patrol car, an officer can simply drive around, and the automatic license plate reader on his hood captures the numbers on every vehicle nearby. If the officer pulls over a driver, instead of having to wait 20 minutes for someone back at the office to manually check records, he can use a hand-held device to instantly call up a mug shot, a Social Security number, the status of the driver's license and any outstanding warrants.

The computer in the cruiser can tell an officer even more about who owns the vehicle, the owner's name and address and criminal history, and who else with a criminal history might live at the same address.

Take a recent case of two officers with the hood-mounted camera equipment who stopped a man driving on a suspended license. One handcuffed him, and the other checked his own PDA. Based on the information that came up, the man was ordered downtown to pay a fine and released as the officers drove off to stop another car.

That wasn't the end of it, though.

A record of that stop - and the details of every other arrest made that night, and every summons written - was automatically transferred to the Memphis Real Time Crime Center, a command center with three walls of streaming surveillance video and analysis capabilities that rival those of an Army command center.

There, the information would be geocoded on a map to produce a visual rendering of crime patterns. This information would help the crime intelligence analysts predict trends so the department could figure out what neighborhoods to swarm with officers and surveillance cameras.

But that was still not the end of it, because the fingerprints from the crime records would also go to the FBI's data campus in Clarksburg, W.Va. There, fingerprints from across the United States are stored, along with others collected by American authorities from prisoners in Saudi Arabia and Yemen, Iraq and Afghanistan.

There are 96 million sets of fingerprints in Clarksburg, a volume that government officials view not as daunting but as an opportunity.

This year for the first time, the FBI, the DHS and the Defense Department are able to search each other's fingerprint databases, said Myra Gray, head of the Defense Department's Biometrics Identity Management Agency, speaking to an industry group recently. "Hopefully in the not-too-distant future," she said, "our relationship with these federal agencies - along with state and local agencies - will be completely symbiotic."


The FBI's 'suspicious' files

At the same time that the FBI is expanding its West Virginia database, it is building a vast repository controlled by people who work in a top-secret vault on the fourth floor of the J. Edgar Hoover FBI Building in Washington. This one stores the profiles of tens of thousands of Americans and legal residents who are not accused of any crime. What they have done is appear to be acting suspiciously to a town sheriff, a traffic cop or even a neighbor.

If the new Nationwide Suspicious Activity Reporting Initiative, or SAR, works as intended, the Guardian database may someday hold files forwarded by all police departments across the country in America's continuing search for terrorists within its borders.

The effectiveness of this database depends, in fact, on collecting the identities of people who are not known criminals or terrorists - and on being able to quickly compile in-depth profiles of them.

"If we want to get to the point where we connect the dots, the dots have to be there," said Richard A. McFeely, special agent in charge of the FBI's Baltimore office.

In response to concerns that information in the database could be improperly used or released, FBI officials say anyone with access has been trained in privacy rules and the penalties for breaking them.

But not everyone is convinced. "It opens a door for all kinds of abuses," said Michael German, a former FBI agent who now leads the American Civil Liberties Union's campaign on national security and privacy matters. "How do we know there are enough controls?"

The government defines a suspicious activity as "observed behavior reasonably indicative of pre-operational planning related to terrorism or other criminal activity" related to terrorism.

State intelligence analysts and FBI investigators use the reports to determine whether a person is buying fertilizer to make a bomb or to plant tomatoes; whether she is plotting to poison a city's drinking water or studying for a metallurgy test; whether, as happened on a Sunday morning in late September, the man snapping a picture of a ferry in the Newport Beach harbor in Southern California simply liked the way it looked or was plotting to blow it up.

Suspicious Activity Report N03821 says a local law enforcement officer observed "a suspicious subject . . . taking photographs of the Orange County Sheriff Department Fire Boat and the Balboa Ferry with a cellular phone camera." The confidential report, marked "For Official Use Only," noted that the subject next made a phone call, walked to his car and returned five minutes later to take more pictures. He was then met by another person, both of whom stood and "observed the boat traffic in the harbor." Next another adult with two small children joined them, and then they all boarded the ferry and crossed the channel.

All of this information was forwarded to the Los Angeles fusion center for further investigation after the local officer ran information about the vehicle and its owner through several crime databases and found nothing.

Authorities would not say what happened to it from there, but there are several paths a suspicious activity report can take:

At the fusion center, an officer would decide to either dismiss the suspicious activity as harmless or forward the report to the nearest FBI terrorism unit for further investigation.

At that unit, it would immediately be entered into the Guardian database, at which point one of three things could happen:

The FBI could collect more information, find no connection to terrorism and mark the file closed, though leaving it in the database.

It could find a possible connection and turn it into a full-fledged case.

Or, as most often happens, it could make no specific determination, which would mean that Suspicious Activity Report N03821 would sit in limbo for as long as five years, during which time many other pieces of information about the man photographing a boat on a Sunday morning could be added to his file: employment, financial and residential histories; multiple phone numbers; audio files; video from the dashboard-mounted camera in the police cruiser at the harbor where he took pictures; and anything else in government or commercial databases "that adds value," as the FBI agent in charge of the database described it.

That could soon include biometric data, if it existed; the FBI is working on a way to attach such information to files. Meanwhile, the bureau will also soon have software that allows local agencies to map all suspicious incidents in their jurisdiction.

The Defense Department is also interested in the database. It recently transferred 100 reports of suspicious behavior into the Guardian system, and over time it expects to add thousands more as it connects 8,000 military law enforcement personnel to an FBI portal that will allow them to send and review reports about people suspected of casing U.S. bases or targeting American personnel.

And the DHS has created a separate way for state and local authorities, private citizens, and businesses to submit suspicious activity reports to the FBI and to the department for analysis.

As of December, there were 161,948 suspicious activity files in the classified Guardian database, mostly leads from FBI headquarters and state field offices. Two years ago, the bureau set up an unclassified section of the database so state and local agencies could send in suspicious incident reports and review those submitted by their counterparts in other states. Some 890 state and local agencies have sent in 7,197 reports so far.

Of those, 103 have become full investigations that have resulted in at least five arrests, the FBI said. There have been no convictions yet. An additional 365 reports have added information to ongoing cases.

But most remain in the uncertain middle, which is why within the FBI and other intelligence agencies there is much debate about the effectiveness of the bottom-up SAR approach, as well as concern over the privacy implications of retaining so much information on U.S. citizens and residents who have not been charged with anything.

The vast majority of terrorism leads in the United States originate from confidential FBI sources and from the bureau's collaboration with federal intelligence agencies, which mainly work overseas. Occasionally a stop by a local police officer has sparked an investigation. Evidence comes from targeted FBI surveillance and undercover operations, not from information and analysis generated by state fusion centers about people acting suspiciously.

"It's really resource-inefficient," said Philip Mudd, a 20-year CIA counterterrorism expert and a top FBI national security official until he retired nine months ago. "If I were to have a dialogue with the country about this . . . it would be about not only how we chase the unknowns, but do you want to do suspicious activity reports across the country? . . . Anyone who is not at least suspected of doing something criminal should not be in a database."

Charles Allen, a longtime senior CIA official who then led the DHS's intelligence office until 2009, said some senior people in the intelligence community are skeptical that SARs are an effective way to find terrorists. "It's more likely that other kinds of more focused efforts by local police will gain you the information that you need about extremist activities," he said.

The DHS can point to some successes: Last year the Colorado fusion center turned up information on Najibullah Zazi, an Afghan-born U.S. resident planning to bomb the New York subway system. In 2007, a Florida fusion center provided the vehicle ownership history used to identify and arrest an Egyptian student who later pleaded guilty to providing material support to terrorism, in this case transporting explosives.

"Ninety-nine percent doesn't pan out or lead to anything" said Richard Lambert Jr., the special agent in charge of the FBI's Knoxville office. "But we're happy to wade through these things."


Expert training?

Ramon Montijo has taught classes on terrorism and Islam to law enforcement officers all over the country.

"Alabama, Colorado, Vermont," said Montijo, a former Army Special Forces sergeant and Los Angeles Police Department investigator who is now a private security consultant. "California, Texas and Missouri," he continued.

What he tells them is always the same, he said: Most Muslims in the United States want to impose sharia law here.

"They want to make this world Islamic. The Islamic flag will fly over the White House - not on my watch!" he said. "My job is to wake up the public, and first, the first responders."

With so many local agencies around the country being asked to help catch terrorists, it often falls to sheriffs or state troopers to try to understand the world of terrorism. They aren't FBI agents, who have years of on-the-job and classroom training.

Instead, they are often people like Lacy Craig, who was a police dispatcher before she became an intelligence analyst at Idaho's fusion center, or the detectives in Minnesota, Michigan and Arkansas who can talk at length about the lineage of gangs or the signs of a crystal meth addict.

Now each of them is a go-to person on terrorism as well.

"The CIA used to train analysts forever before they graduated to be a real analyst," said Allen, the former top CIA and DHS official. "Today we take former law enforcement officers and we call them intelligence officers, and that's not right, because they have not received any training on intelligence analysis."

State fusion center officials say their analysts are getting better with time. "There was a time when law enforcement didn't know much about drugs. This is no different," said Steven W. Hewitt, who runs the Tennessee fusion center, considered one of the best in the country. "Are we experts at the level of [the National Counterterrorism Center]? No. Are we developing an expertise? Absolutely."

But how they do that is usually left up to the local police departments themselves. In their desire to learn more about terrorism, many departments are hiring their own trainers. Some are self-described experts whose extremist views are considered inaccurate and harmful by the FBI and others in the intelligence community

Like Montijo, Walid Shoebat, a onetime Muslim who converted to Christianity, also lectures to local police. He too believes that most Muslims seek to impose sharia law in the United States. To prevent this, he said in an interview, he warns officers that "you need to look at the entire pool of Muslims in a community."

When Shoebat spoke to the first annual South Dakota Fusion Center Conference in Sioux Falls this June, he told them to monitor Muslim student groups and local mosques and, if possible, tap their phones. "You can find out a lot of information that way," he said.

A book expanding on what Shoebat and Montijo believe has just been published by the Center for Security Policy, a Washington-based neoconservative think tank. "Shariah: The Threat to America" describes what its authors call a "stealth jihad" that must be thwarted before it's too late.

The book's co-authors include such notables as former CIA director R. James Woolsey and former deputy undersecretary of defense for intelligence Lt. Gen. William G. Boykin, along with the center's director, a longtime activist. They write that most mosques in the United States already have been radicalized, that most Muslim social organizations are fronts for violent jihadists and that Muslims who practice sharia law seek to impose it in this country.

Frank Gaffney Jr., director of the center, said his team has spoken widely, including to many law enforcement forums.

"Members of our team have been involved in training programs for several years now, many of which have been focused on local law enforcement intelligence, homeland security, state police, National Guard units and the like," Gaffney said. "We're seeing a considerable ramping-up of interest in getting this kind of training."

Government terrorism experts call the views expressed in the center's book inaccurate and counterproductive. They say the DHS should increase its training of local police, using teachers who have evidence-based viewpoints.

DHS spokeswoman Amy Kudwa said the department does not maintain a list of terrorism experts but is working on guidelines for local authorities wrestling with the topic.

So far, the department has trained 1,391 local law enforcement officers in analyzing public information and 400 in analytic thinking and writing skills. Kudwa said the department also offers counterterrorism training through the Federal Emergency Management Agency, which this year enrolled 94 people in a course called "Advanced Criminal Intelligence Analysis to Prevent Terrorism."


A lack of useful information

The DHS also provides local agencies a daily flow of information bulletins.

These reports are meant to inform agencies about possible terror threats. But some officials say they deliver a never-ending stream of information that is vague, alarmist and often useless. "It's like a garage in your house you keep throwing junk into until you can't park your car in it," says Michael Downing, deputy chief of counterterrorism and special operations for the Los Angeles Police Department.

A review of nearly 1,000 DHS reports dating back to 2003 and labeled "For Official Use Only" underscores Downing's description. Typical is one from May 24, 2010, titled "Infrastructure Protection Note: Evolving Threats to the Homeland."

It tells officials to operate "under the premise that other operatives are in the country and could advance plotting with little or no warning." Its list of vulnerable facilities seems to include just about everything: "Commercial Facilities, Government Facilities, Banking and Financial and Transportation . . ."

Bart R. Johnson, who heads the DHS's intelligence and analysis office, defended such reports, saying that threat reporting has "grown and matured and become more focused." The bulletins can't be more specific, he said, because they must be written at the unclassified level.

Recently, the International Association of Chiefs of Police agreed that the information they were receiving had become "more timely and relevant" over the past year.

Downing, however, said the reports would be more helpful if they at least assessed threats within a specific state's boundaries.

States have tried to do that on their own, but with mixed, and at times problematic, results.

In 2009, for instance, after the DHS and the FBI sent out several ambiguous reports about threats to mass-transit systems and sports and entertainment venues, the New Jersey Regional Operations Intelligence Center's Threat Analysis Program added its own information. "New Jersey has a large mass-transit infrastructure," its report warned, and "an NFL stadium and NHL/NBA arenas, a soccer stadium, and several concert venues that attract large crowds."

In Virginia, the state's fusion center published a terrorism threat assessment in 2009 naming historically black colleges as potential hubs for terrorism.

From 2005 to 2007, the Maryland State Police went even further, infiltrating and labeling as terrorists local groups devoted to human rights, antiwar causes and bike lanes.

And in Pennsylvania this year, a local contractor hired to write intelligence bulletins filled them with information about lawful meetings as varied as Pennsylvania Tea Party Patriots Coalition gatherings, antiwar protests and an event at which environmental activists dressed up as Santa Claus and handed out coal-filled stockings.


'We have our own terrorists'

Even if the information were better, it might not make a difference for the simplest of reasons: In many cities and towns across the country, there is just not enough terrorism-related work to do.

In Utah on one recent day, one of five intelligence analysts in the state's fusion center was writing a report about the rise in teenage overdoses of an over-the-counter drug. Another was making sure the visiting president of Senegal had a safe trip. Another had just helped a small town track down two people who were selling magazine subscriptions and pocketing the money themselves.

In the Colorado Information Analysis Center, some investigators were following terrorism leads. Others were looking into illegal Craigslist postings and online "World of Warcraft" gamers.

The vast majority of fusion centers across the country have transformed themselves into analytical hubs for all crimes and are using federal grants, handed out in the name of homeland security, to combat everyday offenses.

This is happening because, after 9/11, local law enforcement groups did what every agency and private company did in Top Secret America: They followed the money.

The DHS helped the Memphis Police Department, for example, purchase 90 surveillance cameras, including 13 that monitor bridges and a causeway. It helped buy the fancy screens on the walls of the Real Time Crime Center, as well as radios, robotic surveillance equipment, a mobile command center and three bomb-sniffing dogs. All came in the name of port security and protection to critical infrastructure.

Since there hasn't been a solid terrorism case in Memphis yet, the equipment's greatest value has been to help drive down city crime. Where the mobile surveillance cameras are set up, criminals scatter, said Lt. Mark Rewalt, who, on a recent Saturday night, scanned the city from an altitude of 1,000 feet.

Flying in a police helicopter, Rewalt pointed out some of the cameras the DHS has funded. They are all over the city, in mall parking lots, in housing projects, at popular street hang-outs. "Cameras are what's happening now," he marveled.

Meanwhile, another post-9/11 unit in Tennessee has had even less terrorism-related work to do.

The Tennessee National Guard 45th Weapons of Mass Destruction Civil Support Team, one of at least 50 such units around the country, was created to respond to what officials still believe is the inevitable release of chemical, biological or radiological material by terrorists.

The unit's 22 hazardous-materials personnel have the best emergency equipment in the state. A fleet of navy-blue vehicles - command, response, detection and tactical operations trucks - is kept polished and ready to roll in a garage at the armory in Smyrna.

The unit practices WMD scenarios constantly. But in real life, the crew uses the equipment very little: twice a year at NASCAR races in nearby Bristol to patrol for suspicious packages. Other than that, said Capt. Matt Hayes, several times a year they respond to hoaxes.

The fact that there has not been much terrorism to worry about is not evident on the Tennessee fusion center's Web site. Click on the incident map, and the state appears to be under attack.

Red icons of explosions dot Tennessee, along with blinking exclamation marks and flashing skulls. The map is labeled: "Terrorism Events and Other Suspicious Activity.

But if you roll over the icons, the explanations that pop up have nothing to do with major terrorist plots: "Johnson City police are investigating three 'bottle bombs' found at homes over the past three days," one description read recently. ". . . The explosives were made from plastic bottles with something inside that reacted chemically and caused the bottles to burst."

Another told a similar story: "The Scott County Courthouse is currently under evacuation after a bomb threat was called in Friday morning. Update: Authorities completed their sweep . . . and have called off the evacuation."

Nine years after 9/11, this map is part of the alternative geography that is Top Secret America, where millions of people are assigned to help stop terrorism. Memphis Police Director Godwin is one of them, and he has his own version of what that means in a city where there have been 86 murders so far this year.

"We have our own terrorists, and they are taking lives every day," Godwin said. "No, we don't have suicide bombers - not yet. But you need to remain vigilant and realize how vulnerable you can be if you let up."

Staff researcher Julie Tate contributed to this story.

© 2011 The Washington Post Company

ARTICLE LINK ... ica/print/
Archived here for strictly non-commercial fair-use purpose of advancing a public discussion and preserving stuff from the Memory Hole.
User avatar
Wombaticus Rex
Posts: 10614
Joined: Wed Nov 08, 2006 6:33 pm
Location: Vermontistan
Blog: View Blog (0)

Re: Data Mining & Intelligence Agencies

Postby Wombaticus Rex » Wed Apr 04, 2012 11:51 pm

AlicetheKurious wrote:Domestic Spying, Inc.

by Tim Shorrock , Special to CorpWatch
November 27th, 2007

A new intelligence institution to be inaugurated soon by the Bush administration will allow government spying agencies to conduct broad surveillance and reconnaissance inside the United States for the first time.

Under a proposal being reviewed by Congress, a National Applications Office (NAO) will be established to coordinate how the Department of Homeland Security (DHS) and domestic law enforcement and rescue agencies use imagery and communications intelligence picked up by U.S. spy satellites. If the plan goes forward, the NAO will create the legal mechanism for an unprecedented degree of domestic intelligence gathering that would make the U.S. one of the world's most closely monitored nations. Until now, domestic use of electronic intelligence from spy satellites was limited to scientific agencies with no responsibility for national security or law enforcement.

The intelligence-sharing system to be managed by the NAO will rely heavily on private contractors including Boeing, BAE Systems, L-3 Communications and Science Applications International Corporation (SAIC). These companies already provide technology and personnel to U.S. agencies involved in foreign intelligence, and the NAO greatly expands their markets. Indeed, at an intelligence conference in San Antonio, Texas, last month, the titans of the industry were actively lobbying intelligence officials to buy products specifically designed for domestic surveillance.

The NAO was created under a plan tentatively approved in May 2007 by Director of National Intelligence Michael McConnell. Specifically, the NAO will oversee how classified information collected by the National Security Agency (NSA), the National Geospatial-Intelligence Agency (NGA) and other key agencies is used within the U.S. during natural disasters, terrorist attacks and other events affecting national security. The most critical intelligence will be supplied by the NSA and the NGA, which are often referred to by U.S. officials as the “eyes” and “ears” of the intelligence community.

The NSA, through a global network of listening posts, surveillance planes, and satellites, captures signals from phone calls, e-mail and Internet traffic, and translates and analyzes them for U.S. military and national intelligence officials.

The National Geospatial-Intelligence Agency (NGA), which was formally inaugurated in 2003, provides overhead imagery and mapping tools that allow intelligence and military analysts to monitor events from the skies and space. The NSA and the NGA have a close relationship with the super-secret National Reconnaissance Agency (NRO), which builds and maintains the U.S. fleet of spy satellites and operates the ground stations where the NSA’s signals and the NGA’s imagery are processed and analyzed. By law, their collection efforts are supposed to be confined to foreign countries and battlefields.

The National Applications Office was conceived in 2005 by the Office of the Director of National Intelligence (ODNI), which Congress created in 2004 to oversee the 16 agencies that make up the U.S. intelligence community. The ODNI, concerned that the legal framework for U.S. intelligence operations had not been updated for the global “war on terror,” turned to Booz Allen Hamilton of McLean, Virginia -- one of the largest contractors in the spy business. The company was tasked with studying how intelligence from spy satellites and photoreconnaissance planes could be better used domestically to track potential threats to security within the U.S.. The Booz Allen study was completed in May of that year, and has since become the basis for the NAO oversight plan. In May 2007, McConnell, the former executive vice president of Booz Allen, signed off on the creation of the NAO as the principal body to oversee the merging of foreign and domestic intelligence collection operations.

The NAO is "an idea whose time has arrived," Charles Allen, a top U.S. intelligence official, told the Wall Street Journal in August 2007 after it broke the news of the creation of the NAO. Allen, the DHS's chief intelligence officer, will head the new program. The announcement came just days after President George W. Bush signed a new law approved by Congress to expand the ability of the NSA to eavesdrop, without warrants, on telephone calls, e-mail and faxes passing through telecommunications hubs in the U.S. when the government suspects agents of a foreign power may be involved. "These [intelligence] systems are already used to help us respond to crises," Allen later told the Washington Post. "We anticipate that we can also use them to protect Americans by preventing the entry of dangerous people and goods into the country, and by helping us examine critical infrastructure for vulnerabilities."

Donald Kerr, a former NRO director who is now the number two at ODNI, recently explained to reporters that the intelligence community was no longer discussing whether or not to spy on U.S. citizens: “Our job now is to engage in a productive debate, which focuses on privacy as a component of appropriate levels of security and public safety,'' Kerr said. ''I think all of us have to really take stock of what we already are willing to give up, in terms of anonymity, but [also] what safeguards we want in place to be sure that giving that doesn't empty our bank account or do something equally bad elsewhere.''

What Will The NAO Do?

The plan for the NAO builds on a domestic security infrastructure that has been in place for at least seven years. After the terrorist attacks of September 11, 2001, the NSA was granted new powers to monitor domestic communications without obtaining warrants from a secret foreign intelligence court established by Congress in 1978 (that warrant-less program ended in January 2007 but was allowed to continue, with some changes, under legislation passed by Congress in August 2007).

Moreover, intelligence and reconnaissance agencies that were historically confined to spying on foreign countries have been used extensively on the home front since 2001. In the hours after the September 11th, 2001 attacks in New York, for example, the Bush administration called on the NGA to capture imagery from lower Manhattan and the Pentagon to help in the rescue and recovery efforts. In 2002, when two deranged snipers terrified the citizens of Washington and its Maryland and Virginia suburbs with a string of fatal shootings, the Federal Bureau of Investigation (FBI) asked the NGA to provide detailed images of freeway interchanges and other locations to help spot the pair.

The NGA was also used extensively during Hurricane Katrina , when the agency provided overhead imagery -- some of it supplied by U-2 photoreconnaissance aircraft -- to federal and state rescue operations. The data, which included mapping of flooded areas in Louisiana and Mississippi, allowed residents of the stricken areas to see the extent of damage to their homes and helped first-responders locate contaminated areas as well as schools, churches and hospitals that might be used in the rescue. More recently, during the October 2007 California wildfires, the Federal Emergency Management Agency (FEMA) asked the NGA to analyze overhead imagery of the fire zones and determine the areas of maximum intensity and damage. In every situation that the NGA is used domestically, it must receive a formal request from a lead domestic agency, according to agency spokesperson David Burpee. That agency is usually FEMA, which is a unit of DHS.

At first blush, the idea of a U.S. intelligence agency serving the public by providing imagery to aid in disaster recovery sounds like a positive development, especially when compared to the Bush administration’s misuse of the NSA and the Pentagon’s Counter-Intelligence Field Activity (CIFA) to spy on American citizens. But the notion of using spy satellites and aircraft for domestic purposes becomes problematic from a civil liberties standpoint when the full capabilities of agencies like the NGA and the NSA are considered.

Imagine, for example, that U.S. intelligence officials have determined, through NSA telephone intercepts, that a group of worshippers at a mosque in Oakland, California, has communicated with an Islamic charity in Saudi Arabia. This is the same group that the FBI and the U.S. Department of the Treasury believe is linked to an organization unfriendly to the United States.

Imagine further that the FBI, as a lead agency, asks and receives permission to monitor that mosque and the people inside using high-resolution imagery obtained from the NGA. Using other technologies, such as overhead traffic cameras in place in many cities, that mosque could be placed under surveillance for months, and -- through cell phone intercepts and overhead imagery -- its suspected worshipers carefully tracked in real-time as they moved almost anywhere in the country.

The NAO, under the plan approved by ODNI’s McConnell, would determine the rules that will guide the DHS and other lead federal agencies when they want to use imagery and signals intelligence in situations like this, as well as during natural disasters. If the organization is established as planned, U.S. domestic agencies will have a vast array of technology at their disposal. In addition to the powerful mapping and signals tools provided by the NGA and the NSA, domestic agencies will also have access to measures and signatures intelligence (MASINT) managed by the Defense Intelligence Agency (DIA), the principal spying agency used by the secretary of defense and the Joint Chiefs of Staff.

(MASINT is a highly classified form of intelligence that uses infrared sensors and other technologies to “sniff” the atmosphere for certain chemicals and electro-magnetic activity and “see” beneath bridges and forest canopies. Using its tools, analysts can detect signs that a nuclear power plant is producing plutonium, determine from truck exhaust what types of vehicles are in a convoy, and detect people and weapons hidden from the view of satellites or photoreconnaissance aircraft.)

Created By Contractors

The study group that established policies for the NAO was jointly funded by the ODNI and the U.S. Geological Survey (USGS), one of only two domestic U.S. agencies that is currently allowed, under rules set in the 1970s, to use classified intelligence from spy satellites. (The other is NASA, the National Aeronautics and Space Administration.) The group was chaired by Keith Hall, a Booz Allen vice president who manages his firm’s extensive contracts with the NGA and previously served as the director of the NRO.

Other members of the group included seven other former intelligence officers working for Booz Allen, as well as retired Army Lieutenant General Patrick M. Hughes, the former director of the DIA and vice president of homeland security for L-3 Communications, a key NSA contractor; and Thomas W. Conroy, the vice president of national security programs for Northrop Grumman, which has extensive contracts with the NSA and the NGA and throughout the intelligence community.

From the start, the study group was heavily weighted toward companies with a stake in both foreign and domestic intelligence. Not surprisingly, its contractor-advisers called for a major expansion in the domestic use of the spy satellites that they sell to the government.
Since the end of the Cold War and particularly since the September 11, 2001 attacks, they said, the “threats to the nation have changed and there is a growing interest in making available the special capabilities of the intelligence community to all parts of the government, to include homeland security and law enforcement entities and on a higher priority basis.”

Contractors are not new to the U.S. spy world. Since the creation of the Central Intelligence Agency (CIA) and the modern intelligence system in 1947, the private sector has been tapped to design and build the technology that facilitates electronic surveillance. Lockheed, for example, built the U-2, the famous surveillance plane that flew scores of spy missions over the Soviet Union and Cuba. During the 1960s, Lockheed was a prime contractor for the Corona system of spy satellites that greatly expanded the CIA’s abilities to photograph secret military installations from space. IBM, Cray Computers and other companies built the super-computers that allowed the NSA to sift through data from millions of telephone calls, and analyze them for intelligence that was passed on to national leaders.

Spending on contracts has increased exponentially in recent years along with intelligence budgets, and the NSA, the NGA and other agencies have turned to the private sector for the latest computer and communications technologies and for intelligence analysts. For example, today about half of staff at the NSA and NGA are private contractors. At the DIA, 70 percent of the workers are contractors. But the most privatized agency of all is the NRO, where a whopping 90 percent of the workforce receive paychecks from corporations. All told the U.S. intelligence agencies spend some 70 percent of their estimated $60 billion annual budget on contracts with private companies, according to documents this reporter obtained in June 2007 from the ODNI.

The plans to increase domestic spying are estimated to be worth billions of dollars in new business for the intelligence contractors. The market potential was on display in October at GEOINT 2007, the annual conference sponsored by the U.S. Geospatial Intelligence Foundation (USGIF), a non-profit organization funded by the largest contractors for the NGA. During the conference, which took place in October at the spacious Henry B. Gonzalez Convention Center in downtown San Antonio, many companies were displaying spying and surveillance tools that had been used in Afghanistan and Iraq and were now being re-branded for potential domestic use.

BAE Systems Inc.

On the first day of the conference, three employees of BAE Systems Inc. who had just returned from a three-week tour of Iraq and Afghanistan with the NGA demonstrated a new software package called SOCET GXP. (BAE Systems Inc. is the U.S. subsidiary of the UK-based BAE, the third-largest military contractor in the world.)

GXP uses Google Earth software as a basis for creating three-dimensional maps that U.S. commanders and soldiers use to conduct intelligence and reconnaissance missions. Eric Bruce, one of the BAE employees back from the Middle East, said his team trained U.S. forces to use the GXP software “to study routes for known terrorist sites” as well as to locate opium fields. “Terrorists use opium to fund their war,” he said. Bruce also said his team received help from Iraqi citizens in locating targets. “Many of the locals can’t read maps, so they tell the analysts, ‘there is a mosque next to a hill,’” he explained.

Bruce said BAE’s new package is designed for defense forces and intelligence agencies, but can also be used for homeland security and by highway departments and airports. Earlier versions of the software were sold to the U.S. Army’s Topographic Engineering Center, where it has been used to collect data on more than 12,000 square kilometers of Iraq, primarily in urban centers and over supply routes.

Another new BAE tool displayed in San Antonio was a program called GOSHAWK, which stands for “Geospatial Operations for a Secure Homeland – Awareness, Workflow, Knowledge.” It was pitched by BAE as a tool to help law enforcement and state and local emergency agencies prepare for, and respond to, “natural disasters and terrorist and criminal incidents.” Under the GOSHAWK program, BAE supplies “agencies and corporations” with data providers and information technology specialists “capable of turning geospatial information into the knowledge needed for quick decisions.” A typical operation might involve acquiring data from satellites, aircraft and sensors in ground vehicles, and integrating those data to support an emergency or security operations center. One of the program’s special attributes, the company says, is its ability to “differentiate levels of classification,” meaning that it can deduce when data are classified and meant only for use by analysts with security clearances.

These two products were just a sampling of what BAE, a major player in the U.S. intelligence market, had to offer. BAE’s services to U.S. intelligence -- including the CIA and the National Counter-Terrorism Center -- are provided through a special unit called the Global Analysis Business Unit. It is located in McLean, Virginia, a stone’s throw from the CIA. The unit is headed by John Gannon, a 25-year veteran of the CIA who reached the agency’s highest analytical ranks as deputy director of intelligence and chairman of the National Intelligence Council. Today, as a private sector contractor for the intelligence community, Gannon manages a staff of more than 800 analysts with security clearances.

A brochure for the Global Analysis unit distributed at GEOINT 2007 explains BAE’s role and, in the process, underscores the degree of outsourcing in U.S. intelligence. “The demand for experienced, skilled, and cleared analysts – and for the best systems to manage them – has never been greater across the Intelligence and Defense Communities, in the field and among federal, state, and local agencies responsible for national and homeland security,” BAE says. The mission of the Global Analysis unit, it says, “is to provide policymakers, warfighters, and law enforcement officials with analysts to help them understand the complex intelligence threats they face, and work force management programs to improve the skills and expertise of analysts.”

At the bottom of the brochure is a series of photographs illustrating BAE’s broad reach: a group of analysts monitoring a bank of computers; three employees studying a map of Europe, the Middle East and the Horn of Africa; the outlines of two related social networks that have been mapped out to show how their members are linked; a bearded man, apparently from the Middle East and presumably a terrorist; the fiery image of a car bomb after it exploded in Iraq; and four white radar domes (known as radomes) of the type used by the NSA to monitor global communications from dozens of bases and facilities around the world.

The brochure may look and sound like typical corporate public relations. But amid BAE’s spy talk were two phrases strategically placed by the company to alert intelligence officials that BAE has an active presence inside the U.S.. The tip-off words were “federal, state and local agencies,” “law enforcement officials” and “homeland security.” By including them, BAE was broadcasting that it is not simply a contractor for agencies involved in foreign intelligence, but has an active presence as a supplier to domestic security agencies, a category that includes the Department of Homeland Security (DHS), the FBI as well as local and state police forces stretching from Maine to Hawaii.

ManTech, Boeing, Harris and L-3

ManTech International, an important NSA contractor based in Fairfax, Virginia, has perfected the art of creating multi-agency software programs for both foreign and domestic intelligence. After the September 11th, 2001 attacks, it developed a classified program for the Defense Intelligence Agency called the Joint Regional Information Exchange System. DIA used it to combine classified and unclassified intelligence on terrorist threats on a single desktop. ManTech then tweaked that software for the Department of Homeland Security and sold it to DHS for its Homeland Security Information Network. According to literature ManTech distributed at GEOINT, that software will “significantly strengthen the exchange of real-time threat information used to combat terrorism.” ManTech, the brochure added, “also provides extensive, advanced information technology support to the National Security Agency” and other agencies.

In a nearby booth, Chicago, Illinois-based Boeing, the world’s second largest defense contractor, was displaying its “information sharing environment” software, which is designed to meet the Office of the Director of National Intelligence’s new requirements on agencies to stop buying “stovepiped” systems that can’t talk to each other. The ODNI wants to focus on products that will allow the NGA and other agencies to easily share their classified imagery with the CIA and other sectors of the community. “To ensure freedom in the world, the United States continues to address the challenges introduced by terrorism,” a Boeing handout said. Its new software, the company said, will allow information to be “shared efficiently and uninterrupted across intelligence agencies, first responders, military and world allies.” Boeing has a reason for publishing boastful material like this: In 2005, it lost a major contract with the NRO to build a new generation of imaging satellites after ringing up billions of dollars in cost-overruns. The New York Times recently called the Boeing project “the most spectacular and expensive failure in the 50-year history of American spy satellite projects.”

Boeing’s geospatial intelligence offerings are provided through its Space and Intelligence Systems unit, which also holds contracts with the NSA. It allows agencies and military units to map global shorelines and create detailed maps of cities and battlefields, complete with digital elevation data that allow users to construct three-dimensional maps. (In an intriguing aside, one Boeing intelligence brochure lists among its “specialized organizations” Jeppesen Government and Military Services. According to a 2006 account by New Yorker reporter Jane Mayer, Jeppesen provided logistical and navigational assistance, including flight plans and clearance to fly over other countries, to the CIA for its “extraordinary rendition” program.)

Although less known as an intelligence contractor than BAE and Boeing, the Harris Corporation has become a major force in providing contracted electronic, satellite and information technology services to the intelligence community, including the NSA and the NRO. In 2007, according to its most recent annual report, the $4.2 billion company, based in Melbourne, Florida, won several new classified contracts. NSA awarded one of them for software to be used by NSA analysts in the agency’s “Rapidly Deployable Integrated Command and Control System,” which is used by the NSA to transmit “actionable intelligence” to soldiers and commanders in the field. Harris also supplies geospatial and imagery products to the NGA. At GEOINT, Harris displayed a new product that allows agencies to analyze live video and audio data imported from UAVs. It was developed, said Fred Poole, a Harris market development manager, “with input from intelligence analysts who were looking for a video and audio analysis tool that would allow them to perform ‘intelligence fusion’” -- combining information from several agencies into a single picture of an ongoing operation.

For many of the contractors at GEOINT, the highlight of the symposium was an “interoperability demonstration” that allowed vendors to show how their products would work in a domestic crisis.

One scenario involved Cuba as a rogue nation supplying spent nuclear fuel to terrorists bent on creating havoc in the U.S.. Implausible as it was, the plot, which involved maritime transportation and ports, allowed the companies to display software that was likely already in use by the Department of Homeland Security and Naval Intelligence. The “plot” involved the discovery by U.S. intelligence of a Cuban ship carrying spent nuclear fuel heading for the U.S. Gulf Coast; an analysis of the social networks of Cuban officials involved with the illicit cargo; and the tracking and interception of the cargo as it departed from Cuba and moved across the Caribbean to Corpus Christi, Texas, a major port on the Gulf Coast. The agencies involved included the NGA, the NSA, Naval Intelligence and the Marines, and some of the key contractors working for those agencies. It illustrated how sophisticated the U.S. domestic surveillance system has become in the six years since the 9/11 attacks.

L-3 Communications, which is based in New York city, was a natural for the exercise: As mentioned earlier, retired Army Lt. General Patrick M. Hughes, its vice president of homeland security, was a member of the Booz Allen Hamilton study group that advised the Bush administration to expand the domestic use of military spy satellites. At GEOINT, L-3 displayed a new program called “multi-INT visualization environment” that combines imagery and signals intelligence data that can be laid over photographs and maps. One example shown during the interoperability demonstration showed how such data would be incorporated into a map of Florida and the waters surrounding Cuba. With L-3 a major player at the NSA, this demonstration software is likely seeing much use as the NSA and the NGA expand their information-sharing relationship.

Over the past two years, for example, the NGA has deployed dozens of employees and contractors to Iraq to support the “surge” of U.S. troops. The NGA teams provide imagery and full-motion video -- much of it beamed to the ground from Unmanned Aerial Vehicles (UAV) -- that help U.S. commanders and soldiers track and destroy insurgents fighting the U.S. occupation. And since 2004, under a memorandum of understanding with the NSA, the NGA has begun to incorporate signals intelligence into its imagery products. The blending technique allows U.S. military units to track and find targets by picking up signals from their cell phones, follow the suspects in real-time using overhead video, and direct fighter planes and artillery units to the exact location of the targets -- and blow them to smithereens.

That’s exactly how U.S. Special Forces tracked and killed Abu Musab al-Zarqawi, the alleged leader of Al Qaeda in Iraq, the NGA’s director, Navy Vice Admiral Robert Murrett, said in 2006. Later, Murrett told reporters during GEOINT 2007, the NSA and the NGA have cooperated in similar fashion in several other fronts of the “war on terror,” including in the Horn of Africa, where the U.S. military has attacked Al Qaeda units in Somalia, and in the Philippines, where U.S. forces are helping the government put down the Muslim insurgent group Abu Sayyaf. “When the NGA and the NSA work together, one plus one equals five,” said Murrett.

Civil Liberty Worries

For U.S. citizens, however, the combination of NGA imagery and NSA signals intelligence in a domestic situation could threaten important constitutional safeguards against unwarranted searches and seizures. Kate Martin, the director of the Center for National Security Studies, a nonprofit advocacy organization, has likened the NAO plan to “Big Brother in the Sky.” The Bush administration, she told the Washington Post, is “laying the bricks one at a time for a police state.”

Some Congress members, too, are concerned. “The enormity of the NAO’s capabilities and the intended use of the imagery received through these satellites for domestic homeland security purposes, and the unintended consequences that may arise, have heightened concerns among the general public, including reputable civil rights and civil liberties organizations,” Bennie G. Thompson, a Democratic member of Congress from Mississippi and the chairman of the House Homeland Security Committee, wrote in a September letter to Secretary of Homeland Security Michael Chertoff. Thompson and other lawmakers reacted with anger after reports of the NAO and the domestic spying plan were first revealed by the Wall Street Journal in August. “There was no briefing, no hearing, and no phone call from anyone on your staff to any member of this committee of why, how, or when satellite imagery would be shared with police and sheriffs’ officers nationwide,” Thompson complained to Chertoff.

At a hastily organized hearing in September, Thompson and others demanded that the opening of the NAO be delayed until further studies were conducted on its legal basis and questions about civil liberties were answered. They also demanded biweekly updates from Chertoff on the activities and progress of the new organization. Others pointed out the potential danger of allowing U.S. military satellites to be used domestically. “It will terrify you if you really understand the capabilities of satellites,” warned Jane Harman, a Democratic member of Congress from California, who represents a coastal area of Los Angeles where many of the nation’s satellites are built. As Harman well knows, military spy satellites are far more flexible, offer greater resolution, and have considerably more power to observe human activity than commercial satellites. “Even if this program is well-designed and executed, someone somewhere else could hijack it,” Harman said during the hearing.

The NAO was supposed to open for business on October 1, 2007. But the Congressional complaints have led the ODNI and DHS to delay their plans. The NAO "has no intention to begin operations until we address your questions," Charles Allen of DHS explained in a letter to Thompson. In an address at the GEOINT conference in San Antonio, Allen said that the ODNI is working with DHS and the Departments of Justice and Interior to draft the charter for the new organization, which he said will face “layers of review” once it is established.

Yet, given the Bush administration’s record of using U.S. intelligence agencies to spy on U.S. citizens, it is difficult to take such promises at face value. Moreover, the extensive corporate role in foreign and domestic intelligence means that the private sector has a great deal to gain in the new plan for intelligence-sharing. Because most private contracts with intelligence agencies are classified, however, the public will have little knowledge of this role. Before Congress signs off on the NAO, it should create a better oversight system that would allow the House of Representatives and the Senate to monitor the new organization and to examine how BAE, Boeing, Harris and its fellow corporations stand to profit from this unprecedented expansion of America’s domestic intelligence system.

Tim ­Shorrock has been writing about U.S. foreign policy and national security for nearly 30 years. His book, Spies for Hire: The Secret World of Outsourced Intelligence, will be published in May 2008 by Simon & Schuster. He can be reached at

American Dream wrote:

Big Increases for Intelligence and Pentagon "Black" Programs in 2010

Continuing along the dark path marked out by his predecessors in the Oval Office, President Barack Obama's Defense and Intelligence budget for Fiscal Year 2010 will greatly expand the reach of unaccountable agencies--and the corporate grifters whom they serve.

According to Aviation Week, "the Pentagon's 'black' operations, including the intelligence budgets nested inside it, are roughly equal in magnitude to the entire defense budgets of the UK, France or Japan, and 10 per cent of the total."

Yes, you read that correctly. The "black" or secret portions of the budget are almost as large as the entire defense outlays of America's allies, hardly slouches when it comes to feeding their own militarist beasts. The U.S. Air Force alone intends to spend approximately $12 billion on "black" programs in 2010 or 36 percent of its entire research and development budget. Aviation Week reveals:

Black-world procurement remains dominated by the single line item that used to be called "Selected Activities," resident in the USAF's "other procurement" section. This year's number stands just above $16 billion. In inflation-adjusted terms, that's 240 per cent more than it was ten years ago.

On the operations side, secret spending has risen 8 per cent over last year, to just over $15 billion--equivalent to more than a third of Air Force operating costs.

What does it all go for? In simple terms, we don't know. It is apparent that much if not all of the intelligence community is funded through the black budget: for example, an $850 million USAF line item is clearly linked to reconnaissance satellites. But even so, the numbers are startling--and get more so year by year.
(Bill Sweetman, "Black budget blows by $50 billion mark," Aviation Week, May 7, 2009)

How's that for change! The Register gives a break down of the numbers for added emphasis:

1) Mainstream US armed forces $490bn-odd
2) UK armed forces $60bn
3) Chinese armed forces $58bn
4) French armed forces $54bn
5) "Black" US forces $50bn+
6) Japanese Self-Defence forces $44bn

While the American government refuses to disclose the CIA or NSA's budget, "both the Agency and other non-military spooks do get money of their own. Some of this is spent on military or quasi-military activities," The Register reports.

Toss in the world-wide deployment of CIA and U.S. Special Operations Command (USSOCOM) paramilitary operatives hidden among a welter of Special Access Programs (SAPs) classified above top secret and pretty soon we're talking real money!

One such program may have been Dick Cheney's "executive assassination ring" disclosed by investigative journalist Seymour Hersh during a "Great Conversations" event at the University of Minnesota in March.

And should pesky investigators from the Government Accountability Office (GAO) have the temerity to probe said "executive assassination ring," or other DoD "black" programs well, their Inspector General's had better think again!

According to the whistleblowing security and intelligence website Cryptome, a May 8, 2009 letter from Susan Ragland, GAO Director of Financial Management and Assurance to Diane Watson (D-CA), Chairwoman of the House Committee on Government Management, Organization and Procurement, lays down the law in no uncertain terms to Congress.

Ms. Ragland wrote: "the IG Act authorizes the heads of six agencies to prohibit their respective IGs from carrying out or completing an audit or investigation, or from issuing any subpoena if the head determines that such prohibition is necessary to prevent either the disclosure of certain sensitive information or significant harm to certain national interests."

Neat, isn't it! Under statutory authority granted the Executive Branch by congressional grifters, Congress amended the IG Act "to establish the Department of Defense (DOD) IG and placed the IG under the authority, direction, and control of the Secretary of Defense with respect to audits or investigations or the issuance of subpoenas that require access to certain information."

What information may be withheld from public scrutiny? Ms. Ragland informs us: "Specifically, the Secretary of Defense may prohibit the DOD IG from initiating, carrying out, or completing such audits or investigations or from issuing a subpoena if the Secretary determines that the prohibition is necessary to preserve the national security interests of the United States." (emphasis added)

The same restrictions to the IG Act that apply to the Defense Department are similarly operative for the Departments of the Treasury, Homeland Security, Justice, the U.S. Postal Service (!), the Federal Reserve Board, and the Central Intelligence Agency. Talk about veritable mountains of dirty laundry--and "black" programs--that can be hidden here!

Space-Based Spies

Among the items nestled within the dark arms of Pentagon war planners is a program called "Imagery Satellite Way Ahead," a joint effort between "the Office of the Director of National Intelligence and the Department of Defense designed to revamp the nation's constellation of spy satellites," Congressional Quarterly reports.

As Antifascist Calling revealed in several investigative pieces in June, October and November 2008, America's fleet of military spy satellites are flown by the secretive National Reconnaissance Office (NRO).

According to the agency's own description, "The NRO is a joint organization engaged in the research and development, acquisition, launch and operation of overhead reconnaissance systems necessary to meet the needs of the Intelligence Community and of the Department of Defense. The NRO conducts other activities as directed by the Secretary of Defense and/or the Director of National Intelligence."

As investigative journalist Tim Shorrock revealed in his essential book, Spies for Hire, some ninety-five percent of NRO employees are contractors working for defense and security firms. Indeed, as Shorrock disclosed, "with an estimated $8 billion annual budget, the largest in the IC, contractors control about $7 billion worth of business at the NRO, giving the spy satellite industry the distinction of being the most privatized part of the Intelligence Community."

While the Office's website is short on information, some of the "other activities" alluded to by NRO spooks include the Department of Homeland Security's National Applications Office (NAO).

As I wrote in October, the NAO will coordinate how domestic law enforcement and "disaster relief" agencies such as FEMA use satellite imagery (IMINT) generated by spy satellites. But based on the available evidence, hard to come by since these programs are classified above top secret, the technological power of these military assets are truly terrifying--and toxic for a democracy.

DHS describes the National Applications Office as "the executive agent to facilitate the use of intelligence community technological assets for civil, homeland security and law enforcement purposes." As Congressional Quarterly reveals, the "classified plan would include new, redesigned 'electro-optical' satellites, which collect data from across the electromagnetic spectrum, as well as the expanded use of commercial satellite imagery. Although the cost is secret, most estimates place it in the multibillion-dollar range."

How these redesigned assets will be deployed hasn't been announced. The more pertinent issue is whether or not DHS, reputedly a civilian agency but one which answers to the militarized Office of the Director of National Intelligence (ODNI), will position these assets to illegally spy on Americans. The available evidence is they will.

DHS avers that "homeland security and law enforcement will also benefit from access to Intelligence Community capabilities." With Pentagon "black" programs already costing taxpayers tens of billions of dollars the question remains, with NAO as the "principal interface" between American spooks, DHS bureaucrats and law enforcement, who will oversee NAO's "more robust access to needed remote sensing information to appropriate customers"?

Certainly not Congress. Investigative journalist Siobhan Gorman writing in The Wall Street Journal documented last year, that despite a highly-critical June 2008 study by the Congressional Research Service (CRS), Congress partially-funded the program "in a little debated $634 billion spending measure."

Indeed, a fully-operational NAO now provides federal, state and local officials "with extensive access to spy-satellite imagery--but no eavesdropping--to assist with emergency response and other domestic-security needs, such as identifying where ports or border areas are vulnerable to terrorism." But as CRS investigators wrote:

Members of Congress and outside groups have raised concerns that using satellites for law enforcement purposes may infringe on the privacy and Fourth Amendment rights of U.S. persons. Other commentators have questioned whether the proposed surveillance will violate the Posse Comitatus Act or other restrictions on military involvement in civilian law enforcement, or would otherwise exceed the statutory mandates of the agencies involved. (Richard A. Best Jr. and Jennifer K. Elsea, "Satellite Surveillance: Domestic Issues," Congressional Research Service, June 27, 2008)

While these serious civil liberties' issues have apparently been swept under the carpet, huge funding outlays by Congress for Pentagon's "black" budget operations indicate that President Obama's promises of "change" in how "government does business" is so much hot-air meant to placate the rubes.

Driven by a Corporatist Agenda

Wholesale spying by the American government on its citizens as numerous investigators have uncovered, is aided and abetted by a host of well-heeled corporate grifters in the defense, intelligence and security industries. These powerful, and influential, private players in the Military-Industrial-Security Complex are largely unaccountable; it can be said that America's intelligence and security needs are driven by firms that benefit directly from the Pentagon's penchant for secrecy.

Federal Computer Week reported in April that the program to revamp America's spy satellites "has the backing of the Obama administration, and the program is expected to win congressional approval, according to a senior intelligence official."

The same anonymous "senior official" told the publication, "given the backing of the Defense Department, ODNI and the Obama administration, lawmakers are expected to approve the plan." And as with other "black" programs, the cost is classified but is expected to run into the billions; a veritable windfall for enterprising defense corporations.

The electro-optical satellite modernization program involves building new satellites that the National Reconnaissance Office (NRO) would operate and expanding the use of imagery from commercial providers, according to a statement the Office of the Director of National Intelligence released April 7. Under the plan, the National Geospatial-Intelligence Agency would continue to integrate imagery products for government customers. (Ben Bain, "Spy satellite tally could increase," Federal Computer Week, April 8, 2009)

While no decision has been reached on the "acquisition approach for the program," ODNI and NRO "would oversee the acquisition strategy for the new government-built satellites and a contract would likely be awarded within months."

In a toss-off statement to justify the enormous outlay of taxpayer dollars for the new initiative, Obama's Director of National Intelligence, Dennis Blair, said last month, "When it comes to supporting our military forces and the safety of Americans, we cannot afford any gaps in collection." Or perhaps "any gaps in collection" on Americans. As Tim Shorrock revealed,

The plans to increase domestic spying are estimated to be worth billions of dollars in new business for the intelligence contractors. The market potential was on display in October at GEOINT 2007, the annual conference sponsored by the U.S. Geospatial Intelligence Foundation (USGIF), a non-profit organization funded by the largest contractors for the NGA. During the conference, which took place in October at the spacious Henry B. Gonzalez Convention Center in downtown San Antonio, many companies were displaying spying and surveillance tools that had been used in Afghanistan and Iraq and were now being re-branded for potential domestic use. ("Domestic Spying, Inc.," CorpWatch, November 27, 2007)

Indeed, according to Shorrock when the NAO program was conceived in 2005, former ODNI director Michael McConnell "turned to Booz Allen Hamilton of McLean, Virginia--one of the largest contractors in the spy business. The company was tasked with studying how intelligence from spy satellites and photoreconnaissance planes could be better used domestically to track potential threats to security within the U.S."

Tellingly, McConnell was a senior vice president with the spooky firm for a decade. Booz Allen Hamilton was acquired by the private equity firm The Carlyle Group in a 2008 deal worth $2.54 billion. In addition to Booz Allen Hamilton, other giant defense and security corporations involved in running Homeland Security's National Applications Office include the scandal-tainted British firm BAE Systems, ManTech, Boeing and L-3 Communications.

Among the firms in the running to land ODNI/NRO new spy satellite contracts are: BAE, Boeing, Lockheed Martin and Northrop Grumman. All of these corporations according to the Project on Government Oversight's (POGO) Federal Contractor Mismanagement Database (FCMD) have "histories of misconduct such as contract fraud and environmental, ethics, and labor violations."

Unsurprisingly, Lockheed Martin, Boeing, BAE and Northrop Grumman lead the pack in "total instances of misconduct" as well as fines levied by the federal government for abusive practices and outright fraud.


Unaccountable federal agencies and corporations will continue the capitalist "security" grift, particularly when it comes to "black" programs run by the Department of Defense and the Office of the Director of National Intelligence. Despite a documented history of serious ethical and constitutional breeches, these programs will persist and expand well into the future. While the Obama administration has said it favors government transparency, it has continued to employ the opaque methods of its predecessors.

From the use of the state secrets privilege to conceal driftnet surveillance of Americans, to its refusal to launch an investigation--and prosecution--of Bush regime torture enablers and war criminals, the "change" administration instead, has delivered "more of the same."
User avatar
Wombaticus Rex
Posts: 10614
Joined: Wed Nov 08, 2006 6:33 pm
Location: Vermontistan
Blog: View Blog (0)

Re: Data Mining & Intelligence Agencies

Postby Wombaticus Rex » Wed Apr 04, 2012 11:54 pm

Data mining project benefits investigators, scares privacy experts
By Susan Taylor Martin, Times Senior Correspondent
In Print: Sunday, January 10, 2010

Data-mining whiz Hank Asher, who has a private jet and a $3 million mansion, rents part of the Boca Raton office park where IBM once made personal computers.

BOCA RATON — At any one time, some 750,000 pedophiles are prowling the Internet, the United Nations says. They might be lurking in chat rooms. Or swapping images of adults having sex with kids.

It's a virtual epidemic of child pornography, and to fight it, law enforcement officers from all over are converging on a cavernous building in South Florida. Here they have access to the most advanced technology for finding pedophiles.

But this isn't run by any government agency. The desks, computers, technology — all are provided free by a former drug smuggler named Hank Asher.

Called a "mad scientist'' by one employee, Asher has made a fortune collecting public records — deeds, lawsuits, voter registrations — and combining them into databases that can be invaluable in locating people. Plug a name into Accurint, Asher's best-known product, and you'll see addresses, possible relatives, licenses held.

It was Asher's technology that helped police find the Washington, D.C., snipers.

Now he is building a super computer and a database "a thousand times more powerful" than anything he has developed yet.

It's a project that worries privacy-rights advocates and other critics. They wonder if Asher's real reason for donating some of his technology to government agencies is to get access to confidential data like firearms registries, tax information, even health records — information that could be a boon to businesses and an unprecedented intrusion into the lives of millions of Americans.

"He wants to have every scrap of personal data that he can acquire on any and everybody,'' says Marion Hammer, a past president of the National Rifle Association. "I know that he has people working to find ways to get data from state agencies and of course there is data that we would never want him to get his hands on.''

Fueling speculation about Asher's motives are his controversial past and the fact he has hired many well-connected individuals. Among them: Bob Butterworth, former head of the Florida Department of Children and Families.

Asher acknowledges that his new database product could earn billions of dollars for his Boca Raton company. But he says he'll continue to provide his predator-tracking technology free to police and the National Center for Missing and Exploited Children.

"No matter what you think about him, he has a great record of philanthropy,'' says John Walsh, who helped start the center after his son Adam was murdered in Hollywood, Fla., in 1981.

Gerald Bailey, head of the Florida Department of Law Enforcement, applauds Asher's efforts on behalf of kids. But FDLE declined his offer of free space and his request that it assign someone to help develop better ways of finding sexual predators.

"I could never get a handle on exactly what the product was or what they wanted from us,'' Bailey said. "I did not want to be obligated to follow through with something when we're not sure where the end is going to be.''

Cocaine to computers

At 58, the thrice-divorced Asher is a long way from the "simple Indiana farm boy'' who moved to Florida and made his first fortune painting Gold Coast high-rises. He bought a house in the Bahamas and, as he has admitted, piloted several cocaine flights in 1980 and '81. (He was not prosecuted, and later cooperated with drug enforcement authorities.)

In the late '80s, Asher began dabbling with computers and learned to combine databases. He discovered he could buy databases not only from public sources like state motor vehicle bureaus, but also banks and other businesses whose databases contained Social Security numbers and other information not generally open to the public.

Asher's 1992 breakthrough was collating this wealth of data into an easily searchable product he dubbed AutoTrack. It proved a boon to police, who previously had to search many sources in doing background checks.

Even Asher was surprised by AutoTrack. Searching on his own name, he got a long list of "associated'' people, including "my ex-wife and her newest victim. I thought, 'What have I done?' ''

Forgoing what he says could have been millions in profits, he limited sales of AutoTrack to what he considered legitimate users like reporters and insurance investigators. The father of two girls, he also offered AutoTrack for free to the National Center for Missing Children.

"I think he shared my frustration that law enforcement agencies don't work together and don't exchange information,'' Walsh says. "I thought, 'Why not give it a go?' It turned out to be a huge boon to (the center) and America's Most Wanted,'' the TV show hosted by Walsh.

In 1999, Asher's drug-smuggling days returned to haunt him. The FBI and Drug Enforcement Administration suspended their use of AutoTrak, worried that his company, DBT, could potentially monitor activities in ongoing cases.

There was no evidence any cases had been compromised, but DBT's directors forced Asher to sell his ownership stake (he walked away with at least $117 million) in order to save the law enforcement contracts.

Asher already had started another company and was developing Accurint, faster and more comprehensive than AutoTrack.

Then came Sept. 11.

Tracking terrorists

Acting on his own a few days after the attacks, Asher wrote a computer program so powerful it culled through data on hundreds of millions of people and flagged 419 possible suspects.

The hijackers' names were not yet public, but the program hit on one: Marwan al-Shehhi, a 9/11 pilot.

For months, Asher was stymied in efforts to demonstrate his program to top Bush administration officials. So in December 2002, court records show, he paid $2 million to Republican Rudy Giuliani, the former New York mayor turned lobbyist. He also gave a $15,000 watch to the wife of a California sheriff serving on a federal homeland security panel.

Barely a month later, with Florida Gov. Jeb Bush introducing him, Asher made his case to Vice President Dick Cheney and Homeland Security director Tom Ridge. Soon afterward, Ridge's office authorized a $12-million pilot project.

MATRIX — for Multistate Anti-Terrorism Information Exchange — was controversial from the start. Several states dropped out, concerned about costs and violation of privacy laws. The American Civil Liberties Union warned that people could be wrongly labeled as potential terrorists.

"There is a lot of scientific evidence that you cannot predict the actions of terrorists or criminals or anyone based on their computer profiles,'' says Chris Calabrese of the ACLU's liberty and technology project. "That's a very dangerous thing that could cause people a lot of harm.''

In 2005, the federal money ran out and the project ended.

"The existence of MATRIX should have been kept a national secret,'' Asher says.

'It's quite scary'

Frustrated by the flap over MATRIX, Asher sold his company in 2004 for $260 million. He helped care for his dying sister and established a cancer research center in conjunction with the Mayo Clinic.

Asher also started another company, TLO, and offered to develop, for free, a means of tracking children in state care. In 2001, 5-year-old Rilya Wilson had disappeared from a Miami foster home. She was missing for 18 months before anyone in the Department of Children and Families realized she was gone.

The idea for a tracking system was pushed from two main quarters: the National Center for Missing Children, of which Asher is a board member and major donor; and DCF, whose secretary, Butterworth, resigned in August 2008 and went to work for Asher.

Two months later, Butterworth's successor, George Sheldon, and other top department officials visited Asher's Boca Raton headquarters to hear what he had to offer.

Sheldon says Asher came across "like a really bright guy'' and demonstrated his search capabilities by pulling up reams of personal data on one member of the DCF group.

"I don't think the average citizen realizes how much information is out there,'' Sheldon says. "The collection of information about individuals can be a slippery slope, and I think government always has to be careful about that.''

Did Asher ask for any data from DCF in exchange for designing a free system?

"I don't think we got that far,'' says Sheldon, who adds that the agency decided to work with a company already doing business with the state.

Soon after that visit, though, several DCF employees met in Orlando with representatives of Asher's company, FDLE and other agencies. The main topic for discussion: a three-page list of databases that the National Center for Missing Children said could be helpful in finding child predators.

Among the databases: financial records, credit card data, even Blockbuster accounts.

From a privacy standpoint, ''it's a lot and it's quite scary,'' says Lee Tien, an attorney with the San Francisco-based Electronic Frontier Foundation.

Butterworth, who helped organize the meeting, insists it was only a "wish list'' and that no one expected to get access to so much confidential data. But he understands why people were spooked.

"Our biggest mistake was that we should have looked at privacy first,'' Butterworth says. "Then we would not have the problem we have now. If we could take that meeting back, things would be entirely different.''

Asher's company tried to organize another "brainstorming" session last year, and urged FDLE to send a representative. But the meeting never got off the ground: An e-mail obtained by the St. Petersburg Times reflects the skepticism of FDLE Commissioner Bailey.

"Bottom line,'' he wrote to his chief deputy, "this is a vendor asking us to help develop a product.''

Grateful police

One of Asher's current "passions,'' as he puts it, is combatting pedophilia.

Canadian researchers say as many as 4 percent of adults are sexually attracted to children. Most never physically act on their fantasies, but they have unprecedented access to pornographic photos, videos, even children themselves.

"The problems of pedophilia have exploded because of the Internet,'' Asher says. "It's an epidemic.''

Since 2008, his company has leased 143,000 square feet in the Boca Raton complex where IBM once made personal computers. Asher has set aside part of that space for investigators from the Palm Beach County State Attorney's Office and other agencies to work on sex crimes against kids.

By law, Internet service providers are required to inform the National Center for Missing Children of any suspected child pornography detected on their networks. Those tips are turned over to law enforcement agencies, many of which use technology developed by Asher and his employees.

"We get requests from all over the world,'' says Flint Waters, a former Wyoming detective. "In 33 countries there are investigators who have access to this (technology). If they work here, they have the newest tools. If they have a need for forensic tools that our tools don't accomplish, we'll write it in a day or two.''

Waters and Asher say those "tools'' have led to thousands of arrests. Where investigators once had to send computers to a lab to see if they contained pornographic images — a process that could take six months, while a predator continued abusing children — "now they pop a disc in and it shows if there's child porn,'' Waters says.

Another tool lets investigators zero in on certain frames without watching all 20 or 30 minutes of a pornographic video.

Security in the law enforcement area appears tight, with surveillance cameras and restricted entry. Though his company provides the phones, desks and computers, Asher says he has no access to data on the hard drives or the server, which is owned by Palm Beach County.

"If the time ever comes that Hank profits from the cops, I hope they rain on him,'' Waters says. ''The commitment he's made to me is that he gives those tools away (to police).''

Commitments aside, there are no contracts with any of the agencies that have people working here, and the center seems to operate on a casual basis.

"I think it's 'at will.' They can kick us out if they want to,'' acknowledges Palm Beach County State Attorney Michael F. McAuliffe.

Still, he says, the free space and access to technology have been good for both taxpayers and children. The county's predator unit has made more than 30 arrests since setting up shop in May.

FDLE and other agencies, however, have declined the no-rent offer.

Despite his crusade against cybercrimes involving kids, Florida Attorney General Bill McCollum did not want to be in a position "where there were any questions about whether the acceptance of that free gift would be appropriate,'' says spokeswoman Sandi Copes.

Tomorrow's risks

Asher says the new database "product'' he is developing — AK, for Accurint Killer — will revolutionize the businesses of assessing risk and investigating fraud. He plans to introduce AK in March at a conference of private investigators in Dallas, where he and John Walsh will be featured speakers.

Asher acknowledges that AK could be very profitable.

"Half-a-trillion dollars of fraud has come to us looking for remedies,'' he says. "I've never seen financial opportunities like this in my life. I think it will do several billion dollars a year.''

Could it also flag potential terrorists, like the Nigerian who got on a U.S.-bound flight Christmas Day despite many signs he was up to no good? Asher, whose MATRIX technology sparked such privacy concerns after 9/11, hints that it could.

"Our new systems have the capacity to address tomorrow's risks and threats. We have built the next generation of what I built before. It's going to be much less safe to be dangerous.''

Times researcher Carolyn Edds contributed to this report. Susan Martin can be contacted at

Fast Facts

Asher's all-stars

Hank Asher, a onetime drug smuggler who went on to create databases used by police and businesses to find people, has hired many well-connected individuals as he develops a new commercial product that he says will revolutionize the investigation and risk industries. Among his hires:

• Bob Butterworth, former Florida attorney general, former secretary of Florida Department of Children and Families.

• Michael Moore, former Mississippi attorney general (worked with Butterworth on landmark lawsuit against tobacco companies).

• Tim Moore, former commissioner, Florida Department of Law Enforcement.

• Martha Barnett, lawyer and past president of the American Bar Association.

• Ken Hunter, former associate U.S. postmaster general.

• Jon A. Sale, former assistant U.S. attorney.

• Dan MacLachlan, former FBI agent.

• Don Hunter, former Collier County sheriff.

• Bill Shrewsbury, former FDLE agent.

• Flint Waters, former Wyoming special agent who developed technology for tracking sex predators.

• Mark Lunsford, who has lobbied for "Jessica's Laws'' since the 2005 murder of his daughter in Citrus County. Asher pays him $104,000 a year.

• Ed Smart, whose daughter Elizabeth was kidnapped in 2002 and held nine months.

[Last modified: Jan 09, 2010 08:09 PM] ... 064010.ece

User avatar
Wombaticus Rex
Posts: 10614
Joined: Wed Nov 08, 2006 6:33 pm
Location: Vermontistan
Blog: View Blog (0)

Re: Data Mining & Intelligence Agencies

Postby Wombaticus Rex » Wed Apr 04, 2012 11:55 pm

Data-mining: terrorism prevention or social control?
Submitted by sosadmin on Thu, 12/22/2011 - 12:11

You may or may not have heard of the CIA's favorite data analysis company, Palantir, which currently operates out of Facebook's old offices in Palo Alto, California. But you likely have heard something about data mining software more generally; it's supposed to be the silver bullet that solves the data-flood problem for the world's spy agencies, which can't seem to know enough about our every movement, thought, purchase, communication, etc.

Software like Palantir is meant to make sense out of the mass of swirling data that clogs databases at the FBI, CIA, DOD, NYPD, LAPD, and increasingly state and local police fusion centers. Those databases contain intimate information about all of us --- and yet the vast majority of us aren't plotting violent schemes, but simply going about our quotidian, daily lives. Palantir and like-programs, the story goes, solve the drowning-in-data problem by "connecting the dots," piecing together seemingly unrelated data points to help intelligence and law enforcement agents distinguish between those people who are planning to bomb something, and those who are not. Palantir does something besides highlight the supposedly dangerous among us, however. As Businessweek reports in a lengthy piece on the company:

An organization like the CIA or FBI can have thousands of different databases, each with its own quirks: financial records, DNA samples, sound samples, video clips, maps, floor plans, human intelligence reports from all over the world. Gluing all that into a coherent whole can take years. Even if that system comes together, it will struggle to handle different types of data—sales records on a spreadsheet, say, plus video surveillance images. What Palantir (pronounced Pal-an-TEER) does, says Avivah Litan, an analyst at Gartner (IT), is “make it really easy to mine these big data sets.” The company’s software pulls off one of the great computer science feats of the era: It combs through all available databases, identifying related pieces of information, and puts everything together in one place.

"Everything together in one place." Sounds creepy, right? It is. And contrary to claims made by Palantir, the CIA and even the Businessweek piece, it doesn't succeed in preventing terrorism. It can't, because data mining and data analysis programs rely on patterns of suspicious behavior in order to determine who is a 'risk'. But as a Homeland Security funded study showed in 2008, predictive terrorism modeling does not work. Why? There is no particular risk profile for people who are likely to commit heinous acts of violence. And furthermore, those people intent on doing real harm will go out of their way to study the latest law enforcement approach, and work diligently to get around it.

We've all heard the basic patterns to look out for: paying cash for one way plane tickets; young men traveling alone; buying large quantities of fertilizer far from a farm, etc. But the 9/11 attacks were so successful precisely because they were so unexpected. What makes the CIA think that the next round of spectacular attacks --- if indeed it comes --- will be anything like what it has seen before? In other words, how do you model for an infinite number of possible approaches?

You can't. So Palantir won't stop terrorism, full stop. But on the other hand, data-mining software like Palantir is very useful for maintaining social control over people who are not constantly trying to evade the surveillance state, who are simply going about their normal lives under it's ever-watchful eye.

The ways in which Palantir can be deployed as a tool for social control are seemingly limitless:

Using Palantir technology, the FBI can now instantly compile thorough dossiers on U.S. citizens, tying together surveillance video outside a drugstore with credit-card transactions, cell-phone call records, e-mails, airplane travel records, and Web search information.

If the police want to know what you are doing and where you are going, they can. But towards what end? Can they really discern from your captured images and web reading habits if you are a threat to society? Could Palantir have predicted and therefore stopped Jared Loughner from taking a gun to the shopping mall in Arizona and shooting it up, killing and injuring many? If so, why didn't it?

Even though DHS found in 2008 that data mining used to predict terrorism doesn't work and is too great an assault on personal privacy even if it did, there are even more basic questions we should ask ourselves before we consent to giving up our most basic rights to privacy and personal integrity to the state.

Foremost among these questions is: Can the government keep us safe from all harm at all times? Furthermore, do we want to live in a society wherein we give up all of our privacy, trading our sacred human dignity for (false) promises of personal safety? And if the true aim of the CIA's use of programs like Palantir is public safety, can the government use the technology to prevent car accidents and domestic homicides, which kill tens of thousands more Americans every year than terrorism?

The answers to these questions are obvious. It's time to say 'no' to the culture of fear that promotes the police state ideology.

Democracy has its risks; we either accept them, or we instead accept the rise of the creeping police state. We cannot have both.


Remarkable that Palantir emerged from PayPal's collection department.

The origins of Palantir go back to PayPal, the online payments pioneer founded in 1998. A hit with consumers and businesses, PayPal also attracted criminals who used the service for money laundering and fraud. By 2000, PayPal looked like “it was just going to go out of business” because of the cost of keeping up with the bad guys, says Peter Thiel, a PayPal co-founder.

The antifraud tools of the time could not keep up with the crooks. PayPal’s engineers would train computers to look out for suspicious transfers—a number of large transactions between U.S. and Russian accounts, for example—and then have human analysts review each flagged deal. But each time PayPal cottoned to a new ploy, the criminals changed tactics. The computers would miss these shifts, and the humans were overwhelmed by the explosion of transactions the company handled.

PayPal’s computer scientists set to work building a software system that would treat each transaction as part of a pattern rather than just an entry in a database. They devised ways to get information about a person’s computer, the other people he did business with, and how all this fit into the history of transactions. These techniques let human analysts see networks of suspicious accounts and pick up on patterns missed by the computers. PayPal could start freezing dodgy payments before they were processed. “It saved hundreds of millions of dollars,” says Bob McGrew, a former PayPal engineer and the current director of engineering at Palantir.

After EBay (EBAY) acquired PayPal in 2002, Thiel left to start a hedge fund, Clarium Capital Management. He and Joe Lonsdale, a Clarium executive who’d been a PayPal intern, decided to turn PayPal’s fraud detection into a business by building a data analysis system that married artificial intelligence software with human skills. Washington, they guessed, would be a natural place to begin selling such technology. “We were watching the government spend tens of billions on information systems that were just horrible,” Lonsdale says. “Silicon Valley had gotten to be a lot more advanced than government contractors, because the government doesn’t have access to the best engineers.”

via ... age_2.html
User avatar
Wombaticus Rex
Posts: 10614
Joined: Wed Nov 08, 2006 6:33 pm
Location: Vermontistan
Blog: View Blog (0)

Re: Data Mining & Intelligence Agencies

Postby Wombaticus Rex » Thu Apr 05, 2012 11:54 pm hugely important. Tim brings so many threads together here it made my head hurt when I first read this.


The Ruling Class of U.S. Intelligence
Posted on April 5, 2012 by Tim Shorrock

On April 4, 2012, I spoke on a panel discussion on National Security, Secrecy and Surveillance in New York City. The event was sponsored by the Open Society Foundations and the Government Accountability Project, and moderated by Steven Aftergood, the reknowned editor of Secrecy News for the Federation of American Scientists. Besides myself, the speakers were Thomas Drake, the courageous former intelligence officer who blew the whistle on National Security Agency/contractor corruption during the Bush administration and was wrongly prosecuted by the Obama administration as a result; his equally courageous attorney, Jessylyn Radek , who is a whistle-blower herself for exposing the barbaric treatment of the so-called “American Taliban” John Walker Lindh in the days after 9/11; and Jameel Jaffer of the ACLU, who has participated in some of the most important national security litigation of the past ten years.

The format was informal; Aftergood posed a series of questions to each panelist, giving us a few minutes to respond, and then posed a series of follow-up queries. After that, the audience got to ask its own questions, and at the end we all gave some final thoughts. The entire event will soon be available on video at the OSF and GAP websites, and I will post it here as soon as I get it.

For me, it was a tremendous honor to speak about my special area of expertise, intelligence contracting, with people who have spent much of the last decade fighting the threat to democracy posed by our national surveillance state. I had prepared a five-minute talk, but what I had written didn’t fit into the Q&A format. So I thought readers of my book SPIES FOR HIRE and my many followers on Twitter would be interested in the notes I made in preparation, and I present them below.

I started by talking about the NSA’s Trailblazer program, a $4 billion corporate boondoggle that Tom Drake, as an NSA whistle-blower, had sought to expose as a massive waste of resources and a threat to our democratic rights:

Trailblazer is highly symbolic of the folly of contracting. It was an enormous, wasteful project that made a lot of people rich while doing nothing to protect Americans and actually helping them lose a little more of their freedom. The culprit was SAIC, one of the nation’s largest defense and intelligence contractors. New Yorkers may know SAIC because it just pled guilty to massive fraud involving the city’s payroll systems and paid a $500 million fine to basically avoid being blacklisted by the government.

In the case of Trailblazer, the company paid zero fines and kept winning new contracts. But it wasn’t only SAIC – the Trailblazer “team” included Northrop Grumman and Booz Allen Hamilton, both longtime NSA contractors, and literally dozens of subcontractors. The entire project was symptomatic of the way the privatized intelligence community operates, without oversight or accountability, and basically in the shadows.

As Tom and Jesslyn have argued, Tom didn’t leak anything secret about Trailblazer: he was merely passing on unclassified information to a Baltimore Sun reporter about one of the worst contract failures – and scandals – in US intelligence history.

So it was interesting to read in Jane Mayer’s excellent New Yorker piece on Tom a quote about this from Jack Goldsmith, one of the Justice Department lawyers who justified Bush’s programs. Instead of prosecuting Tom Drake, he said, the government should have gone after the leakers who talked to Bob Woodward for his four books on Bush’s wars, which he said were “filled with classified information that he could only have received from the top of government.”

That’s true: Woodward, in fact, did rely on top-level leaking – including from George W. Bush himself. One of the most startling parts of his last book THE WAR WITHIN concerns the intelligence technologies used to capture and kill Abu Musab al-Zarqawi and other Al Qaeda leaders in Iraq in 2005 and 2006. In the book, Woodward argues that these technologies were the secret weapons that turned the Iraq War around for Bush.

They were “some of the most highly classified techniques and information in the US government,” he wrote. A Defense Intelligence Agency official who was a top aide at the time to General Stanley McChrystal, the former commander of the Joint Special Operations Command (JSOC), told Woodward that the high-tech operations were so effective they gave him “orgasms.” All Bush would say, when asked about them, was: “JSOC is awesome.” The White House asked Woodward not to publish any details because that “might lead to unraveling of state secrets.”

This really blew Woodward’s mind. In interviews on 60 Minutes, CNN, NBC and other networks in the days after the book was published, he repeatedly said that he’d stumbled on the greatest national security secret since World War II and the Manhattan Project. When he talked to a 4-star general about his findings, he told NBC’s Andrea Mitchell, “the blood literally drained from his face” and “he said ‘you cannot write about this.’” By not disclosing the information, Woodward acted like he was somehow saving the Republic.

Well, this really struck me as odd because Woodward’s information was so familiar to me.

In fact, I’d learned about it as a lowly book writer and reporter two years before! Specifically, I learned about these Manhattan Project-like secrets at GEOINT, the annual conference and exhibition sponsored by the contractor-organized US Geospatial Intelligence Foundation. For intelligence players and aficianados, GEOINT is kind of the holy place where contractors and intelligence officials meet.

So what was Woodward’s big secret? Well, as anybody writing about intelligence at the time was aware, he was talking about how terrorists were found, tracked and targeted by the NSA and the National Geospatial-Intelligence Agency, or NGA, which is responsible for imagery and mapping intelligence.

Basically, these two agencies have learned how to create hybrid intelligence tools that – in official parlance – create “horizontal integration” between the two agencies, defined as “working together from start to finish, using NGA’s ‘eyes’ and NSA ‘ears’” (that’s actually from an NGA press release). They combine intercepts of cellphone calls with overhead imagery gathered by Predators and drones and use this data to track suspected terrorists in real time.

At the GEOINT meeting in 2006, the NGA director at the time, Adm. Robert B. Murrett, disclosed that it was through such technology that the U.S. military was able to locate and bomb the safe house where Zarqawi, the leader of al-Qaida in Iraq, was staying in June 2006. “Eventually, it all comes down to physical location,” he told reporters.

When NSA and NGA data are combined, he added, “the multiplier effect is dramatic.” I knew this was big and wrote about it in an article for Salon – “America Under Surveillance” on Aug. 9, 2007 – a least one year before Woodward’s book came out. Details also appeared in my book, which was also released before Woodward’s.

So why did I learn this huge secret at GEOINT? Because the entities doing the work for the NSA and NGA were (and are) contractors, such as SAIC and Booz Allen. Contractors supply the tracking and surveillance technologies as well as many of the analysts who interpret the intelligence.

A few contractors even took public credit for their tracking and surveillance work on Al Qaeda: George Tenet actually gave an award to SAIC for the Zarqawi hit, an event well-publicized by the company. Another contractor to claim credit was CACI International, which gained notoriety for being the contractor most involved in the Abu Ghraib scandal. In a radio interview in 2006, CACI’s CEO bragged of its “forensic-type work” using information from “overhead imagery, communications satellites, and intercepts” to “determine connections among organizations and cells of people” and bragged that they had used these technologies to get Zarqawi.

Sound familiar?

This points to a broader issue about intelligence: the more it is privatized, the less secret it becomes.

Just as I learned at GEOINT one of the most deeply held secrets of the war on terror, I was able to glean much that was on the Dark Side from corporate websites, interviews with contractors, SEC reports, internal and external corporate info, and conferences with investors and defense contractors. In fact I obtained so much information this way that my book is used at the Naval Intelligence Center for Information Dominance in Pensacola, Florida, to help budding intelligence analysts understand how much can be learned about intelligence from public sources.

Maybe, instead of prosecuting the Tom Drakes of the national security world, the government should go after the contractors.

But that doesn’t mean that contracted intelligence isn’t secret – it most certainly is. Most contracts are classified and there’s no requirement to disclose them. The use of earmarks in the conventional budget process, in which congressmen can secretly insert contractor projects without fear of any disclosure, allows agencies to further hide programs, including highly sensitive Special Access programs, from both congressional overseers and the public. And for the most part, these contractors toil on without recognition of the press and are thus hidden from the public (unless you know what you’re looking for: don’t forget to scroll through my 2010 posting, “The Corporate Intelligence Community: A Photo Exclusive.”)

Worse, Congress has refused to investigate. There’s been only one hearing on intelligence contractors that I can remember – a desultory event sponsored by the Senate Homeland Security Committee last year. And only a couple of concerned lawmakers in the House, Jan Shankowsky of Illinois and David Price of North Carolina – have bothered to ask serious questions about the implications of intelligence contracting. But without Congressional oversight, fraud waste and abuse – as exemplified by Trailblazer – continue. And the real actors in intelligence, the private sector, remain hidden from the American people.

So part of what I want to do here is introduce you to the ruling class of US intelligence. I’ll name three people – three of dozens. Two of them are probably familiar to you. But the first is not. Here they are (and there’s plenty more about them in my book):

RICHARD HAVER - If you read the first few pages of James Risen’s book on the CIA, STATE OF WAR, you’ll understand why. After many years in the darkest parts of naval intelligence, he worked for years at TRW, one of the first intelligence contractors whose work for the CIA was made famous in the Sean Penn movie “The Falcon and the Snowman.” It’s now a key unit of Northrop Grumman, where Haver, until recently, was vice president for intelligence (he’s now at a company called Passur Aerospace, which focuses on “integrated surveillance networks and databases, predictive analytics and business intelligence.”) Haver, who led the CIA investigation into the Aldrich Ames spying case, was for many years an intelligence adviser to both Dick Cheney and Donald Rumsfeld – in two administrations – and led the Cheney-Bush intelligence transition team in 2001. TRW & Northrop Grumman play key roles at the NSA and NGA and in the computerized drone war of today.

MIKE MCCONNELL – When he was nominated as Bush’s Director of National Intelligence in 2006 I wrote a profile in Salon hoping someone in the Senate would look into his role at Booz Allen Hamilton and all the programs it’s been involved in. McConnell, like Haver, started out in Naval intelligence, tracking target for US bombers in Vietnam and Cambodia. Later he was a military intelligence adviser to Colin Powell and Dick Cheney during the first Gulf War. Then, with Cheney’s assistance, he was named NSA director and served under President Clinton.

After that he was hired by Booz Allen, where he ran the company’s extensive programs in military intelligence until Bush appointed him DNI. He ran the warrantless surveillance program and pushed through legislation to exonerate and provide immunity to the telecom providers and the contractors who’d collaborted with the NSA (and, as Jameel Jaffer reminded me, create an entirely new national system of warrantless wiretapping). And now he’s back at Booz, promoting cyberwarfare.

JOHN BRENNAN – As President Obama’s chief intelligence adviser, this man has revolved through the door and back again. Brennan is the former CIA Station Chief in Saudi Arabia and was once director of the CIA’s CounterTerrorism Center. Then he left the CIA and joined a company called The Analysis Corporation before moving on to the Obama administration. Remember the Christmas Day bomber in 2009? Remember Brennan almost going down on his knees apologizing to Obama? Why would an intel adviser do that? Well he had personal responsibility – TAC, his former company, built the database for the National CounterTerrorism Center that failed to track the Nigerian. CACI, by the way, maintains the database.

These men, when they’re wearing their corporate logos, make up the core of the private intelligence industry but should really be consider an essential part of the IC and subject to as much public exposure as high ranking government officials such as the DNI or the Secretary of Defense. Let me read a quote about McConnell that’s true of all these men. It’s from Ellen McCarthy, president of the Intelligence and National Security Alliance, another shadowy part of the IC (I have a lot about INSA in my book), speaking to Bloomberg:

“In many ways Admiral McConnell can be more influential in supporting the intelligence community now than when he was in office. He’s not constrained by the bureaucracy, and is viewed as a senior statesman operating in an advisory capacity.” That’s a very telling.

So, to conclude, what we have here is a revolving door of the highest order. And, as Tom Drake will tell you based on his own experience as an NSA employee and a contractor, it’s all part of a massive transfer of wealth from government to business the likes of which we’ve never seen. As I revealed in SPIES FOR HIRE, 70 percent of the intelligence budget goes to contractors. You can do the math – if the intelligence budget is $100 billion, that’s $70 billion going straight to the private sector.

As you listen to our discussion tonight, keep in mind that everything we talk about has a private component. That includes covert operations, rendition, torture, illegal surveillance and wiretapping, targeted assassinations, drone wars: everything.

And here’s a plea to the other reporters in the room: without considering the private sector component, the concept of intelligence “community” is a misnomer. In reality, the IC is a joint venture, probably the most profitable secret business in the world. And I don’t think we’ve come close to grasping the full implications of these private companies sharing – and making money from – the nation’s most classified programs. For my part, I’m going to do all I can to expose this industry and its part in diminishing our rights as American citizens.

Tim Shorrock
New York City
April 4, 2012
User avatar
Wombaticus Rex
Posts: 10614
Joined: Wed Nov 08, 2006 6:33 pm
Location: Vermontistan
Blog: View Blog (0)

Re: Data Mining & Intelligence Agencies

Postby Wombaticus Rex » Fri Apr 06, 2012 1:16 pm

More Shorrock, this time 2006 vintage...

Watching What You Say
Tim Shorrock | May 11, 2006

Two months after the New York Times revealed that the Bush Administration ordered the National Security Agency to conduct warrantless surveillance of American citizens, only three corporations--AT&T, Sprint and MCI--have been identified by the media as cooperating. If the reports in the Times and other newspapers are true, these companies have allowed the NSA to intercept thousands of telephone calls, fax messages and e-mails without warrants from a special oversight court established by Congress under the 1978 Foreign Intelligence Surveillance Act (FISA). Some companies, according to the same reports, have given the NSA a direct hookup to their huge databases of communications records. The NSA, using the same supercomputers that analyze foreign communications, sifts through this data for key words and phrases that could indicate communication to or from suspected terrorists or terrorist sympathizers and then tracks those individuals and their ever-widening circle of associates. "This is the US version of Echelon," says Albert Gidari, a prominent telecommunications attorney in Seattle, referring to a massive eavesdropping program run by the NSA and its English-speaking counterparts that created a huge controversy in Europe in the late 1990s.

So far, a handful of Democratic lawmakers--Representative John Conyers, the ranking Democrat on the House Judiciary Committee, and Senators Edward Kennedy and Russell Feingold--have attempted to obtain information from companies involved in the domestic surveillance program. But they've largely been rebuffed. Further details about the highly classified program are likely to emerge as the Electronic Frontier Foundation pursues a lawsuit, filed January 31, against AT&T for violating privacy laws by giving the NSA direct access to its telephone records database and Internet transaction logs. On February 16 a federal judge gave the Bush Administration until March 8 to turn over a list of internal documents related to two other lawsuits, filed by the American Civil Liberties Union and the Electronic Privacy Information Center, seeking an injunction to end the program.

Despite the President's rigorous defense of the program, no company has dared to admit its cooperation publicly. Their reticence is understandable: The Justice Department has launched a criminal investigation of the government officials who leaked the NSA story to the Times, and many constitutional scholars and a few lawmakers believe the program is both illegal and unconstitutional. And the companies may be embarrassed at being caught--particularly AT&T, which spent millions advertising its global services during the Winter Olympics. "It's a huge betrayal of the public trust, and they know it," says Bruce Schneier, the founder and chief technology officer of Counterpane Internet Security, a California consulting firm.

Corporations have been cooperating with the NSA for half a century. What's different now is that they appear to be helping the NSA deploy its awesome computing and data-mining powers inside the United States in direct contravention of US law, which specifically bans the agency from collecting information from US citizens living inside the United States. "They wouldn't touch US persons before unless they had a FISA warrant," says a former national security official who read NSA intercepts as part of his work for the State Department and the Pentagon.

This is happening at a time when both the military and its spy agencies are more dependent on the private sector than ever before, and an increasing number of companies are involved. In the 1970s, when Congress acted to stop domestic spying programs like Operation Shamrock, in which the NSA monitored overseas telegrams and phone calls, the communications industry was in its infancy. "It was basically Western Union for cables, and AT&T for the telephone," says James Bamford, who revealed the existence of the NSA in his famous book The Puzzle Palace and is a plaintiff in the ACLU lawsuit. "It's much more complicated now." In fact, today's global telecom market includes dozens of companies that compete with AT&T, Sprint and MCI for telephone and mobile services, as well as scores of Internet service providers like Google, Yahoo! and AOL that offer e-mail, Internet and voice connections to customers around the world. They are served by multinational conglomerates like Apollo, Flag Atlantic and Global Crossing, which own and operate the global system of undersea fiber-optic cables that link the United States to the rest of the world. Any one of them could be among the companies contacted by intelligence officials when President Bush issued his 2002 executive order to obtain surveillance without FISA approval.

Nobody's talking, though. Asked if AT&T, which was recently acquired by SBC Communications, is cooperating with the NSA, AT&T spokesman Walt Sharp said, "We don't comment on national security matters." He referred me to a recent AT&T letter to Representative Conyers, which stated that AT&T "abides by all applicable laws, regulations and statutes in its operations and, in particular, with respect to requests for assistance from governmental authorities." MCI, which was acquired in January by Verizon, and Sprint, which recently merged with Nextel Communications, declined to comment. Attorney Gidari, who has represented Google, T-Mobile, Nextel and Cingular Wireless (now part of AT&T), believes that "some companies, both telecom and Internet," were asked to participate in the NSA program. But he suggests that only a limited number agreed. "The list of those who said no is much longer than most people think," he says.

The NSA, some analysts say, may have sought the assistance of US telecoms because most of the world's cable operators are controlled by foreign corporations. Apollo, for example, is owned by Britain's Cable & Wireless, while Flag Atlantic is owned by the Reliance Group of India. Much of the international "transit traffic" carried by the cable companies flows through the United States (this is particularly true of communications emanating from South America and moving between Asia and Europe). The NSA could get access to this traffic by sending a submarine team to splice the cables in international waters, as the agency once did to the Soviet Union's undersea military cables. But that is an extremely expensive proposition, and politically dicey to boot--which is where the US telecoms come in. "Cooperation with the telcos doesn't make NSA surveillance possible, but it does make it cheaper," says Schneier, the technology consultant.

According to Alan Mauldin, a senior research analyst with TeleGeography Research in Washington, DC, it would be possible for US intelligence operatives to gain access to transit traffic from anywhere in the country with the cooperation of a US company. "You could be inland, at an important city like New York or Washington, DC, where networks interconnect, and you could have the ability to tap into the whole network for not only that city but between that city and the rest of the world," he says. Foreign-owned cable operators, says Gidari, are also required by US law to maintain security offices manned by US citizens, with background checks and security clearances at the landing sites in Oregon, Florida, New Jersey and other states where fiber-optic cables come ashore.

The government has gone to great lengths to insure law-enforcement access to foreign-owned telecom companies. Take the example of Global Crossing, which owns several undersea cable systems and claims to serve more than 700 carriers, mobile operators and ISPs. Three years ago, as Global Crossing was emerging from one of the largest bankruptcies in US history, it was purchased by ST Telemedia, which is partly owned by the government of Singapore. As part of the US approval process (which occurred at a time when Global Crossing was being advised by Richard Perle, then-chairman of Donald Rumsfeld's Defense Policy Board), the company signed an unprecedented Network Security Agreement with the FBI and the Defense Department. Under the agreement, which is on file with the Federal Communications Commission, Global Crossing pledged that "all domestic communications" would pass through a facility "physically located in the United States, from which Electronic Surveillance can be conducted pursuant to lawful US process." (Global Crossing declined to comment.) Legal experts say the wording is significant in the context of the NSA spying flap, but cautioned not to read too much into it. "These agreements are not uncommon in the industry," says James Andrew Lewis, director of the Technology and Public Policy Program at the Center for Strategic and International Studies in Washington. "They provide assurances that US interests won't suffer damage with foreign ownership."

History proves a good guide to how the NSA would go about winning cooperation from a telecom company. When telephone and telegraph companies began assisting the NSA during the 1940s, only one or two executives were in on the secret. That kind of arrangement continued into the 1970s, and is probably how cooperation with the NSA works today, says Kenneth Bass III, a Justice Department official during the Carter Administration. "Once the CEO approved, all the contacts [with the intelligence agencies] would be worked at a lower level," he says. "The telcos have been participating in surveillance activities for decades--pre-FISA, post-FISA--so it's nothing new to them." Bass, who helped craft the FISA law and worked with the NSA to implement it, adds that he "would not be surprised at all" if cooperating executives received from the Bush Administration "the same sort of briefing, but much more detailed and specific than the FISA court got when [the surveillance] was first approved."

For US intelligence officials looking for allies in the industry, AT&T, MCI and Sprint have a lot to offer. In 2002, when the spying program began, AT&T's CEO was C. Michael Armstrong, the former CEO of Hughes Electronic Corp. At the time, Armstrong was also chairman of the Business Roundtable's Security Task Force, where he was instrumental in creating CEO COM LINK, a secure telecommunications system that allows the chief executives of major US corporations to speak directly to senior members of Bush's Cabinet during national emergencies. Randall Stephenson, a former SBC Communications executive who is now AT&T's chief operating officer, is a member of the National Security Telecommunications Advisory Committee, a group of executives from the communications and defense industries who advise the President on security issues related to telecom.

Those executives, all of whom hold security clearances, meet at the White House once a year--Vice President Cheney was the speaker at their last meeting--and hold quarterly conference calls with high-ranking officials. (Asked if the NSA surveillance was ever discussed at these sessions, committee spokesman Stephen Barrett said, "We do not participate in intelligence gathering.") AT&T also makes no bones about its national security work. When SBC was preparing to acquire the company last year, the two companies underscored their ties with US intelligence in joint comments to the FCC. "AT&T's support of the intelligence and defense communities includes the performance of various classified contracts," the companies said, pointing out that AT&T "maintains special secure facilities for the performance of classified work and the safeguarding of classified information."

MCI, too, is a major government contractor and was highly valued by Verizon in part because of its work in defense and intelligence. Nicholas Katzenbach, the former US Attorney General who was appointed chairman of MCI's board after the spectacular collapse of its previous owner, WorldCom, reiterated MCI's intelligence connections in a 2003 statement to the Senate Judiciary Committee. "We are especially proud," he wrote, "of our role in supporting our national-security agencies' infrastructure, and we are gratified by the many positive comments about our service from officials at the US Department of Defense and other national-security agencies." MCI's general counsel--who would presumably have a say in any decision to cooperate with the NSA--is William Barr. He is a former assistant general counsel at the Central Intelligence Agency and served as Attorney General during the Administration of President George H.W. Bush.

Sprint Nextel is top-loaded with executives with long experience in national security and defense. Chairman and CEO Gary Forsee is a member of Bush's telecom council (as is Lawrence Babbio, the vice chairman and president of Verizon). Keith Bane, a company director, recently retired from a twenty-nine-year career with Motorola, which has worked closely with US intelligence for decades. William Conway Jr. and former FCC chairman William Kennard are managing directors of the Carlyle Group, the Washington private equity fund that invests heavily in the military and has extensive contacts in the Bush Administration.

There's another group of companies, largely overlooked, that could also be cooperating with the NSA. These are firms clustered around the Beltway that contract with the agency to provide intelligence analysts, data-mining technologies and equipment used in the NSA's global signals-intelligence operations. The largest of them employ so many former intelligence officials that it's almost impossible to see where the government ends and the private sector begins. Booz Allen Hamilton, the prime contractor for Trailblazer, a huge NSA project updating its surveillance and eavesdropping infrastructure, employs several NSA alumni, including Mike McConnell, its vice president, who retired as NSA director in 1996. (Ralph Shrader, the company's CEO, joined Booz Allen in 1978 after serving in senior positions with Western Union and RCA, both of which cooperated with the NSA on Operation Shamrock.) SI International, a software and systems engineering company with NSA contracts, recently hired Harry Gatanas, the NSA's former director of acquisitions and outsourcing, to oversee its $250-million-a-year business with US intelligence and the Pentagon. Science Applications International Corporation, another big NSA contractor, is run by executives with long histories in military intelligence, including COO Duane Andrews, a former Assistant Secretary of Defense for Command, Control, Communications and Intelligence.

Are firms that cooperate with the NSA legally culpable? Bamford, who is not a lawyer but probably knows more about the NSA than any American outside government, says yes. "The FISA law is very clear," he says. "If you don't have a warrant, you're in violation, and the penalty is five years and you can be sued by the aggrieved parties." Kevin Bankston, an attorney for the Electronic Frontier Foundation, adds that US law "not only prohibits unauthorized wiretapping; it also prohibits unauthorized disclosure or use of illegally wiretapped information. As long as you were doing that, you're potentially liable." Schneier, the technology consultant, harbors no doubts either. "Arguing that this is legal is basically saying we're in a police state."

But Gidari, the Seattle telecom attorney, believes that companies would be insulated from legal challenges if they had assurances from the government that the program was within the law. He also says Congress has passed legislation granting immunity to companies operating under "statutory grants of authority" from the government. "It's not a slamdunk, but it is a good-faith defense," he says. Former Justice Department official Bass agrees but says reliance on oral requests from US officials is another matter: "If they didn't get the type of legal assurances the FISA provides for"--such as a written statement from the Attorney General--"there could be some legal exposure." But a full airing of the legal issues raised by the surveillance program may be a long time coming. "The likelihood of any enforcement absent a change in administration is zero," Bass says.
User avatar
Wombaticus Rex
Posts: 10614
Joined: Wed Nov 08, 2006 6:33 pm
Location: Vermontistan
Blog: View Blog (0)

Re: Data Mining & Intelligence Agencies

Postby Wombaticus Rex » Wed Apr 11, 2012 9:13 pm

Some deep history on 60's era SIGINT projects:

Project Dark Gene and Project Ibex

For many years little if any information could be found anywhere about Project Dark Gene and Project Ibex, athough both projects played a significant role in the ongoing programme of intelligence gathering targeted against the Soviet Union in the late 1960's and 1970's. Now, thanks to the efforts of Tom Cooper and Art Kremzel, some details of these projects are slowly beginning to emerge. There must be many personnel still alive who were involved in supporting these projects and I hope they will also eventually choose to contribute to our knowledge of what actually happened. In particular, I would like to try and obtain copies of any photographs showing the aircraft that actually took part in either project, as well as images of the Tracksman 1 and Tracksman 2 sites in Iran and the elusive USAF Col John Saunders.

Research undertaken by Tom Cooper, Editor, and Art Kremzel

Project Dark Gene

Throughout the Cold War the US conducted almost daily sorties around the edge of the USSR and Warsaw Pact countries, as part of a long-standing programme to identify and classify radars, SAM sites and any other electronic emissions of interest. The information gathered would then be used to try and establish the safest ingress routes to various targets for US and NATO bombers and fighter-bombers should WW3 ever break out. Given the huge landmass of the USSR, it was hardly surprising that some gaps in their radar coverage would be identified by these ELINT sorties and that, once these gaps had been found, it was almost inevitable that they would then be further exploited to try and gather even more useful intelligence.

Iran has a long border with what was then the southern part of the USSR and during the 1970’s, before the revolution that saw the Shah deposed, Iran maintained a good relationship with the USA. As part of this close relationship it was agreed that various long-range radars and listening posts could be established in Iran to enable the USA to monitor activities behind the Iron Curtain. However, it was also realised that, as the border between the USSR and Iran contained a number of significant gaps in overlapping radar cover, a low-flying reconnaissance aircraft could easily get over the border and take some useful photographs of areas of interest. When these cross-border flights were eventually detected they would have the added advantage of stirring up a hornets next of activity by other radar and SAM sites, allowing valuable intelligence on their location and operating frequencies to be scooped up by high-flying ELINT aircraft and listening posts positioned just the other side of the border. This ELINT activity was probably part of ‘Project Ibex’, but a more pro-active reconnaissance programme also took place around the same time.

From 1968 onwards, in recognition of the good relationship between the USA and Iran, 12 Northrop RF-5A aircraft were delivered to the IIAF, however, all was not quite as it seemed. In fact it appears that officially these aircraft never actually existed – their serial numbers were deleted from Northrop’s production list to make them ‘deniable’. In addition, the aircraft were actually flown by USAF pilots until 1971 under an operation known as Dark Gene and were used to make covert reconnaissance sorties across the border into the USSR, gathering mainly ELINT. It is understood that two of these aircraft were actually shot down inside the USSR whilst being flown by USAF pilots – they ejected and, presumably after pleading that they were actually training IIRAF pilots and simply got lost, they were quietly allowed to return to Iran, although this has yet to be confirmed. The RF-5A’s were also ‘A’ wired and had a secondary war role to carry a nuclear weapon and if necessary attack various targets in the USSR.

However, although the sub-sonic RF-5A’s were useful and presumably helped generate some interesting intelligence, it wasn’t really what the USAF pilots wanted to be piloting when they crossed over the border into the USSR – something with a little more grunt was called for and the RF-4 fitted the bill nicely. In addition, the Shah, who presumably was kept informed of the intelligence obtained by the RF-5A overflights, was keen for Iran to play an even more active role in this activity and offered to pay for the RF-4s. A solution was agreed – Iran would pay for the RF-4s and they would be flown by mixed crews of USAF and IIAF personnel, allowing the IIAF crews to gain valuable operational experience. In 1971 the first six RF-4s arrived in Iran, officially these were RF-4Es, however, sources involved have indicated that the airframes were actually highly unusual RF-4Cs. In fact it appears that these aircraft had been specially built for this operation and contained various specialised ELINT equipment and cost over $12 per airframe, making them the most expensive F-4s ever built. To date no authentic photos of these unique RF-5As and the expensive RF-4Cs have been discovered.

A number of these first six aircraft were delivered without the production number being officially listed and are generally referred to as UKIs – Unknown Iranians, some others were probably part of the 72-0266 to 720269 serials later acknowledged as delivered to Iran. Eventually, somewhere between 22 and 25 RF-4 airframes were delivered to Iran, the precise number is impossible to determine. As with the RF-5As, the RF-4Cs were also ‘A’ wired and could if necessary carry a nuclear weapon as a secondary role. Flown by mixed USAF and IIAF crews, the six RF-4Cs averaged two missions a month over the USSR from 1971 through to 1978. If they were shot down, the cover story was that the USAF crewmember was training the IIAF crewman and that they were on a navigation-training sortie, had got lost in bad weather and had inadvertently strayed over the border into the USSR. In actual fact it is understood that at least two of these ‘Iranian’ RF-4Cs were shot down inside the USSR by Soviet fighters, the first in 1973 and the second in 1976.

Some details of the 1973 incident have emerged. On 28 Nov 73 an Iranian RF-4C, flown by an IIAF pilot Maj Shokouhnia with USAF Col John Saunders in the rear seat, was detected inside the USSR. The RF-4C made a run for the border and was at Mach 1.4 when it was intercepted by a MiG-21 flown by Capt Gennady Eliseev. Col Saunders began firing out decoy flares to try and prevent the MiG from locking on a heat-seeking missile, eventually using all the 54 flares carried. The MiG-21 finally managed to launch two R-3S missiles at the RF-4C, but both missed. However, whilst turning hard to limit the chances of the MiG achieving a good lock-on, the RF-4C had lost some airspeed, allowing the MiG-21 to suddenly cut a corner and close up. It is presumed that Capt Gennady Eliseev then made a conscious decision to ram the RF-4C to prevent it escaping, as the MiG-21 aircraft rammed the aircraft from the left and below, near the engine nozzles, probably cutting off the tail of the RF-4C, throwing it into a high-speed dive. Capt Gennady Eliseev was killed in the collision and was posthumously decorated with the ‘Hero of the Soviet Union’ medal. Maj Shokouhnia and Col Saunders both ejected and were captured by Soviet ground forces. They used their cover story as briefed and, as the RF-4C had impacted the ground at something like Mach 2, there was little if any evidence the Soviets could use to prove otherwise.

It was quietly agreed that Maj Shokouhnia and Col Saunders would be returned to Iran in exchange for a cartridge from a Soviet reconnaissance satellite that had accidentally landed in Iran. Both Maj Shokouhnia and Col Saunders were decorated for their exploit. It is believed that Maj Shokouhnia, left Iran during the revolution in 1979, but later returned and was executed in 1980. Col Saunders returned to more normal duties and has never spoken officially about his activities in Iran during this period. In response to these overflights, the USSR began overflying Iran with the MiG-25RBSh and various attempts were made to intercept the aircraft with the IIAF F-4D and F-4E. Apparently, sometime in 1976, one IIAF F-4E eventually managed to hit a MiG-25RBSh, but it made it back over the border into Russia before it crashed. The loss of the second RF-4C over Russia later in 1976 may well have been in response to this incident. The arrival of the IIAF F-14 Tomcat put an end to the overflights of Iran by the MiG-25RBSh, particularly after one had been intercepted over the Caspian Sea in Oct 1978 by two IIAF F-14s, who then maintained a radar lock-on to the MiG-25 for over a minute, no doubt giving the MiG-25 pilot something to think about.

At present that is all that is known about Operation Dark Gene, but hopefully as time goes on more details will emerge of the project in general and about the loss of the two RF-5As and the second IIAF RF-4C in 1978 in particular.

Project Ibex

Project Ibex was launched in around 1974, with Iran and the USA as equal partners, however, Iran paid almost the entire cost of the project, some $500 million, to the main US contractor - Rockwell Inc. Project Ibex was a joint CIA+NSA / Iran enterprise for building and operating a series of observation and listening posts along the Soviet border, as well as for the purchase and operation of a number of reconnaissance aircraft.

There were five ground stations; three jointly run by the IIAF/USAF crews and two operated by the CIA or, more likely, the NSA. The CIA-crewed stations were established at Bushehr (Tracksman 1) and at Kapkan (Tracksman 2). The most distinct functions of these five intelligence-gathering stations were: to monitor the radio and telemetry traffic of the Soviet armed forces in southern USSR, especially to find evidence of heightened military activity; to monitor Soviet missile testing; and to receive high resolution photographs from the orbiting spy satellites.

USAF aircrews flying ELINT missions out of U.S. bases in such places as Okinawa and Alaska were alerted by Tracksmen messages to watch for Soviet missiles. These sites were considered so sophisticated that Stanfield Turner described them as systems built for the 21st century. Certain elements of the IIAF were also included in Project IBEX and were tasked with providing air defense for all five stations; all the stations were also surrounded by barbed-wired and mine fields and could be quickly blown up in the event of some unauthorized personnel managing to gain entry.

Additionally, two DHC-4 Caribou STOL aircraft were purchased solely for the task of supplying logistics and transporting personnel to the remote IBEX stations. With this in mind it’s interesting that, although the IIAF never operated the DHC-4 Caribou, one is still on display at the IIAF Museum, in Mehrabad, Tehran - in full IIAF markings! The fate of the other Caribou is unknown. The original plan for IBEX also called for two of IIAF’s Boeing 707s and two of the six P-3F Orion patrol aircraft to be converted for ELINT roles. However, rather than the P-3F Orion, the IIAF opted instead to purchase four C-130E/Hs, mainly because they were capable of operating from rough airfields at high altitudes, whilst carrying a significant ELINT payload. The four chosen C-130E/Hs were modified by E-Systems, Greenville Division, while work on one Boeing 707-3J9C was started in 1975.


Aircraft equipped with IBEX sub-systems are understood to have had 13 operator positions, an ELINT sub-system capable of detecting and classifying emitters, and an oblique and vertical camera subsystem for surveillance photography. The conversion involved extensive structural modification and reinforcement, including the installation of a range of antennas on the aircraft’s wings and fuselage, a retractable direction-finding array under the belly, a new INS, modified power and cabin-environment systems, and special enlarged pods under the outboards of their outer engines. Hercules of the IBEX project carried oblique long focal length cameras to photograph Soviet radar transmitters from high altitudes, as well as equipment for monitoring and recording radio and radar signals. According to unconfirmed reports, some parts of the IBEX-equipment were built into at least two IIAA (Army Aviation) Aero Commander-560s. Note that both types - the C-130 and Aero Commander-560 - are known to have been used extensively for flying recce sorties along and possibly beyond the Soviet borders, since the early 1960s.

Aside from tracking Soviet missile testing, the Iranian government used the IBEX system to detect/locate, track and identify, and also characterize radar signals originating in most of neighboring countries, particularly Iraq. Prior to Rockwell getting that contract, an Israeli project proposal had been rejected by the IIAF as "insufficiently advanced". Nevertheless, after the installation of IBEX in Iran, a rumour surfaced that the Shah was not particularly pleased with "malfunctioning equipment“, which forced Rockwell to call the CIA for direct help. Additionally, at a very early stage, the project caused great concerns in Washington and elsewhere, because Rockwell employed some ex-NSA specialists to work alongside Iranian trainees, some of whom supposedly showed signs of mental disorder after working for some time with the equipment in the USA.

It remains unclear how the problems with the IBEX were solved, however, the system became operational and all four IBEX C-130H (nicknamed "Khofaash" or Bat whilst in Iranian service) were continuously flown along the Soviet border. Together with the five ground stations, they became part of the IIAF’s Electronic Reconnaissance Wing, under command of Lt. Gen. Abdollah Assrejadid. The rest of this unit seems to have been composed of two squadrons equipped with RF-4Es and RF-5As, respectively. All the Khofaashs and the single Boeing 707 were normally stationed at Mehrabad AB, but there were prominent "forward bases" established at Tabriz and Hamedan, and possibly at other places like Shiraz and Zahedan as well. Although it is certain that dozens of operational missions were flown, details about only one are known: it occurred in October 1978, when a single Boeing 707 escorting two F-14s in an intercept of a Soviet MiG-25R over the Caspian Sea. This engagement was tracked from an immense range (over 200km) without any problems - and the CIA crews, as well as few USN people permitted to see some of result, were highly pleased with intelligence gathered.

The overthrow of the Shah of Iran in 1978 -1979 and the creation of an anti-USA, hard-line Islamic Republic led by Ayatollah Khomeini, quickly brought an end to both projects. What became of the sites is unknown, but given the contingency plans already in place, I imagine both sites, in particular the equipment, were totally destroyed before the personnel were evacuated, probably to Turkey. I presume the planes involved remained part of the new IIAF, but apart from the single DHC-4 Caribou in the IIAF Museum at Mehrabad, Tehran, nothing is known about about their eventual fate. From what has been pieced together to date, it would appear that both projects were a success and, had they continued, they would have more than justified the risk and expense involved.


The only survivor of Project Ibex still flying is the single Iranian ELINT Boeing 707-3J9C (ex 5-8316, c/n 20834) shown here. The aircraft is believed to have been modified in 1975 by E-Systems Greenville Division from its original tanker/transport configuration. After conversion the aircraft was understood to have been equipped with 13 operator positions, an ELINT sub-system capable of detecting and classifying emitters, together with oblique and vertical cameras mounted behind a sliding door under the forward fuselage - the ELINT receivers were mounted in the wing root fairing. During its participation in Project Ibex the aircraft would have flown at high level close to the border with the Soviet Union gathering data on surviellance radars and air defense systems, looking for potential 'holes' that could be exploited by B-52s in the event of WW3.

The aircraft is currently assigned to the 1st Tactical Air Base, Tehran-Mehrabad International Airport. No titles are visible on the aircraft and it is assumed that it is still used to gather ELINT, however, it's not known whether the receiver systems have been continually updated, . Although the Iranians like to believe that their aerospace and electronic capabilities are quite advanced, in reality they lag some way behind the West, so unless they have managed to procure or more likely reverse engineer a relatively current ELINT system, its current capability would be fairly limited.

Definitely raises a question: How new is the phenomenon of private contractors being involved with SAP and black budget projects? Is it really a post-9/11 issue...or just business as usual?
User avatar
Wombaticus Rex
Posts: 10614
Joined: Wed Nov 08, 2006 6:33 pm
Location: Vermontistan
Blog: View Blog (0)

Re: Data Mining & Intelligence Agencies

Postby Wombaticus Rex » Thu Apr 26, 2012 12:28 am


Is Lockheed Martin shadowing you?
How a giant weapons maker became the new Big Brother
By William D. Hartung

Have you noticed that Lockheed Martin, the giant weapons corporation, is shadowing you? No? Then you haven’t been paying much attention. Let me put it this way: If you have a life, Lockheed Martin is likely a part of it.

True, Lockheed Martin doesn’t actually run the U.S. government, but sometimes it seems as if it might as well. After all, it received $36 billion in government contracts in 2008 alone, more than any company in history. It now does work for more than two dozen government agencies from the Department of Defense and the Department of Energy to the Department of Agriculture and the Environmental Protection Agency. It’s involved in surveillance and information processing for the CIA, the FBI, the Internal Revenue Service (IRS), the National Security Agency (NSA), the Pentagon, the Census Bureau, and the Postal Service.

Oh, and Lockheed Martin has even helped train those friendly Transportation Security Administration agents who pat you down at the airport. Naturally, the company produces cluster bombs, designs nuclear weapons, and makes theF-35 Lightning (an overpriced, behind-schedule, underperforming combat aircraft that is slated to be bought by customers in more than a dozen countries) — and when it comes to weaponry, that’s just the start of a long list. In recent times, though, it’s moved beyond anything usually associated with a weapons corporation and has been virtually running its own foreign policy, doing everything from hiring interrogators for U.S. overseas prisons (including at Guantanamo Bay in Cuba and Abu Ghraib in Iraq) to managing a private intelligence network in Pakistan and helping write the Afghan constitution.

A For-Profit Government-in-the-Making

If you want to feel a tad more intimidated, consider Lockheed Martin’s sheer size for a moment. After all, the company receives one of every 14 dollars doled out by the Pentagon. In fact, its government contracts, thought about another way, amount to a “Lockheed Martin tax” of $260 per taxpaying household in the United States, and no weapons contractor has more power or money to wield to defend its turf. It spent $12 million on congressional lobbying and campaign contributions in 2009 alone. Not surprisingly, it’s the top contributor to the incoming House Armed Services Committee chairman, Republican Howard P. “Buck” McKeon of California, giving more than $50,000 in the most recent election cycle. It also tops the list of donors to Sen. Daniel Inouye (D-HI), the powerful chair of the Senate Appropriations Committee, and the self-described “#1 earmarks guy in the U.S. Congress.”

Add to all that its 140,000 employees and its claim to have facilities in 46 states, and the scale of its clout starts to become clearer. While the bulk of its influence-peddling activities may be perfectly legal, the company also has quite a track record when it comes to law-breaking: it ranks number one on the “contractor misconduct” database maintained by the Project on Government Oversight, a Washington-DC-based watchdog group.

How in the world did Lockheed Martin become more than just a military contractor? Its first significant foray outside the world of weaponry came in the early 1990s when plain old Lockheed (not yet merged with Martin Marietta) bought Datacom Inc., a company specializing in providing services for state and city governments, and turned it into the foundation for a new business unit called Lockheed Information Management Services (IMS). In turn, IMS managed to win contracts in 44 states and several foreign countries for tasks ranging from collecting parking fines and tolls to tracking down “deadbeat dads” and running “welfare to work” job-training programs. The result was a number of high profile failures, but hey, you can’t do everything right, can you?

Under pressure from Wall Street to concentrate on its core business — implements of destruction — Lockheed Martin sold IMS in 2001. By then, however, it had developed a taste for non-weapons work, especially when it came to data collection and processing. So it turned to the federal government where it promptly racked up deals with the IRS, the Census Bureau, and the U.S. Postal Service, among other agencies.

As a result, Lockheed Martin is now involved in nearly every interaction you have with the government. Paying your taxes? Lockheed Martin is all over it. The company is even creating a system that provides comprehensive data on every contact taxpayers have with the IRS from phone calls to face-to-face meetings.

Want to stand up and be counted by the U.S. Census? Lockheed Martin will take care of it. The company runs three centers — in Baltimore, Phoenix, and Jeffersonville, Indiana — that processed up to 18 tractor-trailers full of mail per day at the height of the 2010 Census count. For $500 million it is developing the Decennial Response Information Service (DRIS), which will collect and analyze information gathered from any source, from phone calls or the Internet to personal visits. According to Preston Waite, associate director of the Census, the DRIS will be a “big catch net, catching all the data that comes in no matter where it comes from.”

Need to get a package across the country? Lockheed Martin cameras will scan bar codes and recognize addresses, so your package can be sorted “without human intervention,” as the company’s web site puts it.

Plan on committing a crime? Think twice. Lockheed Martin is in charge of the FBI’s Integrated Automatic Fingerprint Identification System (IAFIS), a database of 55 million sets of fingerprints. The company also produces biometric identification devices that will know who you are by scanning your iris, recognizing your face, or coming up with novel ways of collecting your fingerprints or DNA. As the company likes to say, it’s in the business of making everyone’s lives (and so personal data) an “open book,” which is, of course, of great benefit to us all. “Thanks to biometric technology,” the company proclaims, “people don’t have to worry about forgetting a password or bringing multiple forms of identification. Things just got a little easier.”

Are you a New York City resident concerned about a “suspicious package” finding its way onto the subway platform? Lockheed Martin tried to do something about that, too, thanks to a contract from the city’s Metropolitan Transportation Authority (MTA) to install 3,000 security cameras and motion sensors that would spot such packages, as well as the people carrying them, and notify the authorities. Only problem: the cameras didn’t work as advertised and the MTA axed Lockheed Martin and cancelled the $212 million contract.

Collecting Intelligence on You

If it seems a little creepy to you that the same company making ballistic missiles is also processing your taxes, accessing your fingerprints, scanning your packages, ensuring that it’s easier than ever to collect your DNA, and counting you for the census, rest assured: Lockheed Martin’s interest in getting inside your private life via intelligence collection and surveillance has remained remarkably undiminished in the twenty-first century.

Tim Shorrock, author of the seminal book Spies for Hire, has described Lockheed Martin as “the largest defense contractor and private intelligence force in the world.” As far back as 2002, the company plunged into the “Total Information Awareness” (TIA) program that was former National Security Advisor Admiral John Poindexter’s pet project. A giant database to collect telephone numbers, credit cards, and reams of other personal data from U.S. citizens in the name of fighting terrorism, the program was de-funded by Congress the following year, but concerns remain that the National Security Agency is now running a similar secret program.

In the meantime, since at least 2004, Lockheed Martin has been involved in the Pentagon’s Counterintelligence Field Activity (CIFA), which collected personal data on American citizens for storage in a database known as “Threat and Local Observation Notice” (and far more dramatically by the acronym TALON). While Congress shut down the domestic spying aspect of the program in 2007 (assuming, that is, that the Pentagon followed orders), CIFA itself continues to operate. In 2005, Washington Post military and intelligence expert William Arkin revealed that, while the database was theoretically being used to track anyone suspected of terrorism, drug trafficking, or espionage, “some military gumshoe or overzealous commander just has to decide someone is a ‘threat to the military’” for it to be brought into play. Among the “threatening” citizens actually tracked by CIFA were members of antiwar groups. As part of its role in CIFA, Lockheed Martin was not only monitoring intelligence, but also “estimating future threats.” (Not exactly inconvenient for a giant weapons outfit that might see antiwar activism as a threat!)

Lockheed Martin is also intimately bound up in the workings of the National Security Agency, America’s largest spy outfit. In addition to producing spy satellites for the NSA, the company is in charge of “Project Groundbreaker,” a $5 billion, 10-year effort to upgrade the agency’s internal telephone and computer networks.

While Lockheed Martin may well be watching you at home — it’s my personal nominee for twenty-first-century “Big Brother” — it has also been involved in questionable activities abroad that go well beyond supplying weapons to regions in conflict. There were, of course, those interrogators it recruited for America’s offshore prison system from Guantanamo Bay to Afghanistan (and the charges of abuses that so naturally went with them), but the real scandal the company has been embroiled in involves overseeing an assassination program in Pakistan. Initially, it was billed as an information gathering operation using private companies to generate data the CIA and other U.S. intelligence agencies allegedly could not get on their own. Instead, the companies turned out to be supplying targeting information used by U.S. Army Special Forces troops to locate and kill suspected Taliban leaders.

The private firms involved were managed by Lockheed Martin under a $22 million contract from the U.S. Army. As Mark Mazetti of the New York Times has reported, there were just two small problems with the effort: “The American military is largely prohibited from operating in Pakistan. And under Pentagon rules, the army is not allowed to hire contractors for spying.” Much as in the Iran/Contra scandal of the 1980s, when Oliver North set up a network of shell companies to evade the laws against arming right-wing paramilitaries in Nicaragua, the Army used Lockheed Martin to do an end run around rules limiting U.S. military and intelligence activities in Pakistan. It should not, then, be too surprising that one of the people involved in the Lockheed-Martin-managed network was Duane “Dewey” Claridge, an ex-CIA man who had once been knee deep in the Iran/Contra affair.

A Twenty-First Century Big Brother

There has also been a softer side to Lockheed Martin’s foreign policy efforts. It has involved contracts for services that range from recruiting election monitors for Bosnia and the Ukraine and attempting to reform Liberia’s justice system to providing personnel involved in drafting the Afghan constitution. Most of these projects have been carried out by the company’s PAE unit, the successor to a formerly independent firm, Pacific Architects and Engineers, that made its fortune building and maintaining military bases during the Vietnam War.

However, the “soft power” side of Lockheed Martin’s operations (as described on its web site) may soon diminish substantially as the company has put PAE up for sale. Still, the revenues garnered from these activities will undoubtedly be more than offset by a new $5 billion, multi-year contract awarded by the U.S. Army to provide logistics support for U.S. Special Forces in dozens of countries.

Consider all this but a Lockheed Martin précis. A full accounting of its “shadow government” would fill volumes. After all, it’s the number-one contractor not only for the Pentagon, but also for the Department of Energy. It ranks number two for the Department of State, number three for the National Aeronautics and Space Administration, and number four for the Departments of Justice and Housing and Urban Development. Even listing the government and quasi-governmental agencies the company has contracts with is a daunting task, but here’s just a partial run-down: the Department of Agriculture, the Bureau of Land Management, the Census Bureau, the Coast Guard, the Department of Defense (including the Army, the Navy, the Marines, the Air Force and the Missile Defense Agency), the Department of Education, the Department of Energy, the Environmental Protection Agency, the Federal Aviation Administration, the Federal Bureau of Investigation, the Federal Technology Department, the Food and Drug Administration, the General Services Administration, the Geological Survey, the Department of Homeland Security, the Bureau of Indian Affairs, the Internal Revenue Service, the National Aeronautics and Space Administration, the National Institutes of Health, the Department of State, the Social Security Administration, the U.S. Customs Service, the U.S. Postal Service, the Department of Transportation, the Transportation Security Agency, and the Department of Veterans Affairs.

When President Eisenhower warned 50 years ago this month of the dangers of “unwarranted influence, whether sought or unsought, by the military-industrial complex,” he could never have dreamed that one for-profit weapons outfit would so fully insinuate itself into so many aspects of American life. Lockheed Martin has helped turn Eisenhower’s dismal mid-twentieth-century vision into a for-profit military-industrial-surveillance complex fit for the twenty-first century, one in which no governmental activity is now beyond its reach.

I feel safer already.
User avatar
Wombaticus Rex
Posts: 10614
Joined: Wed Nov 08, 2006 6:33 pm
Location: Vermontistan
Blog: View Blog (0)

Re: Data Mining & Intelligence Agencies

Postby Wombaticus Rex » Tue May 22, 2012 4:25 pm

Via: Reuters

(Reuters) - The National Security Agency is trying to expand U.S. cyber expertise needed for secret intelligence operations against adversaries on computer networks through a new cyber-ops program at selected universities.

The cyber-ops curriculum is geared to providing the basic education for jobs in intelligence, military and law enforcement that are so secret they will only be revealed to some students and faculty, who need to pass security clearance requirements, during special summer seminars offered by NSA.

It is not easy to find the right people for cyber operations because the slice of the hacker community that would make a quality cyber operator inside the government is only a sliver.

The "quality cyber operators" the NSA is looking for are few and far between, says Neal Ziring, technical director at the agency's Information Assurance Directorate.

"We're trying to create more of these, and yes they have to know some of the things that hackers know, they have to know a lot of other things too, which is why you really want a good university to create these people for you," Ziring told Reuters in an interview at NSA's headquarters in Maryland.

NSA has two main missions: to protect U.S. government computer networks and to collect foreign intelligence through electronic means like satellites and decode it.

Of 20 universities that applied, only four received this week the new designation of Centers of Academic Excellence in Cyber Operations: Dakota State University, Naval Postgraduate School, Northeastern University and University of Tulsa.

Out of 10 requirements, the two most lacking at many schools were courses on "reverse engineering" - or how to gain knowledge of a technology or product to reproduce it - and cellular communications and mobile technologies, NSA officials said.

"We found a lot of schools weren't emerging with the technology, weren't keeping up," said Captain Jill Newton, who leads NSA's cyber training and education programs.


NSA officials say the program, which is part of President Barack Obama's national initiative to improve cybersecurity through education, aims to prepare students for careers at the U.S. Cyber Command, the NSA's signals intelligence operations and law enforcement agencies investigating cyber crimes.

U.S. officials from the Obama administration and Congress have been banging the drums loudly about the need for greater cybersecurity, accusing China and Russia of hacking U.S. systems for economic gain.

"Right now you hear a lot of talk about foreign countries, China in particular, coming into our networks. They get in, they look around, they see what they might want, they send it home, and you don't know what else they've left behind," Dickie George, a former NSA official, said. "Why wouldn't we want to do the same thing? It's not a one-way game."

Many universities are now focused on web technologies such as how to write applications for the iPhone, which is not what is required for cyber operations to collect intelligence or defend the government's systems, NSA officials said.

That requires knowing "the guts, the internals of the operating systems, having to understand how the hardware actually works," said Steven LaFountain, a senior NSA official who guides academic programs.

Newton said a cyber operation might involve altering computer systems to work to one's advantage and doing that "without being seen or without it being obvious that I was changing the inner workings of the operating system."

"It could be very useful for a defender, so as you see your stuff being adjusted, corrupted, exploited, messed with, and being able to recognize when that is happening to you, to be able to better defend against it," she said.

About 15 years ago, there was a mindset that the computer system being compromised happened rarely and if the security was hardened that would be sufficient to secure it, but the security environment has changed, said Ziring, a computer scientist and the first non-mathematician in his position at NSA.

"What we've realized these days is that's hokum, that doesn't work any more, that systems are under attack constantly," Ziring said.

"For many systems, especially those that for mission reasons have to work in a very exposed space, being under some degree of compromise is sort of their new normal state."

That requires actively defending the systems by blocking and mitigating known problems and hunting for the unknown by looking for anomalies, Ziring said.


One mandatory requirement in the curriculum is covering legal and ethical issues so students understand the limits.

"We still found a lot of schools are still a little reluctant on how they characterize what they are teaching," LaFountain said.

"We are not asking them to teach kids how to break into systems, we're not asking them to teach that. And a lot of them have said they wouldn't teach that," he said. "We're just asking them to teach the hardcore fundamental science that we need students to have when they come to work here."

While the open education provides the basic knowledge, it is not until they arrive at the NSA that newly hired cyber operators get trained in their secret jobs.

"In our operational developmental organization, we would spend up to 12 months to give them the secret sauce, the tradecraft, the really deep technical training so that they could make themselves useful in doing what we need them to do, and that's with that technical underpinning," Newton said.

Ziring said it was important to figure out the next step in threat evolution so the technologies can be built to address it.

"The threat actor's action cycle is speeding up and getting shorter. The defender's cycle has to get shorter. So what technologies can we build that will help that?"
User avatar
Wombaticus Rex
Posts: 10614
Joined: Wed Nov 08, 2006 6:33 pm
Location: Vermontistan
Blog: View Blog (0)

Re: Data Mining & Intelligence Agencies

Postby Wombaticus Rex » Wed Jul 25, 2012 9:57 am

Congress to Examine Data Sellers

via: NYT - 07-25-12

In a move that could lay bare the inner workings of the consumer data industry, eight members of Congress have opened a sweeping investigation into data brokers — companies that collect, collate, analyze and sell billions of details annually about consumers’ offline, online and mobile activities for marketing and other purposes.

Representative Edward J. Markey, Democrat of Massachusetts, and Representative Joe L. Barton, Republican of Texas, co-chairmen of the Bipartisan Congressional Privacy Caucus, along with six other lawmakers, sent letters of inquiry on Tuesday afternoon to nine leading industry players. In the letters, the legislators requested extensive information about how the companies amass, refine, sell and share consumer data.

Data brokers often collect details about people’s financial, retail and recreational activities to help clients like airlines, automakers, banks, credit card issuers and retailers retain their best customers and woo new ones.

The letter’s recipients included marketing services firms like Acxiom and Epsilon; consumer reporting agencies like Experian and Equifax, which have separate credit reporting and consumer analytics divisions; Fair Isaac, now known as FICO, the credit scoring services company; and Intelius, a company that offers reverse phone look-up and background check services. The letter gave the companies three weeks to respond.

The Congressional inquiry heightens the scrutiny of a largely unregulated industry whose companies sell their services to third parties, rarely interacting directly with consumers.

In 2010, the Federal Trade Commission began its own investigation into the practices of more than a dozen data compilers. One of those companies, Spokeo, recently agreed to settle charges with the government that it had violated federal law by selling consumers’ personal data for employment screening. Enforcement actions against several other data brokers are pending, the agency said.

Now Mr. Markey says he wants the Congressional investigation to further expose data broker practices, saying some had the potential to affect people’s access to education, health care, employment or economic opportunities.

But Mr. Markey’s ultimate goal is to determine whether legislators should enact a law regulating the industry. Unlike consumer reporting agencies, which are required by federal law to show people their own credit reports and allow them to correct errors, information brokers are not currently required to show consumers information collected about them for marketing purposes.

“We have gone from an era of data keepers to this new era where data reapers are able to create very complex profiles of every American,” Mr. Markey said in a telephone interview.

He said he was particularly troubled by data broker programs that categorize individual consumers as desirable or undesirable sales prospects, often without their knowledge and consent, a practice that he said raised privacy concerns. “I’m hoping to ratchet up the transparency so we can foster a system of oversight and consumer control over their data.”

The privacy caucus’s letter was prompted by an article last month in The New York Times about Acxiom, based in Little Rock, Ark. Mr. Markey’s office gave The Times a copy of the letter.

Jennifer Barrett Glasgow, the chief privacy officer of Acxiom, said company executives had testified before Congress numerous times to inform legislators about the steps they take to protect consumers. “We are happy to provide whatever information we can to further inform interested parties,” she said.

Other industry representatives did not immediately respond to requests for comment.

The privacy caucus does not have subpoena power. But Mr. Markey said other industries, like cellphone carriers, had complied with his requests in the past. He said he expected similar cooperation from data brokers.

The letter asked each company to provide a list of all of its sources of data; a list of the specific kinds of consumer information, including ethnic, race or religious data, it collects; descriptions of the data collection methods used, like tracking of social network or mobile phone activity; explanations about each product and service the company has marketed to third parties since January 2009, and the type of data used in such products and services; details about whether any of the products or services are federally regulated; explanations about the security measures used to protect consumer data; as well as descriptions of the opt-out, data access, correction and deletion options the company offers consumers.

The direct marketing industry already offers consumers choices about managing marketing pitches sent through the mail. Digital marketers have a program for people who wish to opt out of receiving online ads tailored to their behavior.

But Mr. Markey said consumers also needed greater access to data collected about them so they could make more informed choices.

“You have to make sure that the values of the physical world accompany the transition to the virtual, digital world,” he said.

...days earlier, same publication....

Consumer Data, but Not for Consumers

via: NYT 07-21-12

I recently asked to see the information held about me by the Acxiom Corporation, a database marketing company that collects and sells details about consumers’ financial status, shopping and recreational activities to banks, retailers, automakers and other businesses. In investor presentations and interviews, Acxiom executives have said that the company — the subject of a Sunday Business article last month — has information on about 500 million active consumers worldwide, with about 1,500 data points per person. Acxiom also promotes a program for consumers who wish to see the information the company has on them.

As a former pharmaceuticals industry reporter who has researched all kinds of diseases, drugs and quack cures online, I wanted to learn, for one, whether Acxiom had pegged me as concerned about arthritis, diabetes or allergies. Acxiom also has a proprietary household classification system that places people in one of 70 socioeconomic categories, like “Downtown Dwellers” or “Flush Families,” and I hoped to discover the caste to which it had assigned me.

But after I filled out an online request form and sent a personal check for $5 to cover the processing fee, the company simply sent me a list of some of my previous residential addresses. In other words, rather than learning the details about myself that marketers might use to profile and judge me, I received information I knew already.

It turns out that Acxiom, based in Little Rock, Ark., furnishes consumers only with data related to risk management, like their own prison records, tax liens, bankruptcy filings and residential histories. For a corporate client, the company is able to match customers by name with, say, the social networks or Internet providers they use, but it does not offer consumers the same information about themselves.

Jennifer Barrett Glasgow, Acxiom’s chief privacy officer, said that the company kept consumer data in different databases and that its system was not designed to assemble all the information it had amassed on a single person.

“We do not have the capability to look up an individual’s data in the system,” Ms. Barrett Glasgow said. “We don’t have a search-by-name capability.”

Data brokers like Acxiom have developed advanced techniques to collect and collate information about consumers’ offline, online and mobile behavior. But they have been slow to develop innovative ways for consumers to gain access to the information that companies obtain, share and sell about them for marketing purposes.

Now federal regulators are pressuring data brokers to operate more transparently. In a report earlier this year, the Federal Trade Commission recommended that the industry set up a public Web portal that would display the names and contact information of data brokers, as well as describe consumers’ data access rights and other choices.

Julie Brill, a member of the Federal Trade Commission, said consumers should have access to all the details that data brokers collect on them, as well as any analyses that the companies sell about their behavior.

“I include in that not just the raw data, but also how that information has been analyzed to place the consumer into certain categories for marketing or other purposes,” she said. “I believe that giving consumers this kind of granularity will greatly increase consumer trust in the information flow process and will lead to more accurate marketing.”

At the moment, however, information brokers have wildly different policies. Acxiom lets people opt out of its marketing databases, while Epsilon, another marketing services firm, allows people to opt out of having their data rented to third parties. Epsilon says it will also furnish individuals, upon request, with general information about their past retail transactions — including the categories and years of purchase. But it does not include exact product or retailer names.

Andrew Frawley, the president of Epsilon, says his company has set up a task force to explore giving consumers greater access and choices.

“We agree in principle that more transparency is better,” he said.

But setting up a system for consumers to gain access to their own marketing data could be costly and technically challenging for data brokers, said Stuart Madnick, a professor of information technology at the Massachusetts Institute of Technology. Companies would have to develop security systems to verify a consumer’s identity and to ensure that no one else could have access to that individual’s record, he said. At the same time, they would have to be prepared to respond to people who questioned the accuracy of the records.

“How correct is the information they have and are disseminating on you?” Professor Madnick asked. “How do they know who is asking for it?”

Information security experts said data brokers might be reluctant to make public access easier lest consumers react by wanting to opt out of the data collection process altogether.

In early May, when I first looked at Acxiom’s Web site, the online request form that required consumers to submit their Social Security numbers and other sensitive personal information was not encrypted. (Ms. Barrett Glasgow said the company quickly identified and fixed a broken link that had caused the problem.) After I submitted my application, I didn’t hear back from the company for several weeks. Subsequently, I left a voice mail message on Acxiom’s consumer hot line. Nobody called back.

“It sounds like this form was not a high priority for them,” said Richard M. Smith, the founder of Boston Software Forensics, a consulting firm, and an expert on Internet security. Requiring consumers to mail in a personal check as part of the verification process, he added, seemed old-fashioned and cumbersome. “It’s so last century. Why are you making it so inconvenient?”

After I reported in the article last month that Acxiom had not responded to my data request, a company representative e-mailed me to verify that I was indeed the person who had requested her file. Then Acxiom e-mailed me an encrypted report containing a list of my previous residential addresses.

Several days later, Ms. Barrett Glasgow called to explain the delay in processing: Acxiom receives, on average, fewer than 100 requests a year from consumers, she said, and my check had “ended up on someone’s desk that was on vacation.” She said she would look into why company representatives hadn’t returned my voice mail message.

“We’ll try to take some action to improve and clean up the program,” she said. “We don’t want to make it hard to do, risky to do, or leave a bad impression in the individual’s mind.”

BUT I still wanted to know the financial, retail, travel, health and hobby details that Acxiom might have collected about me. So I e-mailed Ms. Barrett Glasgow last month, asking to see at least some of my data and to find out the socioeconomic category in which Acxiom had placed me.

Ms. Barrett Glasgow was on vacation last week and could not be reached for comment.

Commissioner Brill of the F.T.C. said she could not comment on specific companies. But she said the reluctance of the data broker industry to show consumers their own records reminded her of an earlier era, when consumer reporting agencies — companies that track and sell information about people’s credit histories — protested that it would be too expensive and time-consuming for them to show individuals the same reports that creditors could see. In 1996, Congress updated the Fair Credit Reporting Act of 1970, giving people greater access to the files that those agencies held about them. Today, consumers can easily gain access to their credit reports online.

“What the credit reporting industry did was change their point of view from client-oriented to consumer-oriented, and develop the tools and technology to allow consumers to see what’s in their reports and ensure it is accurate,” Ms. Brill said. “The data broker industry could do the exact same thing.”

...and the earlier piece on Axciom...

via: NYT 06-21-12

You for Sale: Mapping, and Sharing, the Consumer Genome

IT knows who you are. It knows where you live. It knows what you do.

It peers deeper into American life than the F.B.I. or the I.R.S., or those prying digital eyes at Facebook and Google. If you are an American adult, the odds are that it knows things like your age, race, sex, weight, height, marital status, education level, politics, buying habits, household health worries, vacation dreams — and on and on.

Right now in Conway, Ark., north of Little Rock, more than 23,000 computer servers are collecting, collating and analyzing consumer data for a company that, unlike Silicon Valley’s marquee names, rarely makes headlines. It’s called the Acxiom Corporation, and it’s the quiet giant of a multibillion-dollar industry known as database marketing.

Few consumers have ever heard of Acxiom. But analysts say it has amassed the world’s largest commercial database on consumers — and that it wants to know much, much more. Its servers process more than 50 trillion data “transactions” a year. Company executives have said its database contains information about 500 million active consumers worldwide, with about 1,500 data points per person. That includes a majority of adults in the United States.

Such large-scale data mining and analytics — based on information available in public records, consumer surveys and the like — are perfectly legal. Acxiom’s customers have included big banks like Wells Fargo and HSBC, investment services like E*Trade, automakers like Toyota and Ford, department stores like Macy’s — just about any major company looking for insight into its customers.

For Acxiom, based in Little Rock, the setup is lucrative. It posted profit of $77.26 million in its latest fiscal year, on sales of $1.13 billion.

But such profits carry a cost for consumers. Federal authorities say current laws may not be equipped to handle the rapid expansion of an industry whose players often collect and sell sensitive financial and health information yet are nearly invisible to the public. In essence, it’s as if the ore of our data-driven lives were being mined, refined and sold to the highest bidder, usually without our knowledge — by companies that most people rarely even know exist.

Julie Brill, a member of the Federal Trade Commission, says she would like data brokers in general to tell the public about the data they collect, how they collect it, whom they share it with and how it is used. “If someone is listed as diabetic or pregnant, what is happening with this information? Where is the information going?” she asks. “We need to figure out what the rules should be as a society.”

Although Acxiom employs a chief privacy officer, Jennifer Barrett Glasgow, she and other executives declined requests to be interviewed for this article, said Ines Rodriguez Gutzmer, director of corporate communications.

In March, however, Ms. Barrett Glasgow endorsed increased industry openness. “It’s not an unreasonable request to have more transparency among data brokers,” she said in an interview with The New York Times. In marketing materials, Acxiom promotes itself as “a global thought leader in addressing consumer privacy issues and earning the public trust.”

But, in interviews, security experts and consumer advocates paint a portrait of a company with practices that privilege corporate clients’ interests over those of consumers and contradict the company’s stance on transparency. Acxiom’s marketing materials, for example, promote a special security system for clients and associates to encrypt the data they send. Yet cybersecurity experts who examined Acxiom’s Web site for The Times found basic security lapses on an online form for consumers seeking access to their own profiles. (Acxiom says it has fixed the broken link that caused the problem.)

In a fast-changing digital economy, Acxiom is developing even more advanced techniques to mine and refine data. It has recruited talent from Microsoft, Google, and Myspace and is using a powerful, multiplatform approach to predicting consumer behavior that could raise its standing among investors and clients.

Of course, digital marketers already customize pitches to users, based on their past activities. Just think of “cookies,” bits of computer code placed on browsers to keep track of online activity. But Acxiom, analysts say, is pursuing far more comprehensive techniques in an effort to influence consumer decisions. It is integrating what it knows about our offline, online and even mobile selves, creating in-depth behavior portraits in pixilated detail. Its executives have called this approach a “360-degree view” on consumers.

“There’s a lot of players in the digital space trying the same thing,” says Mark Zgutowicz, a Piper Jaffray analyst. “But Acxiom’s advantage is they have a database of offline information that they have been collecting for 40 years and can leverage that expertise in the digital world.”

Yet some prominent privacy advocates worry that such techniques could lead to a new era of consumer profiling.

Jeffrey Chester, executive director of the Center for Digital Democracy, a nonprofit group in Washington, says: “It is Big Brother in Arkansas.”

SCOTT HUGHES, an up-and-coming small-business owner and Facebook denizen, is Acxiom’s ideal consumer. Indeed, it created him.

Mr. Hughes is a fictional character who appeared in an Acxiom investor presentation in 2010. A frequent shopper, he was designed to show the power of Acxiom’s multichannel approach.

In the presentation, he logs on to Facebook and sees that his friend Ella has just become a fan of Bryce Computers, an imaginary electronics retailer and Acxiom client. Ella’s update prompts Mr. Hughes to check out Bryce’s fan page and do some digital window-shopping for a fast inkjet printer.

Such browsing seems innocuous — hardly data mining. But it cues an Acxiom system designed to recognize consumers, remember their actions, classify their behaviors and influence them with tailored marketing.

When Mr. Hughes follows a link to Bryce’s retail site, for example, the system recognizes him from his Facebook activity and shows him a printer to match his interest. He registers on the site, but doesn’t buy the printer right away, so the system tracks him online. Lo and behold, the next morning, while he scans baseball news on, an ad for the printer pops up again.

That evening, he returns to the Bryce site where, the presentation says, “he is instantly recognized” as having registered. It then offers a sweeter deal: a $10 rebate and free shipping.

It’s not a random offer. Acxiom has its own classification system, PersonicX, which assigns consumers to one of 70 detailed socioeconomic clusters and markets to them accordingly. In this situation, it pegs Mr. Hughes as a “savvy single” — meaning he’s in a cluster of mobile, upper-middle-class people who do their banking online, attend pro sports events, are sensitive to prices — and respond to free-shipping offers.

Correctly typecast, Mr. Hughes buys the printer.

But the multichannel system of Acxiom and its online partners is just revving up. Later, it sends him coupons for ink and paper, to be redeemed via his cellphone, and a personalized snail-mail postcard suggesting that he donate his old printer to a nearby school.

Analysts say companies design these sophisticated ecosystems to prompt consumers to volunteer enough personal data — like their names, e-mail addresses and mobile numbers — so that marketers can offer them customized appeals any time, anywhere.

Still, there is a fine line between customization and stalking. While many people welcome the convenience of personalized offers, others may see the surveillance engines behind them as intrusive or even manipulative.

“If you look at it in cold terms, it seems like they are really out to trick the customer,” says Dave Frankland, the research director for customer intelligence at Forrester Research. “But they are actually in the business of helping marketers make sure that the right people are getting offers they are interested in and therefore establish a relationship with the company.”

DECADES before the Internet as we know it, a businessman named Charles Ward planted the seeds of Acxiom. It was 1969, and Mr. Ward started a data processing company in Conway called Demographics Inc., in part to help the Democratic Party reach voters. In a time when Madison Avenue was deploying one-size-fits-all national ad campaigns, Demographics and its lone computer used public phone books to compile lists for direct mailing of campaign material.

Today, Acxiom maintains its own database on about 190 million individuals and 126 million households in the United States. Separately, it manages customer databases for or works with 47 of the Fortune 100 companies. It also worked with the government after the September 2001 terrorist attacks, providing information about 11 of the 19 hijackers.

To beef up its digital services, Acxiom recently mounted an aggressive hiring campaign. Last July, it named Scott E. Howe, a former corporate vice president for Microsoft’s advertising business group, as C.E.O. Last month, it hired Phil Mui, formerly group product manager for Google Analytics, as its chief product and engineering officer.

In interviews, Mr. Howe has laid out a vision of Acxiom as a new-millennium “data refinery” rather than a data miner. That description posits Acxiom as a nimble provider of customer analytics services, able to compete with Facebook and Google, rather than as a stealth engine of consumer espionage.

Still, the more that information brokers mine powerful consumer data, the more they become attractive targets for hackers — and draw scrutiny from consumer advocates.

This year, Advertising Age ranked Epsilon, another database marketing firm, as the biggest advertising agency in the United States, with Acxiom second. Most people know Epsilon, if they know it at all, because it experienced a major security breach last year, exposing the e-mail addresses of millions of customers of Citibank, JPMorgan Chase, Target, Walgreens and others. In 2003, Acxiom had its own security breaches.

But privacy advocates say they are more troubled by data brokers’ ranking systems, which classify some people as high-value prospects, to be offered marketing deals and discounts regularly, while dismissing others as low-value — known in industry slang as “waste.”

Exclusion from a vacation offer may not matter much, says Pam Dixon, the executive director of the World Privacy Forum, a nonprofit group in San Diego, but if marketing algorithms judge certain people as not worthy of receiving promotions for higher education or health services, they could have a serious impact.

“Over time, that can really turn into a mountain of pathways not offered, not seen and not known about,” Ms. Dixon says.

Until now, database marketers operated largely out of the public eye. Unlike consumer reporting agencies that sell sensitive financial information about people for credit or employment purposes, database marketers aren’t required by law to show consumers their own reports and allow them to correct errors. That may be about to change. This year, the F.T.C. published a report calling for greater transparency among data brokers and asking Congress to give consumers the right to access information these firms hold about them.

ACXIOM’S Consumer Data Products Catalog offers hundreds of details — called “elements” — that corporate clients can buy about individuals or households, to augment their own marketing databases. Companies can buy data to pinpoint households that are concerned, say, about allergies, diabetes or “senior needs.” Also for sale is information on sizes of home loans and household incomes.

Clients generally buy this data because they want to hold on to their best customers or find new ones — or both.

A bank that wants to sell its best customers additional services, for example, might buy details about those customers’ social media, Web and mobile habits to identify more efficient ways to market to them. Or, says Mr. Frankland at Forrester, a sporting goods chain whose best customers are 25- to 34-year-old men living near mountains or beaches could buy a list of a million other people with the same characteristics. The retailer could hire Acxiom, he says, to manage a campaign aimed at that new group, testing how factors like consumers’ locations or sports preferences affect responses.

But the catalog also offers delicate information that has set off alarm bells among some privacy advocates, who worry about the potential for misuse by third parties that could take aim at vulnerable groups. Such information includes consumers’ interests — derived, the catalog says, “from actual purchases and self-reported surveys” — like “Christian families,” “Dieting/Weight Loss,” “Gaming-Casino,” “Money Seekers” and “Smoking/Tobacco.” Acxiom also sells data about an individual’s race, ethnicity and country of origin. “Our Race model,” the catalog says, “provides information on the major racial category: Caucasians, Hispanics, African-Americans, or Asians.” Competing companies sell similar data.

Acxiom’s data about race or ethnicity is “used for engaging those communities for marketing purposes,” said Ms. Barrett Glasgow, the privacy officer, in an e-mail response to questions.

There may be a legitimate commercial need for some businesses, like ethnic restaurants, to know the race or ethnicity of consumers, says Joel R. Reidenberg, a privacy expert and a professor at the Fordham Law School.

“At the same time, this is ethnic profiling,” he says. “The people on this list, they are being sold based on their ethnic stereotypes. There is a very strong citizen’s right to have a veto over the commodification of their profile.”

He says the sale of such data is troubling because race coding may be incorrect. And even if a data broker has correct information, a person may not want to be marketed to based on race.

“DO you really know your customers?” Acxiom asks in marketing materials for its shopper recognition system, a program that uses ZIP codes to help retailers confirm consumers’ identities — without asking their permission.

“Simply asking for name and address information poses many challenges: transcription errors, increased checkout time and, worse yet, losing customers who feel that you’re invading their privacy,” Acxiom’s fact sheet explains. In its system, a store clerk need only “capture the shopper’s name from a check or third-party credit card at the point of sale and then ask for the shopper’s ZIP code or telephone number.” With that data Acxiom can identify shoppers within a 10 percent margin of error, it says, enabling stores to reward their best customers with special offers. Other companies offer similar services.

“This is a direct way of circumventing people’s concerns about privacy,” says Mr. Chester of the Center for Digital Democracy.

Ms. Barrett Glasgow of Acxiom says that its program is a “standard practice” among retailers, but that the company encourages its clients to report consumers who wish to opt out.

Acxiom has positioned itself as an industry leader in data privacy, but some of its practices seem to undermine that image. It created the position of chief privacy officer in 1991, well ahead of its rivals. It even offers an online request form, promoted as an easy way for consumers to access information Acxiom collects about them.

But the process turned out to be not so user-friendly for a reporter for The Times.

In early May, the reporter decided to request her record from Acxiom, as any consumer might. Before submitting a Social Security number and other personal information, however, she asked for advice from a cybersecurity expert at The Times. The expert examined Acxiom’s Web site and immediately noticed that the online form did not employ a standard encryption protocol — called https — used by sites like Amazon and American Express. When the expert tested the form, using software that captures data sent over the Web, he could clearly see that the sample Social Security number he had submitted had not been encrypted. At that point, the reporter was advised not to request her file, given the risk that the process might expose her personal information.

Later in May, Ashkan Soltani, an independent security researcher and former technologist in identity protection at the F.T.C., also examined Acxiom’s site and came to the same conclusion. “Parts of the site for corporate clients are encrypted,” he says. “But for consumers, who this information is about and who stand the most to lose from data collection, they don’t provide security.”

Ms. Barrett Glasgow says that the form has always been encrypted with https but that on May 11, its security monitoring system detected a “broken redirect link” that allowed unencrypted access. Since then, she says, Acxiom has fixed the link and determined that no unauthorized person had gained access to information sent using the form.

On May 25, the reporter submitted an online request to Acxiom for her file, along with a personal check, sent by Express Mail, for the $5 processing fee. Three weeks later, no response had arrived.

Regulators at the F.T.C. declined to comment on the practices of individual companies. But Jon Leibowitz, the commission chairman, said consumers should have the right to see and correct personal details about them collected and sold by data aggregators.

After all, he said, “they are the unseen cyberazzi who collect information on all of us.”
User avatar
Wombaticus Rex
Posts: 10614
Joined: Wed Nov 08, 2006 6:33 pm
Location: Vermontistan
Blog: View Blog (0)


Return to Data & Research Compilations

Who is online

Users browsing this forum: No registered users and 4 guests