We NSA Now

[General Patton]

Title changed for personal amusement.

Palantir needs to step up to the plate and create open source tools that empower citizens to take advantage of information to root out corruption and stop terrorism. More on this later.

http://www.kurzweilai.net/intelligence-agencies-turn-to-crowdsourcing
Research firm Applied Research Associates has just launched a website, Global Crowd Intelligence, that invites the public to sign up and try their hand at intelligence forecasting, BBC Future reports.

The website is part of an effort called Aggregative Contingent Estimation, sponsored by the Intelligence Advanced Research Projects Activity (Iarpa), to understand the potential benefits of crowdsourcing for predicting future events by making forecasting more like a game of spy versus spy.

The new website rewards players who successfully forecast future events by giving them privileged access to certain “missions,” and also allowing them to collect reputation points, which can then be used for online bragging rights. When contributors enter the new site, they start off as junior analysts, but eventually progress to higher levels, allowing them to work on privileged missions.

The idea of crowdsourcing geopolitical forecasting is increasing in popularity, and not just for spies. Wikistrat, a private company touted as “the world’s first massively multiplayer online consultancy,” was founded in 2002, and is using crowdsourcing to generate scenarios about future geopolitical events. It recently released a report based on a crowdsourced simulation looking at China’s future naval powers.

Warnaar says that Wikistrat’s approach appears to rely on developing “what-if scenarios,” rather than attaching a probability to a specific event happening, which is the goal of the Iarpa project.

http://bluehost.globalcrowd.com/gci/index.php

http://www.phibetaiota.net/2011/09/paul-fernhout-open-letter-to-the-intelligence-advanced-programs-research-agency-iarpa/
Dear IARPA staff-

The greatest threat facing the USA is the irony inherent in our current defense posture, like for example planning to use nuclear energy embodied in missiles to fight over oil fields that nuclear energy could replace. This irony arises in part because the USA’s current security logic is still based on essentially 19th century and earlier (second millennium) thinking that becomes inappropriate applied to 21st century (third millennium) technological threats and opportunities. That situation represents a systematic intelligence failure of the highest magnitude. There remains time to correct this failure, but time grows short as various exponential trends continue.

To address that pervasive threat from unrecognized irony, it would help to re-envision the CIA as a non-ironic post-scarcity institution. Then the CIA could help others (including in the White House) make more informed decisions to move past this irony as well.

A first step towards that could be for IARPA to support better free software tools for “crowdsourced” public intelligence work involving using a social semantic desktop for sensemaking about open source data and building related open public action plans from that data to make local communities healthier, happier, more intrinsically secure, and also more mutually secure. Secure, healthy, prosperous, and happy local (and virtual) communities then can form together a secure, healthy, prosperous, and happy nation and planet in a non-ironic way. Details on that idea are publicly posted by me here in the form of a Proposal Abstract to the IARPA Incisive Analysis solicitation: “Social Semantic Desktop for Sensemaking on Threats and Opportunities”

And, as will be mentioned below, the greatest threat facing specific CIA staff is heart disease and cancer. These two threats, global and personal, are actually connected in an odd sort of way, both reflecting past adaptive behavior which is no longer very adaptive under new conditions resulting from technological change. The current economic crisis the USA is facing also results from unrecognized underlying exponential trends.
...
“Military robots like drones are ironic because they are created essentially to force humans to work like robots in an industrialized social order. Why not just create industrial robots to do the work instead?

Nuclear weapons are ironic because they are about using space age systems to fight over oil and land. Why not just use advanced materials as found in nuclear missiles to make renewable energy sources (like windmills or solar panels) to replace oil, or why not use rocketry to move into space by building space habitats for more land?

Biological weapons like genetically-engineered plagues are ironic because they are about using advanced life-altering biotechnology to fight over which old-fashioned humans get to occupy the planet. Why not just use advanced biotech to let people pick their skin color, or to create living arkologies and agricultural abundance for everyone everywhere?

These militaristic socio-economic ironies would be hilarious if they were not so deadly serious. …

Likewise, even United States three-letter agencies like the NSA and the CIA, as well as their foreign counterparts, are becoming ironic institutions in many ways. Despite probably having more computing power per square foot than any other place in the world, they seem not to have thought much about the implications of all that computer power and organized information to transform the world into a place of abundance for all. Cheap computing makes possible just about cheap everything else, as does the ability to make better designs through shared computing. …

There is a fundamental mismatch between 21st century reality and 20th century security thinking. Those “security” agencies are using those tools of abundance, cooperation, and sharing mainly from a mindset of scarcity, competition, and secrecy. Given the power of 21st century technology as an amplifier (including as weapons of mass destruction), a scarcity-based approach to using such technology ultimately is just making us all insecure. Such powerful technologies of abundance, designed, organized, and used from a mindset of scarcity could well ironically doom us all whether through military robots, nukes, plagues, propaganda, or whatever else…

[Wombaticus Rex]

Robert David Steele's favorite conversation subject! Reminds me of the good old says when Hugh would call me a sad dupe for buying Steele's limited hangout psyops recruitment material, or, you know, whatever:



http://en.wikipedia.org/wiki/Robert_David_Steele
http://www.oss.net/

...the United States of America finds itself with a military optimized for force-on-force confrontations between nation-states, and a national intelligence community optimized for stealing secrets through technical means, with an extremely narrow range of focus and almost no flexibility.

...the major finding of the National Imagery and Mapping Agency Commission Report of December 1999, concluding that we have been spending tens of billions of dollars on esoteric collection systems, without a commensurate investment in information technologies for Tasking, Processing, Exploitation, and Dissemination (TPED).

The vast majority of the information needed by DoD is unclassified, generally not online nor subject to deep web data mining, generally not in English, and generally not readily identifiable unless a witting willing volunteer from the owning organization “offers it up.” Roughly 90% of the information we need is unclassified, and available only from organizations that will not share that information with secret agencies. Civil affairs is the model to use.

The US focus is on sanitizing classified information of marginal value, or stone-walling the NGOs completely after first getting everything the NGOs have to offer. A much more productive approach is to jointly establish shared requirements, to share what each knows via overt means, and to gradually expand the circle of participants in an overt network so that more and more distinct entities are both contributing original information, and drawing upon the aggregate information, to which DoD can add considerable value by applying generic sense-making tools.

Deep web content acquisition such as is not available from commercial aggregators is a major aspect of the open source information challenge, but it is not the only source. Private databases, especially NGO databases, niche and mainstream publications, gray literature, sermons, and street talk as well as new knowledge created by subject matter experts on demand, are part of this larger global pool that must be addressed in at least 33 languages all the time, and up to 185 languages some of the time.

Modern IO is the seed crystal for a total transformation of the American way of war, a new American way that practices information peacekeeping, and reflects a new commitment by America to stabilize the world intelligently rather than violently. It is a holistic mission that must be accomplished by the J-3 using a civil affairs mind-set, with the J-2 limited to internal validation and support. There are not enough guns on the planet to force our will upon other or to protect our quality of life for future generations. IO is the new way of war, and of peace.

In an era when information converted into intelligence and knowledge is a substitute for wealth, violence, energy, water, and everything else, only the: the United States of America has the power to execute this practical vision. DoD is the catalyst for its achievement.

Edit: This overview / data dump just went up on Phi Beta Iota - http://www.phibetaiota.net/2012/09/2012 ... 2009-2013/

[General Patton]

I was in the process of writing an email to Nathan Gettings, Palantir CTO when I found he had already made a tool for analyzing data.gov, one of the sources I was going to mention:
https://analyzethe.us/

Once again I'm behind the curve

Currently in the process of using it and the demo version of Palantir Government. I bet I can still find a reason to bug him though.

[General Patton]

Amateur Gestapo Software Time:

https://www.cia.gov/library/center-for- ... art11.html
Analysis of competing hypotheses, sometimes abbreviated ACH, is a tool to aid judgment on important issues requiring careful weighing of alternative explanations or conclusions. It helps an analyst overcome, or at least minimize, some of the cognitive limitations that make prescient intelligence analysis so difficult to achieve.

ACH is an eight-step procedure grounded in basic insights from cognitive psychology, decision analysis, and the scientific method. It is a surprisingly effective, proven process that helps analysts avoid common analytic pitfalls. Because of its thoroughness, it is particularly appropriate for controversial issues when analysts want to leave an audit trail to show what they considered and how they arrived at their judgment.

http://en.wikipedia.org/wiki/Analysis_o ... Hypotheses
Weaknesses

The process to create an ACH is time consuming. The ACH matrix can be problematic when analyzing a complex project. It can be cumbersome for an analyst to manage a large database with multiple pieces of evidence.
Especially in intelligence, both governmental and business, analysts must always be aware that the opponent(s) is intelligent and may be generating information intended to deceive.[3][4] Since deception often is the result of a cognitive trap, Elsaesser and Stech use state-based hierarchical plan recognition (see abductive reasoning) to generate causal explanations of observations. The resulting hypotheses are converted to a dynamic Bayesian network and value of information analysis is employed to isolate assumptions implicit in the evaluation of paths in, or conclusions of, particular hypotheses. As evidence in the form of observations of states or assumptions is observed, they can become the subject of separate validation. Should an assumption or necessary state be negated, hypotheses depending on it are rejected. This is a form of root cause analysis.
Evidence also presents a problem if it is unreliable. The evidence used in the matrix is static and therefore it can be a snapshot in time.

Free/OpenSource/Full of kitties and rainbows:
http://competinghypotheses.org/

Analysis of Competing Hypotheses (ACH) is a simple model for how to think about a complex problem when the available information is incomplete or ambiguous, as typically happens in intelligence analysis. The software downloadable here takes an analyst through a process for making a well-reasoned, analytical judgment. It is particularly useful for issues that require careful weighing of alternative explanations of what has happened, is happening, or is likely to happen in the future. It helps the analyst overcome, or at least minimize, some of the cognitive limitations that make prescient intelligence analysis so difficult. ACH is grounded in basic insights from cognitive psychology, decision analysis, and the scientific method. It helps analysts protect themselves from avoidable error, and improves their chances of making a correct judgment.
http://www2.parc.com/istl/projects/ach/ach.html

RapidMiner - About 6% of data miners use it - Can use R as an extension with a GUI
http://rapid-i.com/content/view/281/225/

R - 46% of data miners use this - in some ways better than commercial software - I'm not sure what the limit of this software is, incredibly powerful
http://www.r-project.org/



Network Mapping

Multiple tools - Finding sets of key players in a network - Cultural domain analysis - Network visualization - Software for analyzing ego-network data - Software package for visualizing social networks
http://www.analytictech.com/products.htm

NodeXL is a free, open-source template for Microsoft® Excel® 2007 and 2010 that makes it easy to explore network graphs. With NodeXL, you can enter a network edge list in a worksheet, click a button and see your graph, all in the familiar environment of the Excel window.
http://nodexl.codeplex.com/

Stanford Network Analysis Platform (SNAP) is a general purpose, high performance system for analysis and manipulation of large networks. Graphs consists of nodes and directed/undirected/multiple edges between the graph nodes. Networks are graphs with data on nodes and/or edges of the network.
http://snap.stanford.edu/snap/index.html

NetworkX is a Python language software package for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks.
http://networkx.lanl.gov/

Social Networks Visualizer (SocNetV) is a flexible and user-friendly tool for the analysis and visualization of Social Networks. It lets you construct networks (mathematical graphs) with a few clicks on a virtual canvas or load networks of various formats (GraphViz, GraphML, Adjacency, Pajek, UCINET, etc) and modify them to suit your needs. SocNetV also offers a built-in web crawler, allowing you to automatically create networks from all links found in a given initial URL.
http://socnetv.sourceforge.net/

*ORA is a dynamic meta-network assessment and analysis tool developed by CASOS at Carnegie Mellon. It contains hundreds of social network, dynamic network metrics, trail metrics, procedures for grouping nodes, identifying local patterns, comparing and contrasting networks, groups, and individuals from a dynamic meta-network perspective. *ORA has been used to examine how networks change through space and time, contains procedures for moving back and forth between trail data (e.g. who was where when) and network data (who is connected to whom, who is connected to where …), and has a variety of geo-spatial network metrics, and change detection techniques. *ORA can handle multi-mode, multi-plex, multi-level networks. It can identify key players, groups and vulnerabilities, model network changes over time, and perform COA analysis. It has been tested with large networks (106 nodes per 5 entity classes).Distance based, algorithmic, and statistical procedures for comparing and contrasting networks are part of this toolkit.
http://www.casos.cs.cmu.edu/projects/ora/

SUBDUE is a graph-based knowledge discovery system that finds structural, relational patterns in data representing entities and relationships. SUBDUE represents data using a labeled, directed graph in which entities are represented by labeled vertices or subgraphs, and relationships are represented by labeled edges between the entities. SUBDUE uses the minimum description length (MDL) principle to identify patterns that minimize the number of bits needed to describe the input graph after being compressed by the pattern. SUBDUE can perform several learning tasks, including unsupervised learning, supervised learning, clustering and graph grammar learning. SUBDUE has been successfully applied in a number of areas, including bioinformatics, web structure mining, counter-terrorism, social network analysis, aviation and geology.
http://ailab.wsu.edu/subdue/

A range of tools for social network analysis, including node and graph-level indices, structural distance and covariance methods, structural equivalence detection, p* modeling, random graph generation, and 2D/3D network visualization.(R based)
http://cran.us.r-project.org/web/packag ... index.html

statnet is a suite of software packages for network analysis that implement recent advances in the statistical modeling of networks. The analytic framework is based on Exponential family Random Graph Models (ergm). statnet provides a comprehensive framework for ergm-based network modeling, including tools for model estimation, model evaluation, model-based network simulation, and network visualization. This broad functionality is powered by a central Markov chain Monte Carlo (MCMC) algorithm. (Requires R)
http://statnetproject.org/

Tulip is an information visualization framework dedicated to the analysis and visualization of relational data. Tulip aims to provide the developer with a complete library, supporting the design of interactive information visualization applications for relational data that can be tailored to the problems he or she is addressing.
http://tulip.labri.fr/TulipDrupal/



GraphChi is a spin-off of the GraphLab ( http://www.graphlab.org ) -project from the Carnegie Mellon University. It is based on research by Aapo Kyrola ( http://www.cs.cmu.edu/~akyrola/) and his advisors.

GraphChi can run very large graph computations on just a single machine, by using a novel algorithm for processing the graph from disk (SSD or hard drive). Programs for GraphChi are written in the vertex-centric model, proposed by GraphLab and Google's Pregel. GraphChi runs vertex-centric programs asynchronously (i.e changes written to edges are immediately visible to subsequent computation), and in parallel. GraphChi also supports streaming graph updates and removal of edges from the graph. Section 'Performance' contains some examples of applications implemented for GraphChi and their running times on GraphChi.

The promise of GraphChi is to bring web-scale graph computation, such as analysis of social networks, available to anyone with a modern laptop. It saves you from the hassle and costs of working with a distributed cluster or cloud services. We find it much easier to debug applications on a single computer than trying to understand how a distributed algorithm is executed.

In some cases GraphChi can solve bigger problems in reasonable time than many other available distributed frameworks. GraphChi also runs efficiently on servers with plenty of memory, and can use multiple disks in parallel by striping the data.
https://code.google.com/p/graphchi/

Web Based Stuff:

Play Gestapo from the comfort of your living room:
http://littlesis.org/
http://theyrule.net/

Search Professionals by Name, Company or Title, painfully verbose compared to the above 2 tools
http://www.marketvisual.com/

A tool that uses Palantir Government:
https://analyzethe.us

connected with the following datasets:
http://www.usaspending.gov
http://www.data.gov/
http://www.opensecrets.org/
https://www.epls.gov/
and some misc. others

Broad list of search engines:
http://en.wikipedia.org/wiki/List_of_search_engines
&
http://www.wired.com/business/2009/06/c ... chengines/

Database Listings:
http://www.forecastingprinciples.com/in ... &Itemid=18
http://datamarket.com/
http://www.datawrangling.com/some-datas ... on-the-web



Analytic Methods:

THIS BLOG IS PART OF CLASS PROJECT TO EXPLORE VARIOUS ANALYTIC TECHNIQUES USED BY MODERN INTELLIGENCE ANALYSTS (DELICIOUS ALL CAPS)
http://advat.blogspot.co.uk/

Morphological Analysis - A general method for non-quantified modeling
http://www.swemorph.com/pdf/gma.pdf

Modeling Complex Socio-Technical Systems using Morphological Analysis
http://www.swemorph.com/pdf/it-webart.pdf

Top 5 Intelligence Analysis Methods: Analysis Of Competing Hypotheses
http://sourcesandmethods.blogspot.com/2 ... ds_19.html
(the author scores a 4.4 of 5 on http://www.ratemyprofessors.com/ShowRat ... tid=545372 , 2.4 on the easiness scale)

http://en.wikipedia.org/wiki/Intelligence_analysis#Analytic_tradecraft
Many new analysts find that getting started is the hardest part of their job. Stating the objective, from the consumer's standpoint, is an excellent starting point. If the analyst cannot define the consumer and his needs, how is it possible to provide analysis that complements what the consumer already knows.

"Ambassador Robert D. Blackwill ... seized the attention of the class of some 30 [intelligence community managers] by asserting that as a policy official he never read ... analytic papers. Why? "Because they were nonadhesive." As Blackwill explained, they were written by people who did not know what he was trying to do and, so, could not help him get it done:
"When I was working at State on European affairs, for example, on certain issues I was the Secretary of State. DI analysts did not know that--that I was one of a handful of key decision makers on some very important matters...."

More charitably, he now characterizes his early periods of service at the NSC Staff and in State Department bureaus as ones of "mutual ignorance"

"DI analysts did not have the foggiest notion of what I did; and I did not have a clue as to what they could or should do."[6]
Blackwill explained how he used his time efficiently, which rarely involved reading general CIA reports. "I read a lot. Much of it was press. You have to know how issues are coming across politically to get your job done. Also, cables from overseas for preparing agendas for meetings and sending and receiving messages from my counterparts in foreign governments. Countless versions of policy drafts from those competing for the President's blessing. And dozens of phone calls. Many are a waste of time but have to be answered, again, for policy and political reasons.

"One more minute, please, on what I did not find useful. This is important. My job description called for me to help prepare the President for making policy decisions, including at meetings with foreign counterparts and other officials.... Do you think that after I have spent long weeks shaping the agenda, I have to be told a day or two before the German foreign minister visits Washington why he is coming?"

[General Patton]

Messing around with the Palantir Government suite right now. You have the ability to import/export data, filter access, set up collaborative teams and access to the open archives of the US Gov and some non profits. There are two tiers of users, novice users and power users:

Workspace Operations
Restrictions for Novice Users
Importing data

Novice users can only import data that is correctly mapped to the deployment ontology. Power users are exempt from this restriction.

The maximum number of rows in structured data sources that a Novice user can imported at one time is restricted by the NOVICE_IMPORT_STRUCTURED_MAX_ROWS system property. The default value for this property is 1000.

The maximum size of unstructured data sources that can be imported by a Novice user at one time is restricted by the NOVICE_IMPORT_UNSTRUCTURED_MAX_SIZE_IN_MB system property. The default value for this property is 5 megabytes.
Tagging text

The maximum number of tags that a Novice user can create using the Find and Tag helper is restricted by the system property NOVICE_FIND_AND_TAG_MAX_TAGS. The default setting for this property is 50.

Novice users cannot access the Tag All Occurrences in Tab option in the Browser’s Tag As dialog.
SearchAround search templates

Novice users cannot import SearchAround Templates from XML files.

Novice users cannot publish SearchAround templates for use by the entire deployment, and cannot edit published templates.
All other SearchAround features remain available.
Resolving Nexus Peering data conflicts
The Pending Changes application is available only in the Palantir Enterprise Platform, and is only accessible to Workspace users who belong to the Nexus Peering Data Managers user group.
Nexus Peering Data Managers use the Pending Changes application to check for, analyze, and resolve data conflicts that are not automatically resolved when a local nexus is synchronized with a peered nexus.
Deleting objects

Novice users cannot delete published objects.

Novice users cannot delete objects created or changed by other users.
Resolving objects

The maximum number of objects that Novice users can resolve together at one time is restricted by the NOVICE_RESOLVE_MAX_OBJECTS system property. This restriction does not apply to objects resolved by using existing object resolution suites in the Object Resolution Wizard or during data import.

Novice users may use the Object Resolution Wizard only when using existing object resolution suites. Novice users cannot perform Manual Object Resolution, and cannot record new resolution criteria as an Object Resolution Suite.
To learn more, see Resolving and Unresolving Objects in Workspace: Beyond the Basics.
Map application restrictions
All map metadata tools in the Layers helper are restricted.
Novice users cannot access features that allow sorting of layers by metadata, coloring by metadata, or the creation of new metadata. All other Layer helper functions remain available.

In case you didn't get what I just said, you have access the same tools the FBI and CIA use, except some minor limitations and no access to classified documents. If you have access to Wolfram Alpha/Mathematica and can google for history on your topic of interest then most of the classified files will become redundant.



What about data mining on a budget?

Consider relying on a GPU(s). A CPU is designed to be multitasker that can quickly switch between actions, whereas a Graphical Processing Unit(GPU) is designed to do the same calculations repetitively while giving large increases in performance. The stacks in the listed papers, while giving exponentially higher speeds, did not use modern designs or graphics cards, which hindered them from running even faster.

http://www.azintablog.com/2010/10/16/gp ... ta-mining/
The GPU (Graphics Prossessing Unit) is changing the face of large scale data mining by significantly speeding up the processing of data mining algorithms. For example, using the K-Means clustering algorithm, the GPU-accelerated version was found to be 200x-400x faster than the popular benchmark program MimeBench running on a single core CPU, and 6x-12x faster than a highly optimised CPU-only version running on an 8 core CPU workstation.

These GPU-accelerated performance results also hold for large data sets. For example in 2009 data set with 1 billion 2-dimensional data points and 1,000 clusters, the GPU-accelerated K-Means algorithm took 26 minutes (using a GTX 280 GPU with 240 cores) whilst the CPU-only version running on a single-core CPU workstation, using MimeBench, took close to 6 days (see research paper “Clustering Billions of Data Points using GPUs” by Ren Wu, and Bin Zhang, HP Laboratories). Substantial additional speed-ups are expected were the tests conducted today on the latest Fermi GPUs with 480 cores and 1 TFLOPS performance.

Over the last two years hundreds of research papers have been published, all confirming the substantial improvement in data mining that the GPU delivers. I will identify a further 7 data mining algorithms where substantial GPU acceleration have been achieved in the hope that it will stimulate your interest to start using GPUs to accelerate your data mining projects:

Hidden Markov Models (HMM) have many data mining applications such as financial economics, computational biology, addressing the challenges of financial time series modelling (non-stationary and non-linearity), analysing network intrusion logs, etc. Using parallel HMM algorithms designed for the GPU, researchers (see cuHMM: a CUDA Implementation of Hidden Markov Model Training and Classification by Chaun Lin, May 2009) were able to achieve performance speedup of up to 800x on a GPU compared with the time taken on a single-core CPU workstation.

Sorting is a very important part of many data mining application. Last month Duane Merrill and Andrew Grinshaw (from University of Virginia) reported achieving a very fast implementation of the radix sorting method and was able to exceed 1G keys/sec average sort rate on an the GTX480 (NVidia Fermi GPU). See http://goo.gl/wpra

Density-based Clustering is an important paradigm in clustering since typically it is noise and outlier robust and very good at searching for clusters of arbitrary shape in metric and vector spaces. Tests have shown that the GPU speed-up ranged from 3.5x for 30k points to almost 15x for 2 million data points. A guaranteed GPU speedup factor of at least 10x was obtained on data sets consisting of more than 250k points. (See “Density-based Clustering using Graphics Processors” by Christian Bohm et al).

Similarity Join is an important building block for similarity search and data mining algorithms. Researchers using a special algorithm called Index-supported similarity join for the GPU to outperform the CPU by a factor of 15.9x on 180 Mbytes of data (See “Index-supported Similarity Join on Graphics Processors” by Christian Bohm et al).

Bayesian Mixture Models has applications in many areas and of particular interest is the Bayesian analysis of structured massive multivariate mixtures with large data sets. Recent research work (see “Understanding the GPU Programming for Statistical Computation: Studies in Massively Massive Mixtures” by Marc Suchard et al.) has demonstrated that an old generation GPU (GeForce GTX285 with 240 cores) was able to achieve a 120x speed-up over a quad-core CPU version.

Support Vector Machines (SVM) has many diverse data mining uses including classification and regression analysis. Training SVM and using them for classification remains computationally intensive. The GPU version of a SVM algorithm was found to be 43x-104x faster than SVM CPU version for building classification models and 112x-212x faster over SVM CPU version for building regression models. See “GPU Accelerated Support Vector Machines for Mining High-Throughput Screening Data” by Quan Liao, Jibo Wang, et al.

Kernel Machines. Algorithms based on kernel methods play a central part in data mining including modern machine learning and non-parametric statistics. Central to these algorithms are a number of linear operations on matrices of kernel functions which take as arguments the training and testing data. Recent work (See “GPUML: Graphical processes for speeding up kernel machines” by Balaji Srinivasan et al. 2009) involves transforming these Kernel Machines into parallel kernel algorithms on a GPU and the following are two example where considerable speed-ups were achieved; (1) To estimate the densities of 10,000 data points on 10,000 samples. The CPU implementation took 16 seconds whilst the GPU implementation took 13ms which is a significant speed-up will in excess of 1,230x; (2) In a Gaussian process regression, for regression 8 dimensional data the GPU took 2 seconds to make predictions whist the CPU version took hours to make the same prediction which again is a significant speed-up over the CPU version.
...
If you want to use the GPUs but you do not want to get your hands “dirty” writing CUDA C/C++ code (or other languages bindings such as Python, Java, .NET, Fortran, Perl, or Lau) then consider using MATLAB Parallel Computing Toolbox. This is a powerful solution for those who know MATLAB. Alternatively R now has GPU plugins. A subsequent post will cover using MATLAB and R for GPU accelerated data mining.

These are space whales flying through the sun:

[General Patton]

Working thoughts:
1. Wikileaks can act as a secondary database. What we've seen so far makes it clear that most of the classified material is common knowledge but it could be useful.
2. Steele is right that the humanitarian goodwill approach is superior. We've spent a lot of money in Afghanistan, but most of it was spent in unpopulated areas that were safe, the people who needed it didn't get it. Lots of corruption. A tighter approach could be made.
3. Fiverr and penpal sites can also be useful for general cultural understanding : http://fiverr.com/vann97/answer-10-ques ... ut-vietnam
http://fiverr.com/worryfustion/help-you ... in-vietnam
4. Nearly all current prediction markets operate as zero-sum or negative-sum markets.


Yes it's real and hilarious.
http://www.nsa.gov/public_info/_files/c ... llence.pdf

[General Patton]

More OSINT Links:

City Data:
http://www.city-data.com/

Public Records:
http://publicrecords.onlinesearches.com/

Name/Location Search Engine:
https://pipl.com/

In current IT security environments, files and services are often password protected. In certain situations it is required to get access to files and/or data even when they are protected and the password is unknown.

wyd.pl was born of those two of situations:
• A penetration test should be performed and the default wordlist does not contain a valid password
• During a forensic crime investigation a password protected file must be opened without knowing the the password.

The general idea is to personalize or profile the available data about a "target" person or system and generate a wordlist of possible passwords/passphrases out of the available information. Instead of just using the command 'strings' to extract all the printable characters out of all type of files, we wanted to eliminate as much false-positives as possible.

http://www.social-engineer.org/framewor ... _%28WYD%29

creepy is an application that allows you to gather geolocation related information about users from social networking platforms and image hosting services. The information is presented in a map inside the application where all the retrieved data is shown accompanied with relevant information (i.e. what was posted from that specific location) to provide context to the presentation.
http://ilektrojohn.github.com/creepy/



Detecting Emergent Conflicts through Web Mining and Visualization
https://www.recordedfuture.com/assets/D ... zation.pdf

Maltego is a program that can be used to determine the relationships and real world links between:
People
Groups of people (social networks)
Companies
Organizations
Web sites
Internet infrastructure such as:
Domains
DNS names
Netblocks
IP addresses
Phrases
Affiliations
Documents and files
These entities are linked using open source intelligence.
Maltego is easy and quick to install - it uses Java, so it runs on Windows, Mac and Linux.
Maltego provides you with a graphical interface that makes seeing these relationships instant and accurate - making it possible to see hidden connections.
Using the graphical user interface (GUI) you can see relationships easily - even if they are three or four degrees of separation away.
Maltego is unique because it uses a powerful, flexible framework that makes customizing possible. As such, Maltego can be adapted to your own, unique requirements.

Has a free and paid version:
http://www.paterva.com/web6/

Links for OSINT (Open Source Intelligence) by Randolph Hock
http://www.onstrat.com/osint/

Less than four months ago, the Southern portion of Sudan seceded and formed South Sudan, only the 5th country to be created this century. In this session, we will demonstrate how Palantir can draw from a plethora of Open Source Intelligence (OSINT) data sources (including academic research, blogs, news media, NGO reports and United Nations studies) to rapidly construct an understanding of the conflict underlying this somewhat anomalous 21st Century event. Using a suite of Palantir Helpers developed for OSINT analysis, the video performs relational, temporal, statistical, geospatial, and social network analysis of over a dozen open sources of data.

First off, obviously, I cannot thank you enough for all the brainfood you've passed my way in the past three years.

This thread was especially great, and I've been unpacking it ever since it went up. What grabbed me the most was not even remotely related to AI horizons, though: it's the application of this kind of "structured thinking" to the meandering we do here at RI. We've seen some proto-projects, like Daniel Brandt's NNDB getting hooked up into the Flash-based mapping architecture of TheyRule.net, but the thought of applying Bayesian cartographic techniques to the social map of, say, elite pedo networks...surely there's some bleak gold to be mined there, no?

More soon, this is all very nebulous in my brainpan right now.

National Sex Offender Registry:
http://www.nsopw.gov/
State Sex Offender Listings:
http://www.nsopw.gov/Core/PublicRegistrySites.aspx
Mapped with SO Data:
http://www.city-data.com/so/sexoffenders.html

[psynapz]

General, THANK YOU. Please keep up with the marvelous curation here, and also thanks for keeping the lulz front-and-center, because this shit is enough to drive a monkey into a blind rage if they check their cosmic-level sense of humor at the door.

[General Patton]

http://www.darpa.mil/NewsEvents/Release ... 10/17.aspx
When the team behind DARPA’s Plan X mapped out where it wanted to go with research in the development of cyber capabilities and platforms, it knew the DARPA approach to problem solving included soliciting input from the leading experts in the field. On October 15 and 16, DARPA outlined its plans for Plan X to a packed house of potential developers and performers and solicited their feedback.
)
More than 350 software engineers, cyber researchers and human-machine interface experts attended the event. DARPA officials presented the goals of Plan X in preparation for release of the program’s Broad Agency Announcement (BAA)—anticipated within the next month.

Plan X, announced in May 2012, is the first DARPA program of its kind. It will attempt to create revolutionary technologies for understanding, planning and managing DoD cyber missions in real-time, large-scale and dynamic network environments. Plan X will conduct novel research on the cyber domain. The Plan X program is explicitly not funding research and development efforts in vulnerability analysis or generation of cyberweapons.

“Insights obtained from discussions with government partners and potential performers during the Proposers’ Day workshop will help us finalize our approach to the Plan X program,” said Dan Roelker, DARPA program manager. “The program covers largely unchartered territory as we attempt to formalize cyber mission command and control for the DoD.”

It is anticipated that the BAA for this effort will be posted to http://www.fbo.gov within the next month.

Ooooh man-machine interfaces, how sexy. What does it really mean?

For those who don't know, there is a basic 3d headset being released called Oculus Rift for 3d gaming. The below statement by John Carmack gives you an idea of the requirements for performance, it's pretty standard:

https://twitter.com/ID_AA_Carmack/status/251338927900872705 "The amount of latency reduction needed to turn a typical game into a good VR game is about the same as turning it into a decent cloud game." "It isn't hard to make something "work" with VR gear, but motion-to-photons latency of <20 msec is where you really want to be."

What about gesture recognition and 3d interfaces?

The skeleton mapping in the xbox kinect sensor creates a very primitive stick figure of a person to judge their gestures, this is a software limitation not a hardware one. However several sensors have came out which also improve on the standard hardware.

Current generation xbox kinect sensors can be rigged to create a 3d interface, some commercial companies are also working on their own versions soon to be released:


There are also some Silicon Valley guys working on some EEG style sensors:
http://developer.neurosky.com/

http://www.fastcompany.com/3000672/insi ... x-cyberwar
DARPA is especially interested in the development of custom, secure operating systems and platforms for use in “hostile network environments.” According to the document, these secure operating systems are a priority for use in battle damage monitoring, communication relays, weapon deployment, and adaptive defense. While the agency explicitly stated Plan X will not fund R&D for cyberweapons, another area of development is script development for cyberwarfare efforts, which will effectively partially automate cyberwar exercises. DARPA likens these mission scripts, which will be overseen by human operators, to the “auto-pilot function on modern aircraft.”

The most interesting idea, however, is building giant visual maps for cyberwar. DARPA is seeking assistance in developing visual, intuitive views of large-scale cyber battlespaces for situational awareness in war games. It sounds like, if Plan X works successfully, the Pentagon will have a sweet real-time visual map of large swaths of the Internet within a decade.

In other words, 3d SLAM(Mapping protocol) with the ability to add/edit entities/objects into the mapspace, integrated with sensor nets:
http://www.kartorobotics.com/

Enhanced computer vision is required for this project, work is progressing in that field faster than I can keep up with.



tDCS

DARPA has been using trans-cranial direct current to stimulate areas of the brain to improve pattern recognition and target acquisition for snipers. It has general applications for analysts.

http://www.newscientist.com/article/mg21328501.600-zap-your-brain-into-the-zone-fast-track-to-pure-focus.html?page=3
The mild electrical shock is meant to depolarise the neuronal membranes in the region, making the cells more excitable and responsive to inputs. Like many other neuroscientists working with tDCS, Weisend thinks this accelerates formation of new neural pathways during the time that someone practises a skill. The method he is using on me boosted the speed with which wannabe snipers could detect a threat by a factor of 2.3 (Experimental Brain Research, vol 213, p 9).

Mysteriously, however, these long-term changes also seem to be preceded by a feeling that emerges as soon as the current is switched on and is markedly similar to the flow state. "The number one thing I hear people say after tDCS is that time passed unduly fast," says Weisend. Their movements also seem to become more automatic; they report calm, focused concentration - and their performance improves immediately.

114 academic papers are here, password is "tactical psi" :
http://www.mediafire.com/?x9nwbzsxr7us9s2

Working Memory

http://en.wikipedia.org/wiki/Working_memory
Working memory is the system that actively holds multiple pieces of transitory information in the mind for execution of verbal and nonverbal tasks — such as reasoning and comprehension — and makes them available for further information-processing.[1] It is not the same as short term memory. Working memory tasks require monitoring (i.e. manipulation of information or behaviors) as part of completing goal-directed actions in the setting of interfering processes and distractions. The cognitive processes needed to achieve this include the executive and attention control of short-term memory, which permit interim integration, processing, disposal, and retrieval of information. These processes are sensitive to age: working memory is associated with cognitive development, and research shows that its capacity tends to decline with old age. Working memory is a theoretical concept central both to cognitive psychology and neuroscience. In addition, neurological studies demonstrate a link between working memory and learning and attention.

The average human can store 5-9 objects in their working memory, pros with several years of training can work with around 13. At any given time, your ability to make connections between unconnected objects, say pieces of information on the price of oil, the price of wheat, and drought, you are limited to that number. These factors create interdependent relationships which do not add or subtract, but multiply in complexity. In other words, if your method of studying things involves just reading a lot and seeing what comes out, you aren't going to be able to make any serious connections. This means in any given situation you are managing hundreds if not thousands of potentials, whose data fits multiple mutually exclusive hypotheses. This means that if you have one of the best working memories in the world and you can hold 13 objects, you will only work out 1.5% or less of the total problem in your head. Your goal is to sift through them effectively and quickly. This is not humanly possible. Your problem needs to be solved anyway. To do this you must augment your working memory and awareness**.

This is what makes Morphology and Analysis of Competing Hypothesis approaches effective, it is a method for managing working memory as much as anything else.

Morphological Analysis - A general method for non-quantified modeling
http://www.swemorph.com/pdf/gma.pdf

Modeling Complex Socio-Technical Systems using Morphological Analysis
http://www.swemorph.com/pdf/it-webart.pdf

Analysis of Competing Hypotheses (ACH) is a simple model for how to think about a complex problem when the available information is incomplete or ambiguous, as typically happens in intelligence analysis. The software downloadable here takes an analyst through a process for making a well-reasoned, analytical judgment. It is particularly useful for issues that require careful weighing of alternative explanations of what has happened, is happening, or is likely to happen in the future. It helps the analyst overcome, or at least minimize, some of the cognitive limitations that make prescient intelligence analysis so difficult. ACH is grounded in basic insights from cognitive psychology, decision analysis, and the scientific method. It helps analysts protect themselves from avoidable error, and improves their chances of making a correct judgment.
http://www2.parc.com/istl/projects/ach/ach.html

**This is my problem with Black Swan theory and Resilience : They do not try to augment awareness, which is the root of the problem on why we get caught by surprise. Making a more resilient system is great and should be much more widely practiced, but if you are not aware of all of the connections you will be caught with your pants down anyway. Resilience is a band-aid approach if you are unaware of your situation and cannot efficiently apply resources to manage risks while still gaining exponential returns on your resource pool.



Exoskeleton Enhancement

http://neurogadget.com/2012/10/18/new-h ... eanup/5612
The latest version of HAL has remained brain-controlled but evolved to a full body robot suit that protects against heavy radiation without feeling the weight of the suit. Eventually it could be used by workers dismantling the crippled Fukushima nuclear plant.
The new type of HAL is on display today at the Japan Robot Week exhibition in Tokyo. It will be used by workers at nuclear disaster sites and will be field tested at Fukushima, where a tsunami in March 2011 smashed into the power plant, sparking meltdowns that forced the evacuation of a huge area of northeastern Japan.

HAL – coincidentally the name of the evil supercomputer in Stanley Kubrick’s “2001: A Space Odyssey” – has a network of sensors that monitor the electric signals coming from the wearer’s brain. It uses these to activate the robot’s limbs in concert with the worker’s, taking weight off his or her muscles.

See also:
http://robohub.org/exoskeletons-new-and-older/

[General Patton]

Getting good demographics can help you to quickly understand the context of messages that circulate through different websites. The easiest method for large websites is to look them up on http://www.quantcast.com/

For instance, we an obvious pattern that hispanics and blacks tend to visit conspiracy websites much more often than whites. We also see that many of the sites tend to have older visitors with higher incomes. The exception are data driven websites like Wikileaks, which tend to be lower income but highly educated viewers who are mostly white or asian.




If you can find facebook groups for websites like this, you can cross-check some of the basic information by looking at user photos, names, and ages (keep in mind that facebook users tend to be younger than average):



To get a quick introduction to the character of a website, simply do an imagesearch of it on google, e.g.: site:http://forum.prisonplanet.com/index.php

Search through the websites looking for mentions of states and/or cities using google, e.g.: texas site:infowars.com (don’t add a space between the search command and the website, so use site:websitehere.com). Look for introduction threads or user profiles that list locations. Twitter accounts can also assist in this process.

With this information you can cross-correlate the cities members live in to get an idea of their general make-up, and how it compares to other demographic sources.

If there are a lot of unique images on the website, use google’s image search function to look around for other websites with the same images, which will expand your understanding of the psychographics of the users by finding similar sites and images.



For more google search ideas, look at “How to solve impossible problems: Daniel Russell’s awesome Google search technique”:

http://www.johntedesco.net/blog/2012/06 ... echniques/

If you want to map out keywords and connections, use a graph similar to this:

http://www.touchgraph.com/seo

A basic search gives us something like this:



Which shows us that we can also harvest data from youtube and amazon, as well as the smaller linked websites.

Now we have the basic demographics, we look for commonalities. Search through abstracts of psychology journals using pubmed.gov or google scholar, looking for keywords related to conspiracy theories, demographic information and psychology journals.

We end up with some curious things like this:

http://heb.sagepub.com/content/32/4/474.short

This article examines the endorsement of conspiracy beliefs about birth control (e.g., the belief that birth control is a form of Black genocide) and their association with contraceptive attitudes and behavior among African Americans. The authors conducted a telephone survey with a random sample of 500 African Americans (aged 15-44). Many respondents endorsed birth control conspiracy beliefs, including conspiracy beliefs about Black genocide and the safety of contraceptive methods. Stronger conspiracy beliefs predicted more negative attitudes toward contraceptives. In addition, men with stronger contraceptive safety conspiracy beliefs were less likely to be currently using any birth control. Among current birth control users, women with stronger contraceptive safety conspiracy beliefs were less likely to be using contraceptive methods that must be obtained from a health care provider. Results suggest that conspiracy beliefs are a barrier to pregnancy prevention. Findings point to the need for addressing conspiracy beliefs in public health practice.

And:

http://onlinelibrary.wiley.com/doi/10.1111/0162-895X.00160/abstract

This study used canonical correlation to examine the relationship of 11 individual difference variables to two measures of beliefs in conspiracies. Undergraduates were administered a questionnaire that included these two measures (beliefs in specific conspiracies and attitudes toward the existence of conspiracies) and scales assessing the 11 variables. High levels of anomie, authoritarianism, and powerlessness, along with a low level of self-esteem, were related to beliefs in specific conspiracies, whereas high levels of external locus of control and hostility, along with a low level of trust, were related to attitudes toward the existence of conspiracies in general. These findings support the idea that beliefs in conspiracies are related to feelings of alienation, powerlessness, hostility, and being disadvantaged. There was no support for the idea that people believe in conspiracies because they provide simplified explanations of complex events.

And:
http://www.jstor.org/discover/10.2307/3 ... 1379127527


From this information we can break them into traditional psychographics using stock models:







Now you can create a database that can be used for advanced analytic operations, using R, excel, SAS or a programming language like Python. R tends to be more effective for smaller sets less than 2GB because of it’s memory usage, but it has nearly all statistical functions anyone has thought to use which makes it very useful for experimental projects. SAS is commercial software that is mainly effective for large data sets. Excel is a decent entry level solution. Python is not quite as flexible as R yet, but it’s modules are improving and it can be interfaced with R. This is part of a scientific shift from equation based modeling to simulation based modeling, which relies on computation rather than just analysis. See pages 16-22 of Allen B. Downey's book Think Complexity (PDF format is free):
http://www.greenteapress.com/compmod/

[Wombaticus Rex]

Wow! Had no idea VALS was still in common usage today.

Do you have any recommendations on how image searches can be read with "psychographics" in mind? Are there cheat sheets in effect somewhere?

Also, lest anyone miss it, this was incredible: http://www.johntedesco.net/blog/2012/06 ... echniques/

[DrEvil]

This thread is a goldmine. Thank you General!
Btw: If you use facebook you can do personal analytics on your data with WolframAlpha: http://www.wolframalpha.com/facebook/

Edit: One thing I have been daydreaming about for a while is trying to apply various open source data mining/analytics stuff to this forum. I was thinking, start with something "simple", like extracting names and making connections between threads. See if some obscure name keeps popping up in strange places for instance. Or extract relations between people, and match it with time and geographic location where possible. All kinds of possibilities.

[General Patton]

Wow! Had no idea VALS was still in common usage today.

Do you have any recommendations on how image searches can be read with "psychographics" in mind? Are there cheat sheets in effect somewhere?

Also, lest anyone miss it, this was incredible: http://www.johntedesco.net/blog/2012/06 ... echniques/

You can use whatever stock model you want, it's ultimately more about listening to what the images are showing you about their personality and ideas. You could look for the interplay of yin and yang, liquid and solid, whatever way you want.

What flips their switches? What do the adore, what do they hate? Who are their friends and enemies? If any images stick out you can search around them. Maybe some persons of note in them, or more context for the images on the site page.

Infowars, for instance, is not a Rorschach test:


Another interesting thing - get people from the site, the radical neo-primitivist feminist anarcho-communist christian white nationalists to interpret movies and other works of art. This will provide you with hours of entertainment:

DrEvil One thing I have been daydreaming about for a while is trying to apply various open source data mining/analytics stuff to this forum. I was thinking, start with something "simple", like extracting names and making connections between threads. See if some obscure name keeps popping up in strange places for instance. Or extract relations between people, and match it with time and geographic location where possible. All kinds of possibilities.

Yeah, forums by themselves aren't the best format for projects. No way of working through geotags and correlating, no names/documents/other objects that are tagged for searching around, except with the search method on the top right. No group workspace, just walls of text with some images or video. It can definitely be improved upon.

Also, there has been work on getting R to work with larger data sets (because it's used so heavily in academia/finance):
http://www.mathfinance.cn/handling-large-datasets-in-R/

[DrEvil]

A system for adding meta-data or tags to posts here would definitely be a nice thing to have, provided people used it. A simple wiki-like function for names could also be useful. See an interesting name, click it and get a small window (like the 'spoiler' button in some forums) with links to other posts where that name is mentioned, and possibly a small bio.
A full blown wiki based on/mined from the data stored here could also be interesting, but probably a bit more work involved.

[General Patton]

http://www.pbs.org/newshour/bb/world/july-dec12/dictators_07-16.html
Putin chooses to send tax inspectors or health inspectors to close down or shutter a dissident group.

In Venezuela, laws are written broadly and then used like a scalpel against any group that is deemed a threat. The Chinese Communist Party frequently refers to democracy and makes sure that all of its top leaders only serve two terms. There are all sorts of different ways in which regimes are finding how to move and navigate through forces that challenge their regimes that make them appear to be other than what they are.

…

I was in China about 10 days after Mubarak fell.

And it was an incredible moment, because, on the one hand, there was no visible sign of revolution, but there was a tremendous tension. And there had been a call for people to assemble at different points around China at a particular moment on a particular day.

And the regime knew that. I went to one of those spots at 2:00 p.m. on that Sunday, and it was an incredible thing. You saw the fear that the regime exhibited by just the sheer number of police that were present. But more than the police were the number of plainclothes policeman.

There were moments when I was walking through crowds, and literally, three, four, five people around me, they all had earpieces.

They have really developed excellent crowd control techniques, where they would move the crowds through with street-cleaning equipment. And they would push people through with lots of water, cleaning the same street corner again and again and again.

Mind you, no one is actually coming out to protest. No one is actually declaring, down with the Chinese Communist Party. If you were to do that, then you would be rushed away in no time by security. But, rather, the call had been for people just to come out for a stroll.

And that was a very clever way of going about it, because you can’t really arrest someone for just walking down the street.

…

One of my favorite examples is actually from Egypt before the revolution, where you had members of the April 6 Movement, who they put out a call on Facebook for people to rally around a strike that was going happen in another part of the country.

And they said to people, on that day, just don’t go to work, another easy way to sort of show your protest without actually taking enormous risk. Just stay within your home on that given day. And the support for this was enormous on Facebook. And so the regime’s initial response was to start putting out a ticker on all television broadcasts saying, on April 6, everyone must go to work.

They inadvertently broadcast this message in a way that the members of April 6 never could have imagined. Egypt is a country of 81 million people. The regime had mistakenly, inadvertently communicated this protest to everyone.

http://colonyofcommodus.wordpress.com/2 ... -snitches/
Snitch Spotting 101:

1. Watch people who use drugs or engage in other illegal behavior, if they get charged with anything the police will use that as leverage against them. If they drink or use drugs in a social environment then they will just as likely spill whatever secrets they have anyway.

2. People with large debts generally shouldn’t be trusted, your opponent can offer to help make those debts disappear for the price of turning.

3. If someone is encouraging you to organize for violent actions, they may not be a snitch. But they are unstable enough to get you caught anyway.

4. Watch for people who go around accusing others of being snitches. Accusing others is usually a way for snitches to cover their own tracks.

5. Look for people who magically show up with the critical skills your organization needs at just the right time. Don’t be afraid of subjecting your best friends and best assets to scrutiny

http://colonyofcommodus.wordpress.com/2012/06/05/more-on-snitches/

As detailed earlier, there have been plenty of failed experiments in dealing with the issue of snitches and the current epidemic of snitching cannot be stopped through random beatings or through empty threats. While violence against snitches or collaborators may be necessary (for example, the very public targeting of snitches utilized by the Maquis could prove useful) it is often harmful or useless in ending Snitch Culture. Not only can the State outmatch us in terms of the violence it can expend, rendering moot a wholesale campaign of violent reprisals —as in the case of the RAF— it also seems counter to our politics of freedom to use the idea of terror to coerce people into line and could put off sympathetic or interested individuals —much like the WU did. This should not be read as a dismissal of the tactic of violence in our resistance, but as strong critique of violence as a useful tool in combating Snitch Culture.

What has the greatest possibility of working —although it is currently not practiced to the extent that it needs to be— is community shame coupled with prisoner support. Community shame has the benefit of not being irrevocable —how can you make amends for shooting or beating someone terribly if it turns out they were wrongly accused?— as well as providing a powerful disincentive for snitches by denying them friendship and support. Prisoner support is obviously positive in that it helps remove the power of violence that the State holds over people. Prisoners who feel supported and know they will be cared for have less reason to abandon their principles and betray their friends. Coupled together, a strategy of strong communities of autonomous individuals that will not allow collaborators back in, along with a prisoner support in which the benefits of not snitching far outweigh any measure put forth by the State, seems to be the best course of action. Presently, however, these tactics have proved ineffective in the prevention of Snitch Culture. Snitches know that they will be reviled by some, but they can remain in our communities by moving to where they may be anonymous or because there are people who will not ostracize them and allow them to return. And even though our prisoner support is one of our strongest attributes, it fails to be the linchpin that prevents Snitch Culture, mainly because it is limited in scope.

What this leads us back to is the idea of building stronger communities capable of long-term resistance to the powers of the State. Community shame and prisoner support lack their necessary bite precisely because our Communities of Resistance are fractured, with no real communication or trust amongst groups. In tightly-knit societies like the Basque, snitches have nowhere to go because word will travel and they would face social isolation wherever they go.

Shame only works when communities can communicate in a way that is informative and trustworthy. Without information about snitches, communities cannot take steps to isolate or shame that person; and without trust, communities have no idea if the information is reliable or that others will also take steps to ostracize a snitch.
...
Besides the obvious, this larger support functioned as a way to propel resistance and created a climate of hope —a climate that Snitch Culture does not function very well in. This lack of support may be due to the fact that our definition of Anarchist culture is rather narrow. Although not true in the least, the image of Anarchists as young black-clad punks certainly does persist and is to an extent perpetuated by Anarchist culture. Anarchists should proudly proclaim themselves, but should also move beyond the stereotype and show our many faces: young and old, queer and straight, crusty and freshly-showered.

And obviously, our current prisoner support must continue —and increase— while working on the long-term projects of building infrastructure and widening our circles of support. Prisoner support needs to become something that everyone in our community is working on. Comrades must have the full support of their communities when facing down the courts, the jails, and the prison terms of the State. We need to make prisoner support public and visible, and we need to show comrades on the other side of the wall that they are not forgotten. Murals, benefits, sending letters and birthday cards, attending court dates, and solidarity events are all easy ways to show that we celebrate and stand behind those from our communities who are behind bars. We need to make good on the saying: they are in there for us, so we are out here for them!

http://www.strategypage.com/htmw/htinte ... 70418.aspx
The key to Israeli success was an informant network within the Palestinian community. The Israelis have hundreds of police and military operatives who can pass as Arabs (their families came from Arab countries shortly after Israel was founded in 1947). These Israelis speak fluent Arabic (with a Palestinian accent) and are Arab in appearance. These agents dress as Palestinians and enter Palestinian areas and, backed up by regular troops, grab suspects and hustle them off, or kill them if they resist. But these agents also move about recruit and run Palestinian informants. Many of these Palestinian informants are doing it for the money. Israelis pay for information. They also use other inducements (help with the bureaucracy, medical care, etc). If that fails, they use blackmail and threats. Palestinian terrorist organizations have been unsuccessful in their attempts to shut down the informant networks, and many innocent Palestinians have died simply because they were falsely accused of being informants.

Actually, the Israelis gain a lot of information on terrorists via electronic intelligence work and UAVs that are constantly in the air over Palestinian neighborhoods. They seek to make the terrorists think that its the gadgets, not informants, that is gathering the information. To the Israelis, inducing paranoia among the Palestinians is seen as a successful weapon. All this has kept helped keep the terrorists out of Israel for the last three years, something no one thought was possible.

http://www.newsreview.com/sacramento/PrintFriendly?oid=80311
In an affidavit filed in the case, FBI Special Agent Walker said that the “confidential source” (the government never refers to Anna by name; it’s always the “source”) had been used in at least 12 prior investigations of anarchists and anarchist groups. “Her information has proved accurate and reliable,” Walker said, and she was “granted authority to participate in Tier 1 Otherwise Illegal Activity” as part of the investigation. Tier 1 means investigations of the most dangerous criminals: terrorists. OIA means just that. Anna was allowed to break the law in order to get her collar.
...

“I was on the phone one day, complaining to someone that we only had one volunteer medic,” Del Papa explained.

“I think the phones were tapped, because the next day, this woman Anna shows up, with short blond hair, in these leather pants, with a medic bag.”

But Del Papa said Anna didn’t seem very interested in offering medical care and comfort to protesters. She was more curious about the protest organizers.

“She started asking all of these really specific questions about who was coming and how many people were coming. She got really aggressive about wanting detailed information about our plans.”

During the march, Del Papa said, Anna started recruiting high-school students to stage a sit-in to block traffic, right in front of a large group of Broward County sheriff’s officers in riot gear. Del Papa was sure the provocation would lead to arrests and to the police clearing protesters from the area around the Fort Lauderdale Convention Center where the protest was being held. “It was a trap,” Del Papa told SN&R. That’s when Del Papa was sure that Anna was a government agent. Not that it’s uncommon for political demonstrations and meetings to have an undercover agent or two in their midst.

“What bothers me is that they’ve gone from being information gatherers to being provocateurs. To provoking people into these actions.”

Mark Reichel went to Miami to check it out. He brought a photo of the Miami Anna, taken by organizers there, back to his client in the Sacramento jail. It was the same Anna, McDavid told him.
...
But Anna was getting frustrated. “I guess I’m just different than you guys. I don’t like amorphous crap. … I wish one day we could keep the damned plan. I wish one day you guys could stick to a list. Why can’t one of you guys say, ‘Hey, this is what we want’?”

Here, Reichel said, Anna sensed that the conspiracy was falling apart, and she began trying hard to get them to make statements about specific targets. “Yeah, I would like a damned goal,” she pleaded with the others.

The four agreed that they needed to purchase more supplies to keep experimenting with the explosive. “But at that point, it’s over,” said Reichel. “The whole thing had peaked,” and the conspirators were going to break up and leave Dutch Flat. But then Friday the 13th happened.
...
“She provides the money. She provides the car. She rents the cabin. Oh, and by the way, ‘I used to be a high-school chemistry teacher, so this is second nature to me.’” All that adds up to entrapment and worse, he said. “This is a case where the government is manufacturing crime,” Reichel added

Note: Darby was not a "master manipulator". He is adventurous, volatile and was alienated from others within the groups.

http://theragblog.blogspot.com/2010/03/ ... andon.html
In the first training I did following Brandon’s supposed arrest, Brandon insisted that one of the participants was an undercover cop and demanded that I ask that person to leave. High drama around other people being undercover is behavior I’ve learned to associate with informants as a way to divert attention from them. It also breeds distrust and is destabilizing of collective efforts.

In another intense protest when UT students attempted to block an intersection with a tripod, the police unfortunately were waiting near the intersection and quickly pulled out the legs of a tri-pod, and dropped the person about 15 feet onto the pavement. Brandon who had helped bring props to the site became erratic and started yelling at the police resulting in even more people being arrested, including people who were not intending to risk arrest. Several students left the anti-war movement as a result of this action.

At this time, it became very clear that a key local organizer was being intensely targeted. Her home was broken into repeatedly. She found her vehicle tampered with, was fired from her job, and her cat was poisoned. Coincidentally, also at this time, Brandon began to court her as a mentor, asking her to teach him what she knew about organizing.

The first time she recalled meeting Brandon was the day he was arrested, when he ran up to her yelling that there were undercover cops in the crowd. Following his arrest, Brandon consistently called her, wanting to talk about his arrest and aftermath but rejecting the legal support she was helping organize. Recently, when the Austin Informant Working Group did an open records request on this organizer, the FBI found 600 documents with her name in them (they have not been relinquished by the FBI to date).

Brandon also participated in a protest at the Halliburton shareolders' meeting in Houston. He unexpectedly joined the group intending to commit nonviolent civil disobedience. The group was on edge the night before, and now I understand why. In the planning session the night before the action, Brandon argued strongly that provoking and fighting the police was a tactic to open the eyes of the masses to police brutality, and bring more people into our cause.

He held his ground even when the group strongly disagreed and told him that under no circumstances would the group agree to him provoking or fighting the police. Brandon was a loose cannon and a bully. Even when he said he would agree to nonviolence in the action, it was clear that in his mind his agreement was contingent on the police not “provoking” him. Going into the action the next day was like sitting on a tinderbox waiting to explode.

At some actions, Brandon would show up, all masked up, with a video camera and take a lot of footage. He has continued to do this over the years, including in Minneapolis. I don’t believe he has ever posted or published any of it.

Brandon also befriended a local Palestinian activist, a man named Riad Hamad. In the spring of 2008 his house was raided by the FBI. In April Riad was found bound, gagged, and drowned in Town Lake. The death was ruled a suicide and the FBI is not releasing any information, but it was made clear in David McKay’s trial that Brandon was also involved as an FBI informant on that case.