Searching Open Source Material in the Age of Information
Intelligence analysts are thought to be commensurate experts in writing, research, and analysis, but does the next generation of analysts have the skills necessary to be successful in the intelligence field? One of the greatest challenges for an analyst today is that the amount of information—as well as the means in which it’s shared—is growing exponentially.
Intelligence analysts must be able to gather, correlate, analyze, and evaluate information from a wide variety of sources. These sources can include law enforcement portals and databases, surveillance systems, intelligence networks (various disciplines), geographic information systems (GIS), and private data-mining databases (subscription-based).
In addition to searching these various restricted access databases available to those within the U.S. Intelligence Community, another important source of information for analysts comes from open sources. There are several skills that analysts must master to effectively search and exploit open source material in this age of information.
Gathering Open Source Information
Open source material can be retrieved from a myriad of public sources including radio, television, press agencies, newspapers and other periodicals, as well as books, blogs, social media platforms, and other websites on the internet. Information from these sources is produced very quickly and is buried just as fast as it is disseminated, thus quickly becoming irrelevant.
Open source material also includes gray literature, such as academic sources like dissertations, research papers, technical papers, and information from conference and seminar presentations. Information can also come from government agencies (local, state, and federal), corporations, research centers, associations, and societies, as well as professional organizations.
Many open source websites are free and have open access, while others require an account to be set up or may require you to have a paid subscription. Many higher learning institutions and state and federal government entities provide students and employees with ready access to many of the paid sites.
Mastering the Search
As an intelligence analyst you are expected to be the master of search and research. You must think about how to find nuggets of information that may not be apparent to all. Before you begin a search, you should be aware of what information you are looking for. Conducting a general topic search just to see what you can find by throwing a wide net may be useful initially, but you will eventually need to focus in on the information you need or is usable.
Analysts can go about gathering information in different ways, but it all comes down to their ability to manipulate the search function of the database or portal. All too often, a novice researcher simply plugs in a few key terms—ones they think define what they are looking for—and only investigates the top results of that search. Not often enough does the researcher work to refine the search terms or consider possible abstract terms that may lead to the information they’re looking for.
Beyond choosing the right search terms, how do you connect those terms to further refine your search? Boolean Operators are connectors used to define the relationship between search terms and assist in either narrowing or broadening records or results. The most commonly used Boolean operators are AND, OR and NOT. Search engines in portals or websites may have their additional logic and operators to help you in refining your research.
Searching Social Media
If you are building a dossier on a specific individual, you are going to want to start your search with the most popular social media (SM) platforms such as LinkedIn, Facebook, Instagram, Twitter, and Snapchat. These are just the start of SM and each one alone is not necessarily going produce the desired results. People upload personal information across different SM sites so you need to know what you are and are not looking for. Yes, you can say you don’t know what you are looking for until you find it, but this is not a good mindset to start your search with.
Basic information analysts commonly search for on SM include residential or business addresses, email addresses, phone numbers, IP addresses, domain names, property/assets, marital status… the list can go on. You can also conduct more in depth searches for company ownership, social activities, family or friends, political and/or spiritual leanings, financial status, places they go, or other personal details.
Protecting Your Identity
Intelligence analysts must be cautious and may need to hide their identity while searching. For example, you may want to set up new social media accounts that you use only during investigative inquiry. But be aware that some social media networks prohibit fake accounts and you should read their terms and rules carefully before you sign up and create a fake account.
[Related: An Introduction to the Darknet and Bitcoin]
There are several other ways to hide your identity when searching online and sometimes it’s a good idea to do so. Using Tor Browser is one of several reliable ways to search anonymously. Anonymouseis is another service that allows you to surf the web without revealing any personal information. Then there are proxies. 250 Working Proxies is perhaps the biggest list of anonymous proxies. There are also search engines that don’t track users
Create a Directory of Open Source Sites
Intelligence analysts should have a directory of open source websites at the ready. A directory provides a good starting point to begin your research and should include sites like:
Places to start a general search
- Oscobo (UK-based websearch site)
- Yand ex
- Black Book
- Phonebook of the World
- Public Records
Social Media and Dating Sites
- Google Plus
- MySpace (yes, it is still around)
- Social Mention
- US Tax Court
- Better Business Bureau
- Bloomberg Businessweek
- Cntrl & E European Business Directory
- Corporate Information
- Dun & Bradstreet
- Industry Canada
- Mint Business Information
- Open Corporates
- SEC Company Search
It is strongly suggested you bookmark the sites you find here or on your own and keep track as new sites are created and others are deactivated. Creating a document with a list of the hyperlinked sites on your desktop, laptop or mobile devices is also advisable for quick and easy access.
Log Research and Store Sources
It is always a good idea to keep a log of your research by using Excel. The internet is a vast space of information and that information is very fluid. Information that appears one day may not be there the next so you need to ensure you are capturing your URLs to refer back to. All too often I hear analysts and students complain that they did not properly set up their research. By creating a digital case file or case manager folder, you have the ability to recall websites, photos, and other documents through hyperlinking and embedding, which is never used enough. This is just another part of a well thought out research plan for any competent research analyst.
If you need to return to a website or page that is no longer available, you can make use of the “way back machine” on Archive.org. You should also be capturing screenshots of webpages while you are on them. There are many tools available at your fingertips including SnipIt, SnagIt, and other video or screen capturing software. Smartphones also have the ability to snap a photo of its screen as it appears.
Remember to Verify Information
The last point I want to make is that as with any research, there are limitations. Information may be incorrect, incomplete or inaccurate. You should always double-check your information and results. Anyone and everyone can place information and mis-information out into the ether, so it’s important to know your source and its credibility. If you have any doubts, you should corroborate the information prior to presenting it as fact.
While open source techniques and tools are valuable, they are only as good as the analyst conducting the research. Searching and exploiting open source material are only a starting point and not necessarily the only skills you should be using and honing. It’s important to network with others to find new and useful research methods, tools and techniques. An analyst should be constantly seeking to improve their skills and gain new ones.
About the Author: Charles M. Russo is an instructor in the School of Security and Global Studies at American Military University. He possesses a PhD in Public Safety Leadership from Capella University and a Master’s Degree in Intelligence Studies from American Military University. Charles served in the U.S. Navy for 17 years as an Intelligence Specialist and has taught Criminal Justice, Homeland Security and Intelligence at American Military University, Colorado Technical University and several other state universities. He is a retired U.S. Intelligence Community Intelligence Analyst after serving over 26 years, which included the U.S. Navy, US Air Force, Defense Intelligence Agency, Department of Homeland Security and the Federal Bureau of Investigation. He is the CEO of Intelligence Career Services, a provider of mentoring and assisting individuals looking to become active in the IC. He is also a consultant supporting intelligence, law enforcement and emergency response training and education efforts across state and local government. He currently lives and works in Carson City, Nevada. To reach him, email IPSauthor@apus.edu. For more articles featuring insight from industry experts, subscribe to In Public Safety’s bi-monthly newsletter.