Notes on “Searching Open Source”

2001 Oct 15

In-Q-Tel asks for an ability to search open sources discreetly.

I speculate that by “Open Source” they mean such data bodies as

The sites connected with daily news papers and other news organizations is typically not indexed by the search engines due to its volatility and the fact that they charge for historical access.

If I had the CIA’s money and charter I would want to search such sources while concealing who was searching and what was sought. Google does not promise that queries will be private and indeed they disclose to sites the actual criteria by which that site was listed by Google in response to a query. This information flows via the seeker’s browser which could be changed to omit it.

Google’s substantial snapshot of the web can be strategic as well because it looks into the recent past, which can be strategic when games are played. By this I mean that recent content is available even after events which have caused the site operator to change or remove the information.

Quality of Confinement

Keykos provides much better confinement than any other software platform. It also provides the performance necessary for an application such as Google.

One will presumably ask “Why is this better than merely trusting the Search Enterprise and its software not to disclose the queries from the special client?”. The simple answer is that with a system such as Keykos, a much smaller set of software and team of people need be trusted. The kernel, some security executive software, those who know that security software and those who guard and operate the hardware need to be trusted. This is a much smaller group of people to trust than the entire organization. If other confidential clients are served, then the advantages of Keykos are even more unique and strategic.

Competing Solutions

The most obvious solution is another search engine devoted to the spooks and their associates. This is expensive. Some sites would take steps to avoid being indexed by unknown robots.

Other Services

For a consideration the search service could visit some sites at an increased frequency or archive their material for longer periods. Real time site content delivery could be arranged to client software.

Clients could design and implement custom search algorithms that work from data in extant formats.

Access Security

An SSL portal to the site seems natural. Disguised routing from customer to search site can be arranged but is not something that we have experience with.

Spin Offs

Every one of these features has potential for commercial customers.

Technical Effort

While the Google data base is huge, I imagine that the application is relatively small, compared to running a bank. It is thus probably an order of magnitude easier to support them than customers with large legacy applications written by unknown programmers.

Still it would almost certainly require porting to a 64 bit architecture, MP support and implementation of a few kernel details that we have postponed for years.

This is a big project for Keykos, but a small project compared to any other way of solving the problem.

Related Markets

Other Consumers
In another venue, people that search patent data bases are often very anxious not to reveal what they are looking for. Derwent answers queries concerning chemical patents, often of the form, “What has been patented along these ... lines?”. Their customers typically consider the queries very proprietary.
Other Information Suppliers
News organizations might want to market their historical, but timely, data so that it might be searched uniformly across the variety of sources. Some searches would need access to the raw unindexed data at a price in proportion to the answer yielded by the search—not the volume of the data searched. I put up a recent external web page describing how to combine confinement with payments.

The Global Trust Issue

Who trusts whom is, of course, the complex issue behind the shape of such a business venture. No enterprise is likely to be trusted by all. Some will have clear ties with particular governments, and be untrusted by others as a consequence.

Technical points to make to Search Enterprises

Some will fear that security software will impose an intolerable performance cost. Keykos attempts to transform the hardware as little as possible so as to deliver whatever benefits are inherent in the hardware. In this regard it typically has much superior performance than classic operating systems. The security hooks are very low level and quite efficient.

P.S. 2007: Perhaps page-store.com is in such a business.