RSSSearching the Deep Enterprise
I think it sounds like the name of the next Star Trek TV series, what do you think?
What I am riffing off of, however, is this New York Times article talking about the mythical "Deep Web" and the new efforts that search companies are making to catalog its contents. It's been broadly realized for some time that large chunks of the data available on the Internet are "hidden" from conventional search indexing in databases and data files without any sort of standardized interface format.
This is a bummer for consumers and casual web surfers, of course, but it's not just a problem on the public Internet. While corporate search has experienced considerable growth and some success in the past few years, it also faces this problem: data stored in locations and formats where traditional indexing has little luck penetrating.
The enterprise has at least a theoretical leg up on companies attempting to index the deep web; although there is some fuzz around the edges, most enterprise IT departments have a pretty good idea where their data is and how to access it. But drop-in boxes or internet security software using cutting edge off-the-shelf search technologies, such as the Google appliance pictured above, can't make good use of that expertise. Their own expertise, as deep and successful as it may be, cannot be fully integrated with your internal expertise. Unless the enterprise commits to building or heavily adapting search technology in-house, the result is the same: information you can't get at unless you know that you want it in the first place. Circumventing this was the promise of enterprise search, and it has yet to pay off.
Will these new deep web indexing efforts pay off for corporations internally as well? The potential is certainly there. With most of the startups focused on a piece of the larger Internet pie, though, it may be some years before anyone really explores what the technologies can do inside the corporate firewall.
Greetings from ISEN, a deep web solution. It is a proposed standard not unlike the ISBN is for book and ISSN for journals, but also has a DNS-like component. You might look into what we propose. Not ready for prime time quite yet.
-Matt Theobald
Thanks Matt, I’ll check that out!