information overload, a good thing
6 April, 2007
The way Jeff Jonas gave his talk at etech07 was similar to the content he was presenting. His theme was ‘enterprise amnesia’. Jonas has a history in Las Vegas, where he worked on fraud detection for casinos. The main problem with these organisations is that the left hand does not know about what the right hand is doing. For example, a casino might not know that a dealer and a player at the same table might share the same streetaddress. This could indicate a fraud case. Tying different databases (e.g. the employee and the visitor database) together could solve some problems.
Of course, just tying the databases together does not solve the problem automagically. Data in different places can be slightly different. Therefore, some smart techniques (pdf link) need to be in place to connect data from one record to another. This technique is now featured by IBM (where Jonas is a chief scientist). Basically, all data that shares some elements is compared to connect them into one ‘entity’. A byproduct is that while a database keeps containing more and more records on individuals, the number of individuals grows slower than the database. In other words: the information overload becomes a virtue, more detailed information about individuals is known.
Another interesting point Jonas made is to treat data and queries as the same thing. When someone queries a system, this is also information that enriches the system. Just a simple example would be that a user is looking for information, doesn’t find it, but does find someone else was looking for the same. Having stored the query before, now makes it possible to connect these two individuals. Also, treating new data that enters the database as a query has serious benefits. In this case the new data is used to asked the question: “what does this change to what we already knew?” Jonas calls this ‘perpetual analytics‘.
These techniques can be used for good and bad things. An example where having a sound data storage about individuals might have helped was in the Katrina aftermath.