Sunday, August 22, 2010

The coming black boxes inside business organizations

Google has recently made available a set of tools for analyzing data using machine learning techniques called Google Prediction API.

As you can read in the recent article in MIT Technology Review:

For example, a bank or credit-card company wanting to use machine learning to build systems that make decisions based on historical transactions is unlikely to have the specialized staff and necessary infrastructure for what is a computationally intensive approach. [...]
Google's black box (...) contains a whole suite of different algorithms. When data is uploaded, all of the algorithms are automatically applied to find out which works best for a particular job, and the best algorithm is then used to handle any new information submitted. [...]
Google does, however, get (...) information [about effectiveness of various techniques] that it can use to improve its machine-learning algorithms. "We don't look at users' data, but we do see the same metrics on prediction quality that they do, to help us improve the service," says Green. The engineers running Prediction API will know if a particular algorithm is rarely used, or if a new one needs to be added to the mix to better process certain types of data.
Source: Tom Simonite "Google Offers Cloud-Based Learning Engine", Technology Review, 2010-08-20

As machine learning and other pattern-recognition technologies become more available and effective, companies will use them more often.

We can learn some lessons from the investment area, where algorithmic trading has already overtaken human discretion at least in terms of number of transactions executed on the US stock exchanges.It's true that quite often trading algorithms are still quite simple, but they are getting better as Moore's law increases the computing power available for them. It seems inevitable that one day algorithmic trading will dominate the investment industry, and human investors will share the fate of chess master Garry Kasparov defeated by IBM's Deep Blue.

Similarly, proliferation of quantitative methods in business may lead to some interesting results. First, managers will be often and often required to know these new technology-enabled methods. Second, the decision making processes, starting with operational aspects, may slowly morph into hardly penetrable black boxes, processing increasing amounts of information generated and gathered by business organizations.

Originally posted at: 

No comments: