Moore's Law for Data Integration

The year is 1999, and your company wants to customize its new CRM system so sales people can access contract records from the Finance department's database. The CRM system itself took 9 months to deploy. Everyone's tired of the training classes. The six consultants who implemented the CRM system have lost a few people and gained a few people. Looks like they'll be bringing in someone else to manage this customization, which should take 4-5 months, tops.

Flashforward to 2008. It's September, you're using SugarCRM as your CRM system. You'd like to, again, integrate your CRM system with your Finance system. This time you'd like to do it using MindTouch Deki, a popular open source wiki and collaboration platform, and SnapLogic, an open source data integration framework. Good choice: the entire project, from start to finish, is completed in under two weeks.

No big, expensive consulting contracts. No "tent village" of Big Four consultants camped out by the computer room. An IT manager conceives the project, and within two weeks, it's done.

There's a paradigm shift that's occurred in enterprise IT, and it promises to make the next few years genuinely exciting, despite the downturn.

Over the past decade, application vendors have learned the wisdom of first, opening their APIs, and second, eliminating the complexity of their APIs by adopting a RESTful Web services model for integration.

The result has been a increase in agility for business users and IT department that is proving to be as dramatic, in terms of applications deployments and end user experience, as Moore's law has been for hardware development. Moore's law, you'll recall, is Gordon Moore's observation, first made in 1965, that "the number of transistors that can be placed inexpensively on an integrated circuit has increased exponentially, doubling approximately every two years" (Wikipedia). Thanks to Moore's law, your bookbag can hold a laptop more powerful that a mainframe from a few decades ago. And you watch videos on your MP3 player, which is roughly the size of a pack of gum.

Open source and RESTful APIs (which make use of basic GETs and PUTs, rather than relying on more complex messaging schemes) give enterprise IT organization powerful building blocks for rapidly building new, powerful application solutions—applications that literally would have required man years of programming less than a decade ago.

A year ago, there was a lot of buzz about enterprise mashups that blended two or more data sources. Mashups are still exciting, but what's equally exciting is a kind of mashup occurring at the application level.

Today's announcement by Salesforce.com that its Force.com development platform would work with Facebook APIs, enabling Salesforce.com's 100,000-strong developer community to more easily access the vast library of applications built on Salesforce, is more evidence of this trend.

Other evidence of this trend:

• The ongoing success of the Web site, ProgrammableWeb, which serves as a portal for discovering application APIs and mashups. ProgrammableWeb now has 1,000 APIs in its API directory. Clearly, a lot of companies are publishing APIs.
  
• Once published, APIs often become the dominant channel for accessing an application. As ProgrammableWeb's John Musser points out in a blog post about the 1,000-API milestone:
  
  
  
  • 60% of eBay's listing come from their APIs, rather than through their browser-based interface.
    
  • Twitter's APIs carry 10x the traffic of its Web site.
    
  
  
• REST is becoming the dominant programming model for APIs. 63% of the APIs listed in ProgrammableWeb's directory are RESTful.
  
• New Content Management Systems, such as Alfresco Enterprise 3.0, feature REST interfaces so they can easily access business data from other IT systems.
  
• New applications, such as an increasingly popular network management/IT operations platform offered by an Indian company, use REST APIs to facilitate communication among components. RESTful integration enables one product, for example, a network troubleshooting tool, to easily pass information to a related component, such as a trouble-ticketing application used by a help desk.
  

Enterprise Management Associates analyst Dennis Drogseth is fond of saying that enterprise IT organizations know they need to move to a "lego world." No single IT system has a monopoly on vital data. Best practices call for the automated flow of information seamlessly from one system to another: from a network diagnostic tool to a help desk application, for example, or from a finance system into a wiki, which in turn is embedded in SugarCRM. The key to that flow is having components designed to fit together with other components, even from other vendors.

It sounds like a lot of work. It's dramatically less work than it used to be. Thanks to open APIs, REST, and Web-centric architectures, data integration and application development can beneft from a Moore's law of their own.