Introducing CodeGlide, a New Open Source Mashup and Data Integration Company

Economic downturn or no, the open source data integration market remains active. In late January, Talend announced a $12 million C round of financing. Over the past two years, the company's open source data integration solution has been downloaded 3.3 million times, and the company now boasts hundreds of customers. Other vendors in the market, such as SnapLogic, continue to add connectors and develop their capabilities.

And now comes a new vendor to the scene: CodeGlide, an open source company with a broad business application platform and big ambitions. I spoke recently with Mauro DeGennaro, CodeGlide's co-founder and Vice President of Software Development. He filled me on the company's approach, offerings, and direction.

CodeGlide was founded in May 2007. The company's goal is to offer a comprehensive suite of open source business applications, including:

• Fusion, an open source data integration platform for building mashups
  
• SnapCRM, an open source CRM suite (with a provocative title, since the company competes against SnapLogic)
  
• Rendezvous, an open source collaboration suite, which offers calendaring, document management, and project management features
  

These three platforms are available in an Enterprise Suite bundle, which is the company's most popular download.

Speaking of downloads, CodeGlide seems to be off to a fast start. The company's Web site came on line on January 15. I spoke to Mauro about a week ago, and already the company had racked up 11,000 downloads of its software. Through its partners, the company has sold its Professional Edition to six different companies.

Partners are also lending their expertise to help the company develop other applications, including an E-learning application, an ERP application, and a Platform-as-a-Service (PaaS) offering that will compete against Coghead", LongJump, and other PaaS solution providers.

CodeGlide has a core development team of ten programmers, who are augmented by free-lancers and sub-companies that offer domain expertise in areas such as ERP. Business people and design experts design new features, which the developers then build. Of course, being an open source company, CodeGlide can expect to receive further enhancements and features from its open source community, as well. The company is small, but global: development is based in Argentina, Q.A. in India, sales in Europe and the U.S.

The CodeGlide Approach

CodeGlide's goal is to offer open source business solutions for the SMB market. It wants to make mashups and data integration solutions available to developers without advanced programming skills and even to business people. Accordingly, the company offers a visual tool for building mashups and creating widgets. For example, using Fusion, it's possible to construct a mashup that reads two Excel files and extracts data for a Sales report. A customer is using the product to read VoIP usage reports for billing. You can watch a video of that mashup being built here.

To keep development simple and straightforward, the company has developed a high-level scripting language, X#. The CodeGlide platform converts X# to Java. Of course, creating a new scripting language is risky: anything new or different can be a barrier to adoption. But if the scripting language really is both powerful and simple, and if the design tools minimize even the exposure to X#, CodeGlide may find that its reach really does extend to less technical users, who may have shunned mashups and Web 2.0 development until now.

To build mashups, users drag connectors and logical functions onto a design pane. Right-clicking on a connector or function opens a dialog box for configuring it. To select content or check for conditional status, developers use a filter function. There's also a foreach component for processing input items in turn.

As far as data integration goes, the company boasts an extensive set of connectors, including connectors for popular databases, including MySQL, Oracle, PostgreSQL, and DB2; common file types, such Excel; email services, such as POP3 and IMAP; and Web services, including WSDL-based SOAP services. A "screen-scraper" connector enables developers to access content displayed on Web pages.

Overall, it's a broad feature set for a young company. CodeGlide software is licensed under GLPv3. The Professional Edition of the Fusion mashup and data integration platform is available for $10,000, which is a one-time fee. DeGennaro points out that this price is far below the price of other mashup platforms, such as JackBe.

Low price. Lots of features. Big plans. CodeGlide could do well in an IT environment where business are looking for lots of bang for their buck.

Not All Data Integration Connectors Are Alike

Connectors are a vital part of any data integration solution. No matter what data sources you're integrating—databases, applications, flat files, Web services, etc.—it's awfully handy to have a preconfigured connector or at least a template to minimize the amount of hand-coding required to move data out of or into a particular data source.

The importance of connectors is perfectly clear to customers. In news stories, such SaaS Integration: Real-World Problems, And How CIOs Are Solving Them, which appeared in InformationWeek in October, customers are blunt about their expectations regarding connectors: vendors need to have a lot of them, one for every piece of middleware being integrated, and vendors better know how to make them work.

[H.B. Fuller CIO Steven] John is asking SaaS vendors lots of questions related to middleware, such as whether they have developed plug-ins for a specific middleware package and whether they have direct experience implementing that middleware. "If they say no to either, it's a strike against them," he says.

Recognizing the importance of connectors to prospects, most integration vendors parade their list of connectors on their Web sites.

And certainly, if you walk the tradeshow floor at events like the O'Reilly Web 2.0 conference or the Enterprise 2.0 Conference in Boston, you'll find vendors rattling off the names of the connectors they have.

"SAP? Oh, yeah. We've got a connector for that."

I've written before about the misleading simplicity of this approach. Data connectors aren't like Converse sneakers. You can't simply amass a bunch of them (red, orange, purple, black), and assume you have what you need for every occasion.

Different data sources have different security and access requirements. Applications integrating with protected data need to ensure that the data remains protected. You certainly don't want to bypass all the security and access controls protecting, say, a SAP ERP system, simply so that social platform users can pull ERP data into their wiki pages. These requirements become even more pressing when you're dealing with data in the cloud, where it's outside the perimeter of an internally secured and controlled data center.

Another integration requirement, above mere "connectivity," is transformation. Data might need to be transformed ("groomed") or narrowed before being presented to a group. For example, if I'm pulling in sales numbers from the Tokyo office, I'd probably like to see the amounts in yen converted to dollars. It would be nice to have the integration solution do this, so that I know I'm using a tested conversion tool that the company officially endorses.

Beyond merely connecting, then, connectors may need to work as part of an integration solution that supports access controls, transformation, auditability, and orchestration (e.g., before providing data set X, ensure than operation Y is complete, so that X is valid and up-to-date).

The NetSuite Example

The other fallacy of the "Converse sneaker" approach to connectors is it assumes that all integration endpoints and APIs are more or less alike. There's a DB interface or a middleware API. You write to it. You're done.

Not always.

Some APIs are more interactive and complex.

NetSuite, the hosted provider of mid-market CRM, ERP, and accounting solutions, now has 6,600 active customers. So lots of people have reason to connect to NetSuite data.

Integrating with NetSuite, however, is more involved than integrating with other SaaS applications. Why? Because NetSuite provides a great deal of flexibility in customizing their basic record schema and in defining custom records. You can query meta data to discover some, but not all, aspects of the customization. So a connector cannot rely on an automated process for discovering how an account has been customized and access its data.

Ideally, a connector should mask as much of this complexity as possible from the IT user creating the integration, so a good NetSuite connector will take advantage of the flexibility of the NetSuite API, while hiding as much of its complexity as possible from the user building or using the integration.

SnapLogic, an open source data integration company, offers a NetSuite connector that automates as much of this discovery as possible, while supporting integration that works with custom NetSuite records. Here's an explanation from the SnapLogic documentation pages:

NetSuite allows customization of its schema by allowing users to define custom record types and by adding custom fields to existing records. This extension package can automatically discover all the custom record types in a NetSuite account at install time. However, NetSuite does not provide interfaces which allow the extension package to discover custom fields have been added to all existing record types. The extension package is able to do this discovery for some kinds of records (like entity records, item records and CRM records), but not all. For this reason, a manual approach has been provided for specifying the custom fields of records. The user can use the NetSuite UI to browse the records and find custom fields that are of interest. The user can then use the utility: netsuite/resources/customize_resources.py (provided by the extensions package) to manually add custom fields to the SnapLogic Resources that represent a given NetSuite record types.

The connector automatically discovers of the accounts data schema as possible, then supports one-time additions for custom data. Once this set-up work is done, the user has a collection of ready-to-use, snap-together building blocks for building integration pipelines.

Because the connection reads the NetSuites schema, generates components, and supports customizations, it's able to provide NetSuite customers with a flexible solution for integrating NetSuite with other applications and data sources.

A more perfunctory connector would look just as good in a check list of available connectors, but it wouldn't serve users nearly as well.

Zoho CloudSQL: An interview with Rodrigo Vaca

Earlier this week, Zoho announced CloudSQL, a new SQL interface to Zoho Reports, its popular Web application for online reporting and business intelligence. Zoho applications (in case you haven't heard of them) are credible alternatives to Google Software-as-a-Service (SaaS) applications such as Google Docs. Launched three years ago, the suite of Zoho applications has grown dramatically in number of applications, richness of features, and size of its user base. The company now boasts over 1 million users for its 19 applications. More applications are on the way.


Here's how Rodrigo Vaca, Zoho's Director of Marketing, described CloudSQL in a blog post earlier this week:

Zoho CloudSQL is a middleware technology that allows customers to interact with their business data stored in Zoho through the familiar SQL language. Customers are able to access Zoho cloud data using SQL on both other cloud applications as well as through traditional on-premises software.

At a high-level, Zoho CloudSQL serves as the bridge between the external application and the data stored inside Zoho. It receives the query in SQL, interprets it, delegates queries and aggregates results across the Zoho services.

There are in particular 3 things that stand out about Zoho CloudSQL:

• It's the first technology that allows customers to interact with their data on the cloud, from another cloud application or from an on-premises one through real SQL.


• It supports multiple SQL dialects. We support all the major (and even some not so major) ones: ANSI, Oracle, SQL Server, IBM DB2, MySQL, PostgreSQL and Informix.


• With our JDBC/ODBC drivers, developers can access data in the cloud just as easily as if it were stored in a local database.
  

A Quick Interview
I got in touch with Rodrigo Vaca to ask him a few follow-up questions.

JB: From your announcement, I'm gathering that CloudSQL is a SQL-based service for accessing data in Zoho applications. The interface will be of interest to engineers working on integration projects where they would like to simply work with SQL queries, rather than dealing with JSON or RESTful data access. Is this an accurate characterization?

RV: Yes, that's accurate. Zoho CloudSQL is about making the data in the Zoho cloud more accessible for our customers. SQL is something that most corporate developers know and are familiar with.


JB: The diagram on your December 2 blog post shows CloudSQL being able to access other Web services. Are there non-Zoho Web services you plan to support? Say, any Web services from StrikeIron, ProgrammableWeb, or even Google, etc.?

RV: Ah! You were paying attention! You noticed something that most other people missed. Yes, Zoho CloudSQL can be extended to non-Zoho services. At this point we're not focused or actively pursing that, since we need to first make sure that other Zoho services are accessible through CloudSQL first.

JB: Finally, I was intrigued to see that you're doing entity-mapping, which makes sense. It makes me think of the work Microsoft has been doing in its Project Astoria group (creating a framework now called ADO.NET), where they're using entity mapping to present a non-SQL-based interface to SQL-Server data. Do your RESTful APIs make use of this entity mapping? Does Zoho have plans to publish an Astoria-like interface to Zoho data?

RV: Our REST API should provide all the necessary details for developers, so we don't have plans for entity-mapping like Astoria. We would recommend CloudSQL, as the standard interface for developers, especially as we increase its coverage across Zoho applications.

To learn more about CloudSQL, visit this Zoho wiki page here.

Moore's Law for Data Integration

The year is 1999, and your company wants to customize its new CRM system so sales people can access contract records from the Finance department's database. The CRM system itself took 9 months to deploy. Everyone's tired of the training classes. The six consultants who implemented the CRM system have lost a few people and gained a few people. Looks like they'll be bringing in someone else to manage this customization, which should take 4-5 months, tops.

Flashforward to 2008. It's September, you're using SugarCRM as your CRM system. You'd like to, again, integrate your CRM system with your Finance system. This time you'd like to do it using MindTouch Deki, a popular open source wiki and collaboration platform, and SnapLogic, an open source data integration framework. Good choice: the entire project, from start to finish, is completed in under two weeks.

No big, expensive consulting contracts. No "tent village" of Big Four consultants camped out by the computer room. An IT manager conceives the project, and within two weeks, it's done.

There's a paradigm shift that's occurred in enterprise IT, and it promises to make the next few years genuinely exciting, despite the downturn.

Over the past decade, application vendors have learned the wisdom of first, opening their APIs, and second, eliminating the complexity of their APIs by adopting a RESTful Web services model for integration.

The result has been a increase in agility for business users and IT department that is proving to be as dramatic, in terms of applications deployments and end user experience, as Moore's law has been for hardware development. Moore's law, you'll recall, is Gordon Moore's observation, first made in 1965, that "the number of transistors that can be placed inexpensively on an integrated circuit has increased exponentially, doubling approximately every two years" (Wikipedia). Thanks to Moore's law, your bookbag can hold a laptop more powerful that a mainframe from a few decades ago. And you watch videos on your MP3 player, which is roughly the size of a pack of gum.

Open source and RESTful APIs (which make use of basic GETs and PUTs, rather than relying on more complex messaging schemes) give enterprise IT organization powerful building blocks for rapidly building new, powerful application solutions—applications that literally would have required man years of programming less than a decade ago.

A year ago, there was a lot of buzz about enterprise mashups that blended two or more data sources. Mashups are still exciting, but what's equally exciting is a kind of mashup occurring at the application level.

Today's announcement by Salesforce.com that its Force.com development platform would work with Facebook APIs, enabling Salesforce.com's 100,000-strong developer community to more easily access the vast library of applications built on Salesforce, is more evidence of this trend.

Other evidence of this trend:

• The ongoing success of the Web site, ProgrammableWeb, which serves as a portal for discovering application APIs and mashups. ProgrammableWeb now has 1,000 APIs in its API directory. Clearly, a lot of companies are publishing APIs.
  
• Once published, APIs often become the dominant channel for accessing an application. As ProgrammableWeb's John Musser points out in a blog post about the 1,000-API milestone:
  
  
  
  • 60% of eBay's listing come from their APIs, rather than through their browser-based interface.
    
  • Twitter's APIs carry 10x the traffic of its Web site.
    
  
  
• REST is becoming the dominant programming model for APIs. 63% of the APIs listed in ProgrammableWeb's directory are RESTful.
  
• New Content Management Systems, such as Alfresco Enterprise 3.0, feature REST interfaces so they can easily access business data from other IT systems.
  
• New applications, such as an increasingly popular network management/IT operations platform offered by an Indian company, use REST APIs to facilitate communication among components. RESTful integration enables one product, for example, a network troubleshooting tool, to easily pass information to a related component, such as a trouble-ticketing application used by a help desk.
  

Enterprise Management Associates analyst Dennis Drogseth is fond of saying that enterprise IT organizations know they need to move to a "lego world." No single IT system has a monopoly on vital data. Best practices call for the automated flow of information seamlessly from one system to another: from a network diagnostic tool to a help desk application, for example, or from a finance system into a wiki, which in turn is embedded in SugarCRM. The key to that flow is having components designed to fit together with other components, even from other vendors.

It sounds like a lot of work. It's dramatically less work than it used to be. Thanks to open APIs, REST, and Web-centric architectures, data integration and application development can beneft from a Moore's law of their own.

Data from a Straw

I originally wrote this piece after attending the Enterprise 2.0 Conference in Boston in June. I'm just back from the Web 2.0 Expo in New York, and I see that the ideas still apply. So here are some thoughts on data integration and social computing platforms. -JB

At the Enterprise 2.0 conference in Boston, there was a lot of talk about data. By applying Web 2.0 technology and practices—blogs, wikis, social networks, tagging, RSS, etc.—Enterprise 2.0 would transform enterprise IT infrastructure and foster the collaboration and knowledge-sharing promised by earlier technology practices such as knowledge management. In this new era, users at last will be able to find data easily and discover who else in the company has similar interests and pertinent knowledge. Through collaboration platforms such as Microsoft SharePoint and Jive Software Clearspace, data previously buried in email messages and PC desktops would be published on company blogs and wikis, where it could be found, read, and elaborated upon by coworkers and, if appropriate, by partners and customers.

The software companies creating these portals recognize that a lot of valuable data isn’t found in email or Words documents; instead, it's distributed across data centers and departments in databases and data warehouses. So the portal vendors talk about being able to access Oracle and SAP and other enterprise data sources, in order to pull this data into the collaboration platform.

But as I talked to vendors, I found their views of data access in many cases to be overly simplistic. Their premise seemed to be that all one needs to do is attach a connector to a data source and suck the data out, much as one might stick a straw into a paper cup and extract whatever concoction is sloshing about inside.

If you talk to data integration experts in data centers—or if you talk to security officers for Fortune 1000 companies—you quickly discover that the requirements for data access are much more varied and nuanced. It's rare that you’ll actually want to simply extract data and, say, stage it in an Excel worksheet on a server where it can be accessed by a homogeneous group of authorized users. More likely, you’ll want to apply access controls before the data even reaches a collaboration server, and you’ll need several different views of the data, based on business needs and permissions.

Instead of simply extracting data, it's more useful to think in terms of data access, data transformation, and data delivery. The tables below compare these approaches.

First, here’s the kind of straightforward data access that software vendors often talk about.

Table 1: Simple Data Access

Data Source	Data Access
Customer database	Post customer records as Excel spreadsheet for SharePoint

Next, here's a more realistic scenario, at least for organizations operating under security policies or industry regulations that mandate data security and data governance.

Table 2: Data Access with Support for Data Transformation and Data Delivery

Data Source	Data Access	Data Transformation	Data Delivery
Customer database	Query customer records, presenting only columns 1, 2, 5, and 7	Convert dollars to Euro	Post results to spreadsheet or Web page accessed by EMEA marketing group
	Query customer records, collecting columns 1-5 and 7-9	Add a unique ID to each record for use in this project	Post results to portal used by Private Client Group
	Query customer records, returning columns 1-4		Make this query executable for customer service agents working on the customer service portal

In enterprises operating with strict security and compliance controls, it's rare for data to be simply dumped from a database and made broadly accessible. Policy compliance requires tighter controls over data access (permission to extract the data from its source) and data delivery (the presentation of data to specific users).

Businesses—and software vendors—ought to recognize the critical importance of data transformation: changing, reformatting, or editing data to suit its particular purpose and audience. There's no point in delivering too much data, or financial results in dollars when they should be in yen, or raw data from three sources that end users have to combine for themselves through machinations with spreadsheets. In the real world of harried workers overloaded with information, data transformation is an essential capability for any effective solution for data management and knowledge sharing.

Just as Enterprise 2.0 frees workers from the clutter of irrelevant email messages, so flexible data access and transformation practices can ensure that the right users receive the right data at the right time. The goal should be to get everyone all the data they need—and nothing more.

Conclusion

At the Enterprise 2.0 Conference, it was obvious that software vendors of collaboration and community platforms have made clear progress developing attractive, usable front-ends. Now it's time to apply that same energy and thoughtfulness to developing the back end—data access, transformation, and delivery—in order to realize the full vision of business-ready data platforms for Enterprise 2.0.

Postscript: Since I wrote blog post back in June, SnapLogic, an open source data integration vendor and a client of mine, formed a partnership with MindTouch, an open source wiki company, to create a Customer Relationship Management (CRM) solution building on the kind of custom-tailored data access, transformation, and delivery I described above. In the SnapLogic-MindTouch solution (summarized with a diagram here), CRM applications such as Salesforce.com and SugarCRM are extended with collaborative dashboards based on MindTouch's wiki platform. The wiki is configured with SnapLogic data integration pipelines, enabling CRM users to securely access financial data and customer support records for prospects and customers. No tell-all spreadsheets insecurely posted on servers. Instead, a wealth of account-specific data is made available to authorized users.

I expect will see more partnerships like this one in the coming months.

Integrating SaaS Applications

Software-as-a-Service (SaaS) certainly qualifies as a hot trend in IT. According to Gartner, the SaaS market is growing at twice the rate of the software market overall. SaaS applications such as Salesforce.com are especially popular with SMBs, but even large enterprises with more than 25,000 employees are devoting 11% of their current software budgets to SaaS.

SaaS, then, is big and getting bigger. But getting bigger, too, is the challenge of integrating SaaS applications with the rest of an organization's IT infrastructure. After all, no organization can afford to let the latest SaaS application it subscribes to become another data silo. SaaS applications need to share data with other enterprise applications and IT assets.

Integrating SaaS applications can be difficult, in part because there's an technology mismatch between SaaS application design and the community of SaaS customers. Most SaaS applications feature SOAP Web services interfaces whose complexity and sophistication are far beyond the programming reach of SMBs, the biggest users of SaaS. Even in large organizations with SOAP programmers on staff, SOAP programming often remains a rare skill. SOAP programmers are usually assigned to big internal initiatives. They're usually not available to help a department integrate its data sources with a new, cost-saving SaaS application.

One of my clients, SnapLogic, has published a new white paper on SaaS integration. If you're working with SaaS applications now, or thinking of subscribing to Salesforce.com, SugarCRM, or any other popular SaaS application, the white paper is probably worth a read. You can register for it here.