Introducing CodeGlide, a New Open Source Mashup and Data Integration Company

Economic downturn or no, the open source data integration market remains active. In late January, Talend announced a $12 million C round of financing. Over the past two years, the company's open source data integration solution has been downloaded 3.3 million times, and the company now boasts hundreds of customers. Other vendors in the market, such as SnapLogic, continue to add connectors and develop their capabilities.

And now comes a new vendor to the scene: CodeGlide, an open source company with a broad business application platform and big ambitions. I spoke recently with Mauro DeGennaro, CodeGlide's co-founder and Vice President of Software Development. He filled me on the company's approach, offerings, and direction.

CodeGlide was founded in May 2007. The company's goal is to offer a comprehensive suite of open source business applications, including:

• Fusion, an open source data integration platform for building mashups
  
• SnapCRM, an open source CRM suite (with a provocative title, since the company competes against SnapLogic)
  
• Rendezvous, an open source collaboration suite, which offers calendaring, document management, and project management features
  

These three platforms are available in an Enterprise Suite bundle, which is the company's most popular download.

Speaking of downloads, CodeGlide seems to be off to a fast start. The company's Web site came on line on January 15. I spoke to Mauro about a week ago, and already the company had racked up 11,000 downloads of its software. Through its partners, the company has sold its Professional Edition to six different companies.

Partners are also lending their expertise to help the company develop other applications, including an E-learning application, an ERP application, and a Platform-as-a-Service (PaaS) offering that will compete against Coghead", LongJump, and other PaaS solution providers.

CodeGlide has a core development team of ten programmers, who are augmented by free-lancers and sub-companies that offer domain expertise in areas such as ERP. Business people and design experts design new features, which the developers then build. Of course, being an open source company, CodeGlide can expect to receive further enhancements and features from its open source community, as well. The company is small, but global: development is based in Argentina, Q.A. in India, sales in Europe and the U.S.

The CodeGlide Approach

CodeGlide's goal is to offer open source business solutions for the SMB market. It wants to make mashups and data integration solutions available to developers without advanced programming skills and even to business people. Accordingly, the company offers a visual tool for building mashups and creating widgets. For example, using Fusion, it's possible to construct a mashup that reads two Excel files and extracts data for a Sales report. A customer is using the product to read VoIP usage reports for billing. You can watch a video of that mashup being built here.

To keep development simple and straightforward, the company has developed a high-level scripting language, X#. The CodeGlide platform converts X# to Java. Of course, creating a new scripting language is risky: anything new or different can be a barrier to adoption. But if the scripting language really is both powerful and simple, and if the design tools minimize even the exposure to X#, CodeGlide may find that its reach really does extend to less technical users, who may have shunned mashups and Web 2.0 development until now.

To build mashups, users drag connectors and logical functions onto a design pane. Right-clicking on a connector or function opens a dialog box for configuring it. To select content or check for conditional status, developers use a filter function. There's also a foreach component for processing input items in turn.

As far as data integration goes, the company boasts an extensive set of connectors, including connectors for popular databases, including MySQL, Oracle, PostgreSQL, and DB2; common file types, such Excel; email services, such as POP3 and IMAP; and Web services, including WSDL-based SOAP services. A "screen-scraper" connector enables developers to access content displayed on Web pages.

Overall, it's a broad feature set for a young company. CodeGlide software is licensed under GLPv3. The Professional Edition of the Fusion mashup and data integration platform is available for $10,000, which is a one-time fee. DeGennaro points out that this price is far below the price of other mashup platforms, such as JackBe.

Low price. Lots of features. Big plans. CodeGlide could do well in an IT environment where business are looking for lots of bang for their buck.

Not All Data Integration Connectors Are Alike

Connectors are a vital part of any data integration solution. No matter what data sources you're integrating—databases, applications, flat files, Web services, etc.—it's awfully handy to have a preconfigured connector or at least a template to minimize the amount of hand-coding required to move data out of or into a particular data source.

The importance of connectors is perfectly clear to customers. In news stories, such SaaS Integration: Real-World Problems, And How CIOs Are Solving Them, which appeared in InformationWeek in October, customers are blunt about their expectations regarding connectors: vendors need to have a lot of them, one for every piece of middleware being integrated, and vendors better know how to make them work.

[H.B. Fuller CIO Steven] John is asking SaaS vendors lots of questions related to middleware, such as whether they have developed plug-ins for a specific middleware package and whether they have direct experience implementing that middleware. "If they say no to either, it's a strike against them," he says.

Recognizing the importance of connectors to prospects, most integration vendors parade their list of connectors on their Web sites.

And certainly, if you walk the tradeshow floor at events like the O'Reilly Web 2.0 conference or the Enterprise 2.0 Conference in Boston, you'll find vendors rattling off the names of the connectors they have.

"SAP? Oh, yeah. We've got a connector for that."

I've written before about the misleading simplicity of this approach. Data connectors aren't like Converse sneakers. You can't simply amass a bunch of them (red, orange, purple, black), and assume you have what you need for every occasion.

Different data sources have different security and access requirements. Applications integrating with protected data need to ensure that the data remains protected. You certainly don't want to bypass all the security and access controls protecting, say, a SAP ERP system, simply so that social platform users can pull ERP data into their wiki pages. These requirements become even more pressing when you're dealing with data in the cloud, where it's outside the perimeter of an internally secured and controlled data center.

Another integration requirement, above mere "connectivity," is transformation. Data might need to be transformed ("groomed") or narrowed before being presented to a group. For example, if I'm pulling in sales numbers from the Tokyo office, I'd probably like to see the amounts in yen converted to dollars. It would be nice to have the integration solution do this, so that I know I'm using a tested conversion tool that the company officially endorses.

Beyond merely connecting, then, connectors may need to work as part of an integration solution that supports access controls, transformation, auditability, and orchestration (e.g., before providing data set X, ensure than operation Y is complete, so that X is valid and up-to-date).

The NetSuite Example

The other fallacy of the "Converse sneaker" approach to connectors is it assumes that all integration endpoints and APIs are more or less alike. There's a DB interface or a middleware API. You write to it. You're done.

Not always.

Some APIs are more interactive and complex.

NetSuite, the hosted provider of mid-market CRM, ERP, and accounting solutions, now has 6,600 active customers. So lots of people have reason to connect to NetSuite data.

Integrating with NetSuite, however, is more involved than integrating with other SaaS applications. Why? Because NetSuite provides a great deal of flexibility in customizing their basic record schema and in defining custom records. You can query meta data to discover some, but not all, aspects of the customization. So a connector cannot rely on an automated process for discovering how an account has been customized and access its data.

Ideally, a connector should mask as much of this complexity as possible from the IT user creating the integration, so a good NetSuite connector will take advantage of the flexibility of the NetSuite API, while hiding as much of its complexity as possible from the user building or using the integration.

SnapLogic, an open source data integration company, offers a NetSuite connector that automates as much of this discovery as possible, while supporting integration that works with custom NetSuite records. Here's an explanation from the SnapLogic documentation pages:

NetSuite allows customization of its schema by allowing users to define custom record types and by adding custom fields to existing records. This extension package can automatically discover all the custom record types in a NetSuite account at install time. However, NetSuite does not provide interfaces which allow the extension package to discover custom fields have been added to all existing record types. The extension package is able to do this discovery for some kinds of records (like entity records, item records and CRM records), but not all. For this reason, a manual approach has been provided for specifying the custom fields of records. The user can use the NetSuite UI to browse the records and find custom fields that are of interest. The user can then use the utility: netsuite/resources/customize_resources.py (provided by the extensions package) to manually add custom fields to the SnapLogic Resources that represent a given NetSuite record types.

The connector automatically discovers of the accounts data schema as possible, then supports one-time additions for custom data. Once this set-up work is done, the user has a collection of ready-to-use, snap-together building blocks for building integration pipelines.

Because the connection reads the NetSuites schema, generates components, and supports customizations, it's able to provide NetSuite customers with a flexible solution for integrating NetSuite with other applications and data sources.

A more perfunctory connector would look just as good in a check list of available connectors, but it wouldn't serve users nearly as well.

Data from a Straw

I originally wrote this piece after attending the Enterprise 2.0 Conference in Boston in June. I'm just back from the Web 2.0 Expo in New York, and I see that the ideas still apply. So here are some thoughts on data integration and social computing platforms. -JB

At the Enterprise 2.0 conference in Boston, there was a lot of talk about data. By applying Web 2.0 technology and practices—blogs, wikis, social networks, tagging, RSS, etc.—Enterprise 2.0 would transform enterprise IT infrastructure and foster the collaboration and knowledge-sharing promised by earlier technology practices such as knowledge management. In this new era, users at last will be able to find data easily and discover who else in the company has similar interests and pertinent knowledge. Through collaboration platforms such as Microsoft SharePoint and Jive Software Clearspace, data previously buried in email messages and PC desktops would be published on company blogs and wikis, where it could be found, read, and elaborated upon by coworkers and, if appropriate, by partners and customers.

The software companies creating these portals recognize that a lot of valuable data isn’t found in email or Words documents; instead, it's distributed across data centers and departments in databases and data warehouses. So the portal vendors talk about being able to access Oracle and SAP and other enterprise data sources, in order to pull this data into the collaboration platform.

But as I talked to vendors, I found their views of data access in many cases to be overly simplistic. Their premise seemed to be that all one needs to do is attach a connector to a data source and suck the data out, much as one might stick a straw into a paper cup and extract whatever concoction is sloshing about inside.

If you talk to data integration experts in data centers—or if you talk to security officers for Fortune 1000 companies—you quickly discover that the requirements for data access are much more varied and nuanced. It's rare that you’ll actually want to simply extract data and, say, stage it in an Excel worksheet on a server where it can be accessed by a homogeneous group of authorized users. More likely, you’ll want to apply access controls before the data even reaches a collaboration server, and you’ll need several different views of the data, based on business needs and permissions.

Instead of simply extracting data, it's more useful to think in terms of data access, data transformation, and data delivery. The tables below compare these approaches.

First, here’s the kind of straightforward data access that software vendors often talk about.

Table 1: Simple Data Access

Data Source	Data Access
Customer database	Post customer records as Excel spreadsheet for SharePoint

Next, here's a more realistic scenario, at least for organizations operating under security policies or industry regulations that mandate data security and data governance.

Table 2: Data Access with Support for Data Transformation and Data Delivery

Data Source	Data Access	Data Transformation	Data Delivery
Customer database	Query customer records, presenting only columns 1, 2, 5, and 7	Convert dollars to Euro	Post results to spreadsheet or Web page accessed by EMEA marketing group
	Query customer records, collecting columns 1-5 and 7-9	Add a unique ID to each record for use in this project	Post results to portal used by Private Client Group
	Query customer records, returning columns 1-4		Make this query executable for customer service agents working on the customer service portal

In enterprises operating with strict security and compliance controls, it's rare for data to be simply dumped from a database and made broadly accessible. Policy compliance requires tighter controls over data access (permission to extract the data from its source) and data delivery (the presentation of data to specific users).

Businesses—and software vendors—ought to recognize the critical importance of data transformation: changing, reformatting, or editing data to suit its particular purpose and audience. There's no point in delivering too much data, or financial results in dollars when they should be in yen, or raw data from three sources that end users have to combine for themselves through machinations with spreadsheets. In the real world of harried workers overloaded with information, data transformation is an essential capability for any effective solution for data management and knowledge sharing.

Just as Enterprise 2.0 frees workers from the clutter of irrelevant email messages, so flexible data access and transformation practices can ensure that the right users receive the right data at the right time. The goal should be to get everyone all the data they need—and nothing more.

Conclusion

At the Enterprise 2.0 Conference, it was obvious that software vendors of collaboration and community platforms have made clear progress developing attractive, usable front-ends. Now it's time to apply that same energy and thoughtfulness to developing the back end—data access, transformation, and delivery—in order to realize the full vision of business-ready data platforms for Enterprise 2.0.

Postscript: Since I wrote blog post back in June, SnapLogic, an open source data integration vendor and a client of mine, formed a partnership with MindTouch, an open source wiki company, to create a Customer Relationship Management (CRM) solution building on the kind of custom-tailored data access, transformation, and delivery I described above. In the SnapLogic-MindTouch solution (summarized with a diagram here), CRM applications such as Salesforce.com and SugarCRM are extended with collaborative dashboards based on MindTouch's wiki platform. The wiki is configured with SnapLogic data integration pipelines, enabling CRM users to securely access financial data and customer support records for prospects and customers. No tell-all spreadsheets insecurely posted on servers. Instead, a wealth of account-specific data is made available to authorized users.

I expect will see more partnerships like this one in the coming months.

At Web 2.0 Expo in New York, an Instant Solution for CRM Integration

The first Web 2.0 Expo in New York kicks off today. About 120 vendors will be showing off the latest in software and services related to community, Web-contributed content, and other aspects of Web 2.0.

Among the exhibitors will be SnapLogic, a client of mine, who just announced an OEM deal with MindTouch. The two companies have created a new software solution that extends CRM applications such as SugarCRM to include live data from other business systems, such as finance applications, databases, and more. The diagram below captures the gist of the solution.

Why is this useful? Now a salesperson reviewing customer data in a CRM application can also see relevant data from other business systems. Is this customer due for a maintenance renewal? What's their payment history? Do they typically pay on time? Have other sales, marketing, or support people left comments offering advice about working with this customer? Deki for CRM puts answers to questions like those right in the CRM application window, formatted in a readable, editable wiki workspace.

Five years ago, a solution like this would have entailed tens or hundreds of thousands of dollars and a six- to twelve-month roll-out of a major CRM application like Siebel. SnapLogic and MindTouch put the solution together in a matter of weeks. Customers can have it up and running in a day.

You can read more about Deki for CRM here.

Integrating SaaS Applications

Software-as-a-Service (SaaS) certainly qualifies as a hot trend in IT. According to Gartner, the SaaS market is growing at twice the rate of the software market overall. SaaS applications such as Salesforce.com are especially popular with SMBs, but even large enterprises with more than 25,000 employees are devoting 11% of their current software budgets to SaaS.

SaaS, then, is big and getting bigger. But getting bigger, too, is the challenge of integrating SaaS applications with the rest of an organization's IT infrastructure. After all, no organization can afford to let the latest SaaS application it subscribes to become another data silo. SaaS applications need to share data with other enterprise applications and IT assets.

Integrating SaaS applications can be difficult, in part because there's an technology mismatch between SaaS application design and the community of SaaS customers. Most SaaS applications feature SOAP Web services interfaces whose complexity and sophistication are far beyond the programming reach of SMBs, the biggest users of SaaS. Even in large organizations with SOAP programmers on staff, SOAP programming often remains a rare skill. SOAP programmers are usually assigned to big internal initiatives. They're usually not available to help a department integrate its data sources with a new, cost-saving SaaS application.

One of my clients, SnapLogic, has published a new white paper on SaaS integration. If you're working with SaaS applications now, or thinking of subscribing to Salesforce.com, SugarCRM, or any other popular SaaS application, the white paper is probably worth a read. You can register for it here.