Data from a Straw

I originally wrote this piece after attending the Enterprise 2.0 Conference in Boston in June. I'm just back from the Web 2.0 Expo in New York, and I see that the ideas still apply. So here are some thoughts on data integration and social computing platforms. -JB

At the Enterprise 2.0 conference in Boston, there was a lot of talk about data. By applying Web 2.0 technology and practices—blogs, wikis, social networks, tagging, RSS, etc.—Enterprise 2.0 would transform enterprise IT infrastructure and foster the collaboration and knowledge-sharing promised by earlier technology practices such as knowledge management. In this new era, users at last will be able to find data easily and discover who else in the company has similar interests and pertinent knowledge. Through collaboration platforms such as Microsoft SharePoint and Jive Software Clearspace, data previously buried in email messages and PC desktops would be published on company blogs and wikis, where it could be found, read, and elaborated upon by coworkers and, if appropriate, by partners and customers.

The software companies creating these portals recognize that a lot of valuable data isn’t found in email or Words documents; instead, it's distributed across data centers and departments in databases and data warehouses. So the portal vendors talk about being able to access Oracle and SAP and other enterprise data sources, in order to pull this data into the collaboration platform.

But as I talked to vendors, I found their views of data access in many cases to be overly simplistic. Their premise seemed to be that all one needs to do is attach a connector to a data source and suck the data out, much as one might stick a straw into a paper cup and extract whatever concoction is sloshing about inside.

If you talk to data integration experts in data centers—or if you talk to security officers for Fortune 1000 companies—you quickly discover that the requirements for data access are much more varied and nuanced. It's rare that you’ll actually want to simply extract data and, say, stage it in an Excel worksheet on a server where it can be accessed by a homogeneous group of authorized users. More likely, you’ll want to apply access controls before the data even reaches a collaboration server, and you’ll need several different views of the data, based on business needs and permissions.

Instead of simply extracting data, it's more useful to think in terms of data access, data transformation, and data delivery. The tables below compare these approaches.

First, here’s the kind of straightforward data access that software vendors often talk about.

Table 1: Simple Data Access

Data Source	Data Access
Customer database	Post customer records as Excel spreadsheet for SharePoint

Next, here's a more realistic scenario, at least for organizations operating under security policies or industry regulations that mandate data security and data governance.

Table 2: Data Access with Support for Data Transformation and Data Delivery

Data Source	Data Access	Data Transformation	Data Delivery
Customer database	Query customer records, presenting only columns 1, 2, 5, and 7	Convert dollars to Euro	Post results to spreadsheet or Web page accessed by EMEA marketing group
	Query customer records, collecting columns 1-5 and 7-9	Add a unique ID to each record for use in this project	Post results to portal used by Private Client Group
	Query customer records, returning columns 1-4		Make this query executable for customer service agents working on the customer service portal

In enterprises operating with strict security and compliance controls, it's rare for data to be simply dumped from a database and made broadly accessible. Policy compliance requires tighter controls over data access (permission to extract the data from its source) and data delivery (the presentation of data to specific users).

Businesses—and software vendors—ought to recognize the critical importance of data transformation: changing, reformatting, or editing data to suit its particular purpose and audience. There's no point in delivering too much data, or financial results in dollars when they should be in yen, or raw data from three sources that end users have to combine for themselves through machinations with spreadsheets. In the real world of harried workers overloaded with information, data transformation is an essential capability for any effective solution for data management and knowledge sharing.

Just as Enterprise 2.0 frees workers from the clutter of irrelevant email messages, so flexible data access and transformation practices can ensure that the right users receive the right data at the right time. The goal should be to get everyone all the data they need—and nothing more.

Conclusion

At the Enterprise 2.0 Conference, it was obvious that software vendors of collaboration and community platforms have made clear progress developing attractive, usable front-ends. Now it's time to apply that same energy and thoughtfulness to developing the back end—data access, transformation, and delivery—in order to realize the full vision of business-ready data platforms for Enterprise 2.0.

Postscript: Since I wrote blog post back in June, SnapLogic, an open source data integration vendor and a client of mine, formed a partnership with MindTouch, an open source wiki company, to create a Customer Relationship Management (CRM) solution building on the kind of custom-tailored data access, transformation, and delivery I described above. In the SnapLogic-MindTouch solution (summarized with a diagram here), CRM applications such as Salesforce.com and SugarCRM are extended with collaborative dashboards based on MindTouch's wiki platform. The wiki is configured with SnapLogic data integration pipelines, enabling CRM users to securely access financial data and customer support records for prospects and customers. No tell-all spreadsheets insecurely posted on servers. Instead, a wealth of account-specific data is made available to authorized users.

I expect will see more partnerships like this one in the coming months.

In the Flesh

I had to laugh at one of the visitors to the show floor at Web 2.0 Expo last week.

The scene: The expo floor of the Javits Center on Thursday afternoon. The show is winding down. A couple of exhibitors have begun taking down signs and packing up laptops, though officially the show will remain open for another half hour.

On business for a client, I've been here and there on the show floor, talking to potential business partners. When I return to my client's booth, I find a middle-aged, pot-bellied man standing in front of our demo station, talking to a buddy of his. The pot-bellied man has set his laptop on our table and he's instructing his buddy, who's ill-shaven and looking a little worse for wear in his black suit, about which companies he should go call on. The pot-bellied man is unabashedly treating our booth as his private conference area.

I ask him if he'd like to see a demo. He says sure. I take control of our demo station, and I show him our demo: CRM software integrated with a Wiki. It's been wowing people all week.

He nods, but he's not really interested. I ask what he does. He offers to show me. He flips open his laptop, positions it in the middle of our demo table, and starts running online ads. You've seen them: video ads featuring a well-dressed model who wanders in from the side of the screen, blocks the Web page you're trying to view, and starts talking to you. He has a long list of demo links. I watch nattily dressed salespeople blocking the Web pages of several national retail brands.

I point out that my client sells software primarily to IT people, and that IT people have a low tolerance for anything that smacks of marketing, let alone anything as out-and-out slick and salesy as this. He slaps his business card on our table, makes perfunctory social remarks, and moves on.

Later, it occurred to me. I've always found ads like that intrusive, presumptuous, and annoying. And what kind of person would create and peddle ads like that? Someone who is himself intrusive, presumptuous, and annoying. Someone who would take over your booth at a tradeshow and use it as his personal office, blocking your computer and carrying on.

"By their fruits, ye shall know them." I had just seen a grating interstitial ad in the flesh.

At Web 2.0 Expo in New York, an Instant Solution for CRM Integration

The first Web 2.0 Expo in New York kicks off today. About 120 vendors will be showing off the latest in software and services related to community, Web-contributed content, and other aspects of Web 2.0.

Among the exhibitors will be SnapLogic, a client of mine, who just announced an OEM deal with MindTouch. The two companies have created a new software solution that extends CRM applications such as SugarCRM to include live data from other business systems, such as finance applications, databases, and more. The diagram below captures the gist of the solution.

Why is this useful? Now a salesperson reviewing customer data in a CRM application can also see relevant data from other business systems. Is this customer due for a maintenance renewal? What's their payment history? Do they typically pay on time? Have other sales, marketing, or support people left comments offering advice about working with this customer? Deki for CRM puts answers to questions like those right in the CRM application window, formatted in a readable, editable wiki workspace.

Five years ago, a solution like this would have entailed tens or hundreds of thousands of dollars and a six- to twelve-month roll-out of a major CRM application like Siebel. SnapLogic and MindTouch put the solution together in a matter of weeks. Customers can have it up and running in a day.

You can read more about Deki for CRM here.