Incomplete Analysis

Salesforce.com is providing prospects a study from CSO Insights titled “On-Demand Versus On-Premise CRM: Are There Performance Differences?” The study purports to compare these two types of systems across a number of factors including: performance improvement, implementation timeframe, cost relative to budget, and whether the company would recommend the same solution to others. Is it any surprise that the report shows the on demand solutions to have a distinct advantage in all these categories? It shouldn’t be.

I contacted the authors of this study to learn more about the methodology used for this study, which gives me serious doubts about their conclusion. They started with 1275 companies taking a sales effectiveness study. Of those, about 800 had fully implemented a CRM system. From here the authors selected just the vendors that had “significant client bases”, which pared it down to a survey set of 350. The on-demand results are an aggregation of individual results of Salesforce.com, Siebel on-demand, and NetSuite. The on-premise results are made up of Oracle, Siebel, PeopleSoft, SAP, and Microsoft.

Here’s the problem. Based on the group of on-premise vendors selected I think there are two other very plausible (and seemingly obvious) explanations for why that group had longer implementation time frames, less impact on performance, more cost overruns, etc. With the exception of Microsoft, those on-premise vendors (1) target much larger customers and (2) have much deeper products that are more complex to implement and configure. Without evidence to the contrary, I’m inclined to believe that these two issues have a significant impact on why the on-demand group scored better.

I asked the folks at CSO Insights if might provide a split of the data by customer segment, or including just Microsoft (maybe plus SalesLogix) in the on-premise group. They seem uninterested. I pointed out the other potential conclusions one might draw based on the vendor groups, and suggested their assessment that it’s all about on-demand vs. on-premise would be much more credible if they could eliminate segment and complexity as key drivers. Again, not interested, but I could pay to have this work done.

Charge me? Here I thought I was the one providing the service by suggesting how they might make their conclusions much more credible. Pull out Oracle, Siebel, PeopleSoft, and SAP from the on-premise group and replace them with SalesLogix. Or don’t even replace them; just compare Microsoft to that group. If the on-demand group still scores higher, fine. If it doesn’t, then I understand not wanting to take another look. Salesforce.com no doubt has spent generously for the distribution rights to the study.