Evan on Engineering Software

Yesterday, I had a little fun at Ralph Grabowski's expense, talking about a press release from SolidWorks, where their customer, Vault Structures, was quoted talking about how much money and time they saved using SolidWorks versus AutoCAD.

While I may have poked fun, Ralph did have a very good point:  SolidWorks versus Autodesk Inventor (or Solid Edge, or Think3, or Pro/E, or any other parametric feature-based CAD program) is a technically fairer comparison than SolidWorks versus AutoCAD.

One question I have (and for which I don't have a good answer) is how you'd go about doing a meaningful comparison of SolidWorks and Inventor?

Consider these factors:

• Is the comparison for a specific company, or more general in nature?
• Do you include externalities, such as the availability of trained users, the historical performance of the vendor, or the restrictiveness of the license agreement?
• Do you use a feature checklist, or focus on use cases?
• Do you have vendor application engineers run the benchmarks, or have your own people do it?
• How do you account for bugs?
• How do you measure the differences in productivity curves (i.e, the integral of individual productivity over the  population of users.)
• Do you grade based on cost, or based on return?
• How do you account for time-to-market? (Small differences in delivering products can make big differences in profitability.)
• How do you grade?  (And, even if you think you have a good answer for this, I'd recommend you take a look at Dr. David Ullman's book, Making Robust Decisions.

I think the bottom line here is that these kind of comparisons are really hard.

Let's go back to the example of Vault Structures.  They said that they were able to design and test a vault door 70% faster using SolidWorks compared to AutoCAD, saving $150,000 in prototyping costs. 

That savings estimate is based on what they expected it would have taken with AutoCAD.  But, let me pose a few questions:

• What if they would have hired the *best* AutoCAD guru out there (I can think of a few), and had coupled the use of AutoCAD with tools such as Algor and GraphiCalc and IDX?  How much time and prototyping savings could they have gained with such a best-in-class AutoCAD-centric approach?
• What if they would have hired the best SolidWorks guru out there (again, I can think of a few)?  Would the savings in time and prototype cost be even greater?
• What if they would have built a knowledge-based vault door design system on top of SolidWorks?  The initial design probably would have taken longer, but factoring in a nearly 100% savings in time and prototyping costs for subsequent vault doors would have changed the return on investment dramatically.

I'd guess that, had Vault Structures used Inventor instead of SolidWorks, the results would have been in the same neighborhood.  Depending on human and process factors.  And that's the interesting part.

As interesting as a comparison between products such as SolidWorks and Inventor might be, what I'd really find intriguing would be a comparison between the typical usage of a CAD program, and its usage where human and process factors have been optimized.