Apples to Apples Benchmark? I Don't Think So!

Apples to Apples Benchmark? I Don't Think So!

I don’t normally take the time to respond to any of the various competitive claims out there. But recently in ESNUG 483, item #2, there was a posting entitled “We recently dumped Mentor Calibre for Magma Quartz DRC/LVS” (http://www.deepchip.com/items/0483-02.html) that I feel needs to be addressed because it is misleading. So let me lay out the facts to set the record straight.

Tezzaron Semiconductor Corp., Naperville, IL, became a Calibre customer when it purchased a perpetual license in 2000. In 2001, Tezzaron declined to renew technical support, and since then has not renewed the license, so they only have access to Calibre 2001.2, which was released in early 2001.

In his submission to John Cooley’s ESNUG site, Tezzaron’s Robert Patti, stated:

“We had been using Calibre for many years, but were not satisfied with it. In particular, we were looking for:

• An ability to scale efficiently across a larger # of CPU’s and machines. Calibre is OK on a single machine, but doesn’t do well outside of that. We need to get results faster.
• Better technical support. We do highly specialized designs. DRC customization is the norm rather than an exception for us.
• Native support for TCL. This is really important, as we need to be able to customize both the runsets and tool ourselves. Other EDA tools we use are TCL based. Calibre is based on a legacy runset language, and newer design rules need a better language option.”

I’ll address each of these issues in turn.

Calibre Performance

Eight years ago Calibre’s distributed processing functionality had not yet been released. Since then, Mentor has released numerous Calibre performance innovations, including the ability to thread and distribute rule file operations (Hyperscaling), and the ability to distribute layout data (Hyper Remote). Together, these features consistently provide the best performance and scaling in the market, as demonstrated in many competitive benchmarks. Here’s the data comparing the latest version of Calibre to our 2001 performance:

Calibre Performance 2001 vs 2009

As you can see, there is an order of magnitude performance improvement and great scalability on multiple CPUs.

Calibre Support

If you want technical support, there is none better than Mentor. Mentor is the only Five Star Support award winner in the EDA industry, and we consistently rate higher than our competitors in independently conducted annual customer surveys. As I mentioned above, Tezzaron dropped their Mentor technical support in 2001.

Native Support for TCL

Calibre has supported TCL since 2004 through the TVF syntax. Like the significant scaling improvements, this is now a standard part of the Calibre DRC/DRCH package. Calibre customers with support contracts have had TCL support in Calibre at no additional charge since 2004.

Comparison to Quartz

Mr. Patti asserts in the ESNUG submission:

“In our best ‘apples to apples’ comparison we could do with Calibre (same number of threads for both tools), we found Magma Quartz to be 2-4X faster at a lower number of threads. As the number of threads increase, the gap grows even larger. We have tested from 2 to 16 cores and find the scaling to be very good. When the final top level checks on a 24 chip ‘design’ is ~2 hours, it literally saves us days.”

Tezzaron has the 2001 version of Calibre and does not have appropriate licensing to run more than a single CPU. Since there wasn’t a Calibre version number listed in Mr Patti’s write up, nor were there any Calibre performance times, I can only assume this is a comparison of an eight year old, single-CPU version of Calibre to the current version of Quartz.

With the significant improvements in Calibre runtimes, stemming from engine optimizations, scaling improvements, and hardware and OS platform support, I’m confident that had Mr. Patti tested a current version of Calibre, the results would have been dramatically different.

For further evidence of Calibre’s performance, see the customer review in the same issue of ESUNG, “Calibre nmDRC 6X speed-up from HyperScaling and Hyper Remote” (http://www.deepchip.com/items/0483-07.html ).

Mr. Patti also comments:

“We’ve run our ‘design’ that’s more than 20x larger than a normal design, and Quartz with Direct Read handled it, while Calibre choked.”

Without any background information, it is unclear what this means. Quartz runs on Linux platforms. With the 2001 version of Calibre, 64bit support on similar Linux platforms was not available (64bit Linux computing products were not mainstream in 2001). As a result, Tezzaron’s runs are limited to the 2Gb limit of 32bit processing in Linux machines. Of course, the current version of Calibre supports 64-bit computing, and Calibre customers are verifying some of the largest and most complex designs in the world.

The Bottom Line

All the indications (performance, scaling, availability of TCL, license status) are that this was not a valid comparison of Magma Quartz to Calibre nmDRC. It is a comparison of Magma’s latest product to an eight-year-old version of Calibre.

More Blog Posts

Preparing Recommendations