CARCO delivers higher customer satisfaction and saves $1 Million with NetApp
When your business is built on information, how you manage, store, and protect that information can make all the difference. CARCO is an example. Founded in 1977, Holtsville, New York-based CARCO Group Inc. and its three divisions (collectively, "CARGO") provide investigative, fraud detection, and risk mitigation services to large corporations, producing more than two million reports each year.
CARCO's Investigative & Security Consulting division and Background Screening Services division provide comprehensive background investigations on potential and current employees, vendors, customers, and business partners, as well as drug testing through an alliance partner. CARCO's Inspection Services division provides insurance carriers with pre-insurance vehicle inspection underwriting reports that significantly reduce auto insurance fraud.
CARGO currently stores for its clients more than 40 million documents and images, which are made available online via a custom document management system. Before moving to NetApp storage in 2006, CARGO used an HP optical disk jukebox to store these files.
"The optical disks were fragile and subject to breakage, and the jukebox itself would break down quite often as well," says Jim Owens, president, and CEO, who also serves as the company's CIO. "Five to ten percent of the time the storage was unavailable. Also, performance was terrible. If the picture wasn't already on the front-end caching server, it would take 25 seconds to pull it up. Clients would call and complain." Backing up the jukebox was also a problem. "The backup window was extending past 48 hours, so we couldn't get it done over the weekend," says Owens. "It was pretty much game over for the jukebox at that point,
so we started searching for a solution that would collapse our backup window to a manageable level."
At the same time, CARGO was aggressively pursuing virtualization with VMware®. The company wanted a storage platform that would not only handle its critical image storage but also complement its virtualization efforts - delivering a stronger return on investment.
"By enabling our virtualization strategy and allowing us to run this IT department with just four infrastructure people, the Net App solution has saved us more than $1 million over the past three years."
President and CEO, CARCO Group Inc.
In 2006, NetApp partner Marcum Technology LLC introduced Owens and his staff to the capabilities of NetApp storage and suggested that it might solve CARCO's availability and backup problems. "They set up a NetApp storage system in their lab and showed us how we could do offline, point-in-time backups for our file shares, SQL data, and virtual machines, and we said 'OK, this is what we need.' It was really a no-brainer," says Owens. "We briefly evaluated products from IBM, HP, and EMC, but none could match NetApp's value proposition."
NetApp Professional Services installed a NetApp FAS920 storage system, which it upgraded a year later with a FAS3020 cluster to accommodate CARCO's growing number of virtual machines. "We originally had sought a new storage solution just for the pictures—to replace the jukebox," says Owens. "But then we realized that NetApp is an ideal platform for virtualization, and it offers the valuable option to use iSCSI instead of Fibre Channel."
In 2009, CARCO replaced the FAS3020 cluster with a FAS3140 cluster for even more storage capacity. Two months later, CARCO also replaced the FAS3020 at its disaster recovery (DR) site with a FAS3~40 storage system.
"Our server infrastructure is now completely virtualized on NetApp storage, with the exception of a few fax servers and our VoIP system," says Owens. "The NetApp storage is supporting 80+ production virtual machines on 4 VMware ESX hosts. The combination of NetApp and VMware ESX Server allows 'jack-in-the-box DR'—simple disaster recovery where all the virtual machines are replicated along with the data to the DR site."
An efficient, unified storage architecture The flexibility to use multiple protocols on one box for different storage needs turned out to be critical. "With NetApp, it's all in one," says Owens. "Obviously, we used iSCSI from the beginning. But we quickly saw that the ability to access our millions of departmental files via direct CIFS and at the same time leverage NFS as a rock-solid back end for VMware ESX Server would give us a lot of flexibility."
Using Ethernet storage in combination with high-density, low-cost SATA disk has saved money and staff time. "Ethernet storage is easier to set up, it's more flexible, it's easier to manage, and you're not really losing much in performance," says Owens. "We saved at least $60,000 in hardware costs by going with Gigabit Ethernet over a Fibre Channel SAN, and we didn't have to supplement our staff by adding Fibre Channel expertise."
CARCO is saving 40-50% of the storage capacity that would normally be required for its virtual machines - around $60,000 worth of SATA disk - by using NetApp deduplication technology to eliminate redundant data objects. Owens expects this figure to increase as CARCO adds more virtual machines.
Eliminating the nightly backup window CARCO now relies on NetApp Snapshot`" technology—a feature of the Data ONTAP~ 7G operating system—to backup files, SQL data, and virtual machines in minutes, instead of the hours it would take using tape. NetApp SnapManager~ for Microsoft SQL Server and SnapManager for Virtual Infrastructure automate the point-in-time copy process and verify the integrity of Snapshot data. NetApp SnapRestore~ allows recovery in minutes or even seconds, depending on the amount of data.
CARCO's nightly backup window has been eliminated since systems are continuously available during the backup. And CARCO's IT staff can be certain that all data has been backed up as of a certain point in time, which was impossible with the previous 48-hour rolling window.
"The ability to have instantaneous, coherent backups—and run them parallel to business operations—cannot be overstated in terms of the value to the business," says Owens. "We truly are 24x7 now. We're saving around $100,000 a year in tape costs since we're rolling to tape once a week now instead of every night. Also, we're able to keep 10 days worth of Snapshot copies on hand for fast local recovery of anything that's business critical."
Higher availability improved disaster recovery Because of the reliability and performance of the NetApp storage and the fact that storage can be expanded on the fly, CARCO no longer receives customer complaints about image load times or availability. "With the combination of NetApp and VMware, we're close to four nines uptime," Owens reports. Customers have seen the image load times drastically improve, from 25 seconds to 3 seconds.
CARCO is also using NetApp SnapMirror~ software to asynchronously replicate only changed data blocks over Ethernet to the NetApp FAS3140 storage system at its DR facility. "Before NetApp, we had essentially zero DR capabilities," says Owens. "With SnapMirror, we could probably get our DR site fully operational and business facing within a couple hours of the primary site burning to the ground, with a maximum of 10 minutes of data loss."
Accelerating software development by 30%
By using NetApp FlexClone~ software to create virtual, writable clones of data sets without requiring additional storage space, CARCO has been able to accelerate time to market for internal applications and respond to more specialized customer requests. "We've had instances where customers have requested complete isolation of an environment because the client wants to test only their functionality," explains Eric Vanek, Director of Infrastructure. "We were able to use FlexClone volumes and easily stand up a whole new environment without actually copying anything. IYs like magic. Previously, we would not have been able to accommodate those requests."
The ability to provide developers with thin provisioned FlexClone volumes of production volumes, along with the manageability and performance of NetApp storage, has allowed CARCO to speed up its in-house software development efforts by 30%, says Owens. "The ability to thin provision clones has had a major impact on both infrastructure and development," he states. "We also have environments where developer's access files directly off the NetApp via CIFS—we're able to give them a great deal of flexibility."
A scalable platform for business growth As CARCO's business grows, scalability for its IT solutions becomes more and more important. With NetApp, however, Owens no longer worries about how the company will accommodate that growth. "We actually have even more reserve capacity today than we did a year ago, despite the fact that we're doing more sophisticated things with more complicated business logic," he says. "The NetApp technology is successfully outstripping the growth factor at CARCO."
Complete payback in less than a year. The capability to handle multiple workloads, use Ethernet storage and SATA disk, deduplicate redundant data, accelerate development cycles, and manage the storage environment without adding staff means that return on investment for the NetApp solutions has always come in within one year.
"By enabling our virtualization strategy and allowing us to run this IT department with just four infrastructure people, the NetApp solution has saved us more than $1 million over the past three years," says Owens. "It's literally that big. We'd need a lot more people if we were trying to manage a purely physical environment with direct-attached storage. And the value of the DR aspect is almost incalculable—it's enabling new business. We've sold some major client deals that we would have lost if we lacked the DR capability that we have today. The NetApp environment has changed CARCO a lot over the past few years. We wouldn't be able to do what we're doing, or be the company we are today, without it."
"We actually have more reserve capacity today than we did a year ago, despite the fact that we're doing more sophisticated things. NetApp technology is outstripping the growth factor at CARCO."
President and CEO, CARCO Group Inc.
Improve availability and manageability of IT infrastructure to offer customers better, faster access to information at a lower cost.
Standardize on NetApp unified storage to support virtualized infrastructure and file workloads.
- Eliminated nightly production backup window
- Greater than 99.9% availability with 10-minute recovery point objective
- 30%faster time to market for internal applications
- Up to 50%less disk space used
- Payback in less than one year
- NetApp FAS3140 active-active system
- NetApp FAS3140 system (at DR location)
- NetApp Operations Manager
- NetApp FlexClone, SnapDrive® for
- Windows®, SnapManager for Microsoft
- SQL Server, SnapManager for Virtual
- Infrastructure, SnapMirror, SnapRestore,
- and Snapshot software
NetApp Professional Services
- Storage Infrastructure Implementation Service
- SuppoREdge Premium with Premium
- NAS-NFS (for virtual machine storage)
- NAS-GIPS (for file storage)
- IP SAN-iSCSI (for application storage)
- Applications: Custom document
- management system, Novell
- GroupWise, Veritas NetBackup 6.5
- from Symantec, Microsoft iSCSI
- Software Initiator, Citrix Presentation
- Server, Captaris RightFax
- Databases: Microsoft SQL Server 2005
- Server Platform: Four servers running
- VMware ESX Server 3.5, with 84 virtual
- machines running a mix of Microsoft
- Windows Server 2003 and SUSE Linux®
- Enterprise 10; 10 additional physical
- servers for nonvirtualized workloads
- Users: 200 employees; between 500
- and 1,000 client users on systems at
- any given time.
Marcum Technology LLC