As I’m sure many of your are already aware, yesterday’s extreme weather in Denver (the ‘Bomb Cyclone’ as the meteorologists are calling it) caused a brief system outage for all of RMMI’s hosted services. All of our client-facing services have been fully operational since late yesterday afternoon, and all of our internal and backup systems have now been fully restored. Short of an IT Director who’s running purely on caffeine at this point, there shouldn’t be any further adverse effects.
Now that things are back to normal, we wanted to take a minute to explain the series of events that led to the outage and the steps we take to make sure that your data is safe.
All of RMMI’s hosted services are housed in a Tier 3 Gold Co-location Data Center here in Denver, CO. For those of you unfamiliar with the terminology, a Co-location Data Center (or Colo) is a shared data center that leases rack-space, power, cooling, and data connections to individual companies like us. Sharing the extreme costs involved in building a reliable data center across many clients is the most cost effective way for a company of our size to provide the services we offer. Prior to yesterday’s events, our Colo provider had maintained more that 15 years of up-time without an outage.
Around 10:45 AM yesterday morning, our Colo lost power from both of it’s supplying power grids within minutes of each other. There are battery systems in place that supply temporary power in the event of an outage and give the generators time to come online; however, the battery systems that supply our equipment failed – this resulted in a complete loss of power to all of our equipment.
By 12:00 PM the failed power system had been partially bypassed and a backup generator had been brought online. We were partially able to power up our equipment and had restored access to the Hosted PaperVision system by 12:15 PM
We received a notification from our Colo provider that full power had been restored at 3:45 PM. We were able to bring the rest of our systems online and all client-facing services were fully operational by 4:15 PM.
We know this outage has caused some inconveniences for many of you, and we hope you understand the extreme circumstances we were up against. Our Colo provider has an exemplary track-record, and we have the fullest confidence that despite yesterday’s events, they are still the best provider in the industry.
If you have any questions or concerns stemming from yesterday’s events, please contact me directly at email@example.com
We’re excited to announce that PaperVision Enterprise ver. R84 has been made available for widespread use. The R84 release includes some exciting changes and enhancements, including:
- A fully redesigned user interface
- The ability for end users to request password resets
- An updated and improved PaperVision Assistant (replaces the old Web Assistant)
- The introduction of a ‘Quick Search’ function to make searches across multiple projects much simpler
- The brand new PaperVision Client – a lightweight installed application that offers a new way to search for and access your documents
- Many other improvements ‘under the hood’ to ensure that your documents are safe and secure.
The standard PaperVision by RMMI site will continue to use the old style web interface for now but if you would like to check out the new version it’s available as a separate site:
Your normal login will work and you’ll have access to all of the same documents you have at the main site. You will need version R81 or newer of the PaperVision Web Assistant in order to utilize the new site.
Digitech has put together a ‘Quick Reference Guide’ going over some of the changes, you can view it here:
In the coming months we’ll be releasing more information and training about the new features, but we would love to hear your initial impressions. Please submit any issues, concerns, or questions about the new site to firstname.lastname@example.org.
Here at RMMI we take a lot of pride in the level of service we provide, and we’re constantly looking to our clients to tell us how we can improve. One theme that seemed to be reoccurring across many of our clients was the need for a solution that was easy to implement and offered top-level support, reliability, and security without adding management headaches for already overloaded IT departments. Many of our clients don’t have dedicated IT staff, or share technicians with other departments. Finding a way to deliver our scanning and document management solutions without adding any load to internal or contracted IT personnel moved to the top of our list. To accomplish this goal, early in 2016 we began the steps to implement a hosted PaperVision Enterprise and Capture Scan solution. After many months of planning, touring multiple colocation datacenters, purchasing and configuring hardware, and what seemed like endless testing we went live with our first client in the fall of 2016.
We knew we had several clients who would benefit from the hosted offerings, but we honestly had no idea just how popular the solution would become. We are now hosting over 80 million digitized documents for more than 40 different clients and there are more than 500 end-users who access our systems each week.
In order to keep up with the increased demand, we’ve applied our continuous-improvement model to our hosted solution and have been constantly working on making the system better since our initial roll-out. We’re certainly not done improving, but we wanted to highlight some of the major changes and how they impact you and your team.
The foundation of any electronic document management system is storage. Performance and ease of access mean nothing if your data isn’t protected from deletion (intentional or accidental), loss due to hardware failure, or corruption from malicious attacks. To ensure that we’re meeting these goals, we have implemented a multi-tiered storage and backup system that uses several techniques to make sure your data is secure:
- RMMI’s hosted solution is housed within the top datacenter in Colorado and the only Colocation Datacenter in the world to carry a Tier 3 Gold certification. The datacenter recently celebrated 15 years of uptime without a single outage to any of their major systems (Power, Cooling, Fire-Protection, and Service Connection). Building our hosted solution on top of this excellent foundation allows us to provide the highest level of service.
- At the base level, all of your data rests on high-quality storage arrays that utilize a technology called erasure-coding to spread your data across multiple physical hard-drives – these storage arrays are designed to remain fully functional even in the event that 33% of the disks in the array fail simultaneously and allow our staff to replace potentially failing drives without disrupting service.
- We have a sophisticated monitoring system in place that checks over 8,000 different ‘trigger points’ every minute – this system alerts our technical staff to signs of disk failure far before that disk becomes unusable. The reality is hard-drives fail at an industry average rate of 2.5 – 3% per year and our system utilizes more than 100 drives so a couple drive replacements per year is inevitable. Our monitoring system has given us the tools to make sure that we’ve caught every pending drive-failure in the ‘pre-fail’ warning stage and replaced them before they’ve gone down.
- To prevent against data deletion or corruption we use another technology called snapshotting. Snapshotting allow us to take a virtual ‘snapshot’ of your data at a certain place in time. In the event a document is inadvertently removed or altered in an unwanted way, we can ‘roll-back’ to a previous version of that file. We take snapshots of all of our clients’ PaperVision data every 4 hours and keep those snapshots for 90 days. We can roll back an entire PaperVision entity or a specific file to any point in the last 3 months.
- Another added benefit of utilizing snapshots is it allows to do what’s called ‘snapshot-mirroring’. This means that every time we take a snapshot (every 4 hours) we replicate that data to a second storage array. These backup arrays carry the exact same specifications as the primary arrays and can completely replace them in the event we suffer a hardware failure that is deemed ‘non-recoverable’. All 90 days’ worth of snapshot data is contained on both the primary and secondary arrays.
- To prevent against a total system loss due to fire or natural disaster, we send nightly snapshots to a second datacenter in Palo-Alto California. While this site isn’t equipped to provide hosting services, it means that even in the event of a catastrophic local disaster, your data is secure and we’ll be able to provide a full restore once the local infrastructure is back up and running.
- Our off-site backup system also allows us to write a backup copy of your data to your own contracted storage provider (Google Cloud, Microsoft Azure, Amazon S3, or Backblaze B2). These complete backups are ‘owned’ by you and you can elect to keep as many versions as you would like. If you would like a quote on this service please contact your sales representative.
After reliable storage, uptime and accessibility are our next goals. RMMI’s entire hosted solution is set up for maximum reliability with no single point of failure.
- All of the components of the system from network switches to web-servers are configured in groups. Single failures in any system trigger an automated re-balancing spreading the workload over the remaining operational devices. This redundant configuration has the added benefit of allowing us to perform most maintenance tasks without taking the system offline – meaning we rarely have to file for any ‘maintenance windows’.
- The system is connected to the rest of the internet by a ‘blended-mesh’ of 4 different internet service providers. As long as at least one of these service providers is operational the system is accessible.
- All services and components are actively monitored by our IT staff. Most issues are quickly assessed and resolved before they can cause any noticeable impact to our end-users.
Security is the final part of our configuration. When you trust us with your documents, we want you to rest soundly knowing that not only is your data safe from loss, it’s also safe from unauthorized access. To ensure that all of our client’s security concerns are met, we have utilized the following physical, procedural, and electronic security measures:
- The equipment for the hosted PaperVision system is located at a highly-secure datacenter. Physical access to the system is tightly controlled by two layers of key-card and biometric security, and the server cabinets themselves are protected by combination locks. The entire facility is patrolled 24×7 and is under constant video surveillance.
- Electronic access to the system is limited to RMMI IT administrators only – all IT administrators have completed extensive background checks and have been trained in applicable security practices.
- The entire hosted system resides behind multiple firewall layers and potential cyber-threats and unauthorized access attempts are actively monitored and tracked.
- Each PaperVision entity we host is separated from our other clients by utilizing its own Sql Database and segregated storage shares. Backend data is protected using encryption keys unique to each entity. This ensure that absolutely no data can be inadvertently shared across entities.
- Any and all requests for new user accounts or new access permissions are verified with account administrators prior to completion.
In addition to the changes to the main PaperVision infrastructure, we’ve also been working hard to improve our support system. Live chat support is now embedded directly into the PaperVision system, and we have implemented a complete ticketing and tracking system to make sure that all of your issues are resolved quickly. If you haven’t already had a chance to read my previous post about the changes we made to our support terms and the updated SLA please click here.
We know that there you have a lot of options for document management in today’s market, and we’re truly grateful for the opportunity to serve you. If you have any ideas on how we can improve our products please don’t hesitate to drop us a line at email@example.com. As always, if you have any questions about any of the changes we’ve made and how they directly affect you, please contact you sales representative.
Rick Harper – Director of Technology
December 12, 2017
To our valued customers,
This past year has brought some exciting changes here at RMMI. Our hosted PaperVision Enterprise and Capture Scanning solutions have been a huge success, we’ve added some amazing new software that allow us to integrate with your existing applications, and we’ve partnered with an E-Forms software manufacturer to provide a hosted version of their product. We expect that these solutions will continue to evolve in the coming year and we’re very excited for the opportunity to continue to provide you with the best cutting-edge technology solutions available.
Exceptional customer support has always been one of our core values here at RMMI, and we believe that the time we spend with our clients is one of the biggest reasons for our success. Along with all of the changes to our software portfolio, we’ve also been working diligently to make sure that our support team and infrastructure is up to the challenge of the increased workload. Early in 2017 we implemented a support ‘ticketing system’ and knowledge base that allows us to track and monitor support requests. This system also allows you to provide feedback to our support team – If we’re doing a great job, we would love to hear from you; if we could have done better, we need to hear from you so that we can get it right next time. Once we were able to track the number of support requests we were receiving it became apparent that we were in need of an additional technician, so in October of 2017 we added an additional full-time Technology Support Specialist to our team. We’re always looking for ways we can provide added-value to our clients and our support goals for 2018 include expanding our online knowledge base, adding training and reference materials, and conducting regular webinars showcasing different features of our software. These additions are being driving by customer feedback, so if you have any other suggestions on ways we can improve our service, please let us know.
The changes to our support team have necessitated a slight change to our support terms. Our support has always been included at no charge, and it is our goal to continue this policy wherever it is feasible; however, we have had an increasing number of support tickets arising from issues that are completely out of our control. In order to continue to provide our clients with the best possible support we are implementing the following changes to our support billing for 2018:
- No charges will be incurred for any issues related to software bugs, hosting configuration or outages, server or project miss-configuration, or errors made by RMMI personnel.
- Issues related to workstation miss-configuration, hardware replacement or refreshes, user error, or environmental variables outside of RMMI’s control will be billed in 15 minute increments at the prevailing service rate.
In order to facilitate this change, we are requesting that you provide a list of your end-users who are authorized to initiate a billable support ticket as well as a list of supervisors. In the event that a non-authorized end-user requests support that could incur charges, we will reach out to a supervisor for approval before proceeding. (Non-billable tickets will be accepted from any end-user). Please Click Here to review the new Service Level Agreement and provide your user lists. In order to ensure that there is no disruption to your support, please complete this form by January 1st 2018. The SLA is also posted on the support page of our website: www.rmm-i.com
If you have any questions or concerns about these changes, please contact your sales representative.
We look forward to continuing to serve you and the opportunities that 2018 is going to bring.
Rick Harper – Directory of Technology