After close to two years of effort (see here) the Healthcare IT Stack Exchange Site is closing for good. HealthCare IT is closing:

it simply does not appear that this topic has a strong enough following on our network to support the site long-term

That says it all.

Healthy Architectures – Using CQRS and Event Sourcing for Electronic Medical Records presents a couple of interesting patterns for the management of healthcare data. The two patterns are:

• Command Query Responsibility Segregation (CQRS)
• Event Sourcing (ES)

Here’s another article that provides further clarification on the CQRS pattern and how it compares to ES and Task-Based UIs: CQRS, Task Based UIs, Event Sourcing agh!

These are high level design patterns that result in non-traditional and more complex system architectures when implemented.  The healthcare domain is complex and alternate approaches for providing robust solutions are worth consideration.

My last discussion of Off-The-Shelf software validation only considered the high-level regulatory requirements.  What I want to do now is dig deeper into the strategies for answering question #5:

How do you know it works?

This is the tough one. The other questions are important, but relative to #5, answering them is pretty easy.  How to answer this question (i.e. accomplish this validation) is the source of a lot of confusion.

There are many business and technical considerations that go into the decision to use OTS or SOUP software as part of a medical device. Articles and books are available that include guidance and general OTS validation approaches. e.g. Off-the-Shelf Software: A Broader Picture (warning PDF) is very informative in this regard:

• Define business’ use of the system, ideally including use cases and explicit clarification of in-scope and out-of-scope functionality
• Determine validation deliverables set based on system type, system risk, project scope, and degree of system modification
• Review existing vendor system and validation documentation
• Devise strategy for validation that leverages vendor documentation/systems as applicable
• Create applicable system requirements specification and design documentation
• Generate requirements-traceable validation protocol and execute validation
• Put in place system use, administration, and maintenance procedures to ensure the system is used as intended and remains in a validated state

This is great stuff, but unfortunately it does not help you answer question #5 for a particular type of software. That’s what I want to try to do here.

OTS really implies Commercial off-the-shelf (COTS) software. The “commercial” component is important because it presumes that the software in question is a purchased product (typically in a “shrink-wrapped” package) that is designed, developed, and supported by a real company.  You can presumably find out what design controls and quality systems are in place for the production of their software and incorporate these findings into your own OTS validation.  If not, then the product is essentially SOUP (keep reading).

Contrast OTS with Software of Unknown Provenance (SOUP).  It is very unlikely that you can determine how this software was developed, so it’s up to you to validate that it does what it’s supposed to do.  In some instances this may be legacy custom software, but these days it probably means the integration of an open source program or library into your product.

This following list is by no means complete. It is only meant to provide some typical software categories and the strategies used for validating them.  Some notes:

• I’ve included a Hazard Analysis section in each category because the amount of validation necessary is dependent on the level of concern.
• The example requirements are not comprehensive. I just wanted to give you a flavor for what is expected.
• Always remember, requirements must be testable.  The test protocol has to include a pass/fail criteria for each requirement. This is QA 101, but is often forgotten.
• I have not included any example test protocol steps or reports.  If you’re going to continue reading, you probably don’t need help in that area.

Operating Systems

Examples:

• Windows XP SP3
• Windows 7 32-bit and 64-bit
• Red Hat Linux

Approach:

1. Hazard Analysis: Do a full  assessment of the risks associated with each OS.
• Pay particular attention to the hazards associated with device and device driver interactions.
• List all hazard mitigations.
• Provide a residual Level of Concern (LOC) assessment after mitigation — hopefully this will be negligible.
• If the residual LOC is major, then Special Documentation can still be provided to justify its use.
2. Use your full product verification as proof that the OS meets the OTS requirements. This has validity since your product will probably only be using a small subset of the full capabilities of the OS.  All of the other functionality that the OS provides would be out of scope for your product.
3. This means that a complete re-validation of your product is required for any OS updates.
4. There is no test protocol or report with this approach. The OS is considered validated when the product verification has been successfully completed.

Compilers

Examples:

• Visual Studio .NET 2010  (C# or C++)

Integrated Libraries

Examples:

• Log4Net (Log4J) — Log generation
• Hibernate/NHibernate  — Database Object/Relational Mapping

Version Control Systems

Examples:

• Subversion (+TortoiseSVN)
• Git (+TortoiseGit)

Issue Tracking Tools

Examples:

• Trac
• Jira
• FogBugz

Validation is a lot of work but is necessary to ensure that all of the tools and components used in the development of medical device software meet their intended functionality.

The article Build and Validate Safety in Medical Device Software takes a critical look at the current processes for medical device software and concludes:

The complexity of the software employed in many medical devices has rendered inadequate traditional methods (testing) for demonstrating their safety.

The article then provides examples of the types of analyses that can be performed to better ensure safety.

Interesting read.

Here are some references:

BohrBug: Not necessarily easy to find, but once discovered is reproducible.

Heisenbug: The ever-annoying bug that can not be reliably reproduced.

Spin: An open-source software tool for formal verification of distributed software systems.

I first heard about this from a system message on the Stack Overflow site. The post Protect intellectual property – but not like this explains their position (in particular, SOPA vs. DMCA) and has a lot of good links to more information.

SOPA-Rope-a-dope has a well written explanation of DNS blocking and DNSSEC.

This bill is a really bad idea. A lot of people in the know agree: An Open Letter From Internet Engineers to the U.S. Congress.

Congress to Resume SOPA Hearings Next Week (Wednesday 12/21) so it’s not too late to help stop this bill. If you’re like me and have always wondered why people contact their Congressperson, now we finally have a good reason to do so.  Go to Stop American Censorship and let your voice be heard.

From RWW Cartoon: SOPA Opera:

UPDATE (12/23/11): Bill that could ‘break the Internet’ delayed until 2012. Also see: What You Need to Know About SOPA in 2012

A reader asked me about OTS software tool validation. He says:

It seems to me that the editor and any other tool used to create the software is exactly that, a productivity tool. The end result (compiled binary installed on a validated PC configuration) is still going to go through verification and validation, therefore, it seems validating any of the items used to actually create the binary is unnecessary.

Any thoughts or guidance to help me understand this process?

This is a great question and the source of a lot of confusion.

The bottom line is that all third party tools (and libraries) used to construct or test FDA regulated software need to be validated.

You may think validating a compiler is unnecessary, but the FDA says otherwise — section 6.3 of the FDA Guidance on General Principles of Software Validation discussion includes “off-the-shelf software development tools, such as software compilers, linkers, editors, and operating systems.”

The form of the required documentation is detailed in the Off-The-Shelf Software Use in Medical Devices guidance document.  Section 2.1 has the questions that the OTS software BASIC DOCUMENTATION needs to answer:

1. What is it?
2. What are the Computer System Specifications for the OTS Software?
3. How will you assure appropriate actions are taken by the End User?
4. What does the OTS Software do?
5. How do you know it works?
6. How will you keep track of (control) the OTS Software?