Speech Technology and HealthCare

Recently, I was invited to participate to a meeting of AVIOS (Applied Voice Input Output Society) local chapter in Menlo Park, California to discuss the use of speech technology in HealthCare and Medicine.

Speech technology is not widely used in healthcare at the moment. The healthcare industry is still focusing on efficiency and cost saving by solving systems integration, re-wamping legacy systems and making paper based processes digital.

Dictation and transcription for EMRs

The most successful type of applications is probably dictation/transcription for electronic medical records (EMR).

Speech recognition, enhancing productivity and reducing medical costs as a result. In this domain, Nuance is one of the market leader.

The products and services of Nuance Healthcare are used at more than 5,000 hospitals and by more than 400,000 providers.

Nuance’s on-premise Dictaphone Enterprise Speech System (ESS) components support recording the dictation audio, managing the audio and the resulting transcription process, transcription document distribution, and online reporting.

Dragon medical is another product from Nuance with which physicians dictate in real-time into their EMR in their own words letting them instantly review, sign, and make their notes available for other clinicians. This include Medical Vocabularies, covering nearly 80 specialties and subspecialties and regional accent support.

Potential use for Care and Disease Management

Besides dictation/transcription there are some additional potential uses of speech technology in other domains related to heathcare. Care and Disease Managament is one of them.

Organizations/companies like Lifemasters have call centers where nurses help to help patients with chronic disease achieve optimal health by closing the gaps between recommended and actual care (evidence-based medicine) and encouraging patients to adopt a healthy lifestyle and reducing the overall cost of healthcare. Speech technology could be used to complement and optimize this type of services: virtual operators, voice activation services, automatic text-to-speech combined with voice-over-IP reminder and notifications (e.g. to take drugs, schedule an exam, do certain types of exercices etc ...). Companies like requall and Jott already provide this type of notification infrastructure that can be delivered on mobile phones for example.

Other Areas

Additional research areas in speech technology and healthcare include:

• Using speech/sound recognition to monitor eating behavior

• disease assessment (e.g. parkinson over the phone, although this disease is very complex and voice is probably not the most important factor)

• using speech recognition for medical transcript anonymization to be used as corpora

Conclusion
Overall, there is a big potential for use of Speech Technology in the healthcare industry.

The recent improvements in Text-to-Speech make it very attractive to healthcare, especially to non life critical applications.

Speech recognition accuracy is sometimes still a problem, but it is well suitable for sub-languages with specific vocabulary. It might be more difficult to be used directly by patients/health consumers, especially senior citizens where the language can be altered by their conditions.

HIMSS 2009 - How Consumers use PHRs (KP)

While at HIMSS 2009 in Chicago early this month I did attend a very interesting talk from Judy Derman and Jan Oldenburg from Kaiser Permanente untitled: "How Consumers use Personal Health Records (PHRs) : lessons learned".

Kaiser Permanente (KP) has a very large user based (2.5M consumers/patients and 13,000 healthcare professionels) for their online services (a portal accessible at www.KP.org portal and My Health Manager, a Personal Health Record (PHR) built with the help of EPIC).

In this talk, Judy and Jan described the lessons learned from usage patterns on these web applications.

The consumers/patients are motivated by the fact that they can act on their information. It is not just about looking at a medical record, but act one it. Example of useful actions include:

• start changing health related behavior
• refill a prescription online
• make appointment with their doctor
• send email with care professionals
• access educational programs (some of the most popular are: diabetes, depression, insomnia, weight management)

The consumers will trust these applications and the associated medical information as long as it is :

• transparent
• accessible
• consistent
• secure
• timely and accurate

Kaiser Permanente study shows that the implementation success factors include:

• early adopters are leaders in usage and feedback
• there is a strong leadership from senior leaders
• importance of clearly define explicit and articulated benefits
• allocate appropriate and sufficient resources to do the job
• resource must have appropriate skills
• involve stakeholders / members / patients early in the process
• make decision at the appropriate level
• provide tools to support implementation (marketing material and toolkits)
• use standard prcedures
• use effective and integrated marketing
• simplify the "getting started" mechanism (one step activation + pre-populate heath record)
• drive adoption with multiple channel marketing

The Kaiser PHR was launched in January 2007 and has reached 2.7 users by April 2009.

In 2008, 84% of the users choose the "One step activation" mechanism and 54% of KP members signed-in with 40.8% of user male and 51.2% female.

Here are the top features in the last two years:










Here are some statistics/facts about the Email/secure messaging service of the KP PHR:

• 6M emails in 2008 sent to doctors (13M since the launch in 2007)
• 600,000 emails per months in 2009
• 14% of messages are written by physicians and clinicians
• 14,000 physicians are using emails regularly

One of the benefits of the email services is that the patients using email were 7-10% less likely to schedule an appointment.

Here are some statistics/facts about the Lab/test results service of the KP PHR:

• 16.7M test in 2008
• 61M tests results available online
• 31M since launch
• certain tests are not online (HIV, genetic tests ...)

Here are some statistics/facts about the Presciption refill service of the KP PHR:

• 5.6M refill in 2008
• increase of 26% in 2007-2008
• Up to 23% of refill are done online

HealthCare and the benefits of a Portal Strategy

The idea that software should be componentized (built from prefabricated components) has been proposed more that 40 years ago. However it is only recently, with the advent of Service Oriented Architecture (SOA) and Software as a Service (SaaS) that effective software componentization methods to integrate systems and develop solutions have been available.
These methods help promote code reuse, low-cost system development, contribute to software quality and to more flexible IT infrastructures.

Software componentization can help software companies acquire and maintain a competitive advantage by developing, deploying and maintaining services and solutions faster, cheaper and better than their competitors. Using Web and Mobile based Graphic User Interface (GUI) components is the most common way for users to interact with IT systems.

The ability to create customizable and reusable front-end software components is especially valuable for software companies and their partners. Combined with cost effective deployment solutions such as Cloud Computing, new services can quickly be bundled, offered and tuned for specific targeted sets of customers and later rolled-out to new markets very easily.

Componentization can also help to rapidly make custom changes (such as re-branding or re-skinning) since the presentation layer is isolated from the service layer. Aggregated in web portals, front-end components can be layered on top of existing data structures, enterprise and legacy systems as well as third party services.

In healthcare for example, portals can offer a unified and personalized view for healthcare professionals, and provide real-time access to a selected patient's clinical information. The same portal infrastructure can be also customized for the patients and their care givers.

Portal technology provides ways to integrate information, people and processes across organizational boundaries. It provides a secure unified access point (SSO), often in the form of a web-based user interface, and is designed to aggregate and personalize information through application-specific component or portlets.

Another characteristic of the portal technology is the fact that content management can be decentralized allowing richer content and more efficient update of the data and information presented to the users.

A portal approach brings benefits to your customers/end-users (healthcare professionals and patients), your development, professional service and IT teams, but also your partners!

• benefits for your end-users: Portal solutions also offer rich user experience by leveraging Web 2.0 technologies and specific components (e.g. wikis, blogs, message boards, widgets, social networking, maps, SSO, etc). Portal customization and personalization also offers to the end-users a more personalized experience based on their profiles such as their role in the organization or user group and preferences (e.g. choice of layout and look and feel).
• benefits for your development team: This includes a common architecture for the aggregation of heterogeneous components and services, a clear separation between the presentation layer and the service layer, and the fact that portlets are based on standard technologies (e.g. JSF, Spring, Spring MVC, Hibernate, JSR-168, JSR-286, WSRP 1.0/2.0, AJAX, Java EE, or even Flex).
• benefits for your professional service and IT teams: Portal technology can save substantial costs to both the professional service team in charge of creating solutions using a portal approach including the ability to create and combine quickly customized components that are easy re-branded for various customers is clearly a good return on investment (ROI).
  
• benefits for your IT department: For the IT department, in charge of deploying and maintaining services and applications, to be able to run multiple portal sites, each with a unique domain, on the same portal server reduces avoids to duplicate hardware and image instances. Portlets can be deployed at run-time (hot deployment) reducing down time for the user and facilitate the maintenance of the applications. In addition to this, specific content, branding, layout and skins can be stored and managed independently of the application in a content management system, saving costs during development, deployment and maintenance.
  
  

For the past couple of years, there has been a clear interest in the healthcare industry, not only in the US, but also in Canada and Europe to use portal frameworks as a solution to aggregate heterogeneous services and applications. There are still a lot of important and sensitive issues to address for this type of solution in the healthcare industry (security, auditing capabilities, flexibility, extensibility, maintainability, user and content management, integration, performance, scalability, availability, quality assurance, lifecycle management, maintenance and monetization).

Next week (April 4-8 2009) at HIMSS-09 in Chicago, it will be interesting to see if the trend is confirmed and how many vendors have been repackaged their solutions using portal technology.

Liferay and Flex Ajax Bridge (FABridge)

I am using Liferay to create highly interactive and componentized web applications and solutions for the healthcare industry. Adobe Flex is one of the technology I am using the create portlets for Liferay. One challenge is the communication between portlets and between the core layer of the portlet that access services and wrapping GUI layer (JSP/HTML and Javascript).

Adobe Flex 3.0 SDK now contains the Flex Ajax Bridge (FABridge) developed by Adobe Labs.
Flex Ajax Bridge is a small library that can be help you expose an flex application (Action Script graph) to scripting by Javascript.

To show how FABridge works in Liferay. I have created very simple Flex application that show a button. I then modify the label of the button at runtime via javascript.

The code for the Flex application is very simple:

<?xml version="1.0" encoding="utf-8"?>
<mx:Application
 xmlns:mx="http://www.adobe.com/2006/mxml"
 xmlns:fab="bridge.*">
 
 <fab:FABridge bridgeName="flash" id="flash" /> 
    
 <mx:Button id="button" label="Original Button" width="150"/>
 
</mx:Application>

The Flex application needs to refer to the action script part of the FABridge library (FABridge.as).

Likewise, your Javascript will need to refer to the Javascript part of the library (FABridge.js).


















The build.bat file is a simple command (mxmlc main.mxml -output main.swf) for building the SWF file. But you can also use Flex Builder for this.

Here is how the resulting SWF file looks like inside a very simple Liferay portlet:









The code for the wrapping portlet is contains inside the view.jsp file. I am using SWFObject 2.0 to integrate the SWF file.

I am also using JQuery (I have added manually JQuery 1.2.6 lib - but I assume that I could leverage JQuery that comes with Liferay) to highlight the SWF container in green if the bridge succeeds at changing the content of the button:








If there is an expection, which is the case when the portlet is added in Liferay in Internet Explorer (IE 7.0) - refreshing the page works however! ). Then the container is highlighted in red:








Here is the whole code of the JSP file:

<%@ taglib uri="http://java.sun.com/portlet" prefix="portlet" %>

<portlet:defineObjects />
<script type="text/javascript" src="<%= request.getContextPath() %>/js/swfobject.js"> 
  swfobject.registerObject("myId", "9.0.0", "expressInstall.swf");
</script>
<script type="text/javascript" src="<%= request.getContextPath() %>/js/FABridge.js"></script>
<script type="text/javascript" src="<%=request.getContextPath()%>/js/jquery-1.2.6.js"></script>

<script type="text/javascript">

   accessFlex();
   
   function accessFlex()  {
       
        var initCallback = function() {
            try {
                    var root = FABridge.flash.root();
                    root.getButton().setLabel("Modified by FABridge");                       
                    $("#swf_div").css("border","3px solid green");    
            }
            catch(err) {
                $("#swf_div").css("border","3px solid red");
            }  
        }
        
        FABridge.addInitializationCallback("flash",initCallback);
    }
    
</script>


<div id="swf_div"> 
<object id="myId" classid="clsid:D27CDB6E-AE6D-11cf-96B8-444553540000" width="350" height="70">
  <param name="movie" value="<%= request.getContextPath() %>/flex/main.swf" />
  <!--[if !IE]>-->
  <object type="application/x-shockwave-flash" data="<%= request.getContextPath() %>/flex/main.swf" width="350" height="70">
  <!--<![endif]-->
    <p>Alternative content</p>
  <!--[if !IE]>-->
  </object>
  <!--<![endif]-->
</object>
</div>

Flex Web Service Introspection Wizard and BlazeDS

In my previous post, I mentioned that if you want to fully use Flex 3.0 Web Service introspection wizard, you will need to either use Adobe LifeCycle Data service, or have a cross domain file installed on the server that expose the web services you want to use.

However, if you use only BlazeDS, the web service wizard can still be useful to better understand which type of objects you obtain when calling 3rd party web services (besides looking at the wsdl file and debugging ResultEvent.result content).

In this post I will describe how to use Flex Builder 3.0 to introspect the ICW LifeSensor Web Service API. Then I will build a small Flex based portlet to display information related to a patient who has his medical information stored in the LifeSensor Personal Health Record (PHR).

A. Introspecting the Web Services

For this, you will need to know the WSDL URL of your web services.

In the case of LifeSensor, I am accessing the WSDL file over HTTPS which is protected with a login and password but you can also test the intropection wizard with free available web services available on the internet.

From Flex builder (I am using Flex Eclipse plugin), select "Data/Import Web Service (WSDL)...":






Then select the folder you want to import your classes to, click next, then enter the WSDL URL and click next again:















First you select the list of the operations you want to import. In my case, I just want to import the operation findAccessibleRecords.

You can also change the default value of the packages for the classes that are going to be generated and the main class name.

In my case, I just kept the default values, respectively com.lifesensor and RecordModuleWebServiceImplService.


















It just takes few seconds to generate the proxy classes:




















Even though I am importing only one operation from LifeSensor, a little bit more than 80 classes are generated.




















RecordInfoXto and its dependent classes structure is very close to the object returned by the web service call. Therefore I will be using only the following files:

• AddressXto.as
• CodeSystemXto.as
• CodeXto.as
• DateXto.as
• EmbeddedObjectXto.as
• RecordInfoXto.as

/**
 * RecordInfoXto.as
 * This file was auto-generated from WSDL by the Apache Axis2 generator modified by Adobe
 * Any change made to this file will be overwritten when the code is re-generated.
 */

package com.lifesensor
{
    import mx.utils.ObjectProxy;
    import flash.utils.ByteArray;
    import mx.rpc.soap.types.*;
    /**
     * Wrapper class for a operation required type
     */
   
    public class RecordInfoXto extends com.lifesensor.EmbeddedObjectXto
    {
        /**
         * Constructor, initializes the type class
         */
        public function RecordInfoXto() {}
           
        public var academicTitle:String;
        public var address:com.lifesensor.AddressXto;
        public var birthDate:com.lifesensor.DateXto;
        public var birthPlace:String;
        public var familyName:String;
        public var gender:com.lifesensor.CodeXto;
        public var givenName:String;
        public var middleName:String;
        public var scope:String;
        public var subjectId:String;
    }
}

   public class AddressXto extends com.lifesensor.EmbeddedObjectXto {
  /**
   * Constructor, initializes the type class
   */
  public function AddressXto() {}
          
  public var city:String;
  public var corpus:String;
  public var country:com.lifesensor.CodeXto;
  public var flat:String;
  public var line1:String;
  public var line2:String;
  public var organization:String;
  public var postalCode:String;
  public var state:com.lifesensor.CodeXto;
  public var streetAddressLine:String;
  public var zipCodeExtension:String;
 }

        public class CodeXto extends com.lifesensor.CodeSystemXto
 {
  /**
   * Constructor, initializes the type class
   */
  public function CodeXto() {}
          
  public var key:String;
 }

 public class DateXto extends com.lifesensor.EmbeddedObjectXto
 {
  /**
   * Constructor, initializes the type class
   */
  public function DateXto() {}
          
  public var isoDate:String;
 }

B. Creating the Flex component using BlazeDS

In a previous post, I have described in details how to create a BlazeDS application that uses BlazeDS to access web services. This one is very similar.

The proxy-config.xml describes the web service end-points and channel:

<destination id="ws-lifesensor-record">
        <properties>
            <wsdl>https://record2.us.lifesensor.com/phr/services/v2-5-0/RecordWebService?wsdl</wsdl>
            <remote-username>????????</remote-username>
            <remote-password>????????</remote-password>
            <soap>https://record2.us.lifesensor.com/phr/services/v2-5-0/RecordWebService</soap>
        </properties>
        <adapter ref="soap-proxy"/>
    </destination>

First, I import the generated classes. Then populating the RecordInfoXto object is straightforward:

import com.lifesensor.*;

private function findAccessibleRecords_result(event:ResultEvent):void {

  if (event.result != null) {
    var all_records:ArrayCollection = event.result as ArrayCollection;
    var record:Object = all_records.getItemAt(0);
                    
    // State
    var state:CodeXto = new CodeXto();
    state.key = record.address.state.key;
                    
    // Country
    var country:CodeXto = new CodeXto();
    country.key = record.address.country.key;
                    
    // Address
    var address:AddressXto = new AddressXto();
    address.streetAddressLine = record.address.streetAddressLine;
    address.city = record.address.city;
    address.postalCode = record.address.postalCode;
    address.state = state;
    address.country = country;
                    
    // Gender
    var gender:CodeXto = new CodeXto();
    gender.key = record.gender.key;

    // Birth Date
    var date:DateXto = new DateXto();
    date.isoDate = record.birthDate.isoDate;
                    
    // Record 
    patient_record = new RecordInfoXto();
    patient_record.givenName = record.givenName;
    patient_record.familyName = record.familyName;
    patient_record.gender = gender;
    patient_record.address = address;
    patient_record.birthDate = date;
    }
}

The resulting Flex based portlet is very simple (with a very compact code):

BlazeDS and secure Web Service access

In a previous post, I described how to use Flex/BlazeDS to a access remote Web Services.
This time, I am explaining how to access a secure Web Service that requires basic authentication using the same mechanism. This involves additional changes in the configuration files.

My goal is to create flex components that access ICW Lifesensor public Web services.
The additional complication is that the access to wsdl file required authentication over HTTPS.

Fortunately, I have the login and password of a Lifesensor Account (patient), so I can use them
to access the Reporting Web Services in order to retrieve medical data entries. More information about the Lifesensor APIs and Web services are available on the ICW Developer Network.

To illustrate my point, I will use a very simple Web Service call getVersion that return some general information about the web services such as the Axis version and build date (Lifesensor uses the open source Apache Axis framework to provide Web Services).

In the wsdl file for Version, the port description shows that the getVersion operation does not have any parameter, so the call to the Web Service will be straightforward:

<wsdl:porttype name="Version">
 <wsdl:operation name="getVersion">
  <wsdl:input message="impl:getVersionRequest" name="getVersionRequest"></wsdl:input>
  <wsdl:output message="impl:getVersionResponse" name="getVersionResponse"></wsdl:output>
  </wsdl:operation>
</wsdl:porttype>

BlazeDS offers a Proxy to access remote servers. This is necessary, if you do not have a crossdomain.xml file on your remote server. As a result, there will be two hops. One from the shockwave component on the client to the Proxy, the other one from the Proxy to the remote server where the Web Services reside.

For security reasons, both bops have to be secure, as a result, the initial SWF access and loading has to be done through HTTPS. In my case, my application (and the BlazeDS proxy) is served by Tomcat 6.0 that has been configured for SSL.

The first configuration change is in the proxy-config.xml. Besides the fact that the URL is now using HTTPS protocol, you will need also to specify the soap URL instead of using a wildchard in order to avoid a RPC Fault since "a destination that allows multiple domains or ports does not allow authentication".

Also, since LifeSensor is using basic access authentication, the easiest way to avoid the pop-up window from your browser asking you for the login and password (especially for the first hop, which is not relevant), is to set them in the proxy-config.xml initially via remote-user and remote-password tags.

     <destination id="ws-lifesensor-version">
       <properties>
           <wsdl>https://record2.us.lifesensor.com/phr/services/Version?wsdl</wsdl>
           <remote-username>?????</remote-username>
           <remote-password>?????</remote-password>
           <soap>https://record2.us.lifesensor.com/phr/services/Version</soap>
       </properties>
       <adapter ref="soap-proxy"/>
   </destination>

The MXML flex file is not very different from a Web Service access with no authentication. The following code is just specific to the Lifesensor Web Service API:

<mx:Script>
     <![CDATA[
       ...
       private function getData():void { webService_LS_Version.getVersion.send();}   
       ...
    ]]>
   </mx:Script>
   <mx:WebService id="webService_LS_Version" destination="ws-lifesensor-version" useProxy="true">
       <mx:operation name="getVersion"
               resultFormat="object"
               result="getData_result(event);"
               fault="getData_fault(event);">
       </mx:operation>
   </mx:WebService>

Calling the Version Web Service from LifeSensor (via an application on https://localhost:8443/) returns the following text:

"Apache Axis version: 1.4
Built on Apr 22, 2006 (06:55:48 PDT)"

In addition to this, my recommendations will be to use Flex remote debugging and a HTTP debugging proxy such as Charles or Fiddler which can be very handy to understand and debug AMF and SOAP based HTTP wrapped requests.

Also, Flex Builder 3.0 has very nice Web Service Introspection tool. Unfortunately, you will need to have a cross domain file on the server you want to introspect or have LifeCycle Data service to use the generated proxies. Apparently, if you only use BlazeDS, there are no direct ways to use the generated code out of the box. However, you can use some of the generated classes to store some of the data you obtain from the Web Services. This will be the topic of my next post.

Flex z-index and Liferay Portal

We recently encountered an issue when embedding Flex/Flash applications in Liferay Portlets.

The shockwave (.swf) file was showing on top of the liferay navigation menu:













The usual solution is to specify the z-index for the div layers.
There are even some very cool things you can do with Flex overlay.

Liferay however is a third party portal platform and if you don't want to have to change the source code or extend Liferay, a quick fix is to use the wmode argument for the embedded flex application:

<div>
   <embed wmode="transparent" src="<%= request.getContextPath() %>/flex/clinical_data.swf" height=250 width=500>
</div>

You just have to redeploy your portlet and now the menu appears on top of your portlet: