Modeling Healthcare Logistics in a Virtual World
Progress during Fall 2008
This report is a synopsis of progress in the
Managing
Supplies and Inventory
Higher
Level Languages for Workflows – Lisp and Prolog-based
Smart
Phone Virtual World Plugin
Export
from OpenSim and Second Life
Applications Team
Conveyor Belt
The objective of the conveyor belt project was to build segments of a conveyor belt that could operate together like Lego. A conveyor belt consists of straight segments, curved segments, Y- junctions, and shunts to select a Y direction.
The Second Life physics engine made this project a real challenge – obvious things did not work. Using rotating dowels to move boxes caused the dowel to “capture “ the box and then it rotated around the dowel. A “belt” rotated around its center instead. The first attempts at making boxes responsible for their own movements resulted in boxes as vehicles – when they fell off the conveyor belt, they kept moving along the ground until they reached the edge of the island where they were eventually reaped.
Two conveyor belts were developed.
- Gabe Meisen and Mark Mills built straight and curved segments. Objects were given a movement script when they we placed at the beginning of a segment of conveyor belt and it was removed at the end of a segment. Boxes sometimes fall off the curved section. A possibility is to use way points next time.
- Jeremy Choens built straight, curved, and Y segments as seen below – works pretty well unless there is lag which can cause boxes to fall off – see video.
Mirror Worlds
The objective of this project was to connect changes in the real world to changes in the virtual world. At the RFID Lab, there is a mock store containing RFID tagged apparel. There is a real-time location aware reader antennae array (RFcontrols ITCS system) which reads locations of tags every two minutes with accuracy around one foot. The idea of this project was to capture the tag reads and display the items in the virtual world (a mirror world) so that if an item is moved in the real world, it is also moved to its new location in the virtual world.
Green boxes in the virtual
world represent items on racks in the real world.
When a real world object is
moved, the virtual world object moves too.
Workflow Team
Not much happens in Second Life virtual worlds. In contrast, a real hospital bustles with activity. We need a way to model such activity. Beginning this semester, Fran Hagstrom supplied us with 15 nurse-patient interaction scenarios – mostly based around patient needs (hungry, in pain, cold). For instance,
Workflow:
Patient is thirsty
The patient calls the nurse.
The nurse asks what’s wrong.
The patient says he is thirsty.
The nurse goes to get some water.
The nurse returns with water.
The nurse asks if there is anything else
If so, continue on.
If not, the nurse returns to her post.
Before we describe the projects, we note that Second Life uses Linden Scripting Language – and we had previously created a robot assembly language that is implemented in LSL. The open source implementation of Second Life, OpenSim now called Open Metaverse, uses C#. So we have one set of workflows that involve robots in Second Life and another set that involve avatar-bots in OpenSim.
Face and Body Animations
Humans have emotions. Avatars do too – called animations. Kenny Cason and Danny Cassimatis identified how avatars (and avatar-bots) can use animations in workflows.
Robot Workflow
Nick Farrer has been working on a robot assembly
language. The robots resemble R2D2 and
can navigate the hospital using a waypoint system. Daniel Apon and Evan Ray translated Fran’s
scenarios into Nick’s robot assembly language and developed a scheduler so that
we can run several workflows simultaneously.
See video. Or visit
Example workflow specification in robot assembly language
label
patientIsThirsty
status $var2
" is thirsty"
wp $var1
say
"What is the problem?"
wait listen
"I'm thirsty."
sleep 1
say "Let
me go get you water."
wp shelf2
grab nearest
water
jumpf
thirstyf
remote $var2
" thread p0 default thirstys " $var1
wp $var1
say
"Here's your water."
wait listen
"Thank you for the water."
sleep 1
say
"You're welcome. I'll be over at the nurses' station."
wp shelf1
jump end
label
thirstyf
remote $var2
" thread p0 default thirstyf"
wp $var1
say
"There's no water, sorry."
wp shelf1
jump end
Avatar-bot Workflow
Where avatars are guided by their human, avatar-bots are guided by programs. We also wanted to see if we could develop workflows for avatar-bots playing the role of doctor, nurse, patient, … One huge advantage of avatar-bots is that they can look the part of a doctor, nurse, or patient. Along with that, they can use animations to show if they are happy, sad, surprises, worried, … In one interaction across kinds, the robots nurses help an avatar-bot.
Shawn Marhanka and Taylor Johnson developed ten workflows including when a patient is in pain, thirsty, hungry, cold, depressed, crazy, scared, nurse tantrum, nurse smoke break, nurse wash hands, and nurse get new patient clipboard. For instance, when the patient is thirsty, the patient calls the nurses, says “Nurse, I’m thirsty” and the nurse then goes to get water, then goes to the patient and says “here is your water” then the patient does a drink animation where he brings the virtual water up to his mouth.
One of the animations mixes robots and avatar-bots and involves a robot, nurseZ, and that robot stays in the room right beside my patient’s bed. The patient tells the nurse that he is cold and the nurse gives the command to the robot to go get a blanket for the patient, then the robot goes and gets the blanket and returns it to the patients room, then the patients thanks him and then later the robot returns the blanket to the chest where they are keep (oops – that blanket should have been sanitized – oh well).
In addition to working with Shawn and Taylor, Clinton
Monk developed a Bot monitor GUI which enables users to monitor avatar bots
when the user isn’t even logged in to Second Life. This takes information from the C# code, such
as states and locations, and displays it in a manner so that the user can
easily read information on the bots.
Below is a picture of the finished interface. AIn addition,
And below is a screenshot of the Pain workflow in action:
and another …
My avatar refusing a colonoscopy. (more screenshots here)
Managing Supplies and Inventory
When a nurse needs five blankets, those supplies may not be in just one inventory. Anthony Rosequist developed an inventory system. A nurse requests N of some item and an answer returns the nearest inventories that satisfy the request. The system is integrated with the virtual RFID scanner system (Josh McFarlane, Spring 2008) so when items are removed/returned from/to an inventory, the count goes down/up. The code is here.
Higher Level Languages for Workflows – Lisp and Prolog-based
Writing all workflow steps in LSL or C# seems to lose the idea that workflows are parameterized collections of tasks. It would be nice if, after we executed tasks and subtasks we could record each in a database so that we could later query the tasks, replay them, analyze how long they took and how effective they were. But that will come later. First, we needed a way to represent tasks in a higher level language. We decided to look at Lisp and Prolog for this purpose.
May Zeineldin translated Fran’s English workflows into Prolog. Ralph Nelson translated the same workflows into Lisp. The original idea was to (a) separately develop Prolog and Lisp interpreters that could be embedded in C# (see Infrastructure Team below), (b) run May and Ralph’s workflow code, (c) which would then run the avatar-bots. So far, we have been able to execute May’s Prolog commands but the Lisp2C# wasn’t finished until the 12th hour so we have not tested the Lisp workflow yet.
Infrastructure Team
The infrastructure teams’ objective is to develop tools and capabilities that other teams may use to support their own work in OpenSim and Second Life. This includes tasks that involve browser, cell phone, database, lisp, prolog, and ontology use and integration into the OpenSim environment.
Lisp and Prolog in C#
As mentioned at the end of the workflow section, we wanted to represent workflow using higher level commands and we chose to consider Prolog and Lisp to represent workflows. This meant finding or developing Prolog and Lisp that could embed in C# (which is used by OpenSim) - that is, C# programs that could take Prolog or Lisp syntax as input and could either call lower level Prolog or Lisp or could call the underlying abstract machine, C#. Dameon Rogers worked on Prolog to C# and Jerry Maine worked on Lisp to C#.
Database
The objectives were:
- could we use ODBC to replace MySQL with Oracle 11 in OpenSim – the purpose being to understand DBMS portability.
- could we use Oracle Spatial to query the location of geometric objects – the purpose being to eventually support a wide range of location queries
- could we add a higher level ontology/type facility to name objects so we could find them (because, just looking at the geometry, it is hard to tell a door from a castle.
Tom Censullo and Jamelle Brown succeeded in all three objectives.
Ontology
In an illuminating term paper, Aaron Berkowitz considered how to use ontologies in virtual worlds. His abstract:
In
the current development of internet technologies, one can clearly see two
technology trends: the development of the Semantic Web and the use of virtual
worlds such as Second Life. However, comparatively little work has been done to
date on the integration of these technologies. In this paper we describe two
use cases for an ontology in a virtual world project. The first is the use of
an ontology as a description of and store for the objects in the virtual world.
The second is using an ontology to provide domain knowledge to the virtual
world. We argue that the first use case is problematic, if not impossible in a
meaningful way, and we give examples of the second use case. We go on to
suggest how a sort of intermediate system might be used to aid in the automatic
generation of virtual world objects. Afterwards we discuss work that has been
done in the use of ontologies for medical workflow system and briefly sketch
what sort of work needs to be done to incorporate such a system with the
Modeling Healthcare Logistics in Second Life project. Lastly, we conclude with
some general remarks regarding when it might be appropriate to use an ontology
and future work to be done in this area.
Browser Virtual World Plugin
Second Life architecture is clients on end user workstations connected to remote servers. That is very like the architecture of the web. The difference is, the web is docu-centric but Second Life is a 3D virtual world. Adding one and one, it only makes sense that current web browsers, which already have a plugin architecture, could be extended to also be 3D virtual world browsers.
The goal of our project was to create a browser based plug-in that would allow the user to log-in and access their Second Life account anywhere that a browser would be available. Julien Coutable and Tyler Murray were able to develop a bare-bones plugin for OpenGL helping pave the way for later students to attempt Second Life or OpenSim browser plugins.
Smart Phone Virtual World Plugin
Since smart phones are tomorrow’s laptop and are available across the world even when laptops and desktops are cost prohibitive and since people carry cell phones with them, it seems a no-brainer that access from a cell phone to a virtual world that mirrors the real world would give people a new way to track locations.
The objective of this project was to update the position of an avatar in the second life world based on physical movements of a person in the real world. Moving an avatar in Second Life normally requires a user to sit in front of a computer terminal, and press keys on a keyboard to move around. This team’s goal was to have some sort of application connected to Second Life running on the iPhone which would be aware of any changes in the user’s position.
Warner Skoch, Bradley Simmons, and Dexter Freeman were able to identify some systems that already provide the capability to access Second Life from a cell phone and other applications that provide access to GPS and, after much trouble, got the iPhone development environment working but left the prize for a future team.
Import-Export Team
Our hospital on
Related Work
Chad Ferrari researched existing methods of importing and exporting objects into and out of Second Life to find if they may help our efforts (e.g., Prim.Blender). Along with finding existing solutions, he procured those which seamed feasible - many were closed source or pay solutions, which are not ideal for our purposes. Export is more feasible than import (except for textures). Import from various outside formats is problematic as SL uses its own custom format.
Export from OpenSim and Second Life
Matt Miller developed a way to export prims from opensim to .xml which can be converted to .prims format and opened in prim.blender and converted to KML. Using a Prim-Blender script he created, you also export prims from Second Life which can be opened in prim.blender or converted to .xml and imported to opensim using the ImportExport tool he created. This allows archiving and direct manipulation of existing/already-imported prims.
KMLImporter.cs
Andres Baez and Ben Rowse: Our result is a working command that has been added to libsecondlife called KMLImporter.cs. This command reads values from a text file that correspond to XML prim values and parses an XML file out of these values. This XML file is then imported into OpenSim. Andres drafted a useful prim XML description document.
Parsing KML/Collada
Because most of the
Prim-Splitting Algorithm
Mapping Casey’s COLLADA intermediate parse tree to SL prim was the second half of that translation. Because SL prims are limited in size, John Hart worked on a algorithm to receive vertex information from Prim Creation and divide it into Second-Life friendly sub pieces able to be rendered directly. Currently non-functional due to a system crash at semesters’ end.