Thumbnail for Data Transformation Adventures with InterSystems IRIS by InterSystems Developers

Data Transformation Adventures with InterSystems IRIS

InterSystems Developers

16m 33s2,258 words~12 min read
YouTube auto captions
Transcript source

YouTube auto captions

This transcript was extracted from YouTube's auto-generated caption track. The transcript below is server-rendered so it can be read, searched, cited, and shared without opening the original YouTube player.

Timestamped outline
Pull quotes
[0:00]I would like to introduce our um sales engineer from Indonesia, Merlin, and the other two sales engineer from Singapore Bryan and Martin.
[0:00]And actually, it is our interoperability demo environment and because it's a demo game, so I just want to present it like game.
[0:00]But, uh, back to the basic, to collect data is still the need, so what we are going to talk about is the interoperability.
[0:00]So what he need to do is he need to convert the data he can collect into fire repository so that he can do the research immediately.
Use this transcript
Related transcript hubs

[0:00]Hello everyone, I'm K sales engineer from Singapore. I would like to introduce our um sales engineer from Indonesia, Merlin, and the other two sales engineer from Singapore Bryan and Martin. Today we are going to present our demo game 2025. And actually, it is our interoperability demo environment and because it's a demo game, so I just want to present it like game. So, um, let's talk about the background of this game is that, um, there is a researcher suddenly found out that our fire solution is so good because by making use of this solution, he can just directly use the data on fire converted into immediately plug in the ODC tools, shiny dashboard or other predefined Python library to help them to do the data analysis and do his research immediately. But, uh, back to the basic, to collect data is still the need, so what we are going to talk about is the interoperability. So what he need to do is he need to convert the data he can collect into fire repository so that he can do the research immediately. Of course, we provide him a secret tool called Inter system Magic box. So this is his adventure start. He start the adventure of interoperability to fire with our system magic box. So, his adventure is quite simple. There are six quizzes in his adventure, in each quizzes, he need to collect one kind of data and convert it into the viral repository for her for his research purpose. This is his first task is that, oh, very simple. The data is already on our RS data platform. And how can we convert it into fire? So, let's take a deeper look. So we have an application that will on our IS data platform and we would like to get the data from the data table and then change it in fire format and send it to the fire repository, simple, right? So to facilitate this action, we provide different tools for him to select. So let's see what he select. He chose our fire client, our business process, our DTL and fire model. All good choice. But he also need to have some skill in order to make it do smoothly. For example, some sense in SQL and he know how to use our visual chasing and then some flow chart drawing and relation mapping skill. For the fire client tools is very simple. It can directly core from the uh business operation by clicking a button. And then what they need to do is choose the uh fire client directly from the class list. And then give it the name and then use a few uh simple configuration and we can directly uh connect to the fire repository and send data. So the next thing um the player need to do is set up the business process to tell how we treat the data, what is the data flow look like. And we provide a UI interface for the user to define uh what is the uh source of data. How to do the transformation. After that, we package into stream and then send to the uh fire client business operation to help to send to the fire repository. The next thing is um the player need to uh use a DTL tools with the correct model to transform the data into fire format and they need to have some sense of do the relationship mapping. It's very simple. We just um click on the transformation table and then open the DTL tools and then uh choose the correct um fire model for doing the transformation and choose your database schema that's going to be transformed. So, after that, the player what you need to do is just doing a drag and drop to define the relationship and that's all. So, um, finally is the visual tracing tools uh for the player to understand what happened between the um message flow between the whole uh inside the production.

[4:38]So at the end of the quiz, we can always verify um the data by using a third party tools to see if the data is already sent to the fire repository. So now we can say that um the uh data in the is already uh transformed into fire and put into the fire repository. So um the game workflow will be like this. Um after they finish one task, we will move on to the second quizzes and the next quizzes. And then um for each quizzes they have different target. For example, in the quiz second, we would like to bring the data of a vital sign monitor in the fire repository. So um in this task, we would like to show uh how uh our tools can help to uh connect to um HL7 uh devices and uh we can take care of the HL7 message converting DB or convert into uh fire um data directly. So um in here um the tool sets will be different. For example, now they will need to use the HL7 uh built-in business office and HL7 building business operation. Of course, they need to choose the HL7 uh model for the doing the DTL. So um there will be some introduction of how to get the uh model is very simple, is just out of the balls and by doing some clicking and setting for the business operation and for the um data model is also is very easy, just out of the box and what they need to do is do the drag and job inside the DTL environment. And then for the uh business surface is also very easy, just out of the box and then we just do some um configuration and clicking and then the data pipeline can easily built. And finally, of course, to verify the result if it is uh successfully get the data from the vital sign monitor, get the blood pressure and then put into the fire repository. So we can show that uh we can uh get data from uh HL7 into a vital sign monitor. So the quiz will be move on and move on. So the next quiz is to show we actually can get a lab result from the lab system. And this is a very similar to what we do for the vital sign monitor. Um, send the order to uh the lab system by HL7, get the HL7 feedback and convert into the database as well as into the fire repository. So we actually, we use the same skill set to do this task, so very simple. And then because of the time, we won't go through every quiz one by one, but I would like to share what is the target of each quiz. So, uh, after we uh get data from the uh S, then the next task is, can we get data from um legacy data, uh, into our fire repository. So, our legacy data will be a Excel file. Um, in here, uh, we can use the uh existing uh two skill set, but now I add one more skill which is Python. Why? So, why Python is that in this uh quiz I want to show that is quite easy to uh plug in uh our data platform to any UI framework. For example, this one is in Python, um, we uh have a UI that can um drag and drop the files into it. And then we can choose uh which file we want to do the transformation. For example, it's an Excel file, so we can choose which sheet we are interested to do the uh conversion. And then after that, we choose the sheet and then it will directly write into our database for doing the um data transformation. So, after that, what we need to do is just build the data pipeline as usual, using the skill we have before and then the data will be transformed in fire and then the task is finished. So it's quite easy to um get data uh from a file by making use of a proper UI. So the next quizzes is how about getting data from a six second EMR. So in this case, we would like to show how easy to um connect to a SQL gateway by uh using our built-in tools called um SQL um business surface and SQL business operation. So it's just a simple tool that directly uh called from our library list. And what the user need to do is um to do some configuration to uh show where is the DSN and then also where it what is your SQL language that you would like to query the data.

[10:31]And that's all. So, um, finally, they might need to uh put the data temporary in a document DB um for staging. And they can make use of the staging table to convert into fire and send to the repository then so that we can reuse the data and also this is a way to unify the data from different sources. So, of course, they can test the result by using their own tools. And then, so easy, they already get um data from five sources. So the final, final uh quizzes is just to get data from the watch. So suppose the watch can have a waste API for us to do the query. So, maybe um the API is quite custom made for different kind of um watches or apps. So we can make use of our custom business service and custom business operation to help to achieve to get data directly from um the app or from the watch. So, for the custom business service is just uh you write some code that maybe call some Python library um that already provided by the source or you can write it by yourself. And then compile it and after you compile your um service actually, you can um directly uh call it from the list. And then as you show, you just use it directly. And for the operation is still the same. You write your own call. Call some library or function you have already predefined or that's given by the third party. And then you just add it in by compile and add it in by using our wizard. So what you the things you need to do is just use your skill to build up the pipeline for the data transformation. And then you may build your own UI to do the testing. For example, here I build a UI for testing um how we get the data from um the app of my watch. For example, I let the user to input the time range and then submit and our production will be triggered and then after I refresh the page, the new data will become into um display. Of course, you can visual trace what you have and then um use your client check if the data is already upload to the uh fire repository. So finally, we have data from different sources. After he finished six quizzes, actually what he get is he get a unified fire repository, which have data from different sources uh from EMR on, from um devices, from lab system, from batch file, from SQL gateway and from the waste API variable devices. So, um, of course, everything on a same fire repository is very easy for him to do the research by using the fire to solution to change it in format schema to uh plug in different uh existing tools. But how about uh we can make use of the uh existing fire repository to do something else. For example, this is one of the suggestion is we can provide a UI environment for the user to query the result inside the fire repository. Maybe get some result and populate it into some AI tools to do some analysis just like this example, we populate the observation result of one patient and to consult the open AI for some uh suggestion. Another thing we can do is to make use of our fire SQL builder to uh project our data inside the fire repository from fire format into table format. We have the tools to help us to take a look into the fire repository, different resources and different field and just look into the histogram of um different field to see what this field is meaning and then this will facilitate us to make the decision to see if we need to project this field. So, uh the next thing is that uh the projected data will be by resources into different table. For example, for their encounter resources will be projected in the encounter table and it's look like this.

[15:50]So after it projected into table format, it's very, very easy to uh use um the UI BI tools to uh put it in a pivot and dashboard. So this is example of using our own deep C web to project the data from the projecting encounter table into dashboard and pivot look like this. So thank you very much uh for listening to our presentation and the story is not the end. Because you can put as much sources into the fire repository as you can and do as much as you want. Thank you very much.

Need another transcript?

Paste any YouTube URL to get a clean transcript in seconds.

Get a Transcript