v0.6.2

Updates:

1. Include inline resources and primitive extensions in Fhir-To-Datalake pipeline.

   • a9cd7c7 Omit primitive extension when comparing columns (#309)
   • d0ae7a7 include inline resources and primitive extensions (#304)

2. Add server audience configuration for Dicom-To-Datalake pipeline.

   • dac149d Dicom multiple audiences (#300)

V0.6.1

Updates:

1. Update code namespace

   • e9c1883 Update Namespace (#278)

2. Add JobVersion property to job table

   • e4cc97e Add JobVersion To Both Pipelines (#279)
   • 10e1a0e set processing job start time as next resource time for job v1 (#281)
   • 61c00d6 generate job identifier by selecting fields for different job versions (#277)
   • 41b245b Fix ConfigVersion To Overall Test (#276)
   • ec52f95 add job version to solve version compatibility issues for version update (#274)

3. Commit processing job in sequence order

   • 06dcf24 commit processing job in sequence order (#283)

4. Other refinements

   • 2f5bd64 refine log (#299)
   • 57e197b Handle forbidden statue code reponse (#297)
   • bd62fbc trim white space in filter (#296)
   • 99f1957 remove resource size limitation (#295)
   • 3cd4aa2 Update Job Change Feed Limit To 5000 (#266)
   • c9d5552 Add latency metric (#272)

Fix:

• b6c481c Fix reference bug in doc (#302)
• 04e4b5c Specify Nuget To Resolve Security Vulnerability (#294)
• 91be04b Fix null exception for fhir expression (#292)
• 7ed8368 Fix Bug In Error Metrics (#284)
• 296d4ec Fix unit test for latency metric (#285)
• 6c7f9ec Fix upload task (#264)
• 7091358 Fix the image clean up dependency (#271)
• 3ae07da Update ConfigVersion 2 For Different Data Source (#270)
• 003c3a2 Bump Newtonsoft.Json To 13.0.2 (#265)

Support DICOM Metadata To Data Lake

New Features:

1. Support sync up DICOM metadata to data lake as Parquet files.

   • ee25817 Update readme adding the dicom part (#258)
   • 9e86d1a Refine documents (#257)
   • 45c3fd5 DicomToDatalake Deployment Guidance Page (#255) [ #256 ]
   • 9329a90 Refine DICOM Overall Test (#253)
   • 7d4baf6 Add support for moving DICOM metadata to data lake (#248) [ #232, #229, #231, #233, #234, #235, #240, #239, #245, #244, #246, #247, #250, #249, #252, #251 ]

Refinement:

1. When using Powershell to create External Tables based on the customized schema, will pull customized schema image to local directory with digest as the suffix.

   • ec1c21f Use Digest As Suffix For Customized Schema Directory (#254)

v0.5.0

New Features

1. Implement a generic job management component and scheduler service component to support scaling out to multiple instances.

   • c7c7cc4 refine scheduler service component (#192)
   • fbf582d add config for job concurrency count (#182)
   • 21ff0d5 Update agentName to Queue and Table name (#174)
   • 46ca552 Refine job management to improve scalability (#158) [ #153 ]
   • 22f0327 implement azure storage queue client (#144) [ #153 ]

2. Hide metadate of job status to internal storage to prevent customers' modification.

   • 22f0327 implement azure storage queue client (#144) [ #153 ]

3. Add health check component.

   • c35ccb6 Refine Healthcheck Timeout (#198)
   • bb625f0 Health Check for Scheduler Service (#165)
   • 830e54c Refine HealthChecker (#164)
   • cd0881b Health Check For ACR (#160)
   • cc62c86 Health Check (#132)

4. Support fetching filter config from ACR.

   • e89f4ef Support Fetch Filter Config From ACR (#162)

5. Add external dependencies check stage before pipeline running.

   • 420a607 Add external dependencies check (#210)

6. Add Metrics

   • 2e0f7f4 Add Error Metrics (#187)
   • 516ed5f Refine Health Check Metrics (#197)
   • 04fd562 Add Successful Resource Counts & Data Size Metrics (#169)
   • 270dc5a log metrics (#157)

7. Support container app deployment.

   • 0f4363b Refine Container App Template (#221)
   • 15e2d1f ARM Template To Deploy The Container App (#219)

Refinement

1. Retry Mechanism
   • fde8c1b Retry to get templates from ACR (#184)
   • bf25e28 Retry for containerRegistryTokenException (#218)
2. Performance Improvement
   • 40453b6 Remove _sort search parameter (#185)
   • 4ffe353 Refine memory usage (#236)
3. Lazy Initialize
   • 5fe9b6e Add lazy initialization for capability statement (#195)
   • 33b6da8 Lazy initialization for retrying to get authentication (#171)
4. Refine Logger
   • 4ed119d Refine Log Level & Add Diagnostic Logger (#175)
   • 3802350 Remove the resource content in logs (#161)

Fix

• de398c2 Update the default arrow block size to 30M (#228)
• 6aa0483 Update converter to fix digest bug (#237)
• 68e987e Exclude OperationOutcome (#223)
• 0f19cb3 bug fix: set parquet version to ParquetNativeLibVersion variable (#220)
• 8b17de3 Fix data client (#204)

Bug Fix

Changes:

• aef835e Update azure-pipelines.yml for Azure Pipelines to update version to v0.4.1
• 525b4f7 [bugfix] The job may hang when update job progress (#183)

This list of changes was auto generated.

Support filtering and customized schema for FhirToDatalake

Updates:

Support filtering in FhirToDatalake pipeline

Users can use filtering on the pipeline to filter FHIR data in pipeline

• b84eabd Update documentation for filtering feature (#143)
• ee46696 Merge pull request #122 from microsoft/personal/yanhon/filter

Support customized schema in FhirToDatalake pipeline

Users can use customized schema on the pipeline to handle FHIR Extension or other complex properties.

• 27932b0 add documentation for how to use FHIR extension in Synapse sync agent (#141)
• f1be437 Update script to create customized table (#135)
• eee009f Integrate FHIR-Converter to support customized schema (#127)

Others

• bccc348 upgrade to net6.0 (#137)
• bb3a59b Fix data lake client error when parsing storage account name. (#107)
• 263f09f Fix invalid link and add tool guidence (#105)
• 07d4ed6 Fix FHIR server URL missing trailing slash issue (#104)

v0.3.0

Updates:

1. Add the FHIR to Synapse Sync Agent that enables you to perform Analytics and Machine Learning on FHIR data by moving FHIR data to Azure Data Lake in near real time and making it available to a Synapse workspace. The agent is an Azure function that extracts data from a FHIR server using FHIR Resource APIs, converts it to hierarchical Parquet files, and writes it to Azure Data Lake in near real time. This also contains a script to create external tables and views in Synapse Serverless SQL pool pointing to the Parquet files.
2. Package updates and bug fixes in FHIR to CDM pipeline.

v0.2.0

This is the first public release of FHIR Analytics Pipelines.

New Functionality

• #22 Add CDM to synapse ARM template

Known Limitations

• No support for incremental data load
• Relationships between tables need to be created manually at the destination (Synapse)

v0.1.0

This is the first release of FHIR Analytics Pipelines.

The release contains a FHIR to CDM toolset that can be used to generate an ADF pipeline that exports FHIR data, rectangulaires it, and copies it to a CDM folder on ADLS Gen2. It also contains instructions for subsequently moving the data from CDM folder to Synapse.

Current limitations

• No support for incremental data load
• Relationships between tables need to be created manually at the destination