parser/README.md

## About Healthcare/IO Parser

The Healthcare/IO **parser** is an Electronic Data Interchange (EDI) parser developed at Vanderbilt University Medical Center during Khanhly Nguyen's summer internship 2019. Built in a healthcare setting, the parser focuses (for now) on x12 claims (837) and remittances (835)

This code is intended to extract x12 837 and 835 and format them into portable and human readable format (JSON). This allows the claims to be stored in document data stores such as Mongodb, couchdb or databases that have support for JSON like PostgreSQL

We wrote this frame to be used in both command line or as a library within in your code. The framework is driven by configurations that derviced from X12 standards.

## Features

| Features | Description|
| -------- | --- |
|X12 claims/remits| parsing of {x12} claims/remittances into JSON format with human readible attributes|
|Multi Processing| capable of processing multiple files simultaneously to speed up processing|
|Analytics support| descriptive statistical analytics : distribution, various counts|
|Process Recovery| capable of recovering interrupted runs|
|Export to RDBMS| exports data to relational format (NoSQL -> SQL) 7 supported databases <br> * PostgreSQL,<br> * Redshift, <br>* Neteeza, <br>* Mysql, <br>* Mariadb, <br>* bigquery, <br>* sqlite3 |
|**Issues and Bug reports**| info@the-phi.com


For advanced features visit [Healthcare/IO::Parser](https://healthcareio.the-phi.com/parser) homepage


## Installation

**System requirements**

    - git (latest cli)
    - python3+ (with virtualenv)
    - mongodb (optional)

**Installation command**

    pip install --upgrade git+https://hiplab.mc.vanderbilt.edu/git/lab/parse-edi.git

## Usage 

Healthcare/IO is primarily intended to be used as a command line parser (for now). It is fully written in python 3+

**CLI :**

1. signup to get parsing configuration

    The parser is driven by a configuration file that specifies fields to parse and how to parse them. You need by signing up, to get a copy of the configuration file. 
        #
        # Use sqlite as data-store
        healthcare-io.py --signup <email> [--store <mongo|sqlite>]

2. check version       

    Occasionally the attributes in the configuration file may change, This function will determine if there is a new version available.

        healthcare-io.py --check-update

3. parsing data stored in a folder

    The parser will recursively traverse a directory with claims and or remittances

        healthcare-io.py --parse --folder <path> [--batch <n>] [--resume]
        
        with :
            --parse     tells the engine what to parse claims or remits
            --folder    location of the claims|remits
            --batch     number of processes to spawn to parse the files
            --resume    tells the parser to resume parsing 
                        if all files weren't processed or new files were added into the folder

4. export data to a relational data-store

    The parser will export data into other data-stores as a  relational tables allowing users to construct views to support a variety of studies.

        healthcare-io.py --export <835|837> --export-config <path-export.json>

        with:
            --export-config    configuration to support data-store

    **example**
        1. Exporting to PostgreSQL

            {"provider":"postgresql","database":"healthcareio","schema":"foo"}
    
    **NOTE**

    The output generates a set of tables that are the result of transforming unstructured data to relational structure. The tables can be bound with the attribute **_id**

    The configuration file needed to implement export is modelled after the following template:


        {
            "provider":"<postgresql|redshift|mysql|mariadb>",
            "database":"<name>",["host":"server-name"],["port":5432],
            ["user":"me"],["password":"!@#z4qm"],["schema":"target-schema"]
                
            }

    **parameters:**

        provider    postgresql,redshift,mysql or mariadb (supported providers)
        database    name of the database
    
    **optional:**

        schema      name of the target schema. If not provided we will assume the default
        host        host of the database. If not provided assuming localhost
        port        port value of the database if not provided the default will be used
        user        database user name. If not provided we assume security settings to trust
        password    password of database user. If not set we assume security settings to trust

**Known Limitations**

1. By default it does NOT come with all {X12} Segments. 
2. Does not support an easy way to rename attributes it parses
3. For now can only read {x12} from disk

There is support for additional features and attributes available at [Healthcare/IO::Parser](https://healthcareio.the-phi.com/parser).

**In development**

1. Wizard/UI to enable attribute renaming
2. Dashboard for quick overview
3. Reading {x12} from s3 and other cloud buckets
4. Docker Image

## Credits

This project was realized thanks largely to the effort of  Vanderbilt University Medical Center and 

|  | | |
|--|--|--|
|- Khanhly Nguyen<br>- Heather Dunn<br>- Nina Thousand|- Byun Kang<br>- Gaylon Stanley<br>- Cheng Gao|- Thomas Brown<br>- Brad Malin<br>- Steve Nyemba