hiplab/data-transport
hiplab
/
data-transport
mirror of https://github.com/lnyemba/data-transport
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity
python data transport layer, mongodb, netezza, bigquery, postgresql files
267 Commits 7 Branches 0 Tags 573 KiB
Python 72.6%
Jupyter Notebook 27.4%
Go to file
Steve L. Nyemba aa926f77a3
Merge pull request #20 from lnyemba/v2.2.0 
new provider console and bug fixes with applied commands
 		2024-09-19 11:16:11 -05:00
bin bug fix: registry (more usable) and added to factory method 2024-06-14 15:30:09 -05:00
info new provider console and bug fixes with applied commands 2024-09-19 11:15:13 -05:00
notebooks bug fix, duckdb in-memory handling 2024-09-13 10:15:47 -05:00
transport new provider console and bug fixes with applied commands 2024-09-19 11:15:13 -05:00
.gitignore .. 2023-12-22 14:16:40 -06:00
README.md documentation ... 2024-07-07 11:39:29 -05:00
requirements.txt S3 Requirments file 2017-09-26 16:10:14 -05:00
setup.py bug fixes, using boto3 instead of boto for s3 support 2024-07-10 15:50:46 -05:00

README.md

Introduction

This project implements an abstraction of objects that can have access to a variety of data stores, implementing read/write with a simple and expressive interface. This abstraction works with NoSQL, SQL and Cloud data stores and leverages pandas.

Why Use Data-Transport ?

Mostly data scientists that don't really care about the underlying database and would like a simple and consistent way to read/write and move data are well served. Additionally we implemented lightweight Extract Transform Loading API and command line (CLI) tool. Finally it is possible to add pre/post processing pipeline functions to read/write

  1. Familiarity with pandas data-frames
  2. Connectivity drivers are included
  3. Reading/Writing data from various sources
  4. Useful for data migrations or ETL

Installation

Within the virtual environment perform the following :

pip install git+https://github.com/lnyemba/data-transport.git

Features

- read/write from over a dozen databases
- run ETL jobs seamlessly
- scales and integrates into shared environments like apache zeppelin; jupyterhub; SageMaker; ...

What's new

Unlike older versions 2.0 and under, we focus on collaborative environments like jupyter-x servers; apache zeppelin:

1. Simpler syntax to create reader or writer
2. auth-file registry that can be referenced using a label
3. duckdb support

Learn More

We have available notebooks with sample code to read/write against mongodb, couchdb, Netezza, PostgreSQL, Google Bigquery, Databricks, Microsoft SQL Server, MySQL ... Visit data-transport homepage