Commit Graph

164 Commits

Author SHA1 Message Date
Steve Nyemba 05d990d713 bug fix: table creation with schema (if any is provided) 2023-07-09 21:22:31 -05:00
Steve Nyemba 8f94ea260b bug fix: initialization of fields unnecessary 2023-07-09 20:58:00 -05:00
Steve Nyemba 653d22cd8c bug fix: init function for fields not working 2023-07-09 20:47:54 -05:00
Steve Nyemba 0933e4155f bug fixes has function 2023-07-09 18:03:03 -05:00
Steve Nyemba b5794e0a61 bug fixes has function 2023-07-09 17:58:32 -05:00
Steve Nyemba ad84e24533 version update 2023-07-09 17:51:47 -05:00
Steve Nyemba 1885e3a5f6 bug fix, making table when provided a schema 2023-07-09 17:51:09 -05:00
Steve Nyemba f57cd8d871 version update 2023-07-09 17:46:01 -05:00
Steve Nyemba 5f1592b973 bug fix 2023-07-09 17:37:57 -05:00
Steve Nyemba fb20687a46 version update 2023-07-09 16:37:30 -05:00
Steve Nyemba 712c3d076d bug fix: schema specification in sql handling 2023-07-09 16:32:05 -05:00
Steve Nyemba 687b211b7b bug fix: field_name (typographic error) 2023-07-05 15:58:51 -05:00
Steve Nyemba b27c5a88d4 bug fix: bigquery meta on a project other than the one with service key 2023-07-05 15:53:54 -05:00
Steve Nyemba 74d8b2c6f2 bug fix 2023-07-05 14:51:14 -05:00
Steve Nyemba aaaf64db33 bug fix: versions of sqlalchemy for meta data 2023-06-16 17:39:02 -05:00
Steve Nyemba 18cd1d8b11 bug fix with sql handling sqlalchemy 2023-06-16 16:07:29 -05:00
Steve Nyemba 66f43a98c1 bug fixes: etl, mongodb lexicon 2023-05-30 15:47:10 -05:00
Steve Nyemba 78ad8f3ad8 bug fix: etl transport 2023-05-30 15:19:10 -05:00
Steve Nyemba ed66370fdf bug fix: ETL jobs and streamline 2023-05-30 15:01:51 -05:00
Steve Nyemba 695c10e797 bug fix, added ferretdb 2023-05-25 09:39:51 -05:00
Steve Nyemba 5f9416d2f6 bug fix: removing unused library 2023-01-18 21:13:38 -06:00
Steve Nyemba dcd6a83a35 bug fix: disk - nujson, refactor 2023-01-04 09:12:55 -06:00
Steve Nyemba b92055f018 bug fix: disk - nujson, refactor 2023-01-04 09:12:14 -06:00
Steve Nyemba 98c9348f9c bug fix: mongodb read 2022-12-14 16:48:40 -06:00
Steve Nyemba 1c254eb133 bug fixes, enhancements mongodb 2022-12-09 16:19:39 -06:00
Steve Nyemba b8fb538ec7 bug fix mongodb,bigquery 2022-11-13 15:45:21 -06:00
Steve Nyemba 883a6ef22f bug fix & new feature 2022-09-19 10:01:34 -05:00
Steve Nyemba 3a3946c7d8 bug fix 2022-08-19 15:32:35 -05:00
Steve Nyemba e5f3b19336 bug fix: added chunks and enhacement with global variables (constants) 2022-08-10 09:17:30 -05:00
Steve Nyemba 245b319e7b added partitioning and chunking to support healthy ETL jobs or write functions 2022-08-03 12:07:27 -05:00
Steve Nyemba 8a6beae956 bug fix: sql handler ... 2022-06-26 15:48:29 -05:00
Steve Nyemba ae1a4cec73 bug fix: removing print statements 2022-06-26 12:18:12 -05:00
Steve Nyemba b0c6bb6ca9 bug fix: environment variable on default database 2022-06-25 14:00:22 -05:00
Steve Nyemba 8ca04e8c48 bug fix: environment variable on default database 2022-06-25 13:59:49 -05:00
Steve Nyemba d7c9e41095 bug fix: meta/sql.py 2022-06-12 21:14:46 -05:00
Steve Nyemba 9805264f87 bug fix: meta data 2022-06-12 20:05:03 -05:00
Steve Nyemba 6b78a82e99 bug fix 2022-06-10 10:27:44 -05:00
Steve Nyemba b2a224f4a7 bug fix: sql lock for parallel processing 2022-06-10 10:11:37 -05:00
Steve Nyemba 40bcfc40f9 bug fix: authentication 2022-05-16 13:14:29 -05:00
Steve Nyemba 7c328e5106 bug fix: mongodb mechanism parameter for authentication 2022-05-16 11:59:46 -05:00
Steve Nyemba db1496dd38 bug fixes 2022-05-16 11:54:28 -05:00
Steve Nyemba f64b945245 new: authentication using file on disk, misc bug fixes 2022-05-16 11:27:36 -05:00
Steve Nyemba 8cd34d902a bug fix: ETL logging and rabbitmq-server listener 2022-05-11 11:17:27 -05:00
Steve Nyemba 67cb7de861 bug fix: logging with ETL 2022-05-06 14:30:59 -05:00
Steve Nyemba 899339f11f bug fix: authentication mongodb 2022-05-06 14:25:12 -05:00
Steve Nyemba fd9523e99a enhancement mongodb 2022-04-29 11:15:32 -05:00
Steve Nyemba e9a6779a9b new file and documentation 2022-04-11 23:31:47 -05:00
Steve Nyemba 98eaa99820 bug fixes and enhancements, added console output(for logging) 2022-04-11 18:34:32 -05:00
Steve Nyemba 6c406407b2 bug fix: ... minor update sqlalchemy 2022-03-31 17:13:24 -05:00
Steve Nyemba e5fadc64a0 optimizations mongodb 2022-03-19 00:02:53 -05:00
Steve Nyemba 38e1bce6c2 bug fixes and optimizations 2022-03-12 12:25:29 -06:00
Steve Nyemba 105ff00224 bugfix: ETL multiprocessing 2022-03-07 18:50:29 -06:00
Steve Nyemba f672c04844 bugfix: sqlalchemy 2022-03-07 15:29:03 -06:00
Steve Nyemba cfc683c1b3 bug fix: transport cli and write function for sql 2022-03-07 14:17:27 -06:00
Steve Nyemba fd9442c298 bug fix: write problems, using drivers 2022-03-04 15:00:30 -06:00
Steve Nyemba 14a551e57b Bug fix: sqlalchemy facilities added 2022-03-03 16:08:24 -06:00
Steve Nyemba a7df7bfbce bug fix: has sql 2022-03-02 13:15:35 -06:00
Steve Nyemba b239a5149f bug fixes and simplifying interfaces 2022-01-29 11:15:45 -06:00
Steve Nyemba ccc05acc01 bug fix: etl engine, sqlite inserts 2021-12-09 15:25:58 -06:00
Steve Nyemba 2de6e51bdb bug fix: simplyfying factory interface 2021-11-18 15:21:26 -06:00
Steve Nyemba 44ef71eb92 bug fix 2021-09-03 01:25:45 -05:00
Steve Nyemba 0c0eaf7063 bug fix: imports, testing code removed 2021-09-01 02:33:40 -05:00
Steve Nyemba 83cc1778c7 bug fix: print 2021-07-28 22:51:19 -05:00
Steve Nyemba 10adde7a08 bug fix: sqlwriter 2021-07-28 22:32:07 -05:00
Steve Nyemba 79cdc0c0d0 bug fix, enhancement with pandas 2021-07-23 15:22:23 -05:00
Steve Nyemba 835e1253a5 bug fix: filereader 2021-07-19 19:07:06 -05:00
Steve Nyemba b21ddd6945 bug fix: removing testing code 2021-07-19 18:54:37 -05:00
Steve Nyemba 19ab510125 cli etl tool 2021-07-08 17:31:29 -05:00
Steve Nyemba 7c2e945996 bug fix: support for netezza 2021-06-29 15:12:28 -05:00
Steve Nyemba 4c810fe57a bug fix 2021-06-20 18:56:12 -05:00
Steve Nyemba b5eb5e0991 bug fix: adding types to read function 2021-06-15 12:30:48 -05:00
Steve Nyemba 91434ec32d bug fix: pandas dtypes 2021-06-14 14:31:30 -05:00
Steve Nyemba cebc784494 bug fix: lock, parallel processing bigquery 2021-04-13 17:29:54 -05:00
Steve Nyemba 84e212d5a3 bug fix: lock, parallel processing bigquery 2021-04-13 17:27:23 -05:00
Steve Nyemba afcc5ed690 bug fix ... 2021-03-29 23:30:54 -05:00
Steve Nyemba 687ffec215 bq support 2021-03-15 12:38:54 -05:00
Steve Nyemba 5bf2012c59 bug fix: batch update fixes 2021-01-10 09:55:04 -06:00
Steve Nyemba fa615aa931 bug fix: insert on an etl with data-typing (tricky) 2021-01-02 17:39:27 -06:00
Steve Nyemba eef84d8069 bug fix: sql writer::apply 2021-01-02 05:38:06 -06:00
Steve Nyemba 898455f23c bug fix: sql writer::apply 2021-01-02 05:35:49 -06:00
Steve Nyemba 18c9245ace bug fix: sql writer 2021-01-02 05:29:52 -06:00
Steve Nyemba 99d29f0a1f bug fix: sql writer 2021-01-02 05:24:12 -06:00
Steve Nyemba 504596f838 new: added sql handler for Redshift,PostgreSQL, MySQL/Mariadb 2020-12-31 11:12:34 -06:00
Steve Nyemba 7ba09309d3 bug fix: mongodb commands with values 2020-12-26 12:20:40 -06:00
Steve Nyemba 03e0203c28 bug fix: sqlite lock 2020-12-11 03:22:49 -06:00
Steve Nyemba 6f08d64e49 bug fix: sqlite import pandas 2020-12-11 02:46:31 -06:00
Steve Nyemba 2be9d9db2f bug fix: aws s3 handler 2020-12-10 03:56:08 -06:00
Steve Nyemba c36b07073a bug fix with s3 hierarchy 2020-12-10 03:42:30 -06:00
Steve Nyemba 4b49b0a060 bug fix with mongodb client 2020-11-15 21:19:40 -06:00
Steve Nyemba 9eb46954cd bug fixes: mongodb and s3 bucket 2020-11-04 14:26:46 -06:00
Steve Nyemba e70802ef00 mongo thread/process safe? 2020-10-08 17:14:35 -05:00
Steve Nyemba 18d6ff90fe close function for mongodb 2020-10-06 15:26:06 -05:00
Steve Nyemba e4a1ef8dd7 bug fix 2020-09-26 16:53:33 -05:00
Steve Nyemba cbf641a986 bug fix with reader 2020-09-22 13:30:04 -05:00
Steve Nyemba 5676c4991f json object writing ... 2020-08-11 17:20:46 -05:00
Steve Nyemba 4a985e82dc bug fix with diskwriter (adding return carriage) 2020-08-11 16:42:21 -05:00
Steve Nyemba 617f829e1e bug fix:sqlitewriter 2020-07-27 21:47:02 -05:00
Steve Nyemba 21ef842137 bug fix 2020-05-17 22:53:54 -05:00
Steve Nyemba d5ba648abf support for filters in read 2020-05-17 21:57:18 -05:00
Steve Nyemba 834dc80e79 bug fix (updating pika interface) 2020-03-04 02:23:21 -06:00