ari/minerva
1
0
Fork 0
forked from bellwether/minerva
Code Pull requests Activity
105 commits 1 branch 0 tags 201 KiB
Commit graph

60 commits

Author SHA1 Message Date
Ari Brown
4df9b04b2b whew, got partial reads working with named temporary files. lots of weird oddities were discovered in the process 2024-07-24 18:50:03 -04:00
Ari Brown
d67e398a69 added bare machine support (remote), and making progress with the stdout stderr redirection 2024-07-23 16:34:34 -04:00
Ari Brown
2e215c45cf using select to get partial updates of stdout and stderr from commands 2024-07-22 22:09:17 -04:00
Ari Brown
a3374fd85c whoops, added self 2024-06-17 10:58:15 -04:00
Ari Brown
237c9b8c3c fixed bug in cost 2024-06-17 09:40:19 -04:00
Ari Brown
fdc0fd2ded added machine pool for easy clustering and trying to get glue support 2024-05-29 16:23:21 -04:00
Ari Brown
c2bc9e1028 added support for determining instance costs 2024-05-17 16:47:37 -04:00
Ari Brown
20ae647447 improved table deletion 2024-03-25 15:39:39 -04:00
Ari Brown
fd289257aa oops. bugfix 2024-03-14 09:13:59 -04:00
Ari Brown
e294fc062b bug fix for new sql loader 2024-03-01 13:35:06 -05:00
Ari Brown
af4f8bb999 added sql loading code from metrics calculator to minerva 2024-03-01 13:29:11 -05:00
Ari Brown
98524d3be6 fixed a bug that occurred from refactoring 2024-02-23 10:55:53 -05:00
Ari Brown
b453c9a515 loosened version requirements; added redshift history function; enabled Minerva to work using instance IAM role instead of explicit profile 2024-02-07 12:25:10 -05:00
Ari Brown
40d2eefd81 added a missing dependency; added support for redshift queries to return 0 rows 2024-02-01 14:56:48 -05:00
Ari Brown
5dccce53e9 significant improvement to the readme and verification that all the examples work 2024-01-31 16:18:32 -05:00
Ari Brown
e3c11fb1aa tidying up examples and readme 2024-01-30 18:13:56 -05:00
Ari Brown
819bf7abf3 fixed dataset aggregation; typo; and a bad empty string check 2024-01-26 11:48:35 -05:00
Ari Brown
622d104bcb update ci/cd 2024-01-25 14:43:11 -05:00
Ari Brown
14504accc7 got rid of useless print 2024-01-25 13:49:49 -05:00
Ari Brown
ddcacdb569 fixing local paths and redshift credentials 2024-01-25 13:42:24 -05:00
Ari Brown
ae3173b510 added helpers for local files, loading templates, and an example for canceling queries 2024-01-25 11:10:50 -05:00
Ari Brown
9442c33d14 adding parallelization helpers and query cancelation 2024-01-18 12:38:42 -05:00
Ari Brown
5eb8471081 better redshift support, working on dask stuff 2023-12-21 14:23:58 -05:00
Ari Brown
97c27f25a0 zstd w/ compression level 4 is now standard for athena query results 2023-11-30 16:48:45 -05:00
Ari Brown
fe06b6b808 adding dask examples 2023-11-16 10:33:56 -05:00
Ari Brown
c0ff6af866 specifying port for worker dashboard 2023-11-15 15:09:18 -05:00
Ari Brown
25b0360bed dask worker logs now written to disk hopefully 2023-11-15 14:55:16 -05:00
Ari Brown
10bcd367f6 dask worker logs now written to disk hopefully 2023-11-15 14:50:03 -05:00
Ari Brown
239abf8fc1 dask scheduler apparently doesn't exist sometimes 2023-11-09 14:08:03 -05:00
Ari Brown
65768f6f24 trying to debug why loading large amounts of data fails 2023-11-09 10:46:01 -05:00
Ari Brown
65247844e0 better abstractions 2023-10-26 13:10:49 -04:00
Ari Brown
c9c0ad4422 finally figured out how to stop dask from erroring out. had to call client.close() 2023-10-26 12:13:56 -04:00
Ari Brown
c8a29cfcd3 sped up instance creation 2023-10-12 14:31:39 -04:00
Ari Brown
bfb5dda6d9 added support for distributed dataframes from athena queries 2023-10-10 21:21:22 -04:00
Ari Brown
27a1d75bb3 added monkeypatch to resolve known dask bug 2023-10-10 20:21:56 -04:00
Ari Brown
495d786103 added timing functionality for easy use 2023-10-10 19:24:16 -04:00
Ari Brown
153ab074dd trying to get ebs volume right 2023-10-10 15:42:29 -04:00
Ari Brown
d5532ed36e supply device name 2023-10-10 14:56:14 -04:00
Ari Brown
2caafcc344 added public/private instances and ebs block size 2023-10-10 14:37:17 -04:00
Ari Brown
5eb2e39c69 added dask clustering support 2023-10-02 17:26:02 -04:00
Ari Brown
ffdd27e506 parallel_map briefly stopped working 2023-09-26 17:53:16 -04:00
Ari Brown
efe0f40c04 folded wharfer in and updated __init__.py 2023-09-19 16:52:20 -04:00
Ari Brown
ffe0eddd9b output consolidation; added executable via poetry 2023-09-19 16:12:58 -04:00
Ari Brown
ab344374d9 added binary file for an easy athena console 2023-09-15 10:20:53 -04:00
Ari Brown
383185e6cb better exception info 2023-08-23 08:12:01 -04:00
Ari Brown
209f32a7de add a check to support errors being raised before tempfiles are created 2023-08-21 09:34:16 -04:00
Ari Brown
073de9fb95 version bump for parallel fix 2023-08-18 13:44:41 -04:00
Ari Brown
2613866fa3 added comments and dropped useless file 2023-08-18 12:54:45 -04:00
Ari Brown
972e6dd210 add tempfile deletion to query; ls to s3 2023-08-07 18:10:10 -04:00
Roshan Punnoose
757bc74bfd Adding ResultConfiguration for specific output location for Athena 2023-08-03 14:28:12 -04:00