ari/minerva
1
0
Fork 0
forked from bellwether/minerva
Code Pull requests Activity
84 commits 1 branch 0 tags 201 KiB
Commit graph

84 commits

This branch
This branch
All branches
Author SHA1 Message Date
Ari Brown
0c00d856c0 dask clusters don't work 2024-01-31 16:21:26 -05:00
Ari Brown
138ee44609 updated athena query example 2024-01-31 16:19:24 -05:00
Ari Brown
5dccce53e9 significant improvement to the readme and verification that all the examples work 2024-01-31 16:18:32 -05:00
Ari Brown
e3c11fb1aa tidying up examples and readme 2024-01-30 18:13:56 -05:00
Ari Brown
819bf7abf3 fixed dataset aggregation; typo; and a bad empty string check 2024-01-26 11:48:35 -05:00
Ari Brown
a907be22cc trying to get the ci/cd to work 2024-01-25 14:46:25 -05:00
Ari Brown
622d104bcb update ci/cd 2024-01-25 14:43:11 -05:00
Ari Brown
14504accc7 got rid of useless print 2024-01-25 13:49:49 -05:00
Ari Brown
afd3a0b114 using autoscale gitlab runner 2024-01-25 13:43:24 -05:00
Ari Brown
ddcacdb569 fixing local paths and redshift credentials 2024-01-25 13:42:24 -05:00
Ari Brown
ae3173b510 added helpers for local files, loading templates, and an example for canceling queries 2024-01-25 11:10:50 -05:00
Ari Brown
5bd2218612 version bump 2024-01-18 12:39:06 -05:00
Ari Brown
9442c33d14 adding parallelization helpers and query cancelation 2024-01-18 12:38:42 -05:00
Ari Brown
5eb8471081 better redshift support, working on dask stuff 2023-12-21 14:23:58 -05:00
Ari Brown
e854a93e60 moving cluster scripts to new dir 2023-12-01 09:05:17 -05:00
Ari Brown
97c27f25a0 zstd w/ compression level 4 is now standard for athena query results 2023-11-30 16:48:45 -05:00
Ari Brown
210e7ebd92 improved run_cluster.py; touching up the dask test; upgrade pyarrow dependency for security patch 2023-11-28 11:12:40 -05:00
Ari Brown
c6280a3826 whoops, converting string to int 2023-11-16 13:26:26 -05:00
Ari Brown
902302c9df enhancing CLI usage of starting a cluster 2023-11-16 10:44:12 -05:00
Ari Brown
fe06b6b808 adding dask examples 2023-11-16 10:33:56 -05:00
Ari Brown
c0ff6af866 specifying port for worker dashboard 2023-11-15 15:09:18 -05:00
Ari Brown
25b0360bed dask worker logs now written to disk hopefully 2023-11-15 14:55:16 -05:00
Ari Brown
10bcd367f6 dask worker logs now written to disk hopefully 2023-11-15 14:50:03 -05:00
Ari Brown
239abf8fc1 dask scheduler apparently doesn't exist sometimes 2023-11-09 14:08:03 -05:00
Ari Brown
65768f6f24 trying to debug why loading large amounts of data fails 2023-11-09 10:46:01 -05:00
Ari Brown
65247844e0 better abstractions 2023-10-26 13:10:49 -04:00
Ari Brown
c9c0ad4422 finally figured out how to stop dask from erroring out. had to call client.close() 2023-10-26 12:13:56 -04:00
Ari Brown
937ca168ad updated dask cluster example 2023-10-12 14:59:54 -04:00
Ari Brown
b6b4b4b416 updated dask cluster example 2023-10-12 14:59:24 -04:00
Ari Brown
c2a702fb9d updated readme for another test 2023-10-12 14:58:18 -04:00
Ari Brown
2a2ff94495 updated readme to test 2023-10-12 14:55:13 -04:00
Ari Brown
c8a29cfcd3 sped up instance creation 2023-10-12 14:31:39 -04:00
Ari Brown
bfb5dda6d9 added support for distributed dataframes from athena queries 2023-10-10 21:21:22 -04:00
Ari Brown
27a1d75bb3 added monkeypatch to resolve known dask bug 2023-10-10 20:21:56 -04:00
Ari Brown
495d786103 added timing functionality for easy use 2023-10-10 19:24:16 -04:00
Ari Brown
153ab074dd trying to get ebs volume right 2023-10-10 15:42:29 -04:00
Ari Brown
9c117d6577 updated gitlab ci to always use dind 2023-10-10 14:59:42 -04:00
Ari Brown
d5532ed36e supply device name 2023-10-10 14:56:14 -04:00
Ari Brown
7c8363f5b7 version 2023-10-10 14:37:29 -04:00
Ari Brown
2caafcc344 added public/private instances and ebs block size 2023-10-10 14:37:17 -04:00
Ari Brown
5eb2e39c69 added dask clustering support 2023-10-02 17:26:02 -04:00
Ari Brown
f4dd130266 added examples 2023-09-26 18:36:41 -04:00
Ari Brown
ffdd27e506 parallel_map briefly stopped working 2023-09-26 17:53:16 -04:00
Ari Brown
02d26bbaaf changing python version reqs 2023-09-19 17:05:09 -04:00
Ari Brown
91b05578bb trying more things for ci/cd 2023-09-19 17:03:22 -04:00
Ari Brown
f2354610c2 added dependency for docker and switch gitlab ci/cd to poetry 2023-09-19 16:58:18 -04:00
Ari Brown
fe3313d83b new version for new features 2023-09-19 16:52:34 -04:00
Ari Brown
efe0f40c04 folded wharfer in and updated __init__.py 2023-09-19 16:52:20 -04:00
Ari Brown
bd3c578c06 updated version 2023-09-19 16:13:14 -04:00
Ari Brown
ffe0eddd9b output consolidation; added executable via poetry 2023-09-19 16:12:58 -04:00