ari/minerva
1
0
Fork 0
forked from bellwether/minerva
Code Pull requests Activity

Commit graph

  • bf8f4145f2 Merge remote-tracking branch 'refs/remotes/origin/main' main Ari Brown 2025-01-22 11:16:17 -05:00
  • 1cb4f496e3 updated example as failing Ari Brown 2025-01-22 11:15:55 -05:00
  • bdcd0d1181 added example Ari Brown 2025-01-22 11:14:57 -05:00
  • cd5ce7e16f some updates, cant remember what for Ari Brown 2025-01-22 10:09:36 -05:00
  • 92c769a2b0 okay guess it's suddenly working Ari Brown 2024-10-26 16:51:34 -04:00
  • 1b63d30539 renamed arg Ari Brown 2024-10-25 15:35:02 -04:00
  • 98355b5881 trying to reorganize how commands are done so we can get logfile grouping and tracking Ari Brown 2024-10-25 15:33:42 -04:00
  • 6eb31cab1e finally got streaming output working again Ari Brown 2024-10-18 14:23:27 -04:00
  • 95f87b2e3c updated todo; removed autoclosing of pipes (in case you supply sys.stdout); removed dumb method that gets overwritten anyways Ari Brown 2024-10-17 12:19:00 -04:00
  • 5e604f07f7 fixed repartition Ari Brown 2024-09-24 10:48:47 -04:00
  • e3055333b3 added test files Ari Brown 2024-08-08 14:39:53 -04:00
  • 9f8b67b98f beginning to add web visualization of pools Ari Brown 2024-07-25 09:30:07 -04:00
  • 58f3d7074f fixed watch, ensured flushing of everything Ari Brown 2024-07-25 08:51:47 -04:00
  • 94820beccc cleaning up, adding todo Ari Brown 2024-07-25 08:08:47 -04:00
  • 205ed4bb18 added live output to command running Ari Brown 2024-07-25 08:07:56 -04:00
  • 4df9b04b2b whew, got partial reads working with named temporary files. lots of weird oddities were discovered in the process Ari Brown 2024-07-24 18:50:03 -04:00
  • d67e398a69 added bare machine support (remote), and making progress with the stdout stderr redirection Ari Brown 2024-07-23 16:34:34 -04:00
  • 2e215c45cf using select to get partial updates of stdout and stderr from commands Ari Brown 2024-07-22 22:09:17 -04:00
  • 745919e587 added more examples Ari Brown 2024-07-03 11:04:11 -04:00
  • a3374fd85c whoops, added self Ari Brown 2024-06-17 10:58:15 -04:00
  • 237c9b8c3c fixed bug in cost Ari Brown 2024-06-17 09:40:19 -04:00
  • fdc0fd2ded added machine pool for easy clustering and trying to get glue support Ari Brown 2024-05-29 16:23:21 -04:00
  • c2bc9e1028 added support for determining instance costs Ari Brown 2024-05-17 16:47:37 -04:00
  • 20ae647447 improved table deletion Ari Brown 2024-03-25 15:39:39 -04:00
  • 2eb404d648 version bump Ari Brown 2024-03-14 10:16:39 -04:00
  • fd289257aa oops. bugfix Ari Brown 2024-03-14 09:13:59 -04:00
  • e294fc062b bug fix for new sql loader Ari Brown 2024-03-01 13:35:06 -05:00
  • af4f8bb999 added sql loading code from metrics calculator to minerva Ari Brown 2024-03-01 13:29:11 -05:00
  • 98524d3be6 fixed a bug that occurred from refactoring Ari Brown 2024-02-23 10:55:53 -05:00
  • 64b2cd6c00 version bump Ari Brown 2024-02-07 12:25:23 -05:00
  • b453c9a515 loosened version requirements; added redshift history function; enabled Minerva to work using instance IAM role instead of explicit profile Ari Brown 2024-02-07 12:25:10 -05:00
  • 1e0b73eeaa updated version Ari Brown 2024-02-01 14:57:07 -05:00
  • 40d2eefd81 added a missing dependency; added support for redshift queries to return 0 rows Ari Brown 2024-02-01 14:56:48 -05:00
  • 6e249086d2 missing dependency Ari Brown 2024-02-01 11:04:08 -05:00
  • e3543fe980 dropped lockfile Ari Brown 2024-02-01 10:45:09 -05:00
  • 0a631955c2 trying to get the gitlab runner to work Ari Brown 2024-01-31 16:24:02 -05:00
  • 0c00d856c0 dask clusters don't work Ari Brown 2024-01-31 16:21:26 -05:00
  • 138ee44609 updated athena query example Ari Brown 2024-01-31 16:19:24 -05:00
  • 5dccce53e9 significant improvement to the readme and verification that all the examples work Ari Brown 2024-01-31 16:18:32 -05:00
  • e3c11fb1aa tidying up examples and readme Ari Brown 2024-01-30 18:13:56 -05:00
  • 819bf7abf3 fixed dataset aggregation; typo; and a bad empty string check Ari Brown 2024-01-26 11:48:35 -05:00
  • a907be22cc trying to get the ci/cd to work Ari Brown 2024-01-25 14:46:25 -05:00
  • 622d104bcb update ci/cd Ari Brown 2024-01-25 14:43:11 -05:00
  • 14504accc7 got rid of useless print Ari Brown 2024-01-25 13:49:49 -05:00
  • afd3a0b114 using autoscale gitlab runner Ari Brown 2024-01-25 13:43:24 -05:00
  • ddcacdb569 fixing local paths and redshift credentials Ari Brown 2024-01-25 13:42:24 -05:00
  • ae3173b510 added helpers for local files, loading templates, and an example for canceling queries Ari Brown 2024-01-25 11:10:50 -05:00
  • 5bd2218612 version bump Ari Brown 2024-01-18 12:39:06 -05:00
  • 9442c33d14 adding parallelization helpers and query cancelation Ari Brown 2024-01-18 12:38:42 -05:00
  • 5eb8471081 better redshift support, working on dask stuff Ari Brown 2023-12-21 14:23:58 -05:00
  • e854a93e60 moving cluster scripts to new dir Ari Brown 2023-12-01 09:05:17 -05:00
  • 97c27f25a0 zstd w/ compression level 4 is now standard for athena query results Ari Brown 2023-11-30 16:48:45 -05:00
  • 210e7ebd92 improved run_cluster.py; touching up the dask test; upgrade pyarrow dependency for security patch Ari Brown 2023-11-28 11:12:40 -05:00
  • c6280a3826 whoops, converting string to int Ari Brown 2023-11-16 13:26:26 -05:00
  • 902302c9df enhancing CLI usage of starting a cluster Ari Brown 2023-11-16 10:44:12 -05:00
  • fe06b6b808 adding dask examples Ari Brown 2023-11-16 10:33:56 -05:00
  • c0ff6af866 specifying port for worker dashboard Ari Brown 2023-11-15 15:09:18 -05:00
  • 25b0360bed dask worker logs now written to disk hopefully Ari Brown 2023-11-15 14:55:16 -05:00
  • 10bcd367f6 dask worker logs now written to disk hopefully Ari Brown 2023-11-15 14:50:03 -05:00
  • 239abf8fc1 dask scheduler apparently doesn't exist sometimes Ari Brown 2023-11-09 14:08:03 -05:00
  • 65768f6f24 trying to debug why loading large amounts of data fails Ari Brown 2023-11-09 10:46:01 -05:00
  • 65247844e0 better abstractions Ari Brown 2023-10-26 13:10:49 -04:00
  • c9c0ad4422 finally figured out how to stop dask from erroring out. had to call client.close() Ari Brown 2023-10-26 12:13:56 -04:00
  • 937ca168ad updated dask cluster example Ari Brown 2023-10-12 14:59:54 -04:00
  • b6b4b4b416 updated dask cluster example Ari Brown 2023-10-12 14:59:24 -04:00
  • c2a702fb9d updated readme for another test Ari Brown 2023-10-12 14:58:18 -04:00
  • 2a2ff94495 updated readme to test Ari Brown 2023-10-12 14:55:13 -04:00
  • c8a29cfcd3 sped up instance creation Ari Brown 2023-10-12 14:31:39 -04:00
  • bfb5dda6d9 added support for distributed dataframes from athena queries Ari Brown 2023-10-10 21:21:22 -04:00
  • 27a1d75bb3 added monkeypatch to resolve known dask bug Ari Brown 2023-10-10 20:21:56 -04:00
  • 495d786103 added timing functionality for easy use Ari Brown 2023-10-10 19:24:16 -04:00
  • 153ab074dd trying to get ebs volume right Ari Brown 2023-10-10 15:42:29 -04:00
  • 9c117d6577 updated gitlab ci to always use dind Ari Brown 2023-10-10 14:59:42 -04:00
  • d5532ed36e supply device name Ari Brown 2023-10-10 14:56:14 -04:00
  • 7c8363f5b7 version Ari Brown 2023-10-10 14:37:29 -04:00
  • 2caafcc344 added public/private instances and ebs block size Ari Brown 2023-10-10 14:37:17 -04:00
  • 5eb2e39c69 added dask clustering support Ari Brown 2023-10-02 17:26:02 -04:00
  • f4dd130266 added examples Ari Brown 2023-09-26 18:36:41 -04:00
  • ffdd27e506 parallel_map briefly stopped working Ari Brown 2023-09-26 17:53:16 -04:00
  • 02d26bbaaf changing python version reqs Ari Brown 2023-09-19 17:05:09 -04:00
  • 91b05578bb trying more things for ci/cd Ari Brown 2023-09-19 17:03:22 -04:00
  • f2354610c2 added dependency for docker and switch gitlab ci/cd to poetry Ari Brown 2023-09-19 16:58:18 -04:00
  • fe3313d83b new version for new features Ari Brown 2023-09-19 16:52:34 -04:00
  • efe0f40c04 folded wharfer in and updated __init__.py Ari Brown 2023-09-19 16:52:20 -04:00
  • bd3c578c06 updated version Ari Brown 2023-09-19 16:13:14 -04:00
  • ffe0eddd9b output consolidation; added executable via poetry Ari Brown 2023-09-19 16:12:58 -04:00
  • ab344374d9 added binary file for an easy athena console Ari Brown 2023-09-15 10:20:53 -04:00
  • 383185e6cb better exception info Ari Brown 2023-08-23 08:12:01 -04:00
  • 209f32a7de add a check to support errors being raised before tempfiles are created Ari Brown 2023-08-21 09:34:16 -04:00
  • 073de9fb95 version bump for parallel fix Ari Brown 2023-08-18 13:44:41 -04:00
  • 2613866fa3 added comments and dropped useless file Ari Brown 2023-08-18 12:54:45 -04:00
  • 972e6dd210 add tempfile deletion to query; ls to s3 Ari Brown 2023-08-07 18:10:10 -04:00
  • 1d509f544c Merge branch 'add_result_config' into 'main' Ari Brown 2023-08-03 18:30:19 +00:00
  • 757bc74bfd Adding ResultConfiguration for specific output location for Athena Roshan Punnoose 2023-08-03 14:28:12 -04:00
  • 72fd7fe005 bug fixes Ari Brown 2023-08-03 14:12:09 -04:00
  • dd2edecc64 version bump Ari Brown 2023-08-03 13:46:24 -04:00
  • f6c1a15b4e missing file Ari Brown 2023-08-03 13:46:02 -04:00
  • 6fd32f9bd3 updating version Ari Brown 2023-08-03 11:43:11 -04:00
  • 0e5542cecf better support for params within athena Ari Brown 2023-08-03 11:42:56 -04:00
  • ea31c8e8c0 dropped inaccurate comment Ari Brown 2023-08-02 16:51:22 -04:00