Welcome
Gator is a framework for running a hierarchy of jobs and aggregating logs, metrics, resource utilisation, and artefacts. |
Getting Started
Gator is compatible with Python version 3.8 through 3.11 and is developed using the Poetry packaging tool. To install Gator, check out the repository from GitHub and install it with Poetry:
$> git clone [email protected]:/Intuity/Gator.git
$> cd Gator
$> poetry install
Running Jobs
Before running Gator, you will need to create a job specification as described below. Then invoke the tool as follows:
$> gator my_job_spec.yaml --progress
[17:34:49] [INFO ] Launching task: echo hello
[INFO ] Monitoring task
[INFO ] hello
[INFO ] Task completed with return code 0
[INFO ] Recorded 0 critical, 0 error, 0 warning, 4 info and 2 debug messages
Note
The --progress
switch enables a progress bar which tracks jobs as they
running and the number of passes and failures.
Job Specifications
The complete set of tasks to run are specified with three objects:
- !Job - specifies a single task to run along with the working directory, environment variables, and required resources;
- !JobGroup - groups a collection of different jobs together into a named tier;
- !JobArray - similar to a !JobGroup but it repeats the group of jobs multiple times.
Jobs, groups, and arrays can form dependencies on one another, allowing jobs to be sequenced and only start when a previous job completes (either with success or failure).
A simple example of the syntax can be seen below:
Acknowledgements
Project mascot: Alligator by TRAVIS BIRD from the Noun Project
A lot of the ideas in this project have been taken from the experiments detailed on Rich Porter's blog from 2013 and 2014.