Workers exposes a GRPC-server to communicate with the scheduler. Main job of workers are to run specific jobs requested by the scheduler and report on those jobs. Jobs can be ruby/python/bash scripts or any executables available on the worker machine.
jobs on workers
// jobsMutex is the lock to access jobs map.
// jobs is the map that holds current/past jobs.
// - key: job id
// - value: pointer to the created job object.
var (
jobsMutex = &sync.Mutex{}
jobs = make(map[string]*job)
)
// job holds information about the ongoing or past jobs,
// that were triggered by the scheduler.
// - id: UUID assigned by the worker and sent back to the scheduler.
// - command: command which the scheduler run the job with
// - path: path to the job file/executable sent by the scheduler.
// - outFilePath: file path to where the output of the job will be piped.
// - cmd: pointer to the cmd.Exec command to get job status etc.
// - done: whether if job is done (default false)
// - err: error while running the job (default nil)
type job struct {
id string
command string
path string
outFilePath string
cmd *exec.Cmd
done bool
err error
}
Configuration for the worker is done through the config.toml file.
I wrote about how I approach configuration in Go projects. You can read more hereHandling configuration in Go
grpc_server:
- addr: Address on which the GRPC server will be run.
- use_tls: Whether the GRPC server should use TLS. If true, crt_file and key_file should be provided.
- crt_file: Path to the certificate file for TLS.
- key_file: Path to the key file for TLS.
scheduler:
- addr: Address on which the GRPC server of the scheduler is run.
worker/config.toml
[grpc_server]
addr = "127.0.0.1:50000"
use_tls = false
crt_file = "server.pem"
key_file = "server.key"
[scheduler]
addr = "127.0.0.1:3000"
GRPC Server
Only 3 GRPC requests required in the scheduler GRPC server: to start/stop/query jobs.
Worker GRPC server definitions
service Worker {
rpc StartJob(StartJobReq) returns (StartJobRes) {}
rpc StopJob(StopJobReq) returns (StopJobRes) {}
rpc QueryJob(QueryJobReq) returns (QueryJobRes) {}
}
message StartJobReq {
string command = 1;
string path = 2;
}
message StartJobRes {
string jobID = 1;
}
message StopJobReq {
string jobID = 1;
}
message StopJobRes {
}
message QueryJobReq {
string jobID = 1;
}
message QueryJobRes {
bool done = 1;
bool error = 2;
string errorText = 3;
}
These GRPC requests are translations of HTTP requests into GRPC requests. Read about the http requests in part 2.
