SALT-MASTER job flow:
The Salt master works by always publishing commands to all connected minions and the minions decide if the command is meant for them by checking themselves against the command target.
The typical lifecycle of a salt job from the perspective of the master might be as follows:
- A command is issued on the CLI. For example, ‘salt my_minion test.version’.
- The ‘salt’ command uses LocalClient to generate a request to the salt master by connecting to the ReqServer on TCP:4506 and issuing the job.
- The salt-master ReqServer sees the request and passes it to an available MWorker over workers.ipc.
- A worker picks up the request and handles it. First, it checks to ensure that the requested user has permissions to issue the command. Then, it sends the publish command to all connected minions. For the curious, this happens in ClearFuncs.publish().
- The worker announces on the master event bus that it is about to publish a job to connected minions. This happens by placing the event on the master event bus (master_event_pull.ipc) where the EventPublisher picks it up and distributes it to all connected event listeners on master_event_pub.ipc.
- The message to the minions is encrypted and sent to the Publisher via IPC on publish_pull.ipc.
- Connected minions have a TCP session established with the Publisher on TCP port 4505 where they await commands. When the Publisher receives the job over publish_pull, it sends the jobs across the wire to the minions for processing.
- After the minions receive the request, they decrypt it and perform any requested work, if they determine that they are targeted to do so.
- When the minion is ready to respond, it publishes the result of its job back to the master by sending the encrypted result back to the master on TCP 4506 where it is again picked up by the ReqServer and forwarded to an available MWorker for processing. (Again, this happens by passing this message across workers.ipc to an available worker.)
- When the MWorker receives the job it decrypts it and fires an event onto the master event bus (master_event_pull.ipc). (Again for the curious, this happens in AESFuncs._return().
- The EventPublisher sees this event and re-publishes it on the bus to all connected listeners of the master event bus (on master_event_pub.ipc). This is where the LocalClient has been waiting, listening to the event bus for minion replies. It gathers the job and stores the result.
- When all targeted minions have replied or the timeout has been exceeded, the salt client displays the results of the job to the user on the CLI.
SALT-MINION job flow:
When a salt minion starts up, it attempts to connect to the Publisher and the ReqServer on the salt master. It then attempts to authenticate and once the minion has successfully authenticated, it simply listens for jobs.
Jobs normally come either come from the ‘salt-call’ script run by a local user on the salt minion or they can come directly from a master.
The job flow on a minion, coming from the master via a ‘salt’ command is as follows:
1) A master publishes a job that is received by a minion as outlined by the master’s job flow above.
2) The minion is polling its receive socket that’s connected to the master Publisher (TCP 4505 on master). When it detects an incoming message, it picks it up from the socket and decrypts it.
3) A new minion process or thread is created and provided with the contents of the decrypted message. The _thread_return() method is provided with the contents of the received message.
4) The new minion thread is created. The _thread_return() function starts up and actually calls out to the requested function contained in the job.
5) The requested function runs and returns a result.
6) The result of the function that’s run is encrypted and returned to the master’s ReqServer (TCP 4506 on master).
7) Thread exits. Because the main thread was only blocked for the time that it took to initialize the worker thread, many other requests could have been received and processed during this time.