This is still a work in progress - no stable version has been released yet.
Powerhose turns your CPU-bound tasks into I/O-bound tasks so your Python applications are easier to scale.
Powerhose is an implementation of the Request-Reply Broker pattern in ZMQ.
The three main parts are:
When client sends a job, the broker simply re-routes it to one of its workers, then gets back the result and send it back to the client.
Workers also subscribe to a “heartbeat” socket that receives regular heart beats from the broker. When the beat is lost, workers simply die.
Powerhose uses Circus to manage the life of the broker and the workers, and provide high level APIs for your program.
Workers can be written in Python but also in any language that has a ZeroMQ library.
If you have CPU-Bound tasks that could be performed in a specialized C++ program for example, Powerhose is a library you could use to ease your life.
Before using Powerhose, ask yourself if you really need it.
The overhead of the data exchange can be quite important, so unless your job is a CPU-bound task that takes more than 20 ms to perform, there are high chances using Powerhose will not speed it up.
There’s a bench.py module in the examples you can change to make some tests with your code. It will compare the speed with and without Powerhose in a multi-threaded environment so you can see if you get any benefit.
If you are curious about why we wrote this library see Why Powerhose ?.
Here’s a full example of usage – we want to delegate some work to a specialized worker.
Let’s create a function that just echoes back what was sent to it, and save it in an example module:
def echo(job): return job.data
This function takes a Job instance that contains the data sent by Powerhose. It returns its content immediately.
Let’s run our Powerhose cluster with the powerhose command, by simply pointing the echo() callable:
$ powerhose example.echo [circus] Starting master on pid 51177 [circus] broker started [circus] workers started [circus] Arbiter now waiting for commands
That’s it ! By default one broker and 5 workers are launched, but you can run as many workers as you need, and even add more while the system is running.
Now that the system is up and running, let’s try it out in a Python shell:
>>> from powerhose.client import Client >>> client = Client() >>> client.execute('test') 'test'
Congrats ! You have a Powerhose system up and running !
To learn about all existing commands and their options, see Command-line tools.
Of course, the goal is to keep the broker and its workers up and running on a system. You can use Daemontools, Supervisord or Circus.
Circus is our preferred system. A Circus config can look like this:
[circus] check_delay = 5 endpoint = tcp://127.0.0.1:5555 [watcher:master] cmd = powerhose-broker args = --frontend ipc:///tmp/front --backend ipc:///tmp/backend --heartbeat ipc:///tmp/heartbeat warmup_delay = 0 numprocesses = 1 [watcher:workers] cmd = powerhose-worker args = --backend ipc:///tmp/backend --heartbeat ipc:///tmp/heartbeat echo_worker.echo warmup_delay = 0 numprocesses = 5
This file can then be launched via circusd. See the Circus documentation for details on this.
The simplest way to run a Powerhose system is to use the command line as previously shown, but you can also do everything programmatically via the get_cluster() function.
Here’s an example:
from powerhose import get_cluster from powerhose.client import Client cluster = get_cluster('example.echo', background=True) cluster.start() client = Client() for i in range(10): print client.execute(str(i)) cluster.stop()
Here, the cluster is launched programmatically in the background and the client uses it as before.
To learn more about Powerhose APIs, see Library.