abc - alphabet declarations
load alphabet
typeset /abc
type /abc
Grid is a typeset for alphabet (see
sh-alphabet (1)) which enables allows direct interconnection of remote
and local processing components. It defines one new type, endpoint,
which represents a place in the network to which two parties can connect and
exchange data.
In the following descriptions, if a type is not endpoint or
a type defined in the root typeset (see alphabet-main (1)), it is
assumed to be of type /string.
Modules currently provided within the /grid typeset
include:
- farm [-lnkavA] endpoint addr tasktype [arg...]
-> endpoint
- Farm connects to a grid labour exchange (see scheduler (4))
at addr, starts a new job of type workflow, and passes all
the data read from endpoint to be processed by the currently
available labour. The data is split into records, each one of which will
be processed on a worker node by tasktype, with its associated
arguments. Other than -A, which specifies unauthenticated
access to the scheduler, the various options are all passed verbatim to
workflow: -l causes it to split its input on
newline-separated records; -n specifies that no record separation
is necessary on output; -k specifies that intermediate data for
failed tasks should be kept around; -a specifies that intermediate
data for all tasks should be kept around, and -v specifies that
workflow should produce a wordy description of what it is
doing.
- local endpoint -> /fd
- Local reads everything from endpoint, and writes it to
fd.
- remote [-a addr] fd ->
endpoint
- Remote is the inverse of local: it reads data from fd
and writes it to a newly created endpoint. If -a is given,
addr specifies the network address of an endpoint server on which
to create the new endpoint.
- rexec [-A] endpoint addr cmd ->
endpoint
- Rexec connects to a remote execution server at addr
(unauthenticated if -A is specified), and arranges to execute the
alphabet expression cmd there. The expression should be
compatible with usage fd -> fd. Data from the argument
endpoint will be piped through this expression, and made available
as the resulting endpoint endpoint.
The examples below that a local endpoint is available, and the
following alphabet declarations:
load alphabet
typeset /grid
type /string /endpoint /fd
import /grid/local /grid/remote
autoconvert fd endpoint remote
Set up a rendering pipeline:
-{/read /tmp/somedata |
remote |
rexec tcp!node1!rexec "{(/fd); /filter $1 "{os render_stage1}} |
rexec tcp!node2!rexec "{(/fd); /filter $1 "{os render_stage2}} |
/create /tmp/somedata.result
}
/appl/alphabet/grid.b, /appl/alphabet/gridtypes.b
/appl/cmd/grid/*.b