Pipes and Filters Pattern

The Pipes and Filters architectural pattern provides a structure for systems that process a stream of data. Each processing step is encapsulated in a filter component. Data is passed through pipes between adjacent filters. Recombining filters allows you to build families of related systems. Whether a separation into processing steps is feasible strongly depends on the application domain and the problem to be solved. For example, an interactive, event-driven system does not split into sequential stages.

Example: A parser reading from a file.

The Pipes and Filters architectural pattern divides the task of a system into several sequential processing steps. These steps are connected by the data flow through the system-the output data of a step is the input to the subsequent step. Each processing step is implemented by a filter component. A filter consumes and delivers data incrementally-in contrast to consuming all its input before producing any output. The input to the system is provided by a data source such as a text file. The output flows into a data sink such as a file. The data source, the filters and the data sink are connected sequentially by pipes. Each pipe implements the data flow between adjacent processing steps. The sequence of filters combined by pipes is called a processing pipeline.

A pipeline can output data from the filter or push new input data to the filter (passive filters) or a filter can pull its input from and push its output down the pipeline (active filter).

Pipes denote the connections between filters, between the data source and the first filter, and between the last filter and the data sink. If two active components are joined, the pipe synchronizes them (can buffer too). This synchronization is done with a first-in- first-out buffer. If activity is controlled by one of the adjacent filters, the pipe can be implemented by a direct call from the active to the passive component. Direct calls make filter recombination harder.

Error handling:
Because pipeline components do not share any global state (usually), error handling is hard to address and is often neglected. If a filter detects errors in its input data, it can ignore input until some clearly marked separation occurs. For example, a filter may skip to the next line of input if a line is expected to contain a numerical value and does not. It is usually hard to give a general strategy for error handling with a system based on the Pipes and Filters pattern.

Data transformation overhead:
Using a single data type for all filter input and output to achieve highest flexibility results in data conversion overheads.

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License