Spring Batch - 参考文档（Reference Documentation）

在开源项目及其相关社区把大部分注意力集中在基于 web 和 微服务体系框架时框架中时，基于 Java 的批处理框架却无人问津， 尽管在企业 IT 环境中一直都有这种批处理的需求。但因为缺乏一个标准的、可重用的批处理框架导致在企业客户的 IT 系统中 存在着很多一次编写，一次使用的版本，以及很多不同的内部解决方案。

SpringSource (现被命名为 Pivotal) 和 Accenture（埃森哲）致力于通过合作来改善这种状况。 埃森哲在实现批处理架构上有着丰富的产业实践经验，SpringSource 有深入的技术开发积累， 背靠 Spring 框架提供的编程模型,意味着两者能够结合成为默契且强大的合作伙伴，创造出高质量的、市场认可的企业级 Java 解决方案， 填补这一重要的行业空白。两家公司都与许多通过开发基于Spring的批处理架构解决方案解决类似问题的客户合作。 这提供了一些有用的额外细节和实际约束，有助于确保解决方案可以应用于客户提出的现实问题。

埃森哲为Spring Batch项目贡献了以前专有的批处理体系结构框架，以及提供支持，增强功能和现有功能集的提交者资源。 埃森哲的贡献基于几十年来在使用最新几代平台构建批量架构方面的经验：COBOL / Mainframe，C / Unix以及现在的Java / Anywhere。

埃森哲与SpringSource之间的合作旨在促进软件处理方法，框架和工具的标准化， 在创建批处理应用程序时，企业用户可以始终如一地利用这些方法，框架和工具。希望为其企业IT环境提供标准的， 经过验证的解决方案的公司和政府机构可以从 Spring Batch 中受益。

一般的典型批处理程序：

Spring Batch自动执行此基本批处理迭代，提供处理类似事务的功能，通常在脱机环境中处理，无需任何用户交互。 批处理作业是大多数 IT 项目的一部分，Spring Batch 是唯一提供强大的企业级解决方案的开源框架。

业务场景

技术目标

Spring Batch 设计的时候充分考虑了可扩展性和各类最终用户。 下图显示了 Spring Batch 的架构层次示意图,这种架构层次为最终用户开发者提供了很好的扩展性与易用性。

这个层级体系结构高亮显示了 Spring Batch 的 3 个主要组件：应用（Application），核心（Core）和 基础架构（Infrastructure）。 应用层包含了所有的批量作业和开发者使用 Spring Batch 写的所有自定义代码。批量核心层包含了所有运行和控制批量作业所需必要的运行时类。 同时还包括了有 JobLauncher, Job, 和 Step。应用层和核心层都构建在基础架构层之上。

基础架构层包含了有 读（readers）和 写（writers ）以及服务（services）。例如有针对服务使用， RetryTemplate。 基础架构层的这些东西，这些能够被应用层开发（readers 和 writers， 例如 ItemReader 和 ItemWriter）和批量核心框架（例如，retry，这个是核心层自己的库）所使用。

简单的来说，基础架构层为应用层和批量核心层提供了所需要的的基础内容，是整个 Spring Batch 的基础。我们针对 Spring Batch 的开发绝大部分情况是在应用层完成的。

下面是一些关键的指导原则，可以在构批量处理解决方案可以参考。

为了帮助设计和实现批量处理系统，基本的批量应用是通过块和模式来构建的， 同时也应该能够为程序开发人员和设计人员提供结构的样例和基础的批量处理程序。

当你开始设计一个批量作业任务的时候，商业逻辑应该被拆分一系列的步骤，而这些步骤又是可以通过下面的标准构件块来实现的：

更多的，一个基本的应用外壳应该也能够被针对商业逻辑来提供，这个外壳通常不能通过上面介绍的这些标准模块来完成。

另外的一个主要的构建块，每一个引用通常可以使用下面的一个或者多个标准工具步骤，例如：

批量处理应用程序可以通过下面的输入数据类型来进行分类：

所有批量处理系统的处理基础都是策略（strategy）。对处理策略进行选择产生影响的因素包括有：预估批量处理需要处理的数据量， 在线并发量，和另外一个批量处理系统的在线并发量， 可用的批量处理时间窗口（很多企业都希望系统是能够不间断运行的，基本上来说批量处理可能没有处理时间窗口）。

针对批量处理的标准处理选项包括有（按实现复杂度的递增顺序）：

上面的一些选项或者所有选项能够被商业的任务调度所支持。

在下面的部分，我们将会针对上面的处理选项来对细节进行更多的说明。需要特别注意的是， 批量处理程序使用提交和锁定策略将会根据批量处理的不同而有所不同。作为最佳实践，在线锁策略应该使用相同的原则。 因此，在设计批处理整体架构时不能简单地拍脑袋决定，需要进行详细的分析和论证。

锁定策略可以仅仅使用常见的数据库锁或者你也可以在系统架构中使用其他的自定义锁定服务。 这个锁服务将会跟踪数据库的锁（例如在一个专用的数据库表（db-table）中存储必要的信息），然后在应用程序请求数据库操作时授予权限或拒绝。 重试逻辑应该也需要在系统架构中实现，以避免批量作业中的因资源锁定而导致批量任务被终止。

1. 批量处理作业窗口中的常规处理 针对运行在一个单独批处理窗口中的简单批量处理，更新的数据对在线用户或其他批处理来说并没有实时性要求， 也没有并发问题，在批处理运行完成后执行单次提交即可。

大多数情况下，一种更健壮的方法会更合适。要记住的是，批处理系统会随着时间的流逝而增长，包括复杂度和需要处理的数据量。 如果没有合适的锁定策略，系统仍然依赖于一个单一的提交点，则修改批处理程序会是一件痛苦的事情。 因此，即使是最简单的批处理系统， 也应该为重启-恢复（restart-recovery）选项考虑提交逻辑。针对下面的情况，批量处理就更加复杂了。

2. 并发批量 / 在线处理 批处理程序处理的数据如果会同时被在线用户实时更新，就不应该锁定在线用户需要的所有任何数据（不管是数据库还是文件）， 即使只需要锁定几秒钟的时间。同时还应该每处理一批事务就提交一次数据库。这减少了其他程序不可用的数据数据量，也压缩了数据不可用的时间。

另一个可以使用的方案就是使用逻辑行基本的锁定实现来替代物理锁定。 通过使用乐观锁（Optimistic Locking ）或悲观锁（Pessimistic Locking）模式。

这些模式并不一定适用于批处理，但他们可以被用在并发批处理和在线处理的情况下（例如,数据库不支持行级锁）。作为一般规则，乐观锁更适合于在线应用， 而悲观锁更适合于批处理应用。只要使用了逻辑锁，那么所有访问逻辑锁保护的数据的程序都必须采用同样的方案。

请注意：这两种解决方案都只锁定（address locking）单条记录。但很多情况下我们需要锁定一组相关的记录。如果使用物理锁，你必须非常小心地管理这些以避免潜在的死锁。 如果使用逻辑锁，通常最好的解决办法是创建一个逻辑锁管理器，使管理器能理解你想要保护的逻辑记录分组（groups），并确保连贯和没有死锁（non-deadlocking）。 这种逻辑锁管理器通常使用其私有的表来进行锁管理、争用报告、超时机制 等等。

3. 并行处理 并行处理允许多个批量处理运行（run）/任务（job）同时并行地运行。以使批量处理总运行时间降到最低。 如果多个任务不使用相同的文件、数据表、索引空间时，批量处理这些不算什么问题。如果确实存在共享和竞争，那么这个服务就应该使用分区数据来实现。 另一种选择是使用控制表来构建一个架构模块以维护他们之间的相互依赖关系。控制表应该为每个共享资源分配一行记录，不管这些资源是否被某个程序所使用。 执行并行作业的批处理架构或程序随后将查询这个控制表，以确定是否可以访问所需的资源。

如果解决了数据访问的问题，并行处理就可以通过使用额外的线程来并行实现。在传统的大型主机环境中，并行作业类上通常被用来确保所有进程都有充足的 CPU 时间。 无论如何，解决方案必须足够强劲，以确保所有正在运行的进程都有足够的运行处理时间。

并行处理的其他关键问题还包括负载平衡以及一般系统资源的可用性（如文件、数据库缓冲池等）。请注意，控制表本身也可能很容易变成一个至关重要的资源（有可能发生严重竞争）。

4. 分区 分区技术允许多版本的大型批处理程序并发地（concurrently）运行。这样做的目的是减少超长批处理作业过程所需的时间。 可以成功分区的过程主要是那些可以拆分的输入文件 和/或 主要的数据库表被分区以允许程序使用不同的数据来运行。

此外,被分区的过程必须设计为只处理分配给他的数据集。分区架构与数据库设计和数据库分区策略是密切相关的。 请注意，数据库分区并不一定指数据库需要在物理上实现分区，尽管在大多数情况下这是明智的。 下面的图片展示了分区的方法：

系统架构应该足够灵活，以允许动态配置分区的数量。自动控制和用户配置都应该纳入考虑范围。 自动配置可以根据参数来决定，例如输入文件大小 和/或 输入记录的数量。

4.1 分区方案 下面列出了一些可能的分区方案，至于具体选择哪种分区方案，要根据具体情况来确定：

1. 固定和均衡拆分记录集

这涉及到将输入的记录集合分解成均衡的部分（例如，拆分为 10 份，这样每部分是整个数据集的十分之一）。 每个拆分的部分稍后由一个批处理/提取程序实例来处理。

为了使用这种方案，需要在预处理时候就将记录集进行拆分。拆分的结果有一个最大值和最小值的位置，这两个值可以用作限制每个 批处理/提取程序处理部分的输入。

预处理可能有一个很大的开销，因为它必须计算并确定的每部分数据集的边界。

2. 通过关键字段（Key Column）拆分

这涉及到将输入记录按照某个关键字段来拆分，比如一个地区代码（location code），并将每个键分配给一个批处理实例。为了达到这个目标，也可以使用列值：

在选项 1 下，添加新值意味着手动重新配置批处理/提取以确保将新值添加到特定实例。

在选项 2 下，这可确保通过批处理作业的实例覆盖所有值。但是，一个实例处理的值的数量取决于列值的分布（0000-0999范围内可能有大量位置，1000-1999范围内可能很少）。 在此选项下，数据范围应设计为考虑分区。

在这两个选项下，无法实现记录到批处理实例的最佳均匀分布。没有使用批处理实例数的动态配置。

3. 通过视图（Views）

这种方法基本上是根据键列来分解，但不同的是在数据库级进行分解。它涉及到将记录集分解成视图。这些视图将被批处理程序的各个实例在处理时使用。分解将通过数据分组来完成。

使用这个方法时，批处理的每个实例都必须为其配置一个特定的视图（而非主表）。当然,对于新添加的数据，这个新的数据分组必须被包含在某个视图中。 也没有自动配置功能，实例数量的变化将导致视图需要进行相应的改变。

4. 附加的处理识别器

这涉及到输入表一个附加的新列，它充当一个指示器。在预处理阶段，所有指示器都被标志为未处理。在批处理程序获取记录阶段，只会读取被标记为未处理的记录， 一旦他们被读取（并加锁），它们就被标记为正在处理状态。当记录处理完成，指示器将被更新为完成或错误。 批处理程序的多个实例不需要改变就可以开始，因为附加列确保每条纪录只被处理一次。 在“完成时，指标被标记为完成”的顺序中的一两句话。

使用该选项时，表上的I/O会动态地增长。在批量更新的程序中，这种影响被降低了，因为写操作是必定要进行的。

5. 提取表到无格式文件

这包括将表中的数据提取到一个文件中。然后可以将这个文件拆分成多个部分，作为批处理实例的输入。

使用这个选项时，将数据提取到文件中，并将文件拆分的额外开销，有可能抵消多分区处理（multi-partitioning）的效果。可以通过改变文件分割脚本来实现动态配置。

6. 使用哈希列（Hashing Column）

这个计划需要在数据库表中增加一个哈希列（key/index）来检索驱动（driver）记录。这个哈希列将有一个指示器来确定将由批处理程序的哪个实例处理某个特定的行。 例如，如果启动了三个批处理实例，那么 “A” 指示器将标记某行由实例 1 来处理，“B”将标记着将由实例 2 来处理，“C”将标记着将由实例 3 来处理，以此类推。

稍后用于检索记录的过程（procedure）程序，将有一个额外的 WHERE 子句来选择以一个特定指标标记的所有行。 这个表的插入（insert）需要附加的标记字段，默认值将是其中的某一个实例（例如“A”）。

一个简单的批处理程序将被用来更新不同实例之间的重新分配负载的指标。当添加足够多的新行时，这个批处理会被运行（在任何时间，除了在批处理窗口中）。

批处理应用程序的其他实例只需要像上面这样的批处理程序运行着以重新分配指标，以决定新实例的数量。

4.2 数据库和应用设计原则

如果一个支持多分区（multi-partitioned）的应用程序架构，基于数据库采用关键列（key column）分区方法拆成的多个表，则应该包含一个中心分区仓库来存储分区参数。 这种方式提供了灵活性，并保证了可维护性。这个中心仓库通常只由单个表组成，叫做分区表。

存储在分区表中的信息应该是是静态的，并且只能由 DBA 维护。每个多分区程序对应的单个分区有一行记录，组成这个表。 这个表应该包含这些列：程序 ID 编号，分区编号（分区的逻辑ID），一个分区对应的关键列（key column）的最小值，分区对应的关键列的最大值。

在程序启动时，应用程序架构（Control Processing Tasklet， 控制处理微线程）应该将程序 id 和分区号传递给该程序。 这些变量被用于读取分区表，来确定应用程序应该处理的数据范围（如果使用关键列的话）。另外分区号必须在整个处理过程中用来：

4.3 死锁最小化

当程序并行或分区运行时,会导致数据库资源的争用,还可能会发生死锁（Deadlocks）。其中的关键是数据库设计团队在进行数据库设计时必须考虑尽可能消除潜在的竞争情况。

还要确保设计数据库表的索引时考虑到性能以及死锁预防。

死锁或热点往往发生在管理或架构表上,如日志表、控制表、锁表（lock tables）。这些影响也应该纳入考虑。为了确定架构可能的瓶颈，一个真实的压力测试是至关重要的。

要最小化数据冲突的影响，架构应该提供一些服务，如附加到数据库或遇到死锁时的 等待-重试（wait-and-retry）间隔时间。 这意味着要有一个内置的机制来处理数据库返回码，而不是立即引发错误处理，需要等待一个预定的时间并重试执行数据库操作。

4.4 参数处理和校验

对程序开发人员来说，分区架构应该相对透明。框架以分区模式运行时应该执行的相关任务包括：

验证（validation）要包含必要的检查，以确保：

如果数据库是分区的，可能需要一些额外的验证来保证单个分区不会跨越数据库的片区。

体系架构应该考虑整合分区(partitions).包括以下关键问题:

本发行版本介绍了可以让你通过使用 Micrometer 来监控你的批量作业。在默认的情况下，Spring Batch 将会收集相关批量指标 （包括，作业时间，步骤的时间，读取和写入的项目，以及其他的相关信息），和将这些指标通过 spring.batch 前缀（prefix）注册到 Micrometer 的全局指标中。

这些指标可以发布到任何能够支持 Micrometer 的 监控系统（monitoring system）中。

有关这个新特性的更多细节，请参考 Monitoring and metrics 章节中的内容。

本发行版本添加了一个新的 KafkaItemReader 和 KafkaItemWriter， 用来从 Kafka 的 topics 中读取和写入。 有关更多这个新组建的信息，请参考： Javadoc。

本发行版本添加了一个新的 AvroItemReader 和 AvroItemWriter， 用来从 Avro 资源中读取和写入。 有关更多这个新组建的信息，请参考： Javadoc.

对参考的文档进行更新，以便于与其他 Spring 项目的文档风格保持一致。

This section describes stereotypes relating to the concept of a batch job. A Job is an entity that encapsulates an entire batch process. As is common with other Spring projects, a Job is wired together with either an XML configuration file or Java-based configuration. This configuration may be referred to as the "job configuration". However, Job is just the top of an overall hierarchy, as shown in the following diagram:

In Spring Batch, a Job is simply a container for Step instances. It combines multiple steps that belong logically together in a flow and allows for configuration of properties global to all steps, such as restartability. The job configuration contains:

A Step is a domain object that encapsulates an independent, sequential phase of a batch job. Therefore, every Job is composed entirely of one or more steps. A Step contains all of the information necessary to define and control the actual batch processing. This is a necessarily vague description because the contents of any given Step are at the discretion of the developer writing a Job. A Step can be as simple or complex as the developer desires. A simple Step might load data from a file into the database, requiring little or no code (depending upon the implementations used). A more complex Step may have complicated business rules that are applied as part of the processing. As with a Job, a Step has an individual StepExecution that correlates with a unique JobExecution, as shown in the following image:

An ExecutionContext represents a collection of key/value pairs that are persisted and controlled by the framework in order to allow developers a place to store persistent state that is scoped to a StepExecution object or a JobExecution object. For those familiar with Quartz, it is very similar to JobDataMap. The best usage example is to facilitate restart. Using flat file input as an example, while processing individual lines, the framework periodically persists the ExecutionContext at commit points. Doing so allows the ItemReader to store its state in case a fatal error occurs during the run or even if the power goes out. All that is needed is to put the current number of lines read into the context, as shown in the following example, and the framework will do the rest:

Using the EndOfDay example from the Job Stereotypes section as an example, assume there is one step, 'loadData', that loads a file into the database. After the first failed run, the metadata tables would look like the following example:

In the preceding case, the Step ran for 30 minutes and processed 40,321 'pieces', which would represent lines in a file in this scenario. This value is updated just before each commit by the framework and can contain multiple rows corresponding to entries within the ExecutionContext. Being notified before a commit requires one of the various StepListener implementations (or an ItemStream), which are discussed in more detail later in this guide. As with the previous example, it is assumed that the Job is restarted the next day. When it is restarted, the values from the ExecutionContext of the last run are reconstituted from the database. When the ItemReader is opened, it can check to see if it has any stored state in the context and initialize itself from there, as shown in the following example:

In this case, after the above code runs, the current line is 40,322, allowing the Step to start again from where it left off. The ExecutionContext can also be used for statistics that need to be persisted about the run itself. For example, if a flat file contains orders for processing that exist across multiple lines, it may be necessary to store how many orders have been processed (which is much different from the number of lines read), so that an email can be sent at the end of the Step with the total number of orders processed in the body. The framework handles storing this for the developer, in order to correctly scope it with an individual JobInstance. It can be very difficult to know whether an existing ExecutionContext should be used or not. For example, using the 'EndOfDay' example from above, when the 01-01 run starts again for the second time, the framework recognizes that it is the same JobInstance and on an individual Step basis, pulls the ExecutionContext out of the database, and hands it (as part of the StepExecution) to the Step itself. Conversely, for the 01-02 run, the framework recognizes that it is a different instance, so an empty context must be handed to the Step. There are many of these types of determinations that the framework makes for the developer, to ensure the state is given to them at the correct time. It is also important to note that exactly one ExecutionContext exists per StepExecution at any given time. Clients of the ExecutionContext should be careful, because this creates a shared keyspace. As a result, care should be taken when putting values in to ensure no data is overwritten. However, the Step stores absolutely no data in the context, so there is no way to adversely affect the framework.

It is also important to note that there is at least one ExecutionContext per JobExecution and one for every StepExecution. For example, consider the following code snippet:

As noted in the comment, ecStep does not equal ecJob. They are two different ExecutionContexts. The one scoped to the Step is saved at every commit point in the Step, whereas the one scoped to the Job is saved in between every Step execution.

JobRepository is the persistence mechanism for all of the Stereotypes mentioned above. It provides CRUD operations for JobLauncher, Job, and Step implementations. When a Job is first launched, a JobExecution is obtained from the repository, and, during the course of execution, StepExecution and JobExecution implementations are persisted by passing them to the repository.

JobLauncher represents a simple interface for launching a Job with a given set of JobParameters, as shown in the following example:

It is expected that implementations obtain a valid JobExecution from the JobRepository and execute the Job.

ItemReader is an abstraction that represents the retrieval of input for a Step, one item at a time. When the ItemReader has exhausted the items it can provide, it indicates this by returning null. More details about the ItemReader interface and its various implementations can be found in Readers And Writers.

ItemWriter is an abstraction that represents the output of a Step, one batch or chunk of items at a time. Generally, an ItemWriter has no knowledge of the input it should receive next and knows only the item that was passed in its current invocation. More details about the ItemWriter interface and its various implementations can be found in Readers And Writers.

ItemProcessor is an abstraction that represents the business processing of an item. While the ItemReader reads one item, and the ItemWriter writes them, the ItemProcessor provides an access point to transform or apply other business processing. If, while processing the item, it is determined that the item is not valid, returning null indicates that the item should not be written out. More details about the ItemProcessor interface can be found in Readers And Writers.

Spring 3 brought the ability to configure applications via java in addition to XML. As of Spring Batch 2.2.0, batch jobs can be configured using the same java config. There are two components for the java based configuration: the @EnableBatchProcessing annotation and two builders.

The @EnableBatchProcessing works similarly to the other @Enable* annotations in the Spring family. In this case, @EnableBatchProcessing provides a base configuration for building batch jobs. Within this base configuration, an instance of StepScope is created in addition to a number of beans made available to be autowired:

The core interface for this configuration is the BatchConfigurer. The default implementation provides the beans mentioned above and requires a DataSource as a bean within the context to be provided. This data source will be used by the JobRepository. You can customize any of these beans by creating a custom implementation of the BatchConfigurer interface. Typically, extending the DefaultBatchConfigurer (which is provided if a BatchConfigurer is not found) and overriding the required getter is sufficient. However, implementing your own from scratch may be required. The following example shows how to provide a custom transaction manager:

With the base configuration in place, a user can use the provided builder factories to configure a job. Below is an example of a two step job configured via the JobBuilderFactory and the StepBuilderFactory.

As described in earlier, the JobRepository is used for basic CRUD operations of the various persisted domain objects within Spring Batch, such as JobExecution and StepExecution. It is required by many of the major framework features, such as the JobLauncher, Job, and Step.

The most basic implementation of the JobLauncher interface is the SimpleJobLauncher. Its only required dependency is a JobRepository, in order to obtain an execution:

Once a JobExecution is obtained, it is passed to the execute method of Job, ultimately returning the JobExecution to the caller:

The sequence is straightforward and works well when launched from a scheduler. However, issues arise when trying to launch from an HTTP request. In this scenario, the launching needs to be done asynchronously so that the SimpleJobLauncher returns immediately to its caller. This is because it is not good practice to keep an HTTP request open for the amount of time needed by long running processes such as batch. An example sequence is below:

The SimpleJobLauncher can easily be configured to allow for this scenario by configuring a TaskExecutor:

Any implementation of the spring TaskExecutor interface can be used to control how jobs are asynchronously executed.

At a minimum, launching a batch job requires two things: the Job to be launched and a JobLauncher. Both can be contained within the same context or different contexts. For example, if launching a job from the command line, a new JVM will be instantiated for each Job, and thus every job will have its own JobLauncher. However, if running from within a web container within the scope of an HttpRequest, there will usually be one JobLauncher, configured for asynchronous job launching, that multiple requests will invoke to launch their jobs.

So far, both the JobLauncher and JobRepository interfaces have been discussed. Together, they represent simple launching of a job, and basic CRUD operations of batch domain objects:

A JobLauncher uses the JobRepository to create new JobExecution objects and run them. Job and Step implementations later use the same JobRepository for basic updates of the same executions during the running of a Job. The basic operations suffice for simple scenarios, but in a large batch environment with hundreds of batch jobs and complex scheduling requirements, more advanced access of the meta data is required:

The JobExplorer and JobOperator interfaces, which will be discussed below, add additional functionality for querying and controlling the meta data.

Spring Batch uses a 'Chunk-oriented' processing style within its most common implementation. Chunk oriented processing refers to reading the data one at a time and creating 'chunks' that are written out within a transaction boundary. One item is read in from an ItemReader, handed to an ItemProcessor, and aggregated. Once the number of items read equals the commit interval, the entire chunk is written out by the ItemWriter, and then the transaction is committed. The following image shows the process:

The following code shows the same concepts shown:

Chunk-oriented processing is not the only way to process in a Step. What if a Step must consist of a simple stored procedure call? You could implement the call as an ItemReader and return null after the procedure finishes. However, doing so is a bit unnatural, since there would need to be a no-op ItemWriter. Spring Batch provides the TaskletStep for this scenario.

Tasklet is a simple interface that has one method, execute, which is called repeatedly by the TaskletStep until it either returns RepeatStatus.FINISHED or throws an exception to signal a failure. Each call to a Tasklet is wrapped in a transaction. Tasklet implementors might call a stored procedure, a script, or a simple SQL update statement.

With the ability to group steps together within an owning job comes the need to be able to control how the job "flows" from one step to another. The failure of a Step does not necessarily mean that the Job should fail. Furthermore, there may be more than one type of 'success' that determines which Step should be executed next. Depending upon how a group of Steps is configured, certain steps may not even be processed at all.

Both the XML and flat file examples shown earlier use the Spring Resource abstraction to obtain a file. This works because Resource has a getFile method, which returns a java.io.File. Both XML and flat file resources can be configured using standard Spring constructs, as shown in the following example:

The preceding Resource loads the file from the specified file system location. Note that absolute locations have to start with a double slash (//). In most Spring applications, this solution is good enough, because the names of these resources are known at compile time. However, in batch scenarios, the file name may need to be determined at runtime as a parameter to the job. This can be solved using '-D' parameters to read a system property.

All that would be required for this solution to work would be a system argument (such as -Dinput.file.name="file://outputs/file.txt").

Often, in a batch setting, it is preferable to parametrize the file name in the JobParameters of the job, instead of through system properties, and access them that way. To accomplish this, Spring Batch allows for the late binding of various Job and Step attributes, as shown in the following snippet:

Both the JobExecution and StepExecution level ExecutionContext can be accessed in the same way, as shown in the following examples:

Although a simple concept, an ItemReader is the means for providing data from many different types of input. The most general examples include:

There are many more possibilities, but we focus on the basic ones for this chapter. A complete list of all available ItemReader implementations can be found in Appendix A.

ItemReader is a basic interface for generic input operations, as shown in the following interface definition:

The read method defines the most essential contract of the ItemReader. Calling it returns one item or null if no more items are left. An item might represent a line in a file, a row in a database, or an element in an XML file. It is generally expected that these are mapped to a usable domain object (such as Trade, Foo, or others), but there is no requirement in the contract to do so.

It is expected that implementations of the ItemReader interface are forward only. However, if the underlying resource is transactional (such as a JMS queue) then calling read may return the same logical item on subsequent calls in a rollback scenario. It is also worth noting that a lack of items to process by an ItemReader does not cause an exception to be thrown. For example, a database ItemReader that is configured with a query that returns 0 results returns null on the first invocation of read.

ItemWriter is similar in functionality to an ItemReader but with inverse operations. Resources still need to be located, opened, and closed but they differ in that an ItemWriter writes out, rather than reading in. In the case of databases or queues, these operations may be inserts, updates, or sends. The format of the serialization of the output is specific to each batch job.

As with ItemReader, ItemWriter is a fairly generic interface, as shown in the following interface definition:

As with read on ItemReader, write provides the basic contract of ItemWriter. It attempts to write out the list of items passed in as long as it is open. Because it is generally expected that items are 'batched' together into a chunk and then output, the interface accepts a list of items, rather than an item by itself. After writing out the list, any flushing that may be necessary can be performed before returning from the write method. For example, if writing to a Hibernate DAO, multiple calls to write can be made, one for each item. The writer can then call flush on the hibernate session before returning.

The ItemReader and ItemWriter interfaces are both very useful for their specific tasks, but what if you want to insert business logic before writing? One option for both reading and writing is to use the composite pattern: Create an ItemWriter that contains another ItemWriter or an ItemReader that contains another ItemReader. The following code shows an example:

The preceding class contains another ItemWriter to which it delegates after having provided some business logic. This pattern could easily be used for an ItemReader as well, perhaps to obtain more reference data based upon the input that was provided by the main ItemReader. It is also useful if you need to control the call to write yourself. However, if you only want to 'transform' the item passed in for writing before it is actually written, you need not write yourself. You can just modify the item. For this scenario, Spring Batch provides the ItemProcessor interface, as shown in the following interface definition:

An ItemProcessor is simple. Given one object, transform it and return another. The provided object may or may not be of the same type. The point is that business logic may be applied within the process, and it is completely up to the developer to create that logic. An ItemProcessor can be wired directly into a step. For example, assume an ItemReader provides a class of type Foo and that it needs to be converted to type Bar before being written out. The following example shows an ItemProcessor that performs the conversion:

In the preceding example, there is a class Foo, a class Bar, and a class FooProcessor that adheres to the ItemProcessor interface. The transformation is simple, but any type of transformation could be done here. The BarWriter writes Bar objects, throwing an exception if any other type is provided. Similarly, the FooProcessor throws an exception if anything but a Foo is provided. The FooProcessor can then be injected into a Step, as shown in the following example:

Both ItemReaders and ItemWriters serve their individual purposes well, but there is a common concern among both of them that necessitates another interface. In general, as part of the scope of a batch job, readers and writers need to be opened, closed, and require a mechanism for persisting state. The ItemStream interface serves that purpose, as shown in the following example:

Before describing each method, we should mention the ExecutionContext. Clients of an ItemReader that also implement ItemStream should call open before any calls to read, in order to open any resources such as files or to obtain connections. A similar restriction applies to an ItemWriter that implements ItemStream. As mentioned in Chapter 2, if expected data is found in the ExecutionContext, it may be used to start the ItemReader or ItemWriter at a location other than its initial state. Conversely, close is called to ensure that any resources allocated during open are released safely. update is called primarily to ensure that any state currently being held is loaded into the provided ExecutionContext. This method is called before committing, to ensure that the current state is persisted in the database before commit.

In the special case where the client of an ItemStream is a Step (from the Spring Batch Core), an ExecutionContext is created for each StepExecution to allow users to store the state of a particular execution, with the expectation that it is returned if the same JobInstance is started again. For those familiar with Quartz, the semantics are very similar to a Quartz JobDataMap.

Note that the CompositeItemWriter is an example of the delegation pattern, which is common in Spring Batch. The delegates themselves might implement callback interfaces, such as StepListener. If they do and if they are being used in conjunction with Spring Batch Core as part of a Step in a Job, then they almost certainly need to be registered manually with the Step. A reader, writer, or processor that is directly wired into the Step gets registered automatically if it implements ItemStream or a StepListener interface. However, because the delegates are not known to the Step, they need to be injected as listeners or streams (or both if appropriate), as shown in the following example:

One of the most common mechanisms for interchanging bulk data has always been the flat file. Unlike XML, which has an agreed upon standard for defining how it is structured (XSD), anyone reading a flat file must understand ahead of time exactly how the file is structured. In general, all flat files fall into two types: delimited and fixed length. Delimited files are those in which fields are separated by a delimiter, such as a comma. Fixed Length files have fields that are a set length.

Spring Batch provides transactional infrastructure for both reading XML records and mapping them to Java objects as well as writing Java objects as XML records.

We need to consider how XML input and output works in Spring Batch. First, there are a few concepts that vary from file reading and writing but are common across Spring Batch XML processing. With XML processing, instead of lines of records (FieldSet instances) that need to be tokenized, it is assumed an XML resource is a collection of 'fragments' corresponding to individual records, as shown in the following image:

The 'trade' tag is defined as the 'root element' in the scenario above. Everything between '<trade>' and '</trade>' is considered one 'fragment'. Spring Batch uses Object/XML Mapping (OXM) to bind fragments to objects. However, Spring Batch is not tied to any particular XML binding technology. Typical use is to delegate to Spring OXM, which provides uniform abstraction for the most popular OXM technologies. The dependency on Spring OXM is optional and you can choose to implement Spring Batch specific interfaces if desired. The relationship to the technologies that OXM supports is shown in the following image:

With an introduction to OXM and how one can use XML fragments to represent records, we can now more closely examine readers and writers.

Spring Batch provides support for reading and Writing JSON resources in the following format:

It is assumed that the JSON resource is an array of JSON objects corresponding to individual items. Spring Batch is not tied to any particular JSON library.

It is a common requirement to process multiple files within a single Step. Assuming the files all have the same formatting, the MultiResourceItemReader supports this type of input for both XML and flat file processing. Consider the following files in a directory:

file-1.txt and file-2.txt are formatted the same and, for business reasons, should be processed together. The MultiResourceItemReader can be used to read in both files by using wildcards, as shown in the following example:

The referenced delegate is a simple FlatFileItemReader. The above configuration reads input from both files, handling rollback and restart scenarios. It should be noted that, as with any ItemReader, adding extra input (in this case a file) could cause potential issues when restarting. It is recommended that batch jobs work with their own individual directories until completed successfully.

Like most enterprise application styles, a database is the central storage mechanism for batch. However, batch differs from other application styles due to the sheer size of the datasets with which the system must work. If a SQL statement returns 1 million rows, the result set probably holds all returned results in memory until all rows have been read. Spring Batch provides two types of solutions for this problem:

Batch systems are often used in conjunction with other application styles. The most common is an online system, but it may also support integration or even a thick client application by moving necessary bulk data that each application style uses. For this reason, it is common that many users want to reuse existing DAOs or other services within their batch jobs. The Spring container itself makes this fairly easy by allowing any necessary class to be injected. However, there may be cases where the existing service needs to act as an ItemReader or ItemWriter, either to satisfy the dependency of another Spring Batch class or because it truly is the main ItemReader for a step. It is fairly trivial to write an adapter class for each service that needs wrapping, but because it is such a common concern, Spring Batch provides implementations: ItemReaderAdapter and ItemWriterAdapter. Both classes implement the standard Spring method by invoking the delegate pattern and are fairly simple to set up. The following example uses the ItemReaderAdapter:

One important point to note is that the contract of the targetMethod must be the same as the contract for read: When exhausted, it returns null. Otherwise, it returns an Object. Anything else prevents the framework from knowing when processing should end, either causing an infinite loop or incorrect failure, depending upon the implementation of the ItemWriter. The following example uses the ItemWriterAdapter:

During the course of this chapter, multiple approaches to parsing input have been discussed. Each major implementation throws an exception if it is not 'well-formed'. The FixedLengthTokenizer throws an exception if a range of data is missing. Similarly, attempting to access an index in a RowMapper or FieldSetMapper that does not exist or is in a different format than the one expected causes an exception to be thrown. All of these types of exceptions are thrown before read returns. However, they do not address the issue of whether or not the returned item is valid. For example, if one of the fields is an age, it obviously cannot be negative. It may parse correctly, because it exists and is a number, but it does not cause an exception. Since there are already a plethora of validation frameworks, Spring Batch does not attempt to provide yet another. Rather, it provides a simple interface, called Validator, that can be implemented by any number of frameworks, as shown in the following interface definition:

The contract is that the validate method throws an exception if the object is invalid and returns normally if it is valid. Spring Batch provides an out of the box ValidatingItemProcessor, as shown in the following bean definition:

You can also use the BeanValidatingItemProcessor to validate items annotated with the Bean Validation API (JSR-303) annotations. For example, given the following type Person:

you can validate items by declaring a BeanValidatingItemProcessor bean in your application context and register it as a processor in your chunk-oriented step:

By default, all of the ItemReader and ItemWriter implementations store their current state in the ExecutionContext before it is committed. However, this may not always be the desired behavior. For example, many developers choose to make their database readers 'rerunnable' by using a process indicator. An extra column is added to the input data to indicate whether or not it has been processed. When a particular record is being read (or written) the processed flag is flipped from false to true. The SQL statement can then contain an extra statement in the where clause, such as where PROCESSED_IND = false, thereby ensuring that only unprocessed records are returned in the case of a restart. In this scenario, it is preferable to not store any state, such as the current row number, since it is irrelevant upon restart. For this reason, all readers and writers include the 'saveState' property, as shown in the following example:

The ItemReader configured above does not make any entries in the ExecutionContext for any executions in which it participates.

So far, this chapter has discussed the basic contracts of reading and writing in Spring Batch and some common implementations for doing so. However, these are all fairly generic, and there are many potential scenarios that may not be covered by out-of-the-box implementations. This section shows, by using a simple example, how to create a custom ItemReader and ItemWriter implementation and implement their contracts correctly. The ItemReader also implements ItemStream, in order to illustrate how to make a reader or writer restartable.

In this section, we will introduce you to readers and writers that have not already been discussed in the previous sections.

The simplest way to start parallel processing is to add a TaskExecutor to your Step configuration.

In this example, the taskExecutor is a reference to another bean definition that implements the TaskExecutor interface. TaskExecutor is a standard Spring interface, so consult the Spring User Guide for details of available implementations. The simplest multi-threaded TaskExecutor is a SimpleAsyncTaskExecutor.

The result of the above configuration is that the Step executes by reading, processing, and writing each chunk of items (each commit interval) in a separate thread of execution. Note that this means there is no fixed order for the items to be processed, and a chunk might contain items that are non-consecutive compared to the single-threaded case. In addition to any limits placed by the task executor (such as whether it is backed by a thread pool), there is a throttle limit in the tasklet configuration which defaults to 4. You may need to increase this to ensure that a thread pool is fully utilized.

Note also that there may be limits placed on concurrency by any pooled resources used in your step, such as a DataSource. Be sure to make the pool in those resources at least as large as the desired number of concurrent threads in the step.

There are some practical limitations of using multi-threaded Step implementations for some common batch use cases. Many participants in a Step (such as readers and writers) are stateful. If the state is not segregated by thread, then those components are not usable in a multi-threaded Step. In particular, most of the off-the-shelf readers and writers from Spring Batch are not designed for multi-threaded use. It is, however, possible to work with stateless or thread safe readers and writers, and there is a sample (called parallelJob) in the Spring Batch Samples that shows the use of a process indicator (see Preventing State Persistence) to keep track of items that have been processed in a database input table.

Spring Batch provides some implementations of ItemWriter and ItemReader. Usually, they say in the Javadoc if they are thread safe or not or what you have to do to avoid problems in a concurrent environment. If there is no information in the Javadoc, you can check the implementation to see if there is any state. If a reader is not thread safe, you can decorate it with the provided SynchronizedItemStreamReader or use it in your own synchronizing delegator. You can synchronize the call to read() and as long as the processing and writing is the most expensive part of the chunk, your step may still complete much faster than it would in a single threaded configuration.

As long as the application logic that needs to be parallelized can be split into distinct responsibilities and assigned to individual steps, then it can be parallelized in a single process. Parallel Step execution is easy to configure and use.

The configurable task executor is used to specify which TaskExecutor implementation should be used to execute the individual flows. The default is SyncTaskExecutor, but an asynchronous TaskExecutor is required to run the steps in parallel. Note that the job ensures that every flow in the split completes before aggregating the exit statuses and transitioning.

See the section on Split Flows for more detail.

In remote chunking, the Step processing is split across multiple processes, communicating with each other through some middleware. The following image shows the pattern:

The manager component is a single process, and the workers are multiple remote processes. This pattern works best if the manager is not a bottleneck, so the processing must be more expensive than the reading of items (as is often the case in practice).

The manager is an implementation of a Spring Batch Step with the ItemWriter replaced by a generic version that knows how to send chunks of items to the middleware as messages. The workers are standard listeners for whatever middleware is being used (for example, with JMS, they would be MesssageListener implementations), and their role is to process the chunks of items using a standard ItemWriter or ItemProcessor plus ItemWriter, through the ChunkProcessor interface. One of the advantages of using this pattern is that the reader, processor, and writer components are off-the-shelf (the same as would be used for a local execution of the step). The items are divided up dynamically and work is shared through the middleware, so that, if the listeners are all eager consumers, then load balancing is automatic.

The middleware has to be durable, with guaranteed delivery and a single consumer for each message. JMS is the obvious candidate, but other options (such as JavaSpaces) exist in the grid computing and shared memory product space.

See the section on Spring Batch Integration - Remote Chunking for more detail.

Spring Batch also provides an SPI for partitioning a Step execution and executing it remotely. In this case, the remote participants are Step instances that could just as easily have been configured and used for local processing. The following image shows the pattern:

The Job runs on the left-hand side as a sequence of Step instances, and one of the Step instances is labeled as a manager. The workers in this picture are all identical instances of a Step, which could in fact take the place of the manager, resulting in the same outcome for the Job. The workers are typically going to be remote services but could also be local threads of execution. The messages sent by the manager to the workers in this pattern do not need to be durable or have guaranteed delivery. Spring Batch metadata in the JobRepository ensures that each worker is executed once and only once for each Job execution.

The SPI in Spring Batch consists of a special implementation of Step (called the PartitionStep) and two strategy interfaces that need to be implemented for the specific environment. The strategy interfaces are PartitionHandler and StepExecutionSplitter, and their role is shown in the following sequence diagram:

The Step on the right in this case is the “remote” worker, so, potentially, there are many objects and or processes playing this role, and the PartitionStep is shown driving the execution.

Similar to the multi-threaded step’s throttle-limit attribute, the grid-size attribute prevents the task executor from being saturated with requests from a single step.

There is a simple example that can be copied and extended in the unit test suite for Spring Batch Samples (see Partition*Job.xml configuration).

Spring Batch creates step executions for the partitions called "step1:partition0", and so on. Many people prefer to call the manager step "step1:manager" for consistency. You can use an alias for the step (by specifying the name attribute instead of the id attribute).

Batch processing is about repetitive actions, either as a simple optimization or as part of a job. To strategize and generalize the repetition and to provide what amounts to an iterator framework, Spring Batch has the RepeatOperations interface. The RepeatOperations interface has the following definition:

The callback is an interface, shown in the following definition, that lets you insert some business logic to be repeated:

The callback is executed repeatedly until the implementation determines that the iteration should end. The return value in these interfaces is an enumeration that can either be RepeatStatus.CONTINUABLE or RepeatStatus.FINISHED. A RepeatStatus enumeration conveys information to the caller of the repeat operations about whether there is any more work to do. Generally speaking, implementations of RepeatOperations should inspect the RepeatStatus and use it as part of the decision to end the iteration. Any callback that wishes to signal to the caller that there is no more work to do can return RepeatStatus.FINISHED.

The simplest general purpose implementation of RepeatOperations is RepeatTemplate, as shown in the following example:

In the preceding example, we return RepeatStatus.CONTINUABLE, to show that there is more work to do. The callback can also return RepeatStatus.FINISHED, to signal to the caller that there is no more work to do. Some iterations can be terminated by considerations intrinsic to the work being done in the callback. Others are effectively infinite loops as far as the callback is concerned and the completion decision is delegated to an external policy, as in the case shown in the preceding example.

Inside a RepeatTemplate, the termination of the loop in the iterate method is determined by a CompletionPolicy, which is also a factory for the RepeatContext. The RepeatTemplate has the responsibility to use the current policy to create a RepeatContext and pass that in to the RepeatCallback at every stage in the iteration. After a callback completes its doInIteration, the RepeatTemplate has to make a call to the CompletionPolicy to ask it to update its state (which will be stored in the RepeatContext). Then it asks the policy if the iteration is complete.

Spring Batch provides some simple general purpose implementations of CompletionPolicy. SimpleCompletionPolicy allows execution up to a fixed number of times (with RepeatStatus.FINISHED forcing early completion at any time).

Users might need to implement their own completion policies for more complicated decisions. For example, a batch processing window that prevents batch jobs from executing once the online systems are in use would require a custom policy.

If there is an exception thrown inside a RepeatCallback, the RepeatTemplate consults an ExceptionHandler, which can decide whether or not to re-throw the exception.

The following listing shows the ExceptionHandler interface definition:

A common use case is to count the number of exceptions of a given type and fail when a limit is reached. For this purpose, Spring Batch provides the SimpleLimitExceptionHandler and a slightly more flexible RethrowOnThresholdExceptionHandler. The SimpleLimitExceptionHandler has a limit property and an exception type that should be compared with the current exception. All subclasses of the provided type are also counted. Exceptions of the given type are ignored until the limit is reached, and then they are rethrown. Exceptions of other types are always rethrown.

An important optional property of the SimpleLimitExceptionHandler is the boolean flag called useParent. It is false by default, so the limit is only accounted for in the current RepeatContext. When set to true, the limit is kept across sibling contexts in a nested iteration (such as a set of chunks inside a step).

Often, it is useful to be able to receive additional callbacks for cross-cutting concerns across a number of different iterations. For this purpose, Spring Batch provides the RepeatListener interface. The RepeatTemplate lets users register RepeatListener implementations, and they are given callbacks with the RepeatContext and RepeatStatus where available during the iteration.

The RepeatListener interface has the following definition:

The open and close callbacks come before and after the entire iteration. before, after, and onError apply to the individual RepeatCallback calls.

Note that, when there is more than one listener, they are in a list, so there is an order. In this case, open and before are called in the same order while after, onError, and close are called in reverse order.

Implementations of RepeatOperations are not restricted to executing the callback sequentially. It is quite important that some implementations are able to execute their callbacks in parallel. To this end, Spring Batch provides the TaskExecutorRepeatTemplate, which uses the Spring TaskExecutor strategy to run the RepeatCallback. The default is to use a SynchronousTaskExecutor, which has the effect of executing the whole iteration in the same thread (the same as a normal RepeatTemplate).

Sometimes there is some business processing that you know you want to repeat every time it happens. The classic example of this is the optimization of a message pipeline. It is more efficient to process a batch of messages, if they are arriving frequently, than to bear the cost of a separate transaction for every message. Spring Batch provides an AOP interceptor that wraps a method call in a RepeatOperations object for just this purpose. The RepeatOperationsInterceptor executes the intercepted method and repeats according to the CompletionPolicy in the provided RepeatTemplate.

The preceding example uses a default RepeatTemplate inside the interceptor. To change the policies, listeners, and other details, you can inject an instance of RepeatTemplate into the interceptor.

If the intercepted method returns void, then the interceptor always returns RepeatStatus.CONTINUABLE (so there is a danger of an infinite loop if the CompletionPolicy does not have a finite end point). Otherwise, it returns RepeatStatus.CONTINUABLE until the return value from the intercepted method is null, at which point it returns RepeatStatus.FINISHED. Consequently, the business logic inside the target method can signal that there is no more work to do by returning null or by throwing an exception that is re-thrown by the ExceptionHandler in the provided RepeatTemplate.

To automate retry operations Spring Batch has the RetryOperations strategy. The following interface definition for RetryOperations:

The basic callback is a simple interface that lets you insert some business logic to be retried, as shown in the following interface definition:

The callback runs and, if it fails (by throwing an Exception), it is retried until either it is successful or the implementation aborts. There are a number of overloaded execute methods in the RetryOperations interface. Those methods deal with various use cases for recovery when all retry attempts are exhausted and deal with retry state, which allows clients and implementations to store information between calls (we cover this in more detail later in the chapter).

The simplest general purpose implementation of RetryOperations is RetryTemplate. It can be used as follows:

In the preceding example, we make a web service call and return the result to the user. If that call fails, then it is retried until a timeout is reached.

Inside a RetryTemplate, the decision to retry or fail in the execute method is determined by a RetryPolicy, which is also a factory for the RetryContext. The RetryTemplate has the responsibility to use the current policy to create a RetryContext and pass that in to the RetryCallback at every attempt. After a callback fails, the RetryTemplate has to make a call to the RetryPolicy to ask it to update its state (which is stored in the RetryContext) and then asks the policy if another attempt can be made. If another attempt cannot be made (such as when a limit is reached or a timeout is detected) then the policy is also responsible for handling the exhausted state. Simple implementations throw RetryExhaustedException, which causes any enclosing transaction to be rolled back. More sophisticated implementations might attempt to take some recovery action, in which case the transaction can remain intact.

Spring Batch provides some simple general purpose implementations of stateless RetryPolicy, such as SimpleRetryPolicy and TimeoutRetryPolicy (used in the preceding example).

The SimpleRetryPolicy allows a retry on any of a named list of exception types, up to a fixed number of times. It also has a list of "fatal" exceptions that should never be retried, and this list overrides the retryable list so that it can be used to give finer control over the retry behavior, as shown in the following example:

There is also a more flexible implementation called ExceptionClassifierRetryPolicy, which allows the user to configure different retry behavior for an arbitrary set of exception types though the ExceptionClassifier abstraction. The policy works by calling on the classifier to convert an exception into a delegate RetryPolicy. For example, one exception type can be retried more times before failure than another by mapping it to a different policy.

Users might need to implement their own retry policies for more customized decisions. For instance, a custom retry policy makes sense when there is a well-known, solution-specific classification of exceptions into retryable and not retryable.

When retrying after a transient failure, it often helps to wait a bit before trying again, because usually the failure is caused by some problem that can only be resolved by waiting. If a RetryCallback fails, the RetryTemplate can pause execution according to the BackoffPolicy.

The following code shows the interface definition for the BackOffPolicy interface:

A BackoffPolicy is free to implement the backOff in any way it chooses. The policies provided by Spring Batch out of the box all use Object.wait(). A common use case is to backoff with an exponentially increasing wait period, to avoid two retries getting into lock step and both failing (this is a lesson learned from ethernet). For this purpose, Spring Batch provides the ExponentialBackoffPolicy.

Often, it is useful to be able to receive additional callbacks for cross cutting concerns across a number of different retries. For this purpose, Spring Batch provides the RetryListener interface. The RetryTemplate lets users register RetryListeners, and they are given callbacks with RetryContext and Throwable where available during the iteration.

The following code shows the interface definition for RetryListener:

The open and close callbacks come before and after the entire retry in the simplest case, and onError applies to the individual RetryCallback calls. The close method might also receive a Throwable. If there has been an error, it is the last one thrown by the RetryCallback.

Note that, when there is more than one listener, they are in a list, so there is an order. In this case, open is called in the same order while onError and close are called in reverse order.

Sometimes, there is some business processing that you know you want to retry every time it happens. The classic example of this is the remote service call. Spring Batch provides an AOP interceptor that wraps a method call in a RetryOperations implementation for just this purpose. The RetryOperationsInterceptor executes the intercepted method and retries on failure according to the RetryPolicy in the provided RetryTemplate.

The preceding example uses a default RetryTemplate inside the interceptor. To change the policies or listeners, you can inject an instance of RetryTemplate into the interceptor.

In order for the unit test to run a batch job, the framework must load the job’s ApplicationContext. Two annotations are used to trigger this behavior:

Starting from v4.1, it is also possible to inject Spring Batch test utilities like the JobLauncherTestUtils and JobRepositoryTestUtils in the test context using the @SpringBatchTest annotation.

The following example shows the annotations in use:

'End To End' testing can be defined as testing the complete run of a batch job from beginning to end. This allows for a test that sets up a test condition, executes the job, and verifies the end result.

In the following example, the batch job reads from the database and writes to a flat file. The test method begins by setting up the database with test data. It clears the CUSTOMER table and then inserts 10 new records. The test then launches the Job by using the launchJob() method. The launchJob() method is provided by the JobLauncherTestUtils class. The JobLauncherTestUtils class also provides the launchJob(JobParameters) method, which allows the test to give particular parameters. The launchJob() method returns the JobExecution object, which is useful for asserting particular information about the Job run. In the following case, the test verifies that the Job ended with status "COMPLETED":

For complex batch jobs, test cases in the end-to-end testing approach may become unmanageable. It these cases, it may be more useful to have test cases to test individual steps on their own. The JobLauncherTestUtils class contains a method called launchStep, which takes a step name and runs just that particular Step. This approach allows for more targeted tests letting the test set up data for only that step and to validate its results directly. The following example shows how to use the launchStep method to load a Step by name:

Often, the components that are configured for your steps at runtime use step scope and late binding to inject context from the step or job execution. These are tricky to test as standalone components, unless you have a way to set the context as if they were in a step execution. That is the goal of two components in Spring Batch: StepScopeTestExecutionListener and StepScopeTestUtils.

The listener is declared at the class level, and its job is to create a step execution context for each test method, as shown in the following example:

There are two TestExecutionListeners. One is the regular Spring Test framework, which handles dependency injection from the configured application context to inject the reader. The other is the Spring Batch StepScopeTestExecutionListener. It works by looking for a factory method in the test case for a StepExecution, using that as the context for the test method, as if that execution were active in a Step at runtime. The factory method is detected by its signature (it must return a StepExecution). If a factory method is not provided, then a default StepExecution is created.

Starting from v4.1, the StepScopeTestExecutionListener and JobScopeTestExecutionListener are imported as test execution listeners if the test class is annotated with @SpringBatchTest. The preceding test example can be configured as follows:

The listener approach is convenient if you want the duration of the step scope to be the execution of the test method. For a more flexible but more invasive approach, you can use the StepScopeTestUtils. The following example counts the number of items available in the reader shown in the previous example:

When a batch job writes to the database, it is easy to query the database to verify that the output is as expected. However, if the batch job writes to a file, it is equally important that the output be verified. Spring Batch provides a class called AssertFile to facilitate the verification of output files. The method called assertFileEquals takes two File objects (or two Resource objects) and asserts, line by line, that the two files have the same content. Therefore, it is possible to create a file with the expected output and to compare it to the actual result, as shown in the following example:

Another common issue encountered while writing unit and integration tests for Spring Batch components is how to mock domain objects. A good example is a StepExecutionListener, as illustrated in the following code snippet:

The preceding listener example is provided by the framework and checks a StepExecution for an empty read count, thus signifying that no work was done. While this example is fairly simple, it serves to illustrate the types of problems that may be encountered when attempting to unit test classes that implement interfaces requiring Spring Batch domain objects. Consider the following unit test for the listener’s in the preceding example:

Because the Spring Batch domain model follows good object-oriented principles, the StepExecution requires a JobExecution, which requires a JobInstance and JobParameters, to create a valid StepExecution. While this is good in a solid domain model, it does make creating stub objects for unit testing verbose. To address this issue, the Spring Batch test module includes a factory for creating domain objects: MetaDataInstanceFactory. Given this factory, the unit test can be updated to be more concise, as shown in the following example:

The preceding method for creating a simple StepExecution is just one convenience method available within the factory. A full method listing can be found in its Javadoc.

A common use case is the need for special handling of errors in a step, item by item, perhaps logging to a special channel or inserting a record into a database. A chunk-oriented Step (created from the step factory beans) lets users implement this use case with a simple ItemReadListener for errors on read and an ItemWriteListener for errors on write. The following code snippet illustrates a listener that logs both read and write failures:

Having implemented this listener, it must be registered with a step, as shown in the following example:

Spring Batch provides a stop() method through the JobLauncher interface, but this is really for use by the operator rather than the application programmer. Sometimes, it is more convenient or makes more sense to stop a job execution from within the business logic.

The simplest thing to do is to throw a RuntimeException (one that is neither retried indefinitely nor skipped). For example, a custom exception type could be used, as shown in the following example:

Another simple way to stop a step from executing is to return null from the ItemReader, as shown in the following example:

The previous example actually relies on the fact that there is a default implementation of the CompletionPolicy strategy that signals a complete batch when the item to be processed is null. A more sophisticated completion policy could be implemented and injected into the Step through the SimpleStepFactoryBean, as shown in the following example:

An alternative is to set a flag in the StepExecution, which is checked by the Step implementations in the framework in between item processing. To implement this alternative, we need access to the current StepExecution, and this can be achieved by implementing a StepListener and registering it with the Step. The following example shows a listener that sets the flag:

When the flag is set, the default behavior is for the step to throw a JobInterruptedException. This behavior can be controlled through the StepInterruptionPolicy. However, the only choice is to throw or not throw an exception, so this is always an abnormal ending to a job.

Often, when writing to flat files, a "footer" record must be appended to the end of the file, after all processing has be completed. This can be achieved using the FlatFileFooterCallback interface provided by Spring Batch. The FlatFileFooterCallback (and its counterpart, the FlatFileHeaderCallback) are optional properties of the FlatFileItemWriter and can be added to an item writer as shown in the following example:

The footer callback interface has just one method that is called when the footer must be written, as shown in the following interface definition:

In the chapter on readers and writers, database input using paging was discussed. Many database vendors, such as DB2, have extremely pessimistic locking strategies that can cause issues if the table being read also needs to be used by other portions of the online application. Furthermore, opening cursors over extremely large datasets can cause issues on databases from certain vendors. Therefore, many projects prefer to use a 'Driving Query' approach to reading in data. This approach works by iterating over keys, rather than the entire object that needs to be returned, as the following image illustrates:

As you can see, the example shown in the preceding image uses the same 'FOO' table as was used in the cursor-based example. However, rather than selecting the entire row, only the IDs were selected in the SQL statement. So, rather than a FOO object being returned from read, an Integer is returned. This number can then be used to query for the 'details', which is a complete Foo object, as shown in the following image:

An ItemProcessor should be used to transform the key obtained from the driving query into a full 'Foo' object. An existing DAO can be used to query for the full object based on the key.

While it is usually the case with flat files that each record is confined to a single line, it is common that a file might have records spanning multiple lines with multiple formats. The following excerpt from a file shows an example of such an arrangement:

Everything between the line starting with 'HEA' and the line starting with 'FOT' is considered one record. There are a few considerations that must be made in order to handle this situation correctly:

Because a single record spans multiple lines and because we may not know how many lines there are, the ItemReader must be careful to always read an entire record. In order to do this, a custom ItemReader should be implemented as a wrapper for the FlatFileItemReader, as shown in the following example: