Package com.univocity.parsers.common.processor.core

Class AbstractBatchedColumnProcessor<T extends Context>

java.lang.Object

com.univocity.parsers.common.processor.core.AbstractBatchedColumnProcessor<T>

All Implemented Interfaces:

BatchedColumnReader<java.lang.String>, ColumnReader<java.lang.String>, Processor<T>

Direct Known Subclasses:

BatchedColumnProcessor

public abstract class AbstractBatchedColumnProcessor<T extends Context>
extends java.lang.Object
implements Processor<T>, BatchedColumnReader<java.lang.String>

A Processor implementation that stores values of columns in batches. Use this implementation in favor of AbstractColumnProcessor when processing large inputs to avoid running out of memory. Values parsed in each row will be split into columns of Strings. Each column has its own list of values.

During the execution of the process, the batchProcessed(int) method will be invoked after a given number of rows has been processed.

The user can access the lists with values parsed for all columns using the methods getColumnValuesAsList(), getColumnValuesAsMapOfIndexes() and getColumnValuesAsMapOfNames().

After batchProcessed(int) is invoked, all values will be discarded and the next batch of column values will be accumulated. This process will repeat until there's no more rows in the input.

See Also:

AbstractParser, BatchedColumnReader, Processor

Field Summary

Fields

Modifier and Type	Field	Description
private int	batchCount
private int	batchesProcessed
private int	rowsPerBatch
private ColumnSplitter<java.lang.String>	splitter

Constructor Summary

Constructors

Constructor	Description
AbstractBatchedColumnProcessor(int rowsPerBatch)	Constructs a batched column processor configured to invoke the batchesProcessed method after a given number of rows has been processed.

Method Summary

Modifier and Type	Method	Description
abstract void	batchProcessed(int rowsInThisBatch)	Callback to the user, where the lists with values parsed for all columns can be accessed using the methods ColumnReader.getColumnValuesAsList(), ColumnReader.getColumnValuesAsMapOfIndexes() and ColumnReader.getColumnValuesAsMapOfNames().
int	getBatchesProcessed()	Returns the number of batches already processed
java.util.List<java.lang.String>	getColumn(int columnIndex)	Returns the values of a given column.
java.util.List<java.lang.String>	getColumn(java.lang.String columnName)	Returns the values of a given column.
java.util.List<java.util.List<java.lang.String>>	getColumnValuesAsList()	Returns the values processed for each column
java.util.Map<java.lang.Integer,java.util.List<java.lang.String>>	getColumnValuesAsMapOfIndexes()	Returns a map of column indexes and their respective list of values parsed from the input.
java.util.Map<java.lang.String,java.util.List<java.lang.String>>	getColumnValuesAsMapOfNames()	Returns a map of column names and their respective list of values parsed from the input.
java.lang.String[]	getHeaders()	Returns the column headers.
int	getRowsPerBatch()	Returns the number of rows processed in each batch
void	processEnded(T context)	This method will by invoked by the parser once, after the parsing process stopped and all resources were closed.
void	processStarted(T context)	This method will by invoked by the parser once, when it is ready to start processing the input.
void	putColumnValuesInMapOfIndexes(java.util.Map<java.lang.Integer,java.util.List<java.lang.String>> map)	Fills a given map associating each column index to its list of values
void	putColumnValuesInMapOfNames(java.util.Map<java.lang.String,java.util.List<java.lang.String>> map)	Fills a given map associating each column name to its list o values
void	rowProcessed(java.lang.String[] row, T context)	Invoked by the parser after all values of a valid record have been processed.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

splitter

private final ColumnSplitter<java.lang.String> splitter

rowsPerBatch

private final int rowsPerBatch

batchCount

private int batchCount

batchesProcessed

private int batchesProcessed

Constructor Detail

AbstractBatchedColumnProcessor

public AbstractBatchedColumnProcessor(int rowsPerBatch)

Constructs a batched column processor configured to invoke the batchesProcessed method after a given number of rows has been processed.

Parameters:

rowsPerBatch - the number of rows to process in each batch.

Method Detail

processStarted

public void processStarted(T context)

Description copied from interface: Processor

This method will by invoked by the parser once, when it is ready to start processing the input.

Specified by:

processStarted in interface Processor<T extends Context>

Parameters:

context - A contextual object with information and controls over the current state of the parsing process

rowProcessed

public void rowProcessed(java.lang.String[] row,
                         T context)

Description copied from interface: Processor

Invoked by the parser after all values of a valid record have been processed.

Specified by:

rowProcessed in interface Processor<T extends Context>

Parameters:

row - the data extracted by the parser for an individual record. Note that:

• it will never by null.
• it will never be empty unless explicitly configured using CommonSettings.setSkipEmptyLines(boolean)
• it won't contain lines identified by the parser as comments. To disable comment processing set Format.setComment(char) to '\0'

context - A contextual object with information and controls over the current state of the parsing process

processEnded

public void processEnded(T context)

Description copied from interface: Processor

This method will by invoked by the parser once, after the parsing process stopped and all resources were closed.

It will always be called by the parser: in case of errors, if the end of the input us reached, or if the user stopped the process manually using Context.stop().

Specified by:

processEnded in interface Processor<T extends Context>

Parameters:

context - A contextual object with information and controls over the state of the parsing process

getHeaders

public final java.lang.String[] getHeaders()

Description copied from interface: ColumnReader

Returns the column headers. This can be either the headers defined in CommonSettings.getHeaders() or the headers parsed in the input when CommonSettings.getHeaders() equals to true

Specified by:

getHeaders in interface ColumnReader<T extends Context>

Returns:

the headers of all column parsed.

getColumnValuesAsList

public final java.util.List<java.util.List<java.lang.String>> getColumnValuesAsList()

Description copied from interface: ColumnReader

Returns the values processed for each column

Specified by:

getColumnValuesAsList in interface ColumnReader<T extends Context>

Returns:

a list of lists. The stored lists correspond to the position of the column processed from the input; Each list contains the corresponding values parsed for a column, across multiple rows.

putColumnValuesInMapOfNames

public final void putColumnValuesInMapOfNames(java.util.Map<java.lang.String,java.util.List<java.lang.String>> map)

Description copied from interface: ColumnReader

Fills a given map associating each column name to its list o values

Specified by:

putColumnValuesInMapOfNames in interface ColumnReader<T extends Context>

Parameters:

map - the map to hold the values of each column

putColumnValuesInMapOfIndexes

public final void putColumnValuesInMapOfIndexes(java.util.Map<java.lang.Integer,java.util.List<java.lang.String>> map)

Description copied from interface: ColumnReader

Fills a given map associating each column index to its list of values

Specified by:

putColumnValuesInMapOfIndexes in interface ColumnReader<T extends Context>

Parameters:

map - the map to hold the values of each column

getColumnValuesAsMapOfNames

public final java.util.Map<java.lang.String,java.util.List<java.lang.String>> getColumnValuesAsMapOfNames()

Description copied from interface: ColumnReader

Returns a map of column names and their respective list of values parsed from the input.

Specified by:

getColumnValuesAsMapOfNames in interface ColumnReader<T extends Context>

Returns:

a map of column names and their respective list of values.

getColumnValuesAsMapOfIndexes

public final java.util.Map<java.lang.Integer,java.util.List<java.lang.String>> getColumnValuesAsMapOfIndexes()

Description copied from interface: ColumnReader

Returns a map of column indexes and their respective list of values parsed from the input.

Specified by:

getColumnValuesAsMapOfIndexes in interface ColumnReader<T extends Context>

Returns:

a map of column indexes and their respective list of values.

getColumn

public java.util.List<java.lang.String> getColumn(java.lang.String columnName)

Description copied from interface: ColumnReader

Returns the values of a given column.

Specified by:

getColumn in interface ColumnReader<T extends Context>

Parameters:

columnName - the name of the column in the input.

Returns:

a list with all data stored in the given column

getColumn

public java.util.List<java.lang.String> getColumn(int columnIndex)

Description copied from interface: ColumnReader

Returns the values of a given column.

Specified by:

getColumn in interface ColumnReader<T extends Context>

Parameters:

columnIndex - the position of the column in the input (0-based).

Returns:

a list with all data stored in the given column

getRowsPerBatch

public int getRowsPerBatch()

Description copied from interface: BatchedColumnReader

Returns the number of rows processed in each batch

Specified by:

getRowsPerBatch in interface BatchedColumnReader<T extends Context>

Returns:

the number of rows per batch

getBatchesProcessed

public int getBatchesProcessed()

Description copied from interface: BatchedColumnReader

Returns the number of batches already processed

Specified by:

getBatchesProcessed in interface BatchedColumnReader<T extends Context>

Returns:

the number of batches already processed

batchProcessed

public abstract void batchProcessed(int rowsInThisBatch)

Description copied from interface: BatchedColumnReader

Callback to the user, where the lists with values parsed for all columns can be accessed using the methods ColumnReader.getColumnValuesAsList(), ColumnReader.getColumnValuesAsMapOfIndexes() and ColumnReader.getColumnValuesAsMapOfNames().

Specified by:

batchProcessed in interface BatchedColumnReader<T extends Context>

Parameters:

rowsInThisBatch - the number of rows processed in the current batch. This corresponds to the number of elements of each list of each column.