org.apache.hadoop.fs

Class VectoredReadUtils

java.lang.Object

org.apache.hadoop.fs.VectoredReadUtils

public final class VectoredReadUtils
extends Object

Utility class which implements helper methods used in vectored IO implementation.

Method Summary

Modifier and Type	Method and Description
static boolean	isOrderedDisjoint(List<? extends FileRange> input, int chunkSize, int minimumSeek) Is the given input list.
static List<CombinedFileRange>	mergeSortedRanges(List<? extends FileRange> sortedRanges, int chunkSize, int minimumSeek, int maxSize) Merge sorted ranges to optimize the access from the underlying file system.
static void	readInDirectBuffer(int length, ByteBuffer buffer, Function4RaisingIOE<Integer,byte[],Integer,Integer,Void> operation) Read bytes from stream into a byte buffer using an intermediate byte array.
static CompletableFuture<ByteBuffer>	readRangeFrom(PositionedReadable stream, FileRange range, IntFunction<ByteBuffer> allocate) Synchronously reads a range from the stream dealing with the combinations of ByteBuffers buffers and PositionedReadable streams.
static void	readVectored(PositionedReadable stream, List<? extends FileRange> ranges, IntFunction<ByteBuffer> allocate) This is the default implementation which iterates through the ranges to read each synchronously, but the intent is that subclasses can make more efficient readers.
static long	roundDown(long offset, int chunkSize) Calculates floor value of offset based on chunk size.
static long	roundUp(long offset, int chunkSize) Calculates the ceil value of offset based on chunk size.
static ByteBuffer	sliceTo(ByteBuffer readData, long readOffset, FileRange request) Slice the data that was read to the user's request.
static FileRange[]	sortRanges(List<? extends FileRange> input) Sort the input ranges by offset.
static List<? extends FileRange>	validateNonOverlappingAndReturnSortedRanges(List<? extends FileRange> input) Check if the input ranges are overlapping in nature.
static void	validateRangeRequest(FileRange range) Validate a single range.
static void	validateVectoredReadRanges(List<? extends FileRange> ranges) Validate a list of vectored read ranges.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

validateRangeRequest

public static void validateRangeRequest(FileRange range)
                                 throws EOFException

Validate a single range.

Parameters:

range - file range.

Throws:

EOFException - any EOF Exception.

validateVectoredReadRanges

public static void validateVectoredReadRanges(List<? extends FileRange> ranges)
                                       throws EOFException

Validate a list of vectored read ranges.

Parameters:

ranges - list of ranges.

Throws:

EOFException - any EOF exception.

readVectored

public static void readVectored(PositionedReadable stream,
                                List<? extends FileRange> ranges,
                                IntFunction<ByteBuffer> allocate)

This is the default implementation which iterates through the ranges to read each synchronously, but the intent is that subclasses can make more efficient readers. The data or exceptions are pushed into FileRange.getData().

Parameters:

stream - the stream to read the data from

ranges - the byte ranges to read

allocate - the byte buffer allocation

readRangeFrom

public static CompletableFuture<ByteBuffer> readRangeFrom(PositionedReadable stream,
                                                          FileRange range,
                                                          IntFunction<ByteBuffer> allocate)

Synchronously reads a range from the stream dealing with the combinations of ByteBuffers buffers and PositionedReadable streams.

Parameters:

stream - the stream to read from

range - the range to read

allocate - the function to allocate ByteBuffers

Returns:

the CompletableFuture that contains the read data

readInDirectBuffer

public static void readInDirectBuffer(int length,
                                      ByteBuffer buffer,
                                      Function4RaisingIOE<Integer,byte[],Integer,Integer,Void> operation)
                               throws IOException

Read bytes from stream into a byte buffer using an intermediate byte array.

Parameters:

length - number of bytes to read.

buffer - buffer to fill.

operation - operation to use for reading data.

Throws:

IOException - any IOE.

isOrderedDisjoint

public static boolean isOrderedDisjoint(List<? extends FileRange> input,
                                        int chunkSize,
                                        int minimumSeek)

Is the given input list.

• already sorted by offset
• each range is more than minimumSeek apart
• the start and end of each range is a multiple of chunkSize

Parameters:

input - the list of input ranges.

chunkSize - the size of the chunks that the offset and end must align to.

minimumSeek - the minimum distance between ranges.

Returns:

true if we can use the input list as is.

roundDown

public static long roundDown(long offset,
                             int chunkSize)

Calculates floor value of offset based on chunk size.

Parameters:

offset - file offset.

chunkSize - file chunk size.

Returns:

floor value.

roundUp

public static long roundUp(long offset,
                           int chunkSize)

Calculates the ceil value of offset based on chunk size.

Parameters:

offset - file offset.

chunkSize - file chunk size.

Returns:

ceil value.

validateNonOverlappingAndReturnSortedRanges

public static List<? extends FileRange> validateNonOverlappingAndReturnSortedRanges(List<? extends FileRange> input)

Check if the input ranges are overlapping in nature. We call two ranges to be overlapping when start offset of second is less than the end offset of first. End offset is calculated as start offset + length.

Parameters:

input - list if input ranges.

Returns:

true/false based on logic explained above.

sortRanges

public static FileRange[] sortRanges(List<? extends FileRange> input)

Sort the input ranges by offset.

Parameters:

input - input ranges.

Returns:

sorted ranges.

mergeSortedRanges

public static List<CombinedFileRange> mergeSortedRanges(List<? extends FileRange> sortedRanges,
                                                        int chunkSize,
                                                        int minimumSeek,
                                                        int maxSize)

Merge sorted ranges to optimize the access from the underlying file system. The motivations are that:

• Upper layers want to pass down logical file ranges.
• Fewer reads have better performance.
• Applications want callbacks as ranges are read.
• Some file systems want to round ranges to be at checksum boundaries.

Parameters:

sortedRanges - already sorted list of ranges based on offset.

chunkSize - round the start and end points to multiples of chunkSize

minimumSeek - the smallest gap that we should seek over in bytes

maxSize - the largest combined file range in bytes

Returns:

the list of sorted CombinedFileRanges that cover the input

sliceTo

public static ByteBuffer sliceTo(ByteBuffer readData,
                                 long readOffset,
                                 FileRange request)

Slice the data that was read to the user's request. This function assumes that the user's request is completely subsumed by the read data. This always creates a new buffer pointing to the same underlying data but with its own mark and position fields such that reading one buffer can't effect other's mark and position.

Parameters:

readData - the buffer with the readData

readOffset - the offset in the file for the readData

request - the user's request

Returns:

the readData buffer that is sliced to the user's request