1999-10-16 21:49:28 +02:00
|
|
|
/*-------------------------------------------------------------------------
|
|
|
|
*
|
|
|
|
* logtape.c
|
|
|
|
* Management of "logical tapes" within temporary files.
|
|
|
|
*
|
|
|
|
* This module exists to support sorting via multiple merge passes (see
|
1999-10-18 00:15:09 +02:00
|
|
|
* tuplesort.c). Merging is an ideal algorithm for tape devices, but if
|
|
|
|
* we implement it on disk by creating a separate file for each "tape",
|
1999-10-16 21:49:28 +02:00
|
|
|
* there is an annoying problem: the peak space usage is at least twice
|
2014-05-06 18:12:18 +02:00
|
|
|
* the volume of actual data to be sorted. (This must be so because each
|
1999-10-16 21:49:28 +02:00
|
|
|
* datum will appear in both the input and output tapes of the final
|
2014-05-06 18:12:18 +02:00
|
|
|
* merge pass. For seven-tape polyphase merge, which is otherwise a
|
1999-10-16 21:49:28 +02:00
|
|
|
* pretty good algorithm, peak usage is more like 4x actual data volume.)
|
|
|
|
*
|
|
|
|
* We can work around this problem by recognizing that any one tape
|
|
|
|
* dataset (with the possible exception of the final output) is written
|
2014-05-06 18:12:18 +02:00
|
|
|
* and read exactly once in a perfectly sequential manner. Therefore,
|
1999-10-16 21:49:28 +02:00
|
|
|
* a datum once read will not be required again, and we can recycle its
|
|
|
|
* space for use by the new tape dataset(s) being generated. In this way,
|
|
|
|
* the total space usage is essentially just the actual data volume, plus
|
|
|
|
* insignificant bookkeeping and start/stop overhead.
|
|
|
|
*
|
|
|
|
* Few OSes allow arbitrary parts of a file to be released back to the OS,
|
|
|
|
* so we have to implement this space-recycling ourselves within a single
|
|
|
|
* logical file. logtape.c exists to perform this bookkeeping and provide
|
1999-10-18 00:15:09 +02:00
|
|
|
* the illusion of N independent tape devices to tuplesort.c. Note that
|
1999-10-16 21:49:28 +02:00
|
|
|
* logtape.c itself depends on buffile.c to provide a "logical file" of
|
|
|
|
* larger size than the underlying OS may support.
|
|
|
|
*
|
|
|
|
* For simplicity, we allocate and release space in the underlying file
|
|
|
|
* in BLCKSZ-size blocks. Space allocation boils down to keeping track
|
|
|
|
* of which blocks in the underlying file belong to which logical tape,
|
2006-03-08 00:46:24 +01:00
|
|
|
* plus any blocks that are free (recycled and not yet reused).
|
2016-12-22 17:45:00 +01:00
|
|
|
* The blocks in each logical tape form a chain, with a prev- and next-
|
|
|
|
* pointer in each block.
|
1999-10-16 21:49:28 +02:00
|
|
|
*
|
|
|
|
* The initial write pass is guaranteed to fill the underlying file
|
|
|
|
* perfectly sequentially, no matter how data is divided into logical tapes.
|
|
|
|
* Once we begin merge passes, the access pattern becomes considerably
|
|
|
|
* less predictable --- but the seeking involved should be comparable to
|
|
|
|
* what would happen if we kept each logical tape in a separate file,
|
|
|
|
* so there's no serious performance penalty paid to obtain the space
|
|
|
|
* savings of recycling. We try to localize the write accesses by always
|
|
|
|
* writing to the lowest-numbered free block when we have a choice; it's
|
|
|
|
* not clear this helps much, but it can't hurt. (XXX perhaps a LIFO
|
|
|
|
* policy for free blocks would be better?)
|
|
|
|
*
|
Change the way pre-reading in external sort's merge phase works.
Don't pre-read tuples into SortTuple slots during merge. Instead, use the
memory for larger read buffers in logtape.c. We're doing the same number
of READTUP() calls either way, but managing the pre-read SortTuple slots
is much more complicated. Also, the on-tape representation is more compact
than SortTuples, so we can fit more pre-read tuples into the same amount
of memory this way. And we have better cache-locality, when we use just a
small number of SortTuple slots.
Now that we only hold one tuple from each tape in the SortTuple slots, we
can greatly simplify the "batch memory" management. We now maintain a
small set of fixed-sized slots, to hold the tuples, and fall back to
palloc() for larger tuples. We use this method during all merge phases,
not just the final merge, and also when randomAccess is requested, and
also in the TSS_SORTEDONTAPE case. In other words, it's used whenever we
do an external sort.
Reviewed by Peter Geoghegan and Claudio Freire.
Discussion: <CAM3SWZTpaORV=yQGVCG8Q4axcZ3MvF-05xe39ZvORdU9JcD6hQ@mail.gmail.com>
2016-10-03 12:37:49 +02:00
|
|
|
* To further make the I/Os more sequential, we can use a larger buffer
|
|
|
|
* when reading, and read multiple blocks from the same tape in one go,
|
|
|
|
* whenever the buffer becomes empty. LogicalTapeAssignReadBufferSize()
|
|
|
|
* can be used to set the size of the read buffer.
|
|
|
|
*
|
2006-03-08 00:46:24 +01:00
|
|
|
* To support the above policy of writing to the lowest free block,
|
|
|
|
* ltsGetFreeBlock sorts the list of free block numbers into decreasing
|
|
|
|
* order each time it is asked for a block and the list isn't currently
|
2014-05-06 18:12:18 +02:00
|
|
|
* sorted. This is an efficient way to handle it because we expect cycles
|
2006-03-08 00:46:24 +01:00
|
|
|
* of releasing many blocks followed by re-using many blocks, due to
|
Change the way pre-reading in external sort's merge phase works.
Don't pre-read tuples into SortTuple slots during merge. Instead, use the
memory for larger read buffers in logtape.c. We're doing the same number
of READTUP() calls either way, but managing the pre-read SortTuple slots
is much more complicated. Also, the on-tape representation is more compact
than SortTuples, so we can fit more pre-read tuples into the same amount
of memory this way. And we have better cache-locality, when we use just a
small number of SortTuple slots.
Now that we only hold one tuple from each tape in the SortTuple slots, we
can greatly simplify the "batch memory" management. We now maintain a
small set of fixed-sized slots, to hold the tuples, and fall back to
palloc() for larger tuples. We use this method during all merge phases,
not just the final merge, and also when randomAccess is requested, and
also in the TSS_SORTEDONTAPE case. In other words, it's used whenever we
do an external sort.
Reviewed by Peter Geoghegan and Claudio Freire.
Discussion: <CAM3SWZTpaORV=yQGVCG8Q4axcZ3MvF-05xe39ZvORdU9JcD6hQ@mail.gmail.com>
2016-10-03 12:37:49 +02:00
|
|
|
* the larger read buffer.
|
2006-03-08 00:46:24 +01:00
|
|
|
*
|
1999-10-16 21:49:28 +02:00
|
|
|
* Since all the bookkeeping and buffer memory is allocated with palloc(),
|
|
|
|
* and the underlying file(s) are made with OpenTemporaryFile, all resources
|
|
|
|
* for a logical tape set are certain to be cleaned up even if processing
|
2003-07-25 22:18:01 +02:00
|
|
|
* is aborted by ereport(ERROR). To avoid confusion, the caller should take
|
1999-10-16 21:49:28 +02:00
|
|
|
* care that all calls for a single LogicalTapeSet are made in the same
|
|
|
|
* palloc context.
|
2000-04-12 19:17:23 +02:00
|
|
|
*
|
2017-01-03 19:48:53 +01:00
|
|
|
* Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
|
2000-01-26 06:58:53 +01:00
|
|
|
* Portions Copyright (c) 1994, Regents of the University of California
|
1999-10-16 21:49:28 +02:00
|
|
|
*
|
|
|
|
* IDENTIFICATION
|
2010-09-20 22:08:53 +02:00
|
|
|
* src/backend/utils/sort/logtape.c
|
1999-10-16 21:49:28 +02:00
|
|
|
*
|
|
|
|
*-------------------------------------------------------------------------
|
|
|
|
*/
|
|
|
|
|
|
|
|
#include "postgres.h"
|
|
|
|
|
|
|
|
#include "storage/buffile.h"
|
|
|
|
#include "utils/logtape.h"
|
Change the way pre-reading in external sort's merge phase works.
Don't pre-read tuples into SortTuple slots during merge. Instead, use the
memory for larger read buffers in logtape.c. We're doing the same number
of READTUP() calls either way, but managing the pre-read SortTuple slots
is much more complicated. Also, the on-tape representation is more compact
than SortTuples, so we can fit more pre-read tuples into the same amount
of memory this way. And we have better cache-locality, when we use just a
small number of SortTuple slots.
Now that we only hold one tuple from each tape in the SortTuple slots, we
can greatly simplify the "batch memory" management. We now maintain a
small set of fixed-sized slots, to hold the tuples, and fall back to
palloc() for larger tuples. We use this method during all merge phases,
not just the final merge, and also when randomAccess is requested, and
also in the TSS_SORTEDONTAPE case. In other words, it's used whenever we
do an external sort.
Reviewed by Peter Geoghegan and Claudio Freire.
Discussion: <CAM3SWZTpaORV=yQGVCG8Q4axcZ3MvF-05xe39ZvORdU9JcD6hQ@mail.gmail.com>
2016-10-03 12:37:49 +02:00
|
|
|
#include "utils/memutils.h"
|
1999-10-16 21:49:28 +02:00
|
|
|
|
|
|
|
/*
|
2016-12-22 17:45:00 +01:00
|
|
|
* A TapeBlockTrailer is stored at the end of each BLCKSZ block.
|
|
|
|
*
|
|
|
|
* The first block of a tape has prev == -1. The last block of a tape
|
|
|
|
* stores the number of valid bytes on the block, inverted, in 'next'
|
|
|
|
* Therefore next < 0 indicates the last block.
|
1999-10-16 21:49:28 +02:00
|
|
|
*/
|
2016-12-22 17:45:00 +01:00
|
|
|
typedef struct TapeBlockTrailer
|
1999-10-16 21:49:28 +02:00
|
|
|
{
|
2016-12-22 17:45:00 +01:00
|
|
|
long prev; /* previous block on this tape, or -1 on first
|
|
|
|
* block */
|
|
|
|
long next; /* next block on this tape, or # of valid
|
|
|
|
* bytes on last block (if < 0) */
|
|
|
|
} TapeBlockTrailer;
|
|
|
|
|
|
|
|
#define TapeBlockPayloadSize (BLCKSZ - sizeof(TapeBlockTrailer))
|
|
|
|
#define TapeBlockGetTrailer(buf) \
|
|
|
|
((TapeBlockTrailer *) ((char *) buf + TapeBlockPayloadSize))
|
|
|
|
|
|
|
|
#define TapeBlockIsLast(buf) (TapeBlockGetTrailer(buf)->next < 0)
|
|
|
|
#define TapeBlockGetNBytes(buf) \
|
|
|
|
(TapeBlockIsLast(buf) ? \
|
|
|
|
(- TapeBlockGetTrailer(buf)->next) : TapeBlockPayloadSize)
|
|
|
|
#define TapeBlockSetNBytes(buf, nbytes) \
|
|
|
|
(TapeBlockGetTrailer(buf)->next = -(nbytes))
|
|
|
|
|
1999-10-16 21:49:28 +02:00
|
|
|
|
|
|
|
/*
|
|
|
|
* This data structure represents a single "logical tape" within the set
|
2016-12-22 17:45:00 +01:00
|
|
|
* of logical tapes stored in the same file.
|
|
|
|
*
|
|
|
|
* While writing, we hold the current partially-written data block in the
|
|
|
|
* buffer. While reading, we can hold multiple blocks in the buffer. Note
|
|
|
|
* that we don't retain the trailers of a block when it's read into the
|
|
|
|
* buffer. The buffer therefore contains one large contiguous chunk of data
|
|
|
|
* from the tape.
|
1999-10-16 21:49:28 +02:00
|
|
|
*/
|
|
|
|
typedef struct LogicalTape
|
|
|
|
{
|
|
|
|
bool writing; /* T while in write phase */
|
2005-10-15 04:49:52 +02:00
|
|
|
bool frozen; /* T if blocks should not be freed when read */
|
1999-10-16 21:49:28 +02:00
|
|
|
bool dirty; /* does buffer need to be written? */
|
2000-04-12 19:17:23 +02:00
|
|
|
|
1999-10-16 21:49:28 +02:00
|
|
|
/*
|
2016-12-22 17:45:00 +01:00
|
|
|
* Block numbers of the first, current, and next block of the tape.
|
|
|
|
*
|
|
|
|
* The "current" block number is only valid when writing, or reading from
|
|
|
|
* a frozen tape. (When reading from an unfrozen tape, we use a larger
|
|
|
|
* read buffer that holds multiple blocks, so the "current" block is
|
|
|
|
* ambiguous.)
|
1999-10-16 21:49:28 +02:00
|
|
|
*/
|
2016-12-22 17:45:00 +01:00
|
|
|
long firstBlockNumber;
|
|
|
|
long curBlockNumber;
|
|
|
|
long nextBlockNumber;
|
2000-04-12 19:17:23 +02:00
|
|
|
|
1999-10-16 21:49:28 +02:00
|
|
|
/*
|
2016-12-22 17:45:00 +01:00
|
|
|
* Buffer for current data block(s).
|
1999-10-16 21:49:28 +02:00
|
|
|
*/
|
2006-02-19 06:58:36 +01:00
|
|
|
char *buffer; /* physical buffer (separately palloc'd) */
|
Change the way pre-reading in external sort's merge phase works.
Don't pre-read tuples into SortTuple slots during merge. Instead, use the
memory for larger read buffers in logtape.c. We're doing the same number
of READTUP() calls either way, but managing the pre-read SortTuple slots
is much more complicated. Also, the on-tape representation is more compact
than SortTuples, so we can fit more pre-read tuples into the same amount
of memory this way. And we have better cache-locality, when we use just a
small number of SortTuple slots.
Now that we only hold one tuple from each tape in the SortTuple slots, we
can greatly simplify the "batch memory" management. We now maintain a
small set of fixed-sized slots, to hold the tuples, and fall back to
palloc() for larger tuples. We use this method during all merge phases,
not just the final merge, and also when randomAccess is requested, and
also in the TSS_SORTEDONTAPE case. In other words, it's used whenever we
do an external sort.
Reviewed by Peter Geoghegan and Claudio Freire.
Discussion: <CAM3SWZTpaORV=yQGVCG8Q4axcZ3MvF-05xe39ZvORdU9JcD6hQ@mail.gmail.com>
2016-10-03 12:37:49 +02:00
|
|
|
int buffer_size; /* allocated size of the buffer */
|
1999-10-16 21:49:28 +02:00
|
|
|
int pos; /* next read/write position in buffer */
|
|
|
|
int nbytes; /* total # of valid bytes in buffer */
|
|
|
|
} LogicalTape;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This data structure represents a set of related "logical tapes" sharing
|
|
|
|
* space in a single underlying file. (But that "file" may be multiple files
|
|
|
|
* if needed to escape OS limits on file size; buffile.c handles that for us.)
|
|
|
|
* The number of tapes is fixed at creation.
|
|
|
|
*/
|
|
|
|
struct LogicalTapeSet
|
|
|
|
{
|
2000-04-12 19:17:23 +02:00
|
|
|
BufFile *pfile; /* underlying file for whole tape set */
|
2017-02-01 11:17:38 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* File size tracking. nBlocksWritten is the size of the underlying file,
|
|
|
|
* in BLCKSZ blocks. nBlocksAllocated is the number of blocks allocated
|
|
|
|
* by ltsGetFreeBlock(), and it is always greater than or equal to
|
|
|
|
* nBlocksWritten. Blocks between nBlocksAllocated and nBlocksWritten are
|
|
|
|
* blocks that have been allocated for a tape, but have not been written
|
|
|
|
* to the underlying file yet.
|
|
|
|
*/
|
Phase 2 of pgindent updates.
Change pg_bsd_indent to follow upstream rules for placement of comments
to the right of code, and remove pgindent hack that caused comments
following #endif to not obey the general rule.
Commit e3860ffa4dd0dad0dd9eea4be9cc1412373a8c89 wasn't actually using
the published version of pg_bsd_indent, but a hacked-up version that
tried to minimize the amount of movement of comments to the right of
code. The situation of interest is where such a comment has to be
moved to the right of its default placement at column 33 because there's
code there. BSD indent has always moved right in units of tab stops
in such cases --- but in the previous incarnation, indent was working
in 8-space tab stops, while now it knows we use 4-space tabs. So the
net result is that in about half the cases, such comments are placed
one tab stop left of before. This is better all around: it leaves
more room on the line for comment text, and it means that in such
cases the comment uniformly starts at the next 4-space tab stop after
the code, rather than sometimes one and sometimes two tabs after.
Also, ensure that comments following #endif are indented the same
as comments following other preprocessor commands such as #else.
That inconsistency turns out to have been self-inflicted damage
from a poorly-thought-through post-indent "fixup" in pgindent.
This patch is much less interesting than the first round of indent
changes, but also bulkier, so I thought it best to separate the effects.
Discussion: https://postgr.es/m/E1dAmxK-0006EE-1r@gemulon.postgresql.org
Discussion: https://postgr.es/m/30527.1495162840@sss.pgh.pa.us
2017-06-21 21:18:54 +02:00
|
|
|
long nBlocksAllocated; /* # of blocks allocated */
|
2017-02-01 11:17:38 +01:00
|
|
|
long nBlocksWritten; /* # of blocks used in underlying file */
|
2000-04-12 19:17:23 +02:00
|
|
|
|
1999-10-16 21:49:28 +02:00
|
|
|
/*
|
2005-10-15 04:49:52 +02:00
|
|
|
* We store the numbers of recycled-and-available blocks in freeBlocks[].
|
2006-03-08 00:46:24 +01:00
|
|
|
* When there are no such blocks, we extend the underlying file.
|
2006-03-07 20:06:50 +01:00
|
|
|
*
|
|
|
|
* If forgetFreeSpace is true then any freed blocks are simply forgotten
|
|
|
|
* rather than being remembered in freeBlocks[]. See notes for
|
|
|
|
* LogicalTapeSetForgetFreeSpace().
|
2006-03-08 00:46:24 +01:00
|
|
|
*
|
|
|
|
* If blocksSorted is true then the block numbers in freeBlocks are in
|
|
|
|
* *decreasing* order, so that removing the last entry gives us the lowest
|
2014-05-06 18:12:18 +02:00
|
|
|
* free block. We re-sort the blocks whenever a block is demanded; this
|
2006-03-08 00:46:24 +01:00
|
|
|
* should be reasonably efficient given the expected usage pattern.
|
1999-10-16 21:49:28 +02:00
|
|
|
*/
|
2006-03-07 20:06:50 +01:00
|
|
|
bool forgetFreeSpace; /* are we remembering free blocks? */
|
2006-03-08 00:46:24 +01:00
|
|
|
bool blocksSorted; /* is freeBlocks[] currently in order? */
|
1999-10-16 21:49:28 +02:00
|
|
|
long *freeBlocks; /* resizable array */
|
|
|
|
int nFreeBlocks; /* # of currently free blocks */
|
2005-10-15 04:49:52 +02:00
|
|
|
int freeBlocksLen; /* current allocated length of freeBlocks[] */
|
2000-04-12 19:17:23 +02:00
|
|
|
|
2015-02-21 22:12:14 +01:00
|
|
|
/* The array of logical tapes. */
|
1999-10-16 21:49:28 +02:00
|
|
|
int nTapes; /* # of logical tapes in set */
|
2015-02-21 22:12:14 +01:00
|
|
|
LogicalTape tapes[FLEXIBLE_ARRAY_MEMBER]; /* has nTapes nentries */
|
1999-10-16 21:49:28 +02:00
|
|
|
};
|
|
|
|
|
|
|
|
static void ltsWriteBlock(LogicalTapeSet *lts, long blocknum, void *buffer);
|
|
|
|
static void ltsReadBlock(LogicalTapeSet *lts, long blocknum, void *buffer);
|
|
|
|
static long ltsGetFreeBlock(LogicalTapeSet *lts);
|
|
|
|
static void ltsReleaseBlock(LogicalTapeSet *lts, long blocknum);
|
|
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Write a block-sized buffer to the specified block of the underlying file.
|
|
|
|
*
|
2003-07-25 22:18:01 +02:00
|
|
|
* No need for an error return convention; we ereport() on any error.
|
1999-10-16 21:49:28 +02:00
|
|
|
*/
|
|
|
|
static void
|
|
|
|
ltsWriteBlock(LogicalTapeSet *lts, long blocknum, void *buffer)
|
|
|
|
{
|
2017-02-01 11:17:38 +01:00
|
|
|
/*
|
|
|
|
* BufFile does not support "holes", so if we're about to write a block
|
|
|
|
* that's past the current end of file, fill the space between the current
|
|
|
|
* end of file and the target block with zeros.
|
|
|
|
*
|
|
|
|
* This should happen rarely, otherwise you are not writing very
|
|
|
|
* sequentially. In current use, this only happens when the sort ends
|
|
|
|
* writing a run, and switches to another tape. The last block of the
|
|
|
|
* previous tape isn't flushed to disk until the end of the sort, so you
|
|
|
|
* get one-block hole, where the last block of the previous tape will
|
|
|
|
* later go.
|
|
|
|
*/
|
|
|
|
while (blocknum > lts->nBlocksWritten)
|
|
|
|
{
|
|
|
|
char zerobuf[BLCKSZ];
|
|
|
|
|
|
|
|
MemSet(zerobuf, 0, sizeof(zerobuf));
|
|
|
|
|
|
|
|
ltsWriteBlock(lts, lts->nBlocksWritten, zerobuf);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Write the requested block */
|
1999-10-16 21:49:28 +02:00
|
|
|
if (BufFileSeekBlock(lts->pfile, blocknum) != 0 ||
|
|
|
|
BufFileWrite(lts->pfile, buffer, BLCKSZ) != BLCKSZ)
|
2003-07-25 22:18:01 +02:00
|
|
|
ereport(ERROR,
|
|
|
|
(errcode_for_file_access(),
|
|
|
|
errmsg("could not write block %ld of temporary file: %m",
|
2014-06-13 00:59:06 +02:00
|
|
|
blocknum)));
|
2017-02-01 11:17:38 +01:00
|
|
|
|
|
|
|
/* Update nBlocksWritten, if we extended the file */
|
|
|
|
if (blocknum == lts->nBlocksWritten)
|
|
|
|
lts->nBlocksWritten++;
|
1999-10-16 21:49:28 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Read a block-sized buffer from the specified block of the underlying file.
|
|
|
|
*
|
2014-05-06 18:12:18 +02:00
|
|
|
* No need for an error return convention; we ereport() on any error. This
|
1999-10-16 21:49:28 +02:00
|
|
|
* module should never attempt to read a block it doesn't know is there.
|
|
|
|
*/
|
|
|
|
static void
|
|
|
|
ltsReadBlock(LogicalTapeSet *lts, long blocknum, void *buffer)
|
|
|
|
{
|
|
|
|
if (BufFileSeekBlock(lts->pfile, blocknum) != 0 ||
|
|
|
|
BufFileRead(lts->pfile, buffer, BLCKSZ) != BLCKSZ)
|
2003-07-25 22:18:01 +02:00
|
|
|
ereport(ERROR,
|
|
|
|
(errcode_for_file_access(),
|
|
|
|
errmsg("could not read block %ld of temporary file: %m",
|
|
|
|
blocknum)));
|
1999-10-16 21:49:28 +02:00
|
|
|
}
|
|
|
|
|
Change the way pre-reading in external sort's merge phase works.
Don't pre-read tuples into SortTuple slots during merge. Instead, use the
memory for larger read buffers in logtape.c. We're doing the same number
of READTUP() calls either way, but managing the pre-read SortTuple slots
is much more complicated. Also, the on-tape representation is more compact
than SortTuples, so we can fit more pre-read tuples into the same amount
of memory this way. And we have better cache-locality, when we use just a
small number of SortTuple slots.
Now that we only hold one tuple from each tape in the SortTuple slots, we
can greatly simplify the "batch memory" management. We now maintain a
small set of fixed-sized slots, to hold the tuples, and fall back to
palloc() for larger tuples. We use this method during all merge phases,
not just the final merge, and also when randomAccess is requested, and
also in the TSS_SORTEDONTAPE case. In other words, it's used whenever we
do an external sort.
Reviewed by Peter Geoghegan and Claudio Freire.
Discussion: <CAM3SWZTpaORV=yQGVCG8Q4axcZ3MvF-05xe39ZvORdU9JcD6hQ@mail.gmail.com>
2016-10-03 12:37:49 +02:00
|
|
|
/*
|
|
|
|
* Read as many blocks as we can into the per-tape buffer.
|
|
|
|
*
|
|
|
|
* Returns true if anything was read, 'false' on EOF.
|
|
|
|
*/
|
|
|
|
static bool
|
2016-12-22 17:45:00 +01:00
|
|
|
ltsReadFillBuffer(LogicalTapeSet *lts, LogicalTape *lt)
|
Change the way pre-reading in external sort's merge phase works.
Don't pre-read tuples into SortTuple slots during merge. Instead, use the
memory for larger read buffers in logtape.c. We're doing the same number
of READTUP() calls either way, but managing the pre-read SortTuple slots
is much more complicated. Also, the on-tape representation is more compact
than SortTuples, so we can fit more pre-read tuples into the same amount
of memory this way. And we have better cache-locality, when we use just a
small number of SortTuple slots.
Now that we only hold one tuple from each tape in the SortTuple slots, we
can greatly simplify the "batch memory" management. We now maintain a
small set of fixed-sized slots, to hold the tuples, and fall back to
palloc() for larger tuples. We use this method during all merge phases,
not just the final merge, and also when randomAccess is requested, and
also in the TSS_SORTEDONTAPE case. In other words, it's used whenever we
do an external sort.
Reviewed by Peter Geoghegan and Claudio Freire.
Discussion: <CAM3SWZTpaORV=yQGVCG8Q4axcZ3MvF-05xe39ZvORdU9JcD6hQ@mail.gmail.com>
2016-10-03 12:37:49 +02:00
|
|
|
{
|
|
|
|
lt->pos = 0;
|
|
|
|
lt->nbytes = 0;
|
|
|
|
|
|
|
|
do
|
|
|
|
{
|
2016-12-22 17:45:00 +01:00
|
|
|
char *thisbuf = lt->buffer + lt->nbytes;
|
|
|
|
|
|
|
|
/* Fetch next block number */
|
|
|
|
if (lt->nextBlockNumber == -1L)
|
|
|
|
break; /* EOF */
|
Change the way pre-reading in external sort's merge phase works.
Don't pre-read tuples into SortTuple slots during merge. Instead, use the
memory for larger read buffers in logtape.c. We're doing the same number
of READTUP() calls either way, but managing the pre-read SortTuple slots
is much more complicated. Also, the on-tape representation is more compact
than SortTuples, so we can fit more pre-read tuples into the same amount
of memory this way. And we have better cache-locality, when we use just a
small number of SortTuple slots.
Now that we only hold one tuple from each tape in the SortTuple slots, we
can greatly simplify the "batch memory" management. We now maintain a
small set of fixed-sized slots, to hold the tuples, and fall back to
palloc() for larger tuples. We use this method during all merge phases,
not just the final merge, and also when randomAccess is requested, and
also in the TSS_SORTEDONTAPE case. In other words, it's used whenever we
do an external sort.
Reviewed by Peter Geoghegan and Claudio Freire.
Discussion: <CAM3SWZTpaORV=yQGVCG8Q4axcZ3MvF-05xe39ZvORdU9JcD6hQ@mail.gmail.com>
2016-10-03 12:37:49 +02:00
|
|
|
|
|
|
|
/* Read the block */
|
2016-12-22 17:45:00 +01:00
|
|
|
ltsReadBlock(lts, lt->nextBlockNumber, (void *) thisbuf);
|
Change the way pre-reading in external sort's merge phase works.
Don't pre-read tuples into SortTuple slots during merge. Instead, use the
memory for larger read buffers in logtape.c. We're doing the same number
of READTUP() calls either way, but managing the pre-read SortTuple slots
is much more complicated. Also, the on-tape representation is more compact
than SortTuples, so we can fit more pre-read tuples into the same amount
of memory this way. And we have better cache-locality, when we use just a
small number of SortTuple slots.
Now that we only hold one tuple from each tape in the SortTuple slots, we
can greatly simplify the "batch memory" management. We now maintain a
small set of fixed-sized slots, to hold the tuples, and fall back to
palloc() for larger tuples. We use this method during all merge phases,
not just the final merge, and also when randomAccess is requested, and
also in the TSS_SORTEDONTAPE case. In other words, it's used whenever we
do an external sort.
Reviewed by Peter Geoghegan and Claudio Freire.
Discussion: <CAM3SWZTpaORV=yQGVCG8Q4axcZ3MvF-05xe39ZvORdU9JcD6hQ@mail.gmail.com>
2016-10-03 12:37:49 +02:00
|
|
|
if (!lt->frozen)
|
2016-12-22 17:45:00 +01:00
|
|
|
ltsReleaseBlock(lts, lt->nextBlockNumber);
|
|
|
|
lt->curBlockNumber = lt->nextBlockNumber;
|
Change the way pre-reading in external sort's merge phase works.
Don't pre-read tuples into SortTuple slots during merge. Instead, use the
memory for larger read buffers in logtape.c. We're doing the same number
of READTUP() calls either way, but managing the pre-read SortTuple slots
is much more complicated. Also, the on-tape representation is more compact
than SortTuples, so we can fit more pre-read tuples into the same amount
of memory this way. And we have better cache-locality, when we use just a
small number of SortTuple slots.
Now that we only hold one tuple from each tape in the SortTuple slots, we
can greatly simplify the "batch memory" management. We now maintain a
small set of fixed-sized slots, to hold the tuples, and fall back to
palloc() for larger tuples. We use this method during all merge phases,
not just the final merge, and also when randomAccess is requested, and
also in the TSS_SORTEDONTAPE case. In other words, it's used whenever we
do an external sort.
Reviewed by Peter Geoghegan and Claudio Freire.
Discussion: <CAM3SWZTpaORV=yQGVCG8Q4axcZ3MvF-05xe39ZvORdU9JcD6hQ@mail.gmail.com>
2016-10-03 12:37:49 +02:00
|
|
|
|
2016-12-22 17:45:00 +01:00
|
|
|
lt->nbytes += TapeBlockGetNBytes(thisbuf);
|
|
|
|
if (TapeBlockIsLast(thisbuf))
|
Change the way pre-reading in external sort's merge phase works.
Don't pre-read tuples into SortTuple slots during merge. Instead, use the
memory for larger read buffers in logtape.c. We're doing the same number
of READTUP() calls either way, but managing the pre-read SortTuple slots
is much more complicated. Also, the on-tape representation is more compact
than SortTuples, so we can fit more pre-read tuples into the same amount
of memory this way. And we have better cache-locality, when we use just a
small number of SortTuple slots.
Now that we only hold one tuple from each tape in the SortTuple slots, we
can greatly simplify the "batch memory" management. We now maintain a
small set of fixed-sized slots, to hold the tuples, and fall back to
palloc() for larger tuples. We use this method during all merge phases,
not just the final merge, and also when randomAccess is requested, and
also in the TSS_SORTEDONTAPE case. In other words, it's used whenever we
do an external sort.
Reviewed by Peter Geoghegan and Claudio Freire.
Discussion: <CAM3SWZTpaORV=yQGVCG8Q4axcZ3MvF-05xe39ZvORdU9JcD6hQ@mail.gmail.com>
2016-10-03 12:37:49 +02:00
|
|
|
{
|
2016-12-22 17:45:00 +01:00
|
|
|
lt->nextBlockNumber = -1L;
|
Change the way pre-reading in external sort's merge phase works.
Don't pre-read tuples into SortTuple slots during merge. Instead, use the
memory for larger read buffers in logtape.c. We're doing the same number
of READTUP() calls either way, but managing the pre-read SortTuple slots
is much more complicated. Also, the on-tape representation is more compact
than SortTuples, so we can fit more pre-read tuples into the same amount
of memory this way. And we have better cache-locality, when we use just a
small number of SortTuple slots.
Now that we only hold one tuple from each tape in the SortTuple slots, we
can greatly simplify the "batch memory" management. We now maintain a
small set of fixed-sized slots, to hold the tuples, and fall back to
palloc() for larger tuples. We use this method during all merge phases,
not just the final merge, and also when randomAccess is requested, and
also in the TSS_SORTEDONTAPE case. In other words, it's used whenever we
do an external sort.
Reviewed by Peter Geoghegan and Claudio Freire.
Discussion: <CAM3SWZTpaORV=yQGVCG8Q4axcZ3MvF-05xe39ZvORdU9JcD6hQ@mail.gmail.com>
2016-10-03 12:37:49 +02:00
|
|
|
/* EOF */
|
|
|
|
break;
|
|
|
|
}
|
2016-12-22 17:45:00 +01:00
|
|
|
else
|
|
|
|
lt->nextBlockNumber = TapeBlockGetTrailer(thisbuf)->next;
|
Change the way pre-reading in external sort's merge phase works.
Don't pre-read tuples into SortTuple slots during merge. Instead, use the
memory for larger read buffers in logtape.c. We're doing the same number
of READTUP() calls either way, but managing the pre-read SortTuple slots
is much more complicated. Also, the on-tape representation is more compact
than SortTuples, so we can fit more pre-read tuples into the same amount
of memory this way. And we have better cache-locality, when we use just a
small number of SortTuple slots.
Now that we only hold one tuple from each tape in the SortTuple slots, we
can greatly simplify the "batch memory" management. We now maintain a
small set of fixed-sized slots, to hold the tuples, and fall back to
palloc() for larger tuples. We use this method during all merge phases,
not just the final merge, and also when randomAccess is requested, and
also in the TSS_SORTEDONTAPE case. In other words, it's used whenever we
do an external sort.
Reviewed by Peter Geoghegan and Claudio Freire.
Discussion: <CAM3SWZTpaORV=yQGVCG8Q4axcZ3MvF-05xe39ZvORdU9JcD6hQ@mail.gmail.com>
2016-10-03 12:37:49 +02:00
|
|
|
|
|
|
|
/* Advance to next block, if we have buffer space left */
|
2016-12-22 17:45:00 +01:00
|
|
|
} while (lt->buffer_size - lt->nbytes > BLCKSZ);
|
Change the way pre-reading in external sort's merge phase works.
Don't pre-read tuples into SortTuple slots during merge. Instead, use the
memory for larger read buffers in logtape.c. We're doing the same number
of READTUP() calls either way, but managing the pre-read SortTuple slots
is much more complicated. Also, the on-tape representation is more compact
than SortTuples, so we can fit more pre-read tuples into the same amount
of memory this way. And we have better cache-locality, when we use just a
small number of SortTuple slots.
Now that we only hold one tuple from each tape in the SortTuple slots, we
can greatly simplify the "batch memory" management. We now maintain a
small set of fixed-sized slots, to hold the tuples, and fall back to
palloc() for larger tuples. We use this method during all merge phases,
not just the final merge, and also when randomAccess is requested, and
also in the TSS_SORTEDONTAPE case. In other words, it's used whenever we
do an external sort.
Reviewed by Peter Geoghegan and Claudio Freire.
Discussion: <CAM3SWZTpaORV=yQGVCG8Q4axcZ3MvF-05xe39ZvORdU9JcD6hQ@mail.gmail.com>
2016-10-03 12:37:49 +02:00
|
|
|
|
|
|
|
return (lt->nbytes > 0);
|
|
|
|
}
|
|
|
|
|
2006-03-08 00:46:24 +01:00
|
|
|
/*
|
|
|
|
* qsort comparator for sorting freeBlocks[] into decreasing order.
|
|
|
|
*/
|
|
|
|
static int
|
|
|
|
freeBlocks_cmp(const void *a, const void *b)
|
|
|
|
{
|
|
|
|
long ablk = *((const long *) a);
|
|
|
|
long bblk = *((const long *) b);
|
|
|
|
|
|
|
|
/* can't just subtract because long might be wider than int */
|
|
|
|
if (ablk < bblk)
|
|
|
|
return 1;
|
|
|
|
if (ablk > bblk)
|
|
|
|
return -1;
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
1999-10-16 21:49:28 +02:00
|
|
|
/*
|
|
|
|
* Select a currently unused block for writing to.
|
|
|
|
*/
|
|
|
|
static long
|
|
|
|
ltsGetFreeBlock(LogicalTapeSet *lts)
|
|
|
|
{
|
2000-04-12 19:17:23 +02:00
|
|
|
/*
|
2005-10-15 04:49:52 +02:00
|
|
|
* If there are multiple free blocks, we select the one appearing last in
|
2006-03-08 00:46:24 +01:00
|
|
|
* freeBlocks[] (after sorting the array if needed). If there are none,
|
|
|
|
* assign the next block at the end of the file.
|
1999-10-16 21:49:28 +02:00
|
|
|
*/
|
|
|
|
if (lts->nFreeBlocks > 0)
|
2006-03-08 00:46:24 +01:00
|
|
|
{
|
|
|
|
if (!lts->blocksSorted)
|
|
|
|
{
|
|
|
|
qsort((void *) lts->freeBlocks, lts->nFreeBlocks,
|
|
|
|
sizeof(long), freeBlocks_cmp);
|
|
|
|
lts->blocksSorted = true;
|
|
|
|
}
|
1999-10-16 21:49:28 +02:00
|
|
|
return lts->freeBlocks[--lts->nFreeBlocks];
|
2006-03-08 00:46:24 +01:00
|
|
|
}
|
1999-10-16 21:49:28 +02:00
|
|
|
else
|
2017-02-01 11:17:38 +01:00
|
|
|
return lts->nBlocksAllocated++;
|
1999-10-16 21:49:28 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Return a block# to the freelist.
|
|
|
|
*/
|
|
|
|
static void
|
|
|
|
ltsReleaseBlock(LogicalTapeSet *lts, long blocknum)
|
|
|
|
{
|
2000-04-12 19:17:23 +02:00
|
|
|
int ndx;
|
1999-10-16 21:49:28 +02:00
|
|
|
|
2006-03-07 20:06:50 +01:00
|
|
|
/*
|
|
|
|
* Do nothing if we're no longer interested in remembering free space.
|
|
|
|
*/
|
|
|
|
if (lts->forgetFreeSpace)
|
|
|
|
return;
|
|
|
|
|
1999-10-16 21:49:28 +02:00
|
|
|
/*
|
|
|
|
* Enlarge freeBlocks array if full.
|
|
|
|
*/
|
|
|
|
if (lts->nFreeBlocks >= lts->freeBlocksLen)
|
|
|
|
{
|
|
|
|
lts->freeBlocksLen *= 2;
|
|
|
|
lts->freeBlocks = (long *) repalloc(lts->freeBlocks,
|
2005-10-15 04:49:52 +02:00
|
|
|
lts->freeBlocksLen * sizeof(long));
|
1999-10-16 21:49:28 +02:00
|
|
|
}
|
2000-04-12 19:17:23 +02:00
|
|
|
|
1999-10-16 21:49:28 +02:00
|
|
|
/*
|
2006-10-04 02:30:14 +02:00
|
|
|
* Add blocknum to array, and mark the array unsorted if it's no longer in
|
|
|
|
* decreasing order.
|
1999-10-16 21:49:28 +02:00
|
|
|
*/
|
|
|
|
ndx = lts->nFreeBlocks++;
|
2006-03-08 00:46:24 +01:00
|
|
|
lts->freeBlocks[ndx] = blocknum;
|
2006-10-04 02:30:14 +02:00
|
|
|
if (ndx > 0 && lts->freeBlocks[ndx - 1] < blocknum)
|
2006-03-08 00:46:24 +01:00
|
|
|
lts->blocksSorted = false;
|
1999-10-16 21:49:28 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Create a set of logical tapes in a temporary underlying file.
|
|
|
|
*
|
|
|
|
* Each tape is initialized in write state.
|
|
|
|
*/
|
|
|
|
LogicalTapeSet *
|
|
|
|
LogicalTapeSetCreate(int ntapes)
|
|
|
|
{
|
|
|
|
LogicalTapeSet *lts;
|
2000-04-12 19:17:23 +02:00
|
|
|
LogicalTape *lt;
|
|
|
|
int i;
|
1999-10-16 21:49:28 +02:00
|
|
|
|
|
|
|
/*
|
2015-02-21 22:12:14 +01:00
|
|
|
* Create top-level struct including per-tape LogicalTape structs.
|
1999-10-16 21:49:28 +02:00
|
|
|
*/
|
|
|
|
Assert(ntapes > 0);
|
2015-02-21 22:12:14 +01:00
|
|
|
lts = (LogicalTapeSet *) palloc(offsetof(LogicalTapeSet, tapes) +
|
|
|
|
ntapes * sizeof(LogicalTape));
|
2007-06-07 21:19:57 +02:00
|
|
|
lts->pfile = BufFileCreateTemp(false);
|
2017-02-01 11:17:38 +01:00
|
|
|
lts->nBlocksAllocated = 0L;
|
|
|
|
lts->nBlocksWritten = 0L;
|
2006-03-07 20:06:50 +01:00
|
|
|
lts->forgetFreeSpace = false;
|
2006-03-08 00:46:24 +01:00
|
|
|
lts->blocksSorted = true; /* a zero-length array is sorted ... */
|
1999-10-16 21:49:28 +02:00
|
|
|
lts->freeBlocksLen = 32; /* reasonable initial guess */
|
|
|
|
lts->freeBlocks = (long *) palloc(lts->freeBlocksLen * sizeof(long));
|
|
|
|
lts->nFreeBlocks = 0;
|
|
|
|
lts->nTapes = ntapes;
|
2000-04-12 19:17:23 +02:00
|
|
|
|
1999-10-16 21:49:28 +02:00
|
|
|
/*
|
2016-12-22 17:45:00 +01:00
|
|
|
* Initialize per-tape structs. Note we allocate the I/O buffer and the
|
|
|
|
* first block for a tape only when it is first actually written to. This
|
|
|
|
* avoids wasting memory space when tuplesort.c overestimates the number
|
|
|
|
* of tapes needed.
|
1999-10-16 21:49:28 +02:00
|
|
|
*/
|
|
|
|
for (i = 0; i < ntapes; i++)
|
|
|
|
{
|
2006-02-19 06:58:36 +01:00
|
|
|
lt = <s->tapes[i];
|
1999-10-16 21:49:28 +02:00
|
|
|
lt->writing = true;
|
|
|
|
lt->frozen = false;
|
|
|
|
lt->dirty = false;
|
2016-12-22 17:45:00 +01:00
|
|
|
lt->firstBlockNumber = -1L;
|
|
|
|
lt->curBlockNumber = -1L;
|
2006-02-19 06:58:36 +01:00
|
|
|
lt->buffer = NULL;
|
Change the way pre-reading in external sort's merge phase works.
Don't pre-read tuples into SortTuple slots during merge. Instead, use the
memory for larger read buffers in logtape.c. We're doing the same number
of READTUP() calls either way, but managing the pre-read SortTuple slots
is much more complicated. Also, the on-tape representation is more compact
than SortTuples, so we can fit more pre-read tuples into the same amount
of memory this way. And we have better cache-locality, when we use just a
small number of SortTuple slots.
Now that we only hold one tuple from each tape in the SortTuple slots, we
can greatly simplify the "batch memory" management. We now maintain a
small set of fixed-sized slots, to hold the tuples, and fall back to
palloc() for larger tuples. We use this method during all merge phases,
not just the final merge, and also when randomAccess is requested, and
also in the TSS_SORTEDONTAPE case. In other words, it's used whenever we
do an external sort.
Reviewed by Peter Geoghegan and Claudio Freire.
Discussion: <CAM3SWZTpaORV=yQGVCG8Q4axcZ3MvF-05xe39ZvORdU9JcD6hQ@mail.gmail.com>
2016-10-03 12:37:49 +02:00
|
|
|
lt->buffer_size = 0;
|
1999-10-16 21:49:28 +02:00
|
|
|
lt->pos = 0;
|
|
|
|
lt->nbytes = 0;
|
|
|
|
}
|
|
|
|
return lts;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Close a logical tape set and release all resources.
|
|
|
|
*/
|
2000-04-12 19:17:23 +02:00
|
|
|
void
|
|
|
|
LogicalTapeSetClose(LogicalTapeSet *lts)
|
1999-10-16 21:49:28 +02:00
|
|
|
{
|
2000-04-12 19:17:23 +02:00
|
|
|
LogicalTape *lt;
|
|
|
|
int i;
|
1999-10-16 21:49:28 +02:00
|
|
|
|
|
|
|
BufFileClose(lts->pfile);
|
|
|
|
for (i = 0; i < lts->nTapes; i++)
|
|
|
|
{
|
2006-02-19 06:58:36 +01:00
|
|
|
lt = <s->tapes[i];
|
|
|
|
if (lt->buffer)
|
|
|
|
pfree(lt->buffer);
|
1999-10-16 21:49:28 +02:00
|
|
|
}
|
|
|
|
pfree(lts->freeBlocks);
|
|
|
|
pfree(lts);
|
|
|
|
}
|
|
|
|
|
2006-03-07 20:06:50 +01:00
|
|
|
/*
|
|
|
|
* Mark a logical tape set as not needing management of free space anymore.
|
|
|
|
*
|
|
|
|
* This should be called if the caller does not intend to write any more data
|
2014-05-06 18:12:18 +02:00
|
|
|
* into the tape set, but is reading from un-frozen tapes. Since no more
|
2006-03-07 20:06:50 +01:00
|
|
|
* writes are planned, remembering free blocks is no longer useful. Setting
|
|
|
|
* this flag lets us avoid wasting time and space in ltsReleaseBlock(), which
|
|
|
|
* is not designed to handle large numbers of free blocks.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
LogicalTapeSetForgetFreeSpace(LogicalTapeSet *lts)
|
|
|
|
{
|
|
|
|
lts->forgetFreeSpace = true;
|
|
|
|
}
|
|
|
|
|
1999-10-16 21:49:28 +02:00
|
|
|
/*
|
|
|
|
* Write to a logical tape.
|
|
|
|
*
|
2003-07-25 22:18:01 +02:00
|
|
|
* There are no error returns; we ereport() on failure.
|
1999-10-16 21:49:28 +02:00
|
|
|
*/
|
|
|
|
void
|
|
|
|
LogicalTapeWrite(LogicalTapeSet *lts, int tapenum,
|
|
|
|
void *ptr, size_t size)
|
|
|
|
{
|
2000-04-12 19:17:23 +02:00
|
|
|
LogicalTape *lt;
|
|
|
|
size_t nthistime;
|
1999-10-16 21:49:28 +02:00
|
|
|
|
|
|
|
Assert(tapenum >= 0 && tapenum < lts->nTapes);
|
2006-02-19 06:58:36 +01:00
|
|
|
lt = <s->tapes[tapenum];
|
1999-10-16 21:49:28 +02:00
|
|
|
Assert(lt->writing);
|
|
|
|
|
2016-12-22 17:45:00 +01:00
|
|
|
/* Allocate data buffer and first block on first write */
|
2006-02-19 06:58:36 +01:00
|
|
|
if (lt->buffer == NULL)
|
Change the way pre-reading in external sort's merge phase works.
Don't pre-read tuples into SortTuple slots during merge. Instead, use the
memory for larger read buffers in logtape.c. We're doing the same number
of READTUP() calls either way, but managing the pre-read SortTuple slots
is much more complicated. Also, the on-tape representation is more compact
than SortTuples, so we can fit more pre-read tuples into the same amount
of memory this way. And we have better cache-locality, when we use just a
small number of SortTuple slots.
Now that we only hold one tuple from each tape in the SortTuple slots, we
can greatly simplify the "batch memory" management. We now maintain a
small set of fixed-sized slots, to hold the tuples, and fall back to
palloc() for larger tuples. We use this method during all merge phases,
not just the final merge, and also when randomAccess is requested, and
also in the TSS_SORTEDONTAPE case. In other words, it's used whenever we
do an external sort.
Reviewed by Peter Geoghegan and Claudio Freire.
Discussion: <CAM3SWZTpaORV=yQGVCG8Q4axcZ3MvF-05xe39ZvORdU9JcD6hQ@mail.gmail.com>
2016-10-03 12:37:49 +02:00
|
|
|
{
|
2006-02-19 06:58:36 +01:00
|
|
|
lt->buffer = (char *) palloc(BLCKSZ);
|
Change the way pre-reading in external sort's merge phase works.
Don't pre-read tuples into SortTuple slots during merge. Instead, use the
memory for larger read buffers in logtape.c. We're doing the same number
of READTUP() calls either way, but managing the pre-read SortTuple slots
is much more complicated. Also, the on-tape representation is more compact
than SortTuples, so we can fit more pre-read tuples into the same amount
of memory this way. And we have better cache-locality, when we use just a
small number of SortTuple slots.
Now that we only hold one tuple from each tape in the SortTuple slots, we
can greatly simplify the "batch memory" management. We now maintain a
small set of fixed-sized slots, to hold the tuples, and fall back to
palloc() for larger tuples. We use this method during all merge phases,
not just the final merge, and also when randomAccess is requested, and
also in the TSS_SORTEDONTAPE case. In other words, it's used whenever we
do an external sort.
Reviewed by Peter Geoghegan and Claudio Freire.
Discussion: <CAM3SWZTpaORV=yQGVCG8Q4axcZ3MvF-05xe39ZvORdU9JcD6hQ@mail.gmail.com>
2016-10-03 12:37:49 +02:00
|
|
|
lt->buffer_size = BLCKSZ;
|
|
|
|
}
|
2016-12-22 17:45:00 +01:00
|
|
|
if (lt->curBlockNumber == -1)
|
2006-02-19 06:58:36 +01:00
|
|
|
{
|
2016-12-22 17:45:00 +01:00
|
|
|
Assert(lt->firstBlockNumber == -1);
|
|
|
|
Assert(lt->pos == 0);
|
|
|
|
|
|
|
|
lt->curBlockNumber = ltsGetFreeBlock(lts);
|
|
|
|
lt->firstBlockNumber = lt->curBlockNumber;
|
|
|
|
|
|
|
|
TapeBlockGetTrailer(lt->buffer)->prev = -1L;
|
2006-02-19 06:58:36 +01:00
|
|
|
}
|
|
|
|
|
Change the way pre-reading in external sort's merge phase works.
Don't pre-read tuples into SortTuple slots during merge. Instead, use the
memory for larger read buffers in logtape.c. We're doing the same number
of READTUP() calls either way, but managing the pre-read SortTuple slots
is much more complicated. Also, the on-tape representation is more compact
than SortTuples, so we can fit more pre-read tuples into the same amount
of memory this way. And we have better cache-locality, when we use just a
small number of SortTuple slots.
Now that we only hold one tuple from each tape in the SortTuple slots, we
can greatly simplify the "batch memory" management. We now maintain a
small set of fixed-sized slots, to hold the tuples, and fall back to
palloc() for larger tuples. We use this method during all merge phases,
not just the final merge, and also when randomAccess is requested, and
also in the TSS_SORTEDONTAPE case. In other words, it's used whenever we
do an external sort.
Reviewed by Peter Geoghegan and Claudio Freire.
Discussion: <CAM3SWZTpaORV=yQGVCG8Q4axcZ3MvF-05xe39ZvORdU9JcD6hQ@mail.gmail.com>
2016-10-03 12:37:49 +02:00
|
|
|
Assert(lt->buffer_size == BLCKSZ);
|
1999-10-16 21:49:28 +02:00
|
|
|
while (size > 0)
|
|
|
|
{
|
2016-12-22 17:45:00 +01:00
|
|
|
if (lt->pos >= TapeBlockPayloadSize)
|
1999-10-16 21:49:28 +02:00
|
|
|
{
|
|
|
|
/* Buffer full, dump it out */
|
2016-12-22 17:45:00 +01:00
|
|
|
long nextBlockNumber;
|
|
|
|
|
|
|
|
if (!lt->dirty)
|
1999-10-16 21:49:28 +02:00
|
|
|
{
|
|
|
|
/* Hmm, went directly from reading to writing? */
|
2003-07-25 22:18:01 +02:00
|
|
|
elog(ERROR, "invalid logtape state: should be dirty");
|
1999-10-16 21:49:28 +02:00
|
|
|
}
|
2016-12-22 17:45:00 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* First allocate the next block, so that we can store it in the
|
|
|
|
* 'next' pointer of this block.
|
|
|
|
*/
|
|
|
|
nextBlockNumber = ltsGetFreeBlock(lts);
|
|
|
|
|
|
|
|
/* set the next-pointer and dump the current block. */
|
|
|
|
TapeBlockGetTrailer(lt->buffer)->next = nextBlockNumber;
|
|
|
|
ltsWriteBlock(lts, lt->curBlockNumber, (void *) lt->buffer);
|
|
|
|
|
|
|
|
/* initialize the prev-pointer of the next block */
|
|
|
|
TapeBlockGetTrailer(lt->buffer)->prev = lt->curBlockNumber;
|
|
|
|
lt->curBlockNumber = nextBlockNumber;
|
1999-10-16 21:49:28 +02:00
|
|
|
lt->pos = 0;
|
|
|
|
lt->nbytes = 0;
|
|
|
|
}
|
|
|
|
|
2016-12-22 17:45:00 +01:00
|
|
|
nthistime = TapeBlockPayloadSize - lt->pos;
|
1999-10-16 21:49:28 +02:00
|
|
|
if (nthistime > size)
|
|
|
|
nthistime = size;
|
|
|
|
Assert(nthistime > 0);
|
|
|
|
|
|
|
|
memcpy(lt->buffer + lt->pos, ptr, nthistime);
|
|
|
|
|
|
|
|
lt->dirty = true;
|
|
|
|
lt->pos += nthistime;
|
|
|
|
if (lt->nbytes < lt->pos)
|
|
|
|
lt->nbytes = lt->pos;
|
|
|
|
ptr = (void *) ((char *) ptr + nthistime);
|
|
|
|
size -= nthistime;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2016-10-12 11:05:45 +02:00
|
|
|
* Rewind logical tape and switch from writing to reading.
|
1999-10-16 21:49:28 +02:00
|
|
|
*
|
2016-10-12 11:05:45 +02:00
|
|
|
* The tape must currently be in writing state, or "frozen" in read state.
|
|
|
|
*
|
2016-12-22 17:45:00 +01:00
|
|
|
* 'buffer_size' specifies how much memory to use for the read buffer.
|
|
|
|
* Regardless of the argument, the actual amount of memory used is between
|
|
|
|
* BLCKSZ and MaxAllocSize, and is a multiple of BLCKSZ. The given value is
|
|
|
|
* rounded down and truncated to fit those constraints, if necessary. If the
|
|
|
|
* tape is frozen, the 'buffer_size' argument is ignored, and a small BLCKSZ
|
|
|
|
* byte buffer is used.
|
1999-10-16 21:49:28 +02:00
|
|
|
*/
|
|
|
|
void
|
2016-10-12 11:05:45 +02:00
|
|
|
LogicalTapeRewindForRead(LogicalTapeSet *lts, int tapenum, size_t buffer_size)
|
1999-10-16 21:49:28 +02:00
|
|
|
{
|
2000-04-12 19:17:23 +02:00
|
|
|
LogicalTape *lt;
|
1999-10-16 21:49:28 +02:00
|
|
|
|
|
|
|
Assert(tapenum >= 0 && tapenum < lts->nTapes);
|
2006-02-19 06:58:36 +01:00
|
|
|
lt = <s->tapes[tapenum];
|
1999-10-16 21:49:28 +02:00
|
|
|
|
2016-10-12 11:05:45 +02:00
|
|
|
/*
|
|
|
|
* Round and cap buffer_size if needed.
|
|
|
|
*/
|
|
|
|
if (lt->frozen)
|
|
|
|
buffer_size = BLCKSZ;
|
|
|
|
else
|
1999-10-16 21:49:28 +02:00
|
|
|
{
|
2016-10-12 11:05:45 +02:00
|
|
|
/* need at least one block */
|
|
|
|
if (buffer_size < BLCKSZ)
|
|
|
|
buffer_size = BLCKSZ;
|
Change the way pre-reading in external sort's merge phase works.
Don't pre-read tuples into SortTuple slots during merge. Instead, use the
memory for larger read buffers in logtape.c. We're doing the same number
of READTUP() calls either way, but managing the pre-read SortTuple slots
is much more complicated. Also, the on-tape representation is more compact
than SortTuples, so we can fit more pre-read tuples into the same amount
of memory this way. And we have better cache-locality, when we use just a
small number of SortTuple slots.
Now that we only hold one tuple from each tape in the SortTuple slots, we
can greatly simplify the "batch memory" management. We now maintain a
small set of fixed-sized slots, to hold the tuples, and fall back to
palloc() for larger tuples. We use this method during all merge phases,
not just the final merge, and also when randomAccess is requested, and
also in the TSS_SORTEDONTAPE case. In other words, it's used whenever we
do an external sort.
Reviewed by Peter Geoghegan and Claudio Freire.
Discussion: <CAM3SWZTpaORV=yQGVCG8Q4axcZ3MvF-05xe39ZvORdU9JcD6hQ@mail.gmail.com>
2016-10-03 12:37:49 +02:00
|
|
|
|
2016-10-12 11:05:45 +02:00
|
|
|
/*
|
|
|
|
* palloc() larger than MaxAllocSize would fail (a multi-gigabyte
|
|
|
|
* buffer is unlikely to be helpful, anyway)
|
|
|
|
*/
|
|
|
|
if (buffer_size > MaxAllocSize)
|
|
|
|
buffer_size = MaxAllocSize;
|
Change the way pre-reading in external sort's merge phase works.
Don't pre-read tuples into SortTuple slots during merge. Instead, use the
memory for larger read buffers in logtape.c. We're doing the same number
of READTUP() calls either way, but managing the pre-read SortTuple slots
is much more complicated. Also, the on-tape representation is more compact
than SortTuples, so we can fit more pre-read tuples into the same amount
of memory this way. And we have better cache-locality, when we use just a
small number of SortTuple slots.
Now that we only hold one tuple from each tape in the SortTuple slots, we
can greatly simplify the "batch memory" management. We now maintain a
small set of fixed-sized slots, to hold the tuples, and fall back to
palloc() for larger tuples. We use this method during all merge phases,
not just the final merge, and also when randomAccess is requested, and
also in the TSS_SORTEDONTAPE case. In other words, it's used whenever we
do an external sort.
Reviewed by Peter Geoghegan and Claudio Freire.
Discussion: <CAM3SWZTpaORV=yQGVCG8Q4axcZ3MvF-05xe39ZvORdU9JcD6hQ@mail.gmail.com>
2016-10-03 12:37:49 +02:00
|
|
|
|
2016-10-12 11:05:45 +02:00
|
|
|
/* round down to BLCKSZ boundary */
|
|
|
|
buffer_size -= buffer_size % BLCKSZ;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (lt->writing)
|
|
|
|
{
|
|
|
|
/*
|
2016-12-22 17:45:00 +01:00
|
|
|
* Completion of a write phase. Flush last partial data block, and
|
|
|
|
* rewind for normal (destructive) read.
|
2016-10-12 11:05:45 +02:00
|
|
|
*/
|
|
|
|
if (lt->dirty)
|
2016-12-22 17:45:00 +01:00
|
|
|
{
|
|
|
|
TapeBlockSetNBytes(lt->buffer, lt->nbytes);
|
|
|
|
ltsWriteBlock(lts, lt->curBlockNumber, (void *) lt->buffer);
|
|
|
|
}
|
2016-10-12 11:05:45 +02:00
|
|
|
lt->writing = false;
|
1999-10-16 21:49:28 +02:00
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
/*
|
2016-10-12 11:05:45 +02:00
|
|
|
* This is only OK if tape is frozen; we rewind for (another) read
|
|
|
|
* pass.
|
1999-10-16 21:49:28 +02:00
|
|
|
*/
|
2016-10-12 11:05:45 +02:00
|
|
|
Assert(lt->frozen);
|
|
|
|
}
|
1999-10-16 21:49:28 +02:00
|
|
|
|
2016-10-12 11:05:45 +02:00
|
|
|
/* Allocate a read buffer (unless the tape is empty) */
|
|
|
|
if (lt->buffer)
|
|
|
|
pfree(lt->buffer);
|
|
|
|
lt->buffer = NULL;
|
|
|
|
lt->buffer_size = 0;
|
2016-12-22 17:45:00 +01:00
|
|
|
if (lt->firstBlockNumber != -1L)
|
2016-10-12 11:05:45 +02:00
|
|
|
{
|
|
|
|
lt->buffer = palloc(buffer_size);
|
|
|
|
lt->buffer_size = buffer_size;
|
|
|
|
}
|
Change the way pre-reading in external sort's merge phase works.
Don't pre-read tuples into SortTuple slots during merge. Instead, use the
memory for larger read buffers in logtape.c. We're doing the same number
of READTUP() calls either way, but managing the pre-read SortTuple slots
is much more complicated. Also, the on-tape representation is more compact
than SortTuples, so we can fit more pre-read tuples into the same amount
of memory this way. And we have better cache-locality, when we use just a
small number of SortTuple slots.
Now that we only hold one tuple from each tape in the SortTuple slots, we
can greatly simplify the "batch memory" management. We now maintain a
small set of fixed-sized slots, to hold the tuples, and fall back to
palloc() for larger tuples. We use this method during all merge phases,
not just the final merge, and also when randomAccess is requested, and
also in the TSS_SORTEDONTAPE case. In other words, it's used whenever we
do an external sort.
Reviewed by Peter Geoghegan and Claudio Freire.
Discussion: <CAM3SWZTpaORV=yQGVCG8Q4axcZ3MvF-05xe39ZvORdU9JcD6hQ@mail.gmail.com>
2016-10-03 12:37:49 +02:00
|
|
|
|
2016-10-12 11:05:45 +02:00
|
|
|
/* Read the first block, or reset if tape is empty */
|
2016-12-22 17:45:00 +01:00
|
|
|
lt->nextBlockNumber = lt->firstBlockNumber;
|
2016-10-12 11:05:45 +02:00
|
|
|
lt->pos = 0;
|
|
|
|
lt->nbytes = 0;
|
2016-12-22 17:45:00 +01:00
|
|
|
ltsReadFillBuffer(lts, lt);
|
2016-10-12 11:05:45 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Rewind logical tape and switch from reading to writing.
|
|
|
|
*
|
|
|
|
* NOTE: we assume the caller has read the tape to the end; otherwise
|
2017-01-30 09:51:20 +01:00
|
|
|
* untouched data will not have been freed. We could add more code to free
|
|
|
|
* any unread blocks, but in current usage of this module it'd be useless
|
|
|
|
* code.
|
2016-10-12 11:05:45 +02:00
|
|
|
*/
|
|
|
|
void
|
|
|
|
LogicalTapeRewindForWrite(LogicalTapeSet *lts, int tapenum)
|
|
|
|
{
|
|
|
|
LogicalTape *lt;
|
|
|
|
|
|
|
|
Assert(tapenum >= 0 && tapenum < lts->nTapes);
|
|
|
|
lt = <s->tapes[tapenum];
|
|
|
|
|
|
|
|
Assert(!lt->writing && !lt->frozen);
|
|
|
|
lt->writing = true;
|
|
|
|
lt->dirty = false;
|
2016-12-22 17:45:00 +01:00
|
|
|
lt->firstBlockNumber = -1L;
|
|
|
|
lt->curBlockNumber = -1L;
|
2016-10-12 11:05:45 +02:00
|
|
|
lt->pos = 0;
|
|
|
|
lt->nbytes = 0;
|
|
|
|
if (lt->buffer)
|
|
|
|
pfree(lt->buffer);
|
2016-12-22 17:45:00 +01:00
|
|
|
lt->buffer = NULL;
|
|
|
|
lt->buffer_size = 0;
|
1999-10-16 21:49:28 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Read from a logical tape.
|
|
|
|
*
|
|
|
|
* Early EOF is indicated by return value less than #bytes requested.
|
|
|
|
*/
|
|
|
|
size_t
|
|
|
|
LogicalTapeRead(LogicalTapeSet *lts, int tapenum,
|
|
|
|
void *ptr, size_t size)
|
|
|
|
{
|
2000-04-12 19:17:23 +02:00
|
|
|
LogicalTape *lt;
|
|
|
|
size_t nread = 0;
|
|
|
|
size_t nthistime;
|
1999-10-16 21:49:28 +02:00
|
|
|
|
|
|
|
Assert(tapenum >= 0 && tapenum < lts->nTapes);
|
2006-02-19 06:58:36 +01:00
|
|
|
lt = <s->tapes[tapenum];
|
2000-04-12 19:17:23 +02:00
|
|
|
Assert(!lt->writing);
|
1999-10-16 21:49:28 +02:00
|
|
|
|
|
|
|
while (size > 0)
|
|
|
|
{
|
|
|
|
if (lt->pos >= lt->nbytes)
|
|
|
|
{
|
|
|
|
/* Try to load more data into buffer. */
|
2016-12-22 17:45:00 +01:00
|
|
|
if (!ltsReadFillBuffer(lts, lt))
|
1999-10-16 21:49:28 +02:00
|
|
|
break; /* EOF */
|
|
|
|
}
|
|
|
|
|
|
|
|
nthistime = lt->nbytes - lt->pos;
|
|
|
|
if (nthistime > size)
|
|
|
|
nthistime = size;
|
|
|
|
Assert(nthistime > 0);
|
|
|
|
|
|
|
|
memcpy(ptr, lt->buffer + lt->pos, nthistime);
|
|
|
|
|
|
|
|
lt->pos += nthistime;
|
|
|
|
ptr = (void *) ((char *) ptr + nthistime);
|
|
|
|
size -= nthistime;
|
|
|
|
nread += nthistime;
|
|
|
|
}
|
|
|
|
|
|
|
|
return nread;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* "Freeze" the contents of a tape so that it can be read multiple times
|
|
|
|
* and/or read backwards. Once a tape is frozen, its contents will not
|
|
|
|
* be released until the LogicalTapeSet is destroyed. This is expected
|
|
|
|
* to be used only for the final output pass of a merge.
|
|
|
|
*
|
|
|
|
* This *must* be called just at the end of a write pass, before the
|
|
|
|
* tape is rewound (after rewind is too late!). It performs a rewind
|
2014-05-06 18:12:18 +02:00
|
|
|
* and switch to read mode "for free". An immediately following rewind-
|
1999-10-16 21:49:28 +02:00
|
|
|
* for-read call is OK but not necessary.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
LogicalTapeFreeze(LogicalTapeSet *lts, int tapenum)
|
|
|
|
{
|
2000-04-12 19:17:23 +02:00
|
|
|
LogicalTape *lt;
|
1999-10-16 21:49:28 +02:00
|
|
|
|
|
|
|
Assert(tapenum >= 0 && tapenum < lts->nTapes);
|
2006-02-19 06:58:36 +01:00
|
|
|
lt = <s->tapes[tapenum];
|
1999-10-16 21:49:28 +02:00
|
|
|
Assert(lt->writing);
|
|
|
|
|
|
|
|
/*
|
2016-12-22 17:45:00 +01:00
|
|
|
* Completion of a write phase. Flush last partial data block, and rewind
|
|
|
|
* for nondestructive read.
|
1999-10-16 21:49:28 +02:00
|
|
|
*/
|
|
|
|
if (lt->dirty)
|
2016-12-22 17:45:00 +01:00
|
|
|
{
|
|
|
|
TapeBlockSetNBytes(lt->buffer, lt->nbytes);
|
|
|
|
ltsWriteBlock(lts, lt->curBlockNumber, (void *) lt->buffer);
|
|
|
|
lt->writing = false;
|
|
|
|
}
|
1999-10-16 21:49:28 +02:00
|
|
|
lt->writing = false;
|
|
|
|
lt->frozen = true;
|
Change the way pre-reading in external sort's merge phase works.
Don't pre-read tuples into SortTuple slots during merge. Instead, use the
memory for larger read buffers in logtape.c. We're doing the same number
of READTUP() calls either way, but managing the pre-read SortTuple slots
is much more complicated. Also, the on-tape representation is more compact
than SortTuples, so we can fit more pre-read tuples into the same amount
of memory this way. And we have better cache-locality, when we use just a
small number of SortTuple slots.
Now that we only hold one tuple from each tape in the SortTuple slots, we
can greatly simplify the "batch memory" management. We now maintain a
small set of fixed-sized slots, to hold the tuples, and fall back to
palloc() for larger tuples. We use this method during all merge phases,
not just the final merge, and also when randomAccess is requested, and
also in the TSS_SORTEDONTAPE case. In other words, it's used whenever we
do an external sort.
Reviewed by Peter Geoghegan and Claudio Freire.
Discussion: <CAM3SWZTpaORV=yQGVCG8Q4axcZ3MvF-05xe39ZvORdU9JcD6hQ@mail.gmail.com>
2016-10-03 12:37:49 +02:00
|
|
|
|
|
|
|
/*
|
|
|
|
* The seek and backspace functions assume a single block read buffer.
|
|
|
|
* That's OK with current usage. A larger buffer is helpful to make the
|
|
|
|
* read pattern of the backing file look more sequential to the OS, when
|
|
|
|
* we're reading from multiple tapes. But at the end of a sort, when a
|
|
|
|
* tape is frozen, we only read from a single tape anyway.
|
|
|
|
*/
|
|
|
|
if (!lt->buffer || lt->buffer_size != BLCKSZ)
|
|
|
|
{
|
|
|
|
if (lt->buffer)
|
|
|
|
pfree(lt->buffer);
|
|
|
|
lt->buffer = palloc(BLCKSZ);
|
|
|
|
lt->buffer_size = BLCKSZ;
|
|
|
|
}
|
|
|
|
|
1999-10-16 21:49:28 +02:00
|
|
|
/* Read the first block, or reset if tape is empty */
|
2016-12-22 17:45:00 +01:00
|
|
|
lt->curBlockNumber = lt->firstBlockNumber;
|
1999-10-16 21:49:28 +02:00
|
|
|
lt->pos = 0;
|
|
|
|
lt->nbytes = 0;
|
2016-12-22 17:45:00 +01:00
|
|
|
|
|
|
|
if (lt->firstBlockNumber == -1L)
|
|
|
|
lt->nextBlockNumber = -1L;
|
|
|
|
ltsReadBlock(lts, lt->curBlockNumber, (void *) lt->buffer);
|
|
|
|
if (TapeBlockIsLast(lt->buffer))
|
|
|
|
lt->nextBlockNumber = -1L;
|
|
|
|
else
|
|
|
|
lt->nextBlockNumber = TapeBlockGetTrailer(lt->buffer)->next;
|
|
|
|
lt->nbytes = TapeBlockGetNBytes(lt->buffer);
|
1999-10-16 21:49:28 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2014-05-06 18:12:18 +02:00
|
|
|
* Backspace the tape a given number of bytes. (We also support a more
|
1999-10-16 21:49:28 +02:00
|
|
|
* general seek interface, see below.)
|
|
|
|
*
|
|
|
|
* *Only* a frozen-for-read tape can be backed up; we don't support
|
|
|
|
* random access during write, and an unfrozen read tape may have
|
|
|
|
* already discarded the desired data!
|
|
|
|
*
|
2016-12-22 17:45:00 +01:00
|
|
|
* Returns the number of bytes backed up. It can be less than the
|
|
|
|
* requested amount, if there isn't that much data before the current
|
|
|
|
* position. The tape is positioned to the beginning of the tape in
|
|
|
|
* that case.
|
1999-10-16 21:49:28 +02:00
|
|
|
*/
|
2016-12-22 17:45:00 +01:00
|
|
|
size_t
|
1999-10-16 21:49:28 +02:00
|
|
|
LogicalTapeBackspace(LogicalTapeSet *lts, int tapenum, size_t size)
|
|
|
|
{
|
2000-04-12 19:17:23 +02:00
|
|
|
LogicalTape *lt;
|
2016-12-22 17:45:00 +01:00
|
|
|
size_t seekpos = 0;
|
1999-10-16 21:49:28 +02:00
|
|
|
|
|
|
|
Assert(tapenum >= 0 && tapenum < lts->nTapes);
|
2006-02-19 06:58:36 +01:00
|
|
|
lt = <s->tapes[tapenum];
|
1999-10-16 21:49:28 +02:00
|
|
|
Assert(lt->frozen);
|
Change the way pre-reading in external sort's merge phase works.
Don't pre-read tuples into SortTuple slots during merge. Instead, use the
memory for larger read buffers in logtape.c. We're doing the same number
of READTUP() calls either way, but managing the pre-read SortTuple slots
is much more complicated. Also, the on-tape representation is more compact
than SortTuples, so we can fit more pre-read tuples into the same amount
of memory this way. And we have better cache-locality, when we use just a
small number of SortTuple slots.
Now that we only hold one tuple from each tape in the SortTuple slots, we
can greatly simplify the "batch memory" management. We now maintain a
small set of fixed-sized slots, to hold the tuples, and fall back to
palloc() for larger tuples. We use this method during all merge phases,
not just the final merge, and also when randomAccess is requested, and
also in the TSS_SORTEDONTAPE case. In other words, it's used whenever we
do an external sort.
Reviewed by Peter Geoghegan and Claudio Freire.
Discussion: <CAM3SWZTpaORV=yQGVCG8Q4axcZ3MvF-05xe39ZvORdU9JcD6hQ@mail.gmail.com>
2016-10-03 12:37:49 +02:00
|
|
|
Assert(lt->buffer_size == BLCKSZ);
|
1999-10-16 21:49:28 +02:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Easy case for seek within current block.
|
|
|
|
*/
|
|
|
|
if (size <= (size_t) lt->pos)
|
|
|
|
{
|
|
|
|
lt->pos -= (int) size;
|
2016-12-22 17:45:00 +01:00
|
|
|
return size;
|
1999-10-16 21:49:28 +02:00
|
|
|
}
|
2000-04-12 19:17:23 +02:00
|
|
|
|
1999-10-16 21:49:28 +02:00
|
|
|
/*
|
2016-12-22 17:45:00 +01:00
|
|
|
* Not-so-easy case, have to walk back the chain of blocks. This
|
|
|
|
* implementation would be pretty inefficient for long seeks, but we
|
|
|
|
* really aren't doing that (a seek over one tuple is typical).
|
1999-10-16 21:49:28 +02:00
|
|
|
*/
|
2016-12-22 17:45:00 +01:00
|
|
|
seekpos = (size_t) lt->pos; /* part within this block */
|
|
|
|
while (size > seekpos)
|
1999-10-16 21:49:28 +02:00
|
|
|
{
|
2016-12-22 17:45:00 +01:00
|
|
|
long prev = TapeBlockGetTrailer(lt->buffer)->prev;
|
1999-10-16 21:49:28 +02:00
|
|
|
|
2016-12-22 17:45:00 +01:00
|
|
|
if (prev == -1L)
|
1999-10-16 21:49:28 +02:00
|
|
|
{
|
2016-12-22 17:45:00 +01:00
|
|
|
/* Tried to back up beyond the beginning of tape. */
|
|
|
|
if (lt->curBlockNumber != lt->firstBlockNumber)
|
|
|
|
elog(ERROR, "unexpected end of tape");
|
|
|
|
lt->pos = 0;
|
|
|
|
return seekpos;
|
1999-10-16 21:49:28 +02:00
|
|
|
}
|
2016-12-22 17:45:00 +01:00
|
|
|
|
|
|
|
ltsReadBlock(lts, prev, (void *) lt->buffer);
|
|
|
|
|
|
|
|
if (TapeBlockGetTrailer(lt->buffer)->next != lt->curBlockNumber)
|
|
|
|
elog(ERROR, "broken tape, next of block %ld is %ld, expected %ld",
|
|
|
|
prev,
|
|
|
|
TapeBlockGetTrailer(lt->buffer)->next,
|
|
|
|
lt->curBlockNumber);
|
|
|
|
|
|
|
|
lt->nbytes = TapeBlockPayloadSize;
|
|
|
|
lt->curBlockNumber = prev;
|
|
|
|
lt->nextBlockNumber = TapeBlockGetTrailer(lt->buffer)->next;
|
|
|
|
|
|
|
|
seekpos += TapeBlockPayloadSize;
|
1999-10-16 21:49:28 +02:00
|
|
|
}
|
2016-12-22 17:45:00 +01:00
|
|
|
|
|
|
|
/*
|
|
|
|
* 'seekpos' can now be greater than 'size', because it points to the
|
|
|
|
* beginning the target block. The difference is the position within the
|
|
|
|
* page.
|
|
|
|
*/
|
|
|
|
lt->pos = seekpos - size;
|
|
|
|
return size;
|
1999-10-16 21:49:28 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Seek to an arbitrary position in a logical tape.
|
|
|
|
*
|
|
|
|
* *Only* a frozen-for-read tape can be seeked.
|
|
|
|
*
|
2016-12-22 17:45:00 +01:00
|
|
|
* Must be called with a block/offset previously returned by
|
|
|
|
* LogicalTapeTell().
|
1999-10-16 21:49:28 +02:00
|
|
|
*/
|
2016-12-22 17:45:00 +01:00
|
|
|
void
|
1999-10-16 21:49:28 +02:00
|
|
|
LogicalTapeSeek(LogicalTapeSet *lts, int tapenum,
|
|
|
|
long blocknum, int offset)
|
|
|
|
{
|
2000-04-12 19:17:23 +02:00
|
|
|
LogicalTape *lt;
|
1999-10-16 21:49:28 +02:00
|
|
|
|
|
|
|
Assert(tapenum >= 0 && tapenum < lts->nTapes);
|
2006-02-19 06:58:36 +01:00
|
|
|
lt = <s->tapes[tapenum];
|
1999-10-16 21:49:28 +02:00
|
|
|
Assert(lt->frozen);
|
2016-12-22 17:45:00 +01:00
|
|
|
Assert(offset >= 0 && offset <= TapeBlockPayloadSize);
|
Change the way pre-reading in external sort's merge phase works.
Don't pre-read tuples into SortTuple slots during merge. Instead, use the
memory for larger read buffers in logtape.c. We're doing the same number
of READTUP() calls either way, but managing the pre-read SortTuple slots
is much more complicated. Also, the on-tape representation is more compact
than SortTuples, so we can fit more pre-read tuples into the same amount
of memory this way. And we have better cache-locality, when we use just a
small number of SortTuple slots.
Now that we only hold one tuple from each tape in the SortTuple slots, we
can greatly simplify the "batch memory" management. We now maintain a
small set of fixed-sized slots, to hold the tuples, and fall back to
palloc() for larger tuples. We use this method during all merge phases,
not just the final merge, and also when randomAccess is requested, and
also in the TSS_SORTEDONTAPE case. In other words, it's used whenever we
do an external sort.
Reviewed by Peter Geoghegan and Claudio Freire.
Discussion: <CAM3SWZTpaORV=yQGVCG8Q4axcZ3MvF-05xe39ZvORdU9JcD6hQ@mail.gmail.com>
2016-10-03 12:37:49 +02:00
|
|
|
Assert(lt->buffer_size == BLCKSZ);
|
1999-10-16 21:49:28 +02:00
|
|
|
|
2016-12-22 17:45:00 +01:00
|
|
|
if (blocknum != lt->curBlockNumber)
|
1999-10-16 21:49:28 +02:00
|
|
|
{
|
2016-12-22 17:45:00 +01:00
|
|
|
ltsReadBlock(lts, blocknum, (void *) lt->buffer);
|
|
|
|
lt->curBlockNumber = blocknum;
|
|
|
|
lt->nbytes = TapeBlockPayloadSize;
|
|
|
|
lt->nextBlockNumber = TapeBlockGetTrailer(lt->buffer)->next;
|
1999-10-16 21:49:28 +02:00
|
|
|
}
|
2000-04-12 19:17:23 +02:00
|
|
|
|
2016-12-22 17:45:00 +01:00
|
|
|
if (offset > lt->nbytes)
|
|
|
|
elog(ERROR, "invalid tape seek position");
|
1999-10-16 21:49:28 +02:00
|
|
|
lt->pos = offset;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Obtain current position in a form suitable for a later LogicalTapeSeek.
|
|
|
|
*
|
|
|
|
* NOTE: it'd be OK to do this during write phase with intention of using
|
2014-05-06 18:12:18 +02:00
|
|
|
* the position for a seek after freezing. Not clear if anyone needs that.
|
1999-10-16 21:49:28 +02:00
|
|
|
*/
|
|
|
|
void
|
|
|
|
LogicalTapeTell(LogicalTapeSet *lts, int tapenum,
|
|
|
|
long *blocknum, int *offset)
|
|
|
|
{
|
2000-04-12 19:17:23 +02:00
|
|
|
LogicalTape *lt;
|
1999-10-16 21:49:28 +02:00
|
|
|
|
|
|
|
Assert(tapenum >= 0 && tapenum < lts->nTapes);
|
2006-02-19 06:58:36 +01:00
|
|
|
lt = <s->tapes[tapenum];
|
Change the way pre-reading in external sort's merge phase works.
Don't pre-read tuples into SortTuple slots during merge. Instead, use the
memory for larger read buffers in logtape.c. We're doing the same number
of READTUP() calls either way, but managing the pre-read SortTuple slots
is much more complicated. Also, the on-tape representation is more compact
than SortTuples, so we can fit more pre-read tuples into the same amount
of memory this way. And we have better cache-locality, when we use just a
small number of SortTuple slots.
Now that we only hold one tuple from each tape in the SortTuple slots, we
can greatly simplify the "batch memory" management. We now maintain a
small set of fixed-sized slots, to hold the tuples, and fall back to
palloc() for larger tuples. We use this method during all merge phases,
not just the final merge, and also when randomAccess is requested, and
also in the TSS_SORTEDONTAPE case. In other words, it's used whenever we
do an external sort.
Reviewed by Peter Geoghegan and Claudio Freire.
Discussion: <CAM3SWZTpaORV=yQGVCG8Q4axcZ3MvF-05xe39ZvORdU9JcD6hQ@mail.gmail.com>
2016-10-03 12:37:49 +02:00
|
|
|
|
|
|
|
/* With a larger buffer, 'pos' wouldn't be the same as offset within page */
|
|
|
|
Assert(lt->buffer_size == BLCKSZ);
|
|
|
|
|
1999-10-16 21:49:28 +02:00
|
|
|
*blocknum = lt->curBlockNumber;
|
|
|
|
*offset = lt->pos;
|
|
|
|
}
|
2005-10-19 00:59:37 +02:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Obtain total disk space currently used by a LogicalTapeSet, in blocks.
|
|
|
|
*/
|
|
|
|
long
|
|
|
|
LogicalTapeSetBlocks(LogicalTapeSet *lts)
|
|
|
|
{
|
2017-02-01 11:17:38 +01:00
|
|
|
return lts->nBlocksAllocated;
|
2005-10-19 00:59:37 +02:00
|
|
|
}
|