mirror of
https://git.postgresql.org/git/postgresql.git
synced 2024-08-22 08:23:22 +02:00
619 lines
18 KiB
Plaintext
619 lines
18 KiB
Plaintext
<!--
|
|
$Header: /cvsroot/pgsql/doc/src/sgml/lobj.sgml,v 1.30 2003/08/31 17:32:19 petere Exp $
|
|
-->
|
|
|
|
<chapter id="largeObjects">
|
|
<title id="largeObjects-title">Large Objects</title>
|
|
|
|
<indexterm zone="largeobjects"><primary>large object</></>
|
|
<indexterm><primary>BLOB</><see>large object</></>
|
|
|
|
<para>
|
|
In <productname>PostgreSQL</productname> releases prior to 7.1,
|
|
the size of any row in the database could not exceed the size of a
|
|
data page. Since the size of a data page is 8192 bytes (the
|
|
default, which can be raised up to 32768), the upper limit on the
|
|
size of a data value was relatively low. To support the storage of
|
|
larger atomic values, <productname>PostgreSQL</productname>
|
|
provided and continues to provide a large object interface. This
|
|
interface provides file-oriented access to user data that is stored in
|
|
a special large-object structure.
|
|
</para>
|
|
|
|
<para>
|
|
This chapter describes the implementation and the programming and
|
|
query language interfaces to <productname>PostgreSQL</productname>
|
|
large object data. We use the <application>libpq</application> C
|
|
library for the examples in this chapter, but most programming
|
|
interfaces native to <productname>PostgreSQL</productname> support
|
|
equivalent functionality. Other interfaces may use the large
|
|
object interface internally to provide generic support for large
|
|
values. This is not described here.
|
|
</para>
|
|
|
|
<sect1 id="lo-history">
|
|
<title>History</title>
|
|
|
|
<para>
|
|
<productname>POSTGRES 4.2</productname>, the indirect predecessor
|
|
of <productname>PostgreSQL</productname>, supported three standard
|
|
implementations of large objects: as files external to the
|
|
<productname>POSTGRES</productname> server, as external files
|
|
managed by the <productname>POSTGRES</productname> server, and as
|
|
data stored within the <productname>POSTGRES</productname>
|
|
database. This caused considerable confusion among users. As a
|
|
result, only support for large objects as data stored within the
|
|
database is retained in <productname>PostgreSQL</productname>.
|
|
Even though this is slower to access, it provides stricter data
|
|
integrity. For historical reasons, this storage scheme is
|
|
referred to as <firstterm>Inversion large
|
|
objects</firstterm>. (You will see the term Inversion used
|
|
occasionally to mean the same thing as large object.) Since
|
|
<productname>PostgreSQL 7.1</productname>, all large objects are
|
|
placed in one system table called
|
|
<classname>pg_largeobject</classname>.
|
|
</para>
|
|
|
|
<para>
|
|
<indexterm><primary>TOAST</></>
|
|
<indexterm><primary>sliced bread</><see>TOAST</></indexterm>
|
|
<productname>PostgreSQL 7.1</productname> introduced a mechanism
|
|
(nicknamed <quote><acronym>TOAST</acronym></quote>) that allows
|
|
data rows to be much larger than individual data pages. This
|
|
makes the large object interface partially obsolete. One
|
|
remaining advantage of the large object interface is that it allows values up
|
|
to 2 GB in size, whereas <acronym>TOAST</acronym> can only handle 1 GB.
|
|
</para>
|
|
|
|
</sect1>
|
|
|
|
<sect1 id="lo-implementation">
|
|
<title>Implementation Features</title>
|
|
|
|
<para>
|
|
The large object implementation breaks large
|
|
objects up into <quote>chunks</quote> and stores the chunks in
|
|
rows in the database. A B-tree index guarantees fast
|
|
searches for the correct chunk number when doing random
|
|
access reads and writes.
|
|
</para>
|
|
</sect1>
|
|
|
|
<sect1 id="lo-interfaces">
|
|
<title>Client Interfaces</title>
|
|
|
|
<para>
|
|
This section describes the facilities that
|
|
<productname>PostgreSQL</productname> client interface libraries
|
|
provide for accessing large objects. All large object
|
|
manipulation using these functions <emphasis>must</emphasis> take
|
|
place within an SQL transaction block. (This requirement is
|
|
strictly enforced as of <productname>PostgreSQL 6.5</>, though it
|
|
has been an implicit requirement in previous versions, resulting
|
|
in misbehavior if ignored.)
|
|
The <productname>PostgreSQL</productname> large object interface is modeled after
|
|
the <acronym>Unix</acronym> file-system interface, with analogues of
|
|
<function>open</function>, <function>read</function>,
|
|
<function>write</function>,
|
|
<function>lseek</function>, etc.
|
|
</para>
|
|
|
|
<para>
|
|
Client applications which use the large object interface in
|
|
<application>libpq</application> should include the header file
|
|
<filename>libpq/libpq-fs.h</filename> and link with the
|
|
<application>libpq</application> library.
|
|
</para>
|
|
|
|
<sect2>
|
|
<title>Creating a Large Object</title>
|
|
|
|
<para>
|
|
The function
|
|
<synopsis>
|
|
Oid lo_creat(PGconn *conn, int mode);
|
|
</synopsis>
|
|
<indexterm><primary>lo_creat</></>
|
|
creates a new large object.
|
|
<replaceable class="parameter">mode</replaceable> is a bit mask
|
|
describing several different attributes of the new
|
|
object. The symbolic constants listed here are defined
|
|
in the header file <filename>libpq/libpq-fs.h</filename>.
|
|
The access type (read, write, or both) is controlled by
|
|
or'ing together the bits <symbol>INV_READ</symbol> and
|
|
<symbol>INV_WRITE</symbol>. The low-order sixteen bits of the mask have
|
|
historically been used at Berkeley to designate the storage manager number on which the large object
|
|
should reside. These
|
|
bits should always be zero now.
|
|
The return value is the OID that was assigned to the new large object.
|
|
</para>
|
|
|
|
<para>
|
|
An example:
|
|
<programlisting>
|
|
inv_oid = lo_creat(INV_READ|INV_WRITE);
|
|
</programlisting>
|
|
</para>
|
|
</sect2>
|
|
|
|
<sect2>
|
|
<title>Importing a Large Object</title>
|
|
|
|
<para>
|
|
To import an operating system file as a large object, call
|
|
<synopsis>
|
|
Oid lo_import(PGconn *conn, const char *filename);
|
|
</synopsis>
|
|
<indexterm><primary>lo_import</></>
|
|
<replaceable class="parameter">filename</replaceable>
|
|
specifies the operating system name of
|
|
the file to be imported as a large object.
|
|
The return value is the OID that was assigned to the new large object.
|
|
</para>
|
|
</sect2>
|
|
|
|
<sect2>
|
|
<title>Exporting a Large Object</title>
|
|
|
|
<para>
|
|
To export a large object
|
|
into an operating system file, call
|
|
<synopsis>
|
|
int lo_export(PGconn *conn, Oid lobjId, const char *filename);
|
|
</synopsis>
|
|
<indexterm><primary>lo_export</></>
|
|
The <parameter>lobjId</parameter> argument specifies the OID of the large
|
|
object to export and the <parameter>filename</parameter> argument specifies
|
|
the operating system name name of the file.
|
|
</para>
|
|
</sect2>
|
|
|
|
<sect2>
|
|
<title>Opening an Existing Large Object</title>
|
|
|
|
<para>
|
|
To open an existing large object, call
|
|
<synopsis>
|
|
int lo_open(PGconn *conn, Oid lobjId, int mode);
|
|
</synopsis>
|
|
<indexterm><primary>lo_open</></>
|
|
The <parameter>lobjId</parameter> argument specifies the OID of the large
|
|
object to open. The <parameter>mode</parameter> bits control whether the
|
|
object is opened for reading (<symbol>INV_READ</>), writing (<symbol>INV_WRITE</symbol>), or
|
|
both.
|
|
A large object cannot be opened before it is created.
|
|
<function>lo_open</function> returns a large object descriptor
|
|
for later use in <function>lo_read</function>, <function>lo_write</function>,
|
|
<function>lo_lseek</function>, <function>lo_tell</function>, and
|
|
<function>lo_close</function>. The descriptor is only valid for
|
|
the duration of the current transaction.
|
|
</para>
|
|
</sect2>
|
|
|
|
<sect2>
|
|
<title>Writing Data to a Large Object</title>
|
|
|
|
<para>
|
|
The function
|
|
<synopsis>
|
|
int lo_write(PGconn *conn, int fd, const char *buf, size_t len);
|
|
</synopsis>
|
|
<indexterm><primary>lo_write</></> writes
|
|
<parameter>len</parameter> bytes from <parameter>buf</parameter>
|
|
to large object <parameter>fd</>. The <parameter>fd</parameter>
|
|
argument must have been returned by a previous
|
|
<function>lo_open</function>. The number of bytes actually
|
|
written is returned. In the event of an error, the return value
|
|
is negative.
|
|
</para>
|
|
</sect2>
|
|
|
|
<sect2>
|
|
<title>Reading Data from a Large Object</title>
|
|
|
|
<para>
|
|
The function
|
|
<synopsis>
|
|
int lo_read(PGconn *conn, int fd, char *buf, size_t len);
|
|
</synopsis>
|
|
<indexterm><primary>lo_read</></> reads
|
|
<parameter>len</parameter> bytes from large object
|
|
<parameter>fd</parameter> into <parameter>buf</parameter>. The
|
|
<parameter>fd</parameter> argument must have been returned by a
|
|
previous <function>lo_open</function>. The number of bytes
|
|
actually read is returned. In the event of an error, the return
|
|
value is negative.
|
|
</para>
|
|
</sect2>
|
|
|
|
<sect2>
|
|
<title>Seeking on a Large Object</title>
|
|
|
|
<para>
|
|
To change the current read or write location on a large
|
|
object, call
|
|
<synopsis>
|
|
int lo_lseek(PGconn *conn, int fd, int offset, int whence);
|
|
</synopsis>
|
|
<indexterm><primary>lo_lseek</></> This function moves the
|
|
current location pointer for the large object described by
|
|
<parameter>fd</> to the new location specified by
|
|
<parameter>offset</>. The valid values for <parameter>whence</>
|
|
are <symbol>SEEK_SET</> (seek from object start),
|
|
<symbol>SEEK_CUR</> (seek from current position), and
|
|
<symbol>SEEK_END</> (seek from object end). The return value is
|
|
the new location pointer.
|
|
</para>
|
|
</sect2>
|
|
|
|
<sect2>
|
|
<title>Obtaining the Seek Position of a Large Object</title>
|
|
|
|
<para>
|
|
To obtain the current read or write location of a large object,
|
|
call
|
|
<synopsis>
|
|
int lo_tell(PGconn *conn, int fd);
|
|
</synopsis>
|
|
<indexterm><primary>lo_tell</></> If there is an error, the
|
|
return value is negative.
|
|
</para>
|
|
</sect2>
|
|
|
|
<sect2>
|
|
<title>Closing a Large Object Descriptor</title>
|
|
|
|
<para>
|
|
A large object may be closed by calling
|
|
<synopsis>
|
|
int lo_close(PGconn *conn, int fd);
|
|
</synopsis>
|
|
<indexterm><primary>lo_close</></> where <parameter>fd</> is a
|
|
large object descriptor returned by <function>lo_open</function>.
|
|
On success, <function>lo_close</function> returns zero. On
|
|
error, the return value is negative.
|
|
</para>
|
|
|
|
<para>
|
|
Any large object descriptors that remain open at the end of a
|
|
transaction will be closed automatically.
|
|
</para>
|
|
</sect2>
|
|
|
|
<sect2>
|
|
<title>Removing a Large Object</title>
|
|
|
|
<para>
|
|
To remove a large object from the database, call
|
|
<synopsis>
|
|
int lo_unlink(PGconn *conn, Oid lobjId);
|
|
</synopsis>
|
|
<indexterm><primary>lo_unlink</></> The
|
|
<parameter>lobjId</parameter> argument specifies the OID of the
|
|
large object to remove. In the event of an error, the return
|
|
value is negative.
|
|
</para>
|
|
</sect2>
|
|
|
|
|
|
</sect1>
|
|
|
|
<sect1 id="lo-funcs">
|
|
<title>Server-side Functions</title>
|
|
|
|
<para>
|
|
There are two built-in server-side functions,
|
|
<function>lo_import</function><indexterm><primary>lo_import</></>
|
|
and
|
|
<function>lo_export</function>,<indexterm><primary>lo_export</></>
|
|
for large object access, which are available for use in
|
|
<acronym>SQL</acronym> commands. Here is an example of their
|
|
use:
|
|
<programlisting>
|
|
CREATE TABLE image (
|
|
name text,
|
|
raster oid
|
|
);
|
|
|
|
INSERT INTO image (name, raster)
|
|
VALUES ('beautiful image', lo_import('/etc/motd'));
|
|
|
|
SELECT lo_export(image.raster, '/tmp/motd') FROM image
|
|
WHERE name = 'beautiful image';
|
|
</programlisting>
|
|
</para>
|
|
|
|
<para>
|
|
These functions read and write files in the server's filesystem, using the
|
|
permissions of the database's owning user. Therefore, their use is restricted
|
|
to superusers. (In contrast, the client-side import and export functions
|
|
read and write files in the client's filesystem, using the permissions of
|
|
the client program. Their use is not restricted.)
|
|
</para>
|
|
</sect1>
|
|
|
|
<sect1 id="lo-examplesect">
|
|
<title>Example Program</title>
|
|
|
|
<para>
|
|
<xref linkend="lo-example"> is a sample program which shows how the large object
|
|
interface
|
|
in <application>libpq</> can be used. Parts of the program are
|
|
commented out but are left in the source for the reader's
|
|
benefit. This program can also be found in
|
|
<filename>src/test/examples/testlo.c</filename> in the source distribution.
|
|
</para>
|
|
|
|
<example id="lo-example">
|
|
<title>Large Objects with <application>libpq</application> Example Program</title>
|
|
<programlisting>
|
|
/*--------------------------------------------------------------
|
|
*
|
|
* testlo.c--
|
|
* test using large objects with libpq
|
|
*
|
|
* Copyright (c) 1994, Regents of the University of California
|
|
*
|
|
*--------------------------------------------------------------
|
|
*/
|
|
#include <stdio.h>
|
|
#include "libpq-fe.h"
|
|
#include "libpq/libpq-fs.h"
|
|
|
|
#define BUFSIZE 1024
|
|
|
|
/*
|
|
* importFile
|
|
* import file "in_filename" into database as large object "lobjOid"
|
|
*
|
|
*/
|
|
Oid
|
|
importFile(PGconn *conn, char *filename)
|
|
{
|
|
Oid lobjId;
|
|
int lobj_fd;
|
|
char buf[BUFSIZE];
|
|
int nbytes,
|
|
tmp;
|
|
int fd;
|
|
|
|
/*
|
|
* open the file to be read in
|
|
*/
|
|
fd = open(filename, O_RDONLY, 0666);
|
|
if (fd < 0)
|
|
{ /* error */
|
|
fprintf(stderr, "can't open unix file %s\n", filename);
|
|
}
|
|
|
|
/*
|
|
* create the large object
|
|
*/
|
|
lobjId = lo_creat(conn, INV_READ | INV_WRITE);
|
|
if (lobjId == 0)
|
|
fprintf(stderr, "can't create large object\n");
|
|
|
|
lobj_fd = lo_open(conn, lobjId, INV_WRITE);
|
|
|
|
/*
|
|
* read in from the Unix file and write to the inversion file
|
|
*/
|
|
while ((nbytes = read(fd, buf, BUFSIZE)) > 0)
|
|
{
|
|
tmp = lo_write(conn, lobj_fd, buf, nbytes);
|
|
if (tmp < nbytes)
|
|
fprintf(stderr, "error while reading large object\n");
|
|
}
|
|
|
|
(void) close(fd);
|
|
(void) lo_close(conn, lobj_fd);
|
|
|
|
return lobjId;
|
|
}
|
|
|
|
void
|
|
pickout(PGconn *conn, Oid lobjId, int start, int len)
|
|
{
|
|
int lobj_fd;
|
|
char *buf;
|
|
int nbytes;
|
|
int nread;
|
|
|
|
lobj_fd = lo_open(conn, lobjId, INV_READ);
|
|
if (lobj_fd < 0)
|
|
{
|
|
fprintf(stderr, "can't open large object %d\n",
|
|
lobjId);
|
|
}
|
|
|
|
lo_lseek(conn, lobj_fd, start, SEEK_SET);
|
|
buf = malloc(len + 1);
|
|
|
|
nread = 0;
|
|
while (len - nread > 0)
|
|
{
|
|
nbytes = lo_read(conn, lobj_fd, buf, len - nread);
|
|
buf[nbytes] = ' ';
|
|
fprintf(stderr, ">>> %s", buf);
|
|
nread += nbytes;
|
|
}
|
|
free(buf);
|
|
fprintf(stderr, "\n");
|
|
lo_close(conn, lobj_fd);
|
|
}
|
|
|
|
void
|
|
overwrite(PGconn *conn, Oid lobjId, int start, int len)
|
|
{
|
|
int lobj_fd;
|
|
char *buf;
|
|
int nbytes;
|
|
int nwritten;
|
|
int i;
|
|
|
|
lobj_fd = lo_open(conn, lobjId, INV_READ);
|
|
if (lobj_fd < 0)
|
|
{
|
|
fprintf(stderr, "can't open large object %d\n",
|
|
lobjId);
|
|
}
|
|
|
|
lo_lseek(conn, lobj_fd, start, SEEK_SET);
|
|
buf = malloc(len + 1);
|
|
|
|
for (i = 0; i < len; i++)
|
|
buf[i] = 'X';
|
|
buf[i] = ' ';
|
|
|
|
nwritten = 0;
|
|
while (len - nwritten > 0)
|
|
{
|
|
nbytes = lo_write(conn, lobj_fd, buf + nwritten, len - nwritten);
|
|
nwritten += nbytes;
|
|
}
|
|
free(buf);
|
|
fprintf(stderr, "\n");
|
|
lo_close(conn, lobj_fd);
|
|
}
|
|
|
|
/*
|
|
* exportFile * export large object "lobjOid" to file "out_filename"
|
|
*
|
|
*/
|
|
void
|
|
exportFile(PGconn *conn, Oid lobjId, char *filename)
|
|
{
|
|
int lobj_fd;
|
|
char buf[BUFSIZE];
|
|
int nbytes,
|
|
tmp;
|
|
int fd;
|
|
|
|
/*
|
|
* create an inversion "object"
|
|
*/
|
|
lobj_fd = lo_open(conn, lobjId, INV_READ);
|
|
if (lobj_fd < 0)
|
|
{
|
|
fprintf(stderr, "can't open large object %d\n",
|
|
lobjId);
|
|
}
|
|
|
|
/*
|
|
* open the file to be written to
|
|
*/
|
|
fd = open(filename, O_CREAT | O_WRONLY, 0666);
|
|
if (fd < 0)
|
|
{ /* error */
|
|
fprintf(stderr, "can't open unix file %s\n",
|
|
filename);
|
|
}
|
|
|
|
/*
|
|
* read in from the Unix file and write to the inversion file
|
|
*/
|
|
while ((nbytes = lo_read(conn, lobj_fd, buf, BUFSIZE)) > 0)
|
|
{
|
|
tmp = write(fd, buf, nbytes);
|
|
if (tmp < nbytes)
|
|
{
|
|
fprintf(stderr, "error while writing %s\n",
|
|
filename);
|
|
}
|
|
}
|
|
|
|
(void) lo_close(conn, lobj_fd);
|
|
(void) close(fd);
|
|
|
|
return;
|
|
}
|
|
|
|
void
|
|
exit_nicely(PGconn *conn)
|
|
{
|
|
PQfinish(conn);
|
|
exit(1);
|
|
}
|
|
|
|
int
|
|
main(int argc, char **argv)
|
|
{
|
|
char *in_filename,
|
|
*out_filename;
|
|
char *database;
|
|
Oid lobjOid;
|
|
PGconn *conn;
|
|
PGresult *res;
|
|
|
|
if (argc != 4)
|
|
{
|
|
fprintf(stderr, "Usage: %s database_name in_filename out_filename\n",
|
|
argv[0]);
|
|
exit(1);
|
|
}
|
|
|
|
database = argv[1];
|
|
in_filename = argv[2];
|
|
out_filename = argv[3];
|
|
|
|
/*
|
|
* set up the connection
|
|
*/
|
|
conn = PQsetdb(NULL, NULL, NULL, NULL, database);
|
|
|
|
/* check to see that the backend connection was successfully made */
|
|
if (PQstatus(conn) == CONNECTION_BAD)
|
|
{
|
|
fprintf(stderr, "Connection to database '%s' failed.\n", database);
|
|
fprintf(stderr, "%s", PQerrorMessage(conn));
|
|
exit_nicely(conn);
|
|
}
|
|
|
|
res = PQexec(conn, "begin");
|
|
PQclear(res);
|
|
|
|
printf("importing file %s\n", in_filename);
|
|
/* lobjOid = importFile(conn, in_filename); */
|
|
lobjOid = lo_import(conn, in_filename);
|
|
/*
|
|
printf("as large object %d.\n", lobjOid);
|
|
|
|
printf("picking out bytes 1000-2000 of the large object\n");
|
|
pickout(conn, lobjOid, 1000, 1000);
|
|
|
|
printf("overwriting bytes 1000-2000 of the large object with X's\n");
|
|
overwrite(conn, lobjOid, 1000, 1000);
|
|
*/
|
|
|
|
printf("exporting large object to file %s\n", out_filename);
|
|
/* exportFile(conn, lobjOid, out_filename); */
|
|
lo_export(conn, lobjOid, out_filename);
|
|
|
|
res = PQexec(conn, "end");
|
|
PQclear(res);
|
|
PQfinish(conn);
|
|
exit(0);
|
|
}
|
|
</programlisting>
|
|
</example>
|
|
|
|
</sect1>
|
|
</chapter>
|
|
|
|
<!-- Keep this comment at the end of the file
|
|
Local variables:
|
|
mode:sgml
|
|
sgml-omittag:nil
|
|
sgml-shorttag:t
|
|
sgml-minimize-attributes:nil
|
|
sgml-always-quote-attributes:t
|
|
sgml-indent-step:1
|
|
sgml-indent-data:t
|
|
sgml-parent-document:nil
|
|
sgml-default-dtd-file:"./reference.ced"
|
|
sgml-exposed-tags:nil
|
|
sgml-local-catalogs:("/usr/lib/sgml/catalog")
|
|
sgml-local-ecat-files:nil
|
|
End:
|
|
-->
|