>you will see changes for pg_config_paths.h and for thread safety. Can
>you look at those changes, merge them into bcc.mak. There might be
>other changes in there you want to add to bcc.mak too.
>
>
>
OK, having no M$ diff, I attach files src\interfaces\libpq\bcc32.mak and
src\interfaces\libpq\win32.h
Steve Holdoway
(.dylib format) on Mac OS X, while not messing up loadable modules for
the backend (which are the same kind of animal as a shared library on
every other platform, but not here). Also get the naming convention
to match OS X practice, viz libFOO.version.so not libFOO.so.version.
In support of that last, refactor code in Makefile.shlib to make it
easier to have platform-specific shlib naming conventions.
This patch is loosely based on the Fink project's current postgresql.patch.
Tested by yours truly on OS X 10.3.4; does anyone have 10.2.* to check
it on?
> user must initialize the lock. The problem are concurrent "first" users
> - the pthread_mutex_t initialization must be synchronized.
> The current implementation is broken, the attached patches fixes that:
> mutex_initlock is a spinlock. If the pthread_mutex_t mutex is not
> initialized, then the spinlock is acquired, if the pthread_mutex_t is
> initialized if it's not yet initialized and then the spinlock is dropped.
Manfred Spraul
user must initialize the lock. The problem are concurrent "first" users
- the pthread_mutex_t initialization must be synchronized.
The current implementation is broken, the attached patches fixes that:
mutex_initlock is a spinlock. If the pthread_mutex_t mutex is not
initialized, then the spinlock is acquired, if the pthread_mutex_t is
initialized if it's not yet initialized and then the spinlock is
dropped.
Manfred Spraul
extensive change then what was suggested. I found the file path.c that
contained a lot of "Unix/Windows" agnostic functions so I added a function
there instead and removed the PATHSEP declaration in exec.c altogether. All
to keep things from scattering all over the code.
I also took the liberty of changing the name of the functions
"first_path_sep" and "last_path_sep". Where I come from (and I'm apparently
not alone given the former macro name PATHSEP), they should be called
"first_dir_sep" and "last_dir_sep". The new function I introduced, that
actually finds path separators, is now the "first_path_sep". The patch
contains changes on all affected places of course.
I also changed the documentation on dynamic_library_path to reflect the
chagnes.
Thomas Hallgren
environment variable processing to libpq.
The patch also adds code to our client apps so we set the environment
variable directly based on our binary location, unless it is already
set. This will allow our applications to emit proper locale messages
that are generated in libpq.
the four functions.
> Also, please justify the temp-related changes. I was not aware that we
> had any breakage there.
patch-tmp-schema.txt contains the following bits:
*) Changes pg_namespace_aclmask() so that the superuser is always able
to create objects in the temp namespace.
*) Changes pg_namespace_aclmask() so that if this is a temp namespace,
objects are only allowed to be created in the temp namespace if the
user has TEMP privs on the database. This encompasses all object
creation, not just TEMP tables.
*) InitTempTableNamespace() checks to see if the current user, not the
session user, has access to create a temp namespace.
The first two changes are necessary to support the third change. Now
it's possible to revoke all temp table privs from non-super users and
limiting all creation of temp tables/schemas via a function that's
executed with elevated privs (security definer). Before this change,
it was not possible to have a setuid function to create a temp
table/schema if the session user had no TEMP privs.
patch-area-path.txt contains:
*) Can now determine the area of a closed path.
patch-dfmgr.txt contains:
*) Small tweak to add the library path that's being expanded.
I was using $lib/foo.so and couldn't easily figure out what the error
message, "invalid macro name in dynamic library path" meant without
looking through the source code. With the path in there, at least I
know where to start looking in my config file.
Sean Chittenden
several different module Makefiles with it. Also, do any adjustment
of installation paths during configure, rather than every time Makefile.global
is read.
conversion of basic ASCII letters. Remove all uses of strcasecmp and
strncasecmp in favor of new functions pg_strcasecmp and pg_strncasecmp;
remove most but not all direct uses of toupper and tolower in favor of
pg_toupper and pg_tolower. These functions use the same notions of
case folding already developed for identifier case conversion. I left
the straight locale-based folding in place for situations where we are
just manipulating user data and not trying to match it to built-in
strings --- for example, the SQL upper() function is still locale
dependent. Perhaps this will prove not to be what's wanted, but at
the moment we can initdb and pass regression tests in Turkish locale.
Allow additional thread flags to be added via port templates.
Change thread flag names to PTHREAD_CFLAGS and PTHREAD_LIBS to match new
configure script.
* removed a few redundant defines
* get_user_name safe under win32
* rationalized pipe read EOF for win32 (UPDATED PATCH USED)
* changed all backend instances of sleep() to pg_usleep
- except for the SLEEP_ON_ASSERT in assert.c, as it would exceed a
32-bit long [Note to patcher: If a SLEEP_ON_ASSERT of 2000 seconds is
acceptable, please replace with pg_usleep(2000000000L)]
I added a comment to that part of the code:
/*
* It would be nice to use pg_usleep() here, but only does 2000 sec
* or 33 minutes, which seems too short.
*/
sleep(1000000);
Claudio Natoli
Currently, src/interfaces/libpq/win32.mak builds a statically-linked
library "libpq.lib", a debug dll "libpq.dll", import library for the
debug dll "libpqdll.lib", a release dll "libpq.dll", import library for
the release dll "libpqdll.lib". To avoid naming clashes, I would make
the debug dll and import libraries "libpqd.dll" and "libpqddll.lib".
Basically, the debug build uses the cl flags: "/MDd /D _DEBUG", and the
release build uses the cl flags "/MD /D NDEBUG". Usually the debug
build has a "D" suffix on the file name, so for example:
libpqd.dll libpq, debug build
libpqd.lib libpq, debug build, import library
libpq.dll libpq, release build
libpq.lib libpq, release build, import library
David Turner
* configure + Makefile changes
* shared memory attaching in EXEC_BACKEND case (+ minor fix for apparent
cygwin bug under cygwin/EXEC_BACKEND case only)
* PATH env var separator differences
* missing win32 rand functions added
* placeholder replacements for sync etc under port.h
To those who are really interested, and there are a few of you: the attached
patch + file will allow the source base to be compiled (and, for some
definition, "run") under MingW, with the following caveats (I wanted to
first properly fix all but the last of these, but y'all won't quit asking
for a patch :-):
* child death: SIGCHLD not yet sent, so as a minimum, you'll need to
put in some sort of delay after StartupDatabase, and handle setting
StartupPID to 0 etc (ie. the stuff the reaper() signal function is supposed
to do)
* dirmod.c: comment out the elog calls
* dfmgr.c: some hackage required to substitute_libpath_macro
* slru/xact.c: comment out the errno checking after the readdir
(fixed by next version of MingW)
Again, this is only if you *really* want to see postgres compile and start,
and is a nice leg-up for working on the other Win32 TODO list items. Just
don't expect too much else from it at this point...
Claudio Natoli
ignore SIGPIPE from send() in libpq, but terminate on any other SIGPIPE,
unless the user installs their own signal handler.
This is a minor fix because the only time you get SIGPIPE from libpq's
send() is when the backend dies.
pointer type when it is not necessary to do so.
For future reference, casting NULL to a pointer type is only necessary
when (a) invoking a function AND either (b) the function has no prototype
OR (c) the function is a varargs function.
behavior of malloc and realloc when request size is 0. Fix escape
sequence recognizer so that only valid 3-digit octal sequences are
treated as escape sequences ... isdigit() is not a correct test.
will downcase the supplied field name unless it is double-quoted. Also,
upgrade the routine's handling of double quotes to match the backend,
in particular support doubled double quotes within quoted identifiers.
Per pgsql-interfaces discussion a couple weeks ago.
processing the request; this ensures that the request won't be taken
to cancel a subsequently-issued query. Race condition originally
noted by Oliver Jowett in the context of JDBC, but libpq has it too.
o allow configure to see include/port/win32 include files
o add matching Win32 accept() prototype
o allow pg_id to compile with native Win32 API
o fix invalide mbvalidate() function calls (existing bug)
o allow /scripts to compile with native Win32 API
o add win32.c to Win32 compiles (already in *.mak files)
libpq, talking to an old server, should assume SQL_ASCII as the default
client encoding, because that is what the server will actually use (not
the server encoding).
via extended query protocol, because it sends Sync right after Execute
without realizing that the command to be executed is COPY. There seems
to be no reasonable way for it to realize that, either, so the best fix
seems to be to make the backend ignore Sync during copy-in mode. Bit of
a wart on the protocol, but little alternative. Also, libpq must send
another Sync after terminating the COPY, if the command was issued via
Execute.
libpq users to perform Bind/Execute of previously prepared statements.
Per yesterday's discussion, this offers enough performance improvement
to justify bending the 'no new features during beta' rule.
completely useless error message in 'allow' case, don't retry connection
at the sendauth stage (by then the server will either let us in or not,
no point in wasting cycles on another try in the other SSL state).
psql4win32.patch - changes in the psql source code
psql-ref.patch - changes in the documentation psql-ref.sgml
(for new builtin variable WIN32_CONSOLE)
To apply them use "patch -p 1" in the root directory of the
postgres source directory.
These patches fix the following problems of psql on Win32
(all changes only have effect #ifdef WIN32):
a) Problem: Static library libpq.a did not work
Solution: Added WSAStartup() in fe-connect.c
b) Problem: Secret Password was echoed by psql
Solution: Password echoing disabled in sprompt.c
c) Problem: 8bit characters were displayed/interpreted wrong in psql
This is due to the fact that the Win32 "console" uses a
different encoding than the rest of the Windows system
Solution: Introduced a new psql variable WIN32_CONSOLE
When set with "\set WIN32_console", the function OemToChar()
is applied after reading input and CharToOem() before
displaying Output
Christoph Dalitz
modes (and replace the requiressl boolean). The four options were first
spelled out by Magnus Hagander <mha@sollentuna.net> on 2000-08-23 in email
to pgsql-hackers, archived here:
http://archives.postgresql.org/pgsql-hackers/2000-08/msg00639.php
My original less-flexible patch and the ensuing thread are archived at:
http://dbforums.com/t623845.html
Attached is a new patch, including documentation.
To sum up, there's a new client parameter "sslmode" and environment
variable "PGSSLMODE", with these options:
sslmode description
------- -----------
disable Unencrypted non-SSL only
allow Negotiate, prefer non-SSL
prefer Negotiate, prefer SSL (default)
require Require SSL
The only change to the server is a new pg_hba.conf line type,
"hostnossl", for specifying connections that are not allowed to use SSL
(for example, to prevent servers on a local network from accidentally
using SSL and wasting cycles). Thus the 3 pg_hba.conf line types are:
pg_hba.conf line types
----------------------
host applies to either SSL or regular connections
hostssl applies only to SSL connections
hostnossl applies only to regular connections
These client and server options, the postgresql.conf ssl = false option,
and finally the possibility of compiling with no SSL support at all,
make quite a range of combinations to test. I threw together a test
script to try many of them out. It's in a separate tarball with its
config files, a patch to psql so it'll announce SSL connections even in
absence of a tty, and the test output. The test is especially informative
when run on the same tty the postmaster was started on, so the FATAL:
errors during negotiation are interleaved with the psql client output.
I saw Tom write that new submissions for 7.4 have to be in before midnight
local time, and since I'm on the east coast in the US, this just makes it
in before the bell. :)
Jon Jensen
was modified for IPv6. Use a robust definition of struct sockaddr_storage,
do a proper configure test to see if ss_len exists, don't assume that
getnameinfo() will handle AF_UNIX sockets, don't trust getaddrinfo to
return the protocol we ask for, etc. This incorporates several outstanding
patches from Kurt Roeckx, but I'm to blame for anything that doesn't
work ...
> http://developer.postgresql.org/cvsweb.cgi/pgsql-server/src/include/libpq/pqcomm.h.diff?r1=1.85&r2=1.86
>
> modified SockAddr, but no corresponding change was made here
> (fe-auth.c:612):
>
> case AUTH_REQ_KRB5:
> #ifdef KRB5
> if (pg_krb5_sendauth(PQerrormsg, conn->sock, &conn->laddr.in,
> &conn->raddr.in,
> hostname) != STATUS_OK)
>
> It's not obvious to me what the change ought to be though.
This patch should hopefully fix both kerberos 4 and 5.
Kurt Roeckx
minute and a half to decode a 500Kb on a fairly fast machine. I think the
culprit is sscanf.
I attach a patch that replaces the function with one used to perform the same
task in pyPgSQL (a Python interface to PostgreSQL). This code was written by
Billy Allie, author of pyPgSQL. I've changed a few variable names to match
those in the original code and removed a bit of Pythonness.
Billy has kindly looked at the code and points out that it is slightly
stricter than the original implementation and if it encounters an invalid
bytea such as '\12C' it drops the unescape '\' and outputs '12C'.
The code is licensed by the author under a BSD license.
I've performed limited testing of the function by putting JPEGs into
PostgreSQL, extracting them using them using the new function and diffing
against the original files.
The new function is significantly faster on my machine with the JPEGs being
decoded in less than a second. I attach a modified libpq example program that
I used for my testing.
Ben Lamb.
protocol 3, then falls back to 2 if postmaster rejects the startup packet
with an old-format error message. A side benefit of the rewrite is that
SSL-encrypted connections can now be made without blocking. (I think,
anyway, but do not have a good way to test.)
Win32 port is now called 'win32' rather than 'win'
add -lwsock32 on Win32
make gethostname() be only used when kerberos4 is enabled
use /port/getopt.c
new /port/opendir.c routines
disable GUC unix_socket_group on Win32
convert some keywords.c symbols to KEYWORD_P to prevent conflict
create new FCNTL_NONBLOCK macro to turn off socket blocking
create new /include/port.h file that has /port prototypes, move
out of c.h
new /include/port/win32_include dir to hold missing include files
work around ERROR being defined in Win32 includes
handle multiple 'formats' for data I/O. Restructure CommandDest and
DestReceiver stuff one more time (it's finally starting to look a bit
clean though). Code now matches latest 3.0 protocol document as far
as message formats go --- but there is no support for binary I/O yet.
of Describe on a prepared statement. This was in the original 3.0
protocol proposal, but I took it out for reasons that seemed good at
the time. Put it back per yesterday's pghackers discussion.
for tableID/columnID in RowDescription. (The latter isn't really
implemented yet though --- the backend always sends zeroes, and libpq
just throws away the data.)
initial values and runtime changes in selected parameters. This gets
rid of the need for an initial 'select pg_client_encoding()' query in
libpq, bringing us back to one message transmitted in each direction
for a standard connection startup. To allow server version to be sent
using the same GUC mechanism that handles other parameters, invent the
concept of a never-settable GUC parameter: you can 'show server_version'
but it's not settable by any GUC input source. Create 'lc_collate' and
'lc_ctype' never-settable parameters so that people can find out these
settings without need for pg_controldata. (These side ideas were all
discussed some time ago in pgsql-hackers, but not yet implemented.)
rewritten and the protocol is changed, but most elog calls are still
elog calls. Also, we need to contemplate mechanisms for controlling
all this functionality --- eg, how much stuff should appear in the
postmaster log? And what API should libpq expose for it?
have length words. COPY OUT reimplemented per new protocol: it doesn't
need \. anymore, thank goodness. COPY BINARY to/from frontend works,
at least as far as the backend is concerned --- libpq's PQgetline API
is not up to snuff, and will have to be replaced with something that is
null-safe. libpq uses message length words for performance improvement
(no cycles wasted rescanning long messages), but not yet for error
recovery.
with variable-width fields. No more truncation of long user names.
Also, libpq can now send its environment-variable-driven SET commands
as part of the startup packet, saving round trips to server.
problems in applications that may have a large number of files open,
such that libpq's socket number exceeds the range supported by fd_set.
From Chris Brown.
correctly. See following thread for more details.
Subject: [HACKERS] client_encoding directive is ignored in postgresql.conf
From: Tatsuo Ishii <t-ishii@sra.co.jp>
Date: Wed, 29 Jan 2003 22:24:04 +0900 (JST)
believe I didn't notice this before -- once 64k was sent to/from the
server the client would crash. Basicly, in 7.3 the server SSL code set
the initial state to "about to renegotiate" without actually starting
the renegotiation. In addition, the server and client didn't properly
handle the SSL_ERROR_WANT_(READ|WRITE) error. This is fixed in the
second patch.
Nathan Mueller
first, that I missed when checking over 7.3.1, was that the client
method was switched to SSLv23 along with the server. The SSLv23 client
method does SSLv2 by default, but can also understand SSLv3. In our
situation the SSLv2 backwords compatibility is really only needed on the
server. This is the first patch.
The last thing is that I found a way for the server to understand SSLv2
HELLO messages (sent by pre-7.3 clients) but then get them to talk
SSLv3. This is the last one.
Nathan Mueller
"SSLv23_method(void), SSLv23_server_method(void), SSLv23_client_method(void)
A TLS/SSL connection established with these methods will understand the SSLv2,
SSLv3, and TLSv1 protocol. A client will send out SSLv2 client hello messages
and will indicate that it also understands SSLv3 and TLSv1. A server will
understand SSLv2, SSLv3, and TLSv1 client hello messages. This is the best
choice when compatibility is a concern."
This will maintain backwards compatibility for those us that don't use
TLS connections ...
postgresql version 7.3, but yea... this patch adds full IPv6
support to postgres. I've tested it out on 7.2.3 and has
been running perfectly stable.
CREDITS:
The KAME Project (Initial patch)
Nigel Kukard <nkukard@lbsd.net>
Johan Jordaan <johanj@lando.co.za>
that tv_sec is signed; return a useful error message on timeout failure;
honor PGCONNECT_TIMEOUT environment variable in PQsetdbLogin; make code
obey documentation statement that timeout=0 means no timeout.
client
utilities (libpq.dll and psql.exe) for win32 (missing defines,
adjustments to
includes, pedantic casting, non-existent functions) per:
http://developer.postgresql.org/docs/postgres/install-win32.html.
It compiles cleanly under Windows 2000 using Visual Studio .net. Also
compiles clean and passes all regression tests (regular and contrib)
under Linux.
In addition to a review by the usual suspects, it would be very
desirable for someone well versed in the peculiarities of win32 to take
a look.
Joe Conway
already fixed by You. However there were a few left and attached patch
should fix the rest of them.
I used StringInfo only in 2 places and both of them are inside debug
ifdefs. Only performance penalty will come from using strlen() like all
the other code does.
I also modified some of the already patched parts by changing
snprintf(buf, 2 * BUFSIZE, ... style lines to
snprintf(buf, sizeof(buf), ... where buf is an array.
Jukka Holappa
> > > > where is the code to verify that PGPASSWORDFILE points at a
> > > > non-world-readable file? That needs to be there now, not later, or
> > > > we'll have people moaning about backward compatibility when we finally
> > > > do plug that hole.
Alvaro Herrera
value '-2' is used to indicate a variable-width type whose width is
computed as strlen(datum)+1. Everything that looks at typlen is updated
except for array support, which Joe Conway is working on; at the moment
it wouldn't work to try to create an array of cstring.
PGPASSWORDFILE environment variable. I have modified libpq to make use
of this variable. I present the first cut here.
Currently the format for the file should be
host:port:database:user:password
Alvaro Herrera
.
So i took the opportunity to fix some stuff:
1. Made the thing compile (typos & needed definitions) with the new pqsecure_* s
tuff, and added fe-secure.c to the win32.mak makefile.
2. Fixed some MULTIBYTE compile errors (when building without MB support).
3. Made it do that you can build with debug info: "nmake -f win32.mak DEBUG=1".
4. Misc small compiler speedup changes.
The resulting .dll has been tested in production, and everything seems ok.
I CC:ed -hackers because i'm not sure about two things:
1. In libpq-int.h I typedef ssize_t as an int because Visual C (v6.0)
doesn't de fine ssize_t. Is that ok, or is there any standard about what
type should be use d for ssize_t?
2. To keep the .dll api consistent regarding MULTIBYTE I just return -1
in fe-connect.c:PQsetClientEncoding() instead of taking away the whole
function. I wonder if i should do any compares with the
conn->client_encoding and return 0 if not hing would have changed (if so
how do i check that?).
Regards
Magnus Naeslund
conversion procs and conversions are added in initdb. Currently
supported conversions are:
UTF-8(UNICODE) <--> SQL_ASCII, ISO-8859-1 to 16, EUC_JP, EUC_KR,
EUC_CN, EUC_TW, SJIS, BIG5, GBK, GB18030, UHC,
JOHAB, TCVN
EUC_JP <--> SJIS
EUC_TW <--> BIG5
MULE_INTERNAL <--> EUC_JP, SJIS, EUC_TW, BIG5
Note that initial contents of pg_conversion system catalog are created
in the initdb process. So doing initdb required is ideal, it's
possible to add them to your databases by hand, however. To accomplish
this:
psql -f your_postgresql_install_path/share/conversion_create.sql your_database
So I did not bump up the version in cataversion.h.
TODO:
Add more conversion procs
Add [CASCADE|RESTRICT] to DROP CONVERSION
Add tuples to pg_depend
Add regression tests
Write docs
Add SQL99 CONVERT command?
--
Tatsuo Ishii
files rather than a header file where they belong. Pay some modicum
of attention to picking global routine names that aren't likely to
conflict with surrounding applications.
I am no longer pursuing a total non-blocking implementation. I haven't
found a good way to test it with the type of work that I do with
PostgreSQL. I do use blocking SSL sockets with this mod and have had no
problem whatsoever. The bug that I fixed in this patch is exceptionally
hard to reproduce reliably.
Jack Bates
on the server, if DebugLvl >= 2.
The patch also includes a late addition to the last patch
(X509_check_private_key()). I'm not sure why it the currect
revision wasn't tagged.
Bear Giles
If the user has certificates in $HOME/.postgresql/postgresql.crt
and $HOME/.postgresql/postgresql.key exist, they are provided
to the server. The certificate used to sign this cert must be
known to the server, in $DataDir/root.crt. If successful, the
cert's "common name" is logged.
Client certs are not used for authentication, but they could be
via the port->peer (X509 *), port->peer_dn (char *) or
port->peer_cn (char *) fields. Or any other function could be
used, e.g., many sites like the issuer + serial number hash.
Bear Giles
As the comment headers in be-secure.c discusses, EPH preserves
confidentiality even if the static private key (which is usually
kept unencrypted) is compromised.
Because of the value of this, common default values are hard-coded
to protect the confidentiality of the data even if an attacker
successfully deletes or modifies the external file.
Bear Giles
Attached are a revised set of SSL patches. Many of these patches
are motivated by security concerns, it's not just bug fixes. The key
differences (from stock 7.2.1) are:
*) almost all code that directly uses the OpenSSL library is in two
new files,
src/interfaces/libpq/fe-ssl.c
src/backend/postmaster/be-ssl.c
in the long run, it would be nice to merge these two files.
*) the legacy code to read and write network data have been
encapsulated into read_SSL() and write_SSL(). These functions
should probably be renamed - they handle both SSL and non-SSL
cases.
the remaining code should eliminate the problems identified
earlier, albeit not very cleanly.
*) both front- and back-ends will send a SSL shutdown via the
new close_SSL() function. This is necessary for sessions to
work properly.
(Sessions are not yet fully supported, but by cleanly closing
the SSL connection instead of just sending a TCP FIN packet
other SSL tools will be much happier.)
*) The client certificate and key are now expected in a subdirectory
of the user's home directory. Specifically,
- the directory .postgresql must be owned by the user, and
allow no access by 'group' or 'other.'
- the file .postgresql/postgresql.crt must be a regular file
owned by the user.
- the file .postgresql/postgresql.key must be a regular file
owned by the user, and allow no access by 'group' or 'other'.
At the current time encrypted private keys are not supported.
There should also be a way to support multiple client certs/keys.
*) the front-end performs minimal validation of the back-end cert.
Self-signed certs are permitted, but the common name *must*
match the hostname used by the front-end. (The cert itself
should always use a fully qualified domain name (FDQN) in its
common name field.)
This means that
psql -h eris db
will fail, but
psql -h eris.example.com db
will succeed. At the current time this must be an exact match;
future patches may support any FQDN that resolves to the address
returned by getpeername(2).
Another common "problem" is expiring certs. For now, it may be
a good idea to use a very-long-lived self-signed cert.
As a compile-time option, the front-end can specify a file
containing valid root certificates, but it is not yet required.
*) the back-end performs minimal validation of the client cert.
It allows self-signed certs. It checks for expiration. It
supports a compile-time option specifying a file containing
valid root certificates.
*) both front- and back-ends default to TLSv1, not SSLv3/SSLv2.
*) both front- and back-ends support DSA keys. DSA keys are
moderately more expensive on startup, but many people consider
them preferable than RSA keys. (E.g., SSH2 prefers DSA keys.)
*) if /dev/urandom exists, both client and server will read 16k
of randomization data from it.
*) the server can read empheral DH parameters from the files
$DataDir/dh512.pem
$DataDir/dh1024.pem
$DataDir/dh2048.pem
$DataDir/dh4096.pem
if none are provided, the server will default to hardcoded
parameter files provided by the OpenSSL project.
Remaining tasks:
*) the select() clauses need to be revisited - the SSL abstraction
layer may need to absorb more of the current code to avoid rare
deadlock conditions. This also touches on a true solution to
the pg_eof() problem.
*) the SIGPIPE signal handler may need to be revisited.
*) support encrypted private keys.
*) sessions are not yet fully supported. (SSL sessions can span
multiple "connections," and allow the client and server to avoid
costly renegotiations.)
*) makecert - a script that creates back-end certs.
*) pgkeygen - a tool that creates front-end certs.
*) the whole protocol issue, SASL, etc.
*) certs are fully validated - valid root certs must be available.
This is a hassle, but it means that you *can* trust the identity
of the server.
*) the client library can handle hardcoded root certificates, to
avoid the need to copy these files.
*) host name of server cert must resolve to IP address, or be a
recognized alias. This is more liberal than the previous
iteration.
*) the number of bytes transferred is tracked, and the session
key is periodically renegotiated.
*) basic cert generation scripts (mkcert.sh, pgkeygen.sh). The
configuration files have reasonable defaults for each type
of use.
Bear Giles
are motivated by security concerns, it's not just bug fixes. The key
differences (from stock 7.2.1) are:
*) almost all code that directly uses the OpenSSL library is in two
new files,
src/interfaces/libpq/fe-ssl.c
src/backend/postmaster/be-ssl.c
in the long run, it would be nice to merge these two files.
*) the legacy code to read and write network data have been
encapsulated into read_SSL() and write_SSL(). These functions
should probably be renamed - they handle both SSL and non-SSL
cases.
the remaining code should eliminate the problems identified
earlier, albeit not very cleanly.
*) both front- and back-ends will send a SSL shutdown via the
new close_SSL() function. This is necessary for sessions to
work properly.
(Sessions are not yet fully supported, but by cleanly closing
the SSL connection instead of just sending a TCP FIN packet
other SSL tools will be much happier.)
*) The client certificate and key are now expected in a subdirectory
of the user's home directory. Specifically,
- the directory .postgresql must be owned by the user, and
allow no access by 'group' or 'other.'
- the file .postgresql/postgresql.crt must be a regular file
owned by the user.
- the file .postgresql/postgresql.key must be a regular file
owned by the user, and allow no access by 'group' or 'other'.
At the current time encrypted private keys are not supported.
There should also be a way to support multiple client certs/keys.
*) the front-end performs minimal validation of the back-end cert.
Self-signed certs are permitted, but the common name *must*
match the hostname used by the front-end. (The cert itself
should always use a fully qualified domain name (FDQN) in its
common name field.)
This means that
psql -h eris db
will fail, but
psql -h eris.example.com db
will succeed. At the current time this must be an exact match;
future patches may support any FQDN that resolves to the address
returned by getpeername(2).
Another common "problem" is expiring certs. For now, it may be
a good idea to use a very-long-lived self-signed cert.
As a compile-time option, the front-end can specify a file
containing valid root certificates, but it is not yet required.
*) the back-end performs minimal validation of the client cert.
It allows self-signed certs. It checks for expiration. It
supports a compile-time option specifying a file containing
valid root certificates.
*) both front- and back-ends default to TLSv1, not SSLv3/SSLv2.
*) both front- and back-ends support DSA keys. DSA keys are
moderately more expensive on startup, but many people consider
them preferable than RSA keys. (E.g., SSH2 prefers DSA keys.)
*) if /dev/urandom exists, both client and server will read 16k
of randomization data from it.
*) the server can read empheral DH parameters from the files
$DataDir/dh512.pem
$DataDir/dh1024.pem
$DataDir/dh2048.pem
$DataDir/dh4096.pem
if none are provided, the server will default to hardcoded
parameter files provided by the OpenSSL project.
Remaining tasks:
*) the select() clauses need to be revisited - the SSL abstraction
layer may need to absorb more of the current code to avoid rare
deadlock conditions. This also touches on a true solution to
the pg_eof() problem.
*) the SIGPIPE signal handler may need to be revisited.
*) support encrypted private keys.
*) sessions are not yet fully supported. (SSL sessions can span
multiple "connections," and allow the client and server to avoid
costly renegotiations.)
*) makecert - a script that creates back-end certs.
*) pgkeygen - a tool that creates front-end certs.
*) the whole protocol issue, SASL, etc.
*) certs are fully validated - valid root certs must be available.
This is a hassle, but it means that you *can* trust the identity
of the server.
*) the client library can handle hardcoded root certificates, to
avoid the need to copy these files.
*) host name of server cert must resolve to IP address, or be a
recognized alias. This is more liberal than the previous
iteration.
*) the number of bytes transferred is tracked, and the session
key is periodically renegotiated.
*) basic cert generation scripts (mkcert.sh, pgkeygen.sh). The
configuration files have reasonable defaults for each type
of use.
Bear Giles
> Changes to avoid collisions with WIN32 & MFC names...
> 1. Renamed:
> a. PROC => PGPROC
> b. GetUserName() => GetUserNameFromId()
> c. GetCurrentTime() => GetCurrentDateTime()
> d. IGNORE => IGNORE_DTF in include/utils/datetime.h & utils/adt/datetim
>
> 2. Added _P to some lex/yacc tokens:
> CONST, CHAR, DELETE, FLOAT, GROUP, IN, OUT
Jan
work on all win9x machines, so i made it go thru a l ookup table
instead, using the DLL as last resort. I also moved this out of the
fe-misc.c file because of the size of the lookup ta ble. Who knows, we
might add more other win32 specific code there in the future.
I also fixed a small typo in the pg_config.h.win32 that made the
compiler compla in about the gnu snprintf declaration.
I tried to make this patch with psql coding style. I've successfully
tested this on win2k and win98 and it works fine (i.e. the mes sage
shows on win98 too, it didn't with the old implementation).
Magnus Naeslund
In summary, if a software writer implements timer events or other events
which generate a signal with a timing fast enough to occur while libpq
is inside connect(), then connect returns -EINTR. The code following
the connect call does not handle this and generates an error message.
The sum result is that the pg_connect() fails. If the timer or other
event is right on the window of the connect() completion time, the
pg_connect() may appear to work sporadically. If the event is too slow,
pg_connect() will appear to always work and if the event is too fast,
pg_connect() will always fail.
David Ford
This is necessary for mulibyte character sequences.
See "[HACKERS] PQescapeBytea is not multibyte aware" thread posted around
2002/04/05 for more details.
o Change all current CVS messages of NOTICE to WARNING. We were going
to do this just before 7.3 beta but it has to be done now, as you will
see below.
o Change current INFO messages that should be controlled by
client_min_messages to NOTICE.
o Force remaining INFO messages, like from EXPLAIN, VACUUM VERBOSE, etc.
to always go to the client.
o Remove INFO from the client_min_messages options and add NOTICE.
Seems we do need three non-ERROR elog levels to handle the various
behaviors we need for these messages.
Regression passed.
queries over non-blocking connections with libpq. "Larger" here
basically means that it doesn't fit into the output buffer.
The basic strategy is to fix pqFlush and pqPutBytes.
The problem with pqFlush as it stands now is that it returns EOF when an
error occurs or when not all data could be sent. The latter case is
clearly not an error for a non-blocking connection but the caller can't
distringuish it from an error very well.
The first part of the fix is therefore to fix pqFlush. This is done by
to renaming it to pqSendSome which only differs from pqFlush in its
return values to allow the caller to make the above distinction and a
new pqFlush which is implemented in terms of pqSendSome and behaves
exactly like the old pqFlush.
The second part of the fix modifies pqPutBytes to use pqSendSome instead
of pqFlush and to either send all the data or if not all data can be
sent on a non-blocking connection to at least put all data into the
output buffer, enlarging it if necessary. The callers of pqPutBytes
don't have to be changed because from their point of view pqPutBytes
behaves like before. It either succeeds in queueing all output data or
fails with an error.
I've also added a new API function PQsendSome which analogously to
PQflush just calls pqSendSome. Programs using non-blocking queries
should use this new function. The main difference is that this function
will have to be called repeatedly (calling select() properly in between)
until all data has been written.
AFAICT, the code in CVS HEAD hasn't changed with respect to non-blocking
queries and this fix should work there, too, but I haven't tested that
yet.
Bernhard Herzog
Magnus Hagander that DLL only contains error strings for the Net***
functions, *not* WinSock. We need to look for a workable solution for
older Windows flavors ... but it won't happen for PG 7.2.
one fuzzy translation fix, some
other messages tweaking. Theoretically,
should be up-to-date by now.
Please apply to /src/interfaces/libpq/ru.po
--
Serguei A. Mokhov
an already installed iODBC or unixODBC driver manager. In particular,
use the include files provided by the driver manager over our own,
and use the odbcinst library of the driver manager rather than gpps.c.
Migrate portability sections common to several files into psqlodbc.h.
>
> 1. Now outputs '\\' instead of '\134' when using encode(bytea, 'escape')
> Note that I ended up leaving \0 as \000 so that there are no ambiguities
> when decoding something like, for example, \0123.
>
> 2. Fixed bug in byteain which allowed input values which were not valid
> octals (e.g. \789), to be parsed as if they were octals.
>
> Joe
>
Here's rev 2 of the bytea string support patch. Changes:
1. Added missing declaration for MatchBytea function
2. Added PQescapeBytea to fe-exec.c
3. Applies cleanly on cvs tip from this afternoon
I'm hoping that someone can review/approve/apply this before beta starts, so
I guess I'd vote (not that it counts for much) to delay beta a few days :-)
Joe Conway
> null bytes to be literally '\0', the following can happen:
> 1. User inputs string value as "<null byte>##" where ## are digits in the
> range of 0 to 7.
> 2. PQescapeString converts this to "\0##"
> 3. Escaped string is used in a context that causes "\0##" to be evaluated as
> an octal escape sequence.
I agree that this is a problem, though it is not possible to do
anything harmful with it. In addition, it only occurs if there are
any NUL characters in its input, which is very unlikely if you are
using C strings.
The patch below addresses the issue by removing escaping of \0
characters entirely.
> If the goal is to "safely" encode null bytes, and preserve the rest of the
> string as it was entered, I think the null bytes should be escaped as \\000
> (note that if you simply use \000 the same string truncation problem
> occurs).
We can't do that, this would require 4n + 1 bytes of storage for the
result, breaking the interface.
Florian Weimer
discussion on pgsql-hackers (especially the frightening memory dump in
<12273.999562219@sss.pgh.pa.us>), we decided that it is best not to
use identifiers from an untrusted source at all. Therefore, all
claims of the suitability of PQescapeString() for identifiers have
been removed.
Florian Weimer
-------------------------------------------------------------------
Subject: Re: [PATCHES] encoding names
From: Karel Zak <zakkr@zf.jcu.cz>
To: Peter Eisentraut <peter_e@gmx.net>
Cc: pgsql-patches <pgsql-patches@postgresql.org>
Date: Fri, 31 Aug 2001 17:24:38 +0200
On Thu, Aug 30, 2001 at 01:30:40AM +0200, Peter Eisentraut wrote:
> > - convert encoding 'name' to 'id'
>
> I thought we decided not to add functions returning "new" names until we
> know exactly what the new names should be, and pending schema
Ok, the patch not to add functions.
> better
>
> ...(): encoding name too long
Fixed.
I found new bug in command/variable.c in parse_client_encoding(), nobody
probably never see this error:
if (pg_set_client_encoding(encoding))
{
elog(ERROR, "Conversion between %s and %s is not supported",
value, GetDatabaseEncodingName());
}
because pg_set_client_encoding() returns -1 for error and 0 as true.
It's fixed too.
IMHO it can be apply.
Karel
PS:
* following files are renamed:
src/utils/mb/Unicode/KOI8_to_utf8.map -->
src/utils/mb/Unicode/koi8r_to_utf8.map
src/utils/mb/Unicode/WIN_to_utf8.map -->
src/utils/mb/Unicode/win1251_to_utf8.map
src/utils/mb/Unicode/utf8_to_KOI8.map -->
src/utils/mb/Unicode/utf8_to_koi8r.map
src/utils/mb/Unicode/utf8_to_WIN.map -->
src/utils/mb/Unicode/utf8_to_win1251.map
* new file:
src/utils/mb/encname.c
* removed file:
src/utils/mb/common.c
--
Karel Zak <zakkr@zf.jcu.cz>
http://home.zf.jcu.cz/~zakkr/
C, PostgreSQL, PHP, WWW, http://docs.linux.cz, http://mape.jcu.cz
two additional files win32.mak and libpgtcl.def.
This patch allows to compile libpgtcl.dll on Windows
with tcl > 8.0. I've tested it on WinNT (VC6.0), SUSE Linux (7.0)
and Solaris 2.6 with tcl 8.3.3.
Mikhail Terekhov
really played it totally safe in my last suggestion, the system table might
pick up the msg but not the netmsg.dll, so better try both.
I also added a hex printout of the "errno" appended to all messages, that's
nicer.
If anyone hate my coding style, or that i'm using goto constructs, just tell
me, and i'll rework it into a nested if () thing.
Magnus Naeslund(f)
Client headers are no longer in a subdirectory, since they have been made
namespace-clean.
Internal libpq headers are in a private subdirectory.
Server headers are in a private subdirectory. pg_config has a new option
to point there.
> It seems that win9x doesn't have the "netmsg.dll" so it defaults to "normal"
> FormatMessage.
> I wonder if one could load wsock32.dll or winsock.dll on those systems
> instead of netmsg.dll.
>
> Mikhail, could you please test this code on your nt4 system?
> Could someone else test this code on a win98/95 system?
>
> It works on win2k over here.
It works on win2k here too but not on win98/95 or winNT.
Anyway, attached is the patch which uses Magnus's my_sock_strerror
function (renamed to winsock_strerror). The only difference is that
I put the code to load and unload netmsg.dll in the libpqdll.c
(is this OK Magnus?).
Mikhail Terekhov
Allow pg_shadow to be MD5 encrypted.
Add ENCRYPTED/UNENCRYPTED option to CREATE/ALTER user.
Add password_encryption postgresql.conf option.
Update wire protocol version to 2.1.
changes on this new source to make non-blocking connection work. I
tested it, and PQSendQuery and PQGetResult are working fine.
In win32.h I added one line:
#define snprintf _snprintf
Darko Prenosil
functions do not set errno, so some normal conditions are treated as
fatal errors. e.g. fetching large tuples fails, as at some point recv()
returns EWOULDBLOCK. here's a patch, which replaces errno with
WSAGetLastError(). i've tried to to affect non-win32 code.
Dmitry Yurtaev
that not many people actually use libpq on Win32; I have found another bug. Some
functions that are defined in libpq-fe.h aren't exported in the DLL version of
the library. I have added them to src/interfaces/libpq/libpqdll.def. The new
complete file is attached.
Gerhard H?ring
ready. It appears that most (all?) Unixen will consider a socket to
be read or write ready if it has an error condition, but of course
Microsoft does things differently.
are now separate files "postgres.h" and "postgres_fe.h", which are meant
to be the primary include files for backend .c files and frontend .c files
respectively. By default, only include files meant for frontend use are
installed into the installation include directory. There is a new make
target 'make install-all-headers' that adds the whole content of the
src/include tree to the installed fileset, for use by people who want to
develop server-side code without keeping the complete source tree on hand.
Cleaned up a whole lot of crufty and inconsistent header inclusions.
and two 'win32.mak'. Addresses the following:
1) Oops. Spelled fcntl.h wrong in the last one. D'uh.
2) PG_VERSION changed to be defined with " around it. psql/command.c failed
to compile without that.
3) Changed makefiles to use "/MD" and link both psql and libpq.dll against
MSVCRT.DLL instead of a static library. This takes care of the
crash-upon-free in psql.
I *think* this is what is on the "Open 7.1 Items" list as "Magnus Hagander
ODBC Issues?". It has nothing to do with ODBC, but it's the only issue I've
been involved with...
Magnus Hagander
If pghost == "" and pgport == "" then PQsetdbLogin() fails with a
error message:
Is the postmaster running locally
and accepting connections on Unix socket '/tmp/.s.PGSQL.0'?
I see many applications such as PHP fails due to this behavior.
Now if pgport == "", then it is assumed to be a DEF_PGPORT_STR. This
is the same behavior as the version prior 7.1.
might change it. Experimentation shows that the signal handler call
mechanism does not save/restore errno for you, at least not on Linux
or HPUX, so this is definitely a real risk.
Context diff this time.
Remove -m486 compile args for FreeBSD-i386, compile -O2 on i386.
Compile with only -O on alpha for codegen safety.
Make the port use the TEST_AND_SET for alpha and i386 on FreeBSD.
Fix a lot of bogus string formats for outputting pointers (cast to int
and %u/%x replaced with no cast and %p), and 'Size'(size_t) are now
cast to 'unsigned long' and output with %lu/
Remove an unused variable.
Alfred Perlstein
hosting product, on both shared and dedicated machines. We currently
offer Oracle and MySQL, and it would be a nice middle-ground.
However, as shipped, PostgreSQL lacks the following features we need
that MySQL has:
1. The ability to listen only on a particular IP address. Each
hosting customer has their own IP address, on which all of their
servers (http, ftp, real media, etc.) run.
2. The ability to place the Unix-domain socket in a mode 700 directory.
This allows us to automatically create an empty database, with an
empty DBA password, for new or upgrading customers without having
to interactively set a DBA password and communicate it to (or from)
the customer. This in turn cuts down our install and upgrade times.
3. The ability to connect to the Unix-domain socket from within a
change-rooted environment. We run CGI programs chrooted to the
user's home directory, which is another reason why we need to be
able to specify where the Unix-domain socket is, instead of /tmp.
4. The ability to, if run as root, open a pid file in /var/run as
root, and then setuid to the desired user. (mysqld -u can almost
do this; I had to patch it, too).
The patch below fixes problem 1-3. I plan to address #4, also, but
haven't done so yet. These diffs are big enough that they should give
the PG development team something to think about in the meantime :-)
Also, I'm about to leave for 2 weeks' vacation, so I thought I'd get
out what I have, which works (for the problems it tackles), now.
With these changes, we can set up and run PostgreSQL with scripts the
same way we can with apache or proftpd or mysql.
In summary, this patch makes the following enhancements:
1. Adds an environment variable PGUNIXSOCKET, analogous to MYSQL_UNIX_PORT,
and command line options -k --unix-socket to the relevant programs.
2. Adds a -h option to postmaster to set the hostname or IP address to
listen on instead of the default INADDR_ANY.
3. Extends some library interfaces to support the above.
4. Fixes a few memory leaks in PQconnectdb().
The default behavior is unchanged from stock 7.0.2; if you don't use
any of these new features, they don't change the operation.
David J. MacKenzie
the -l options. (This was not the case when using the OpenSSL or Kerberos
options.) Also make sure that shared library links get to see all the -L
options. Get Kerberos 5 support to compile on Redhat 7.0. Add OpenSSL and
-lsocket (if used/found) to libpq link.
* Makefile: Add more standard targets. Improve shell redirection in GNU
make detection.
* src/backend/access/transam/rmgr.c: Fix incorrect(?) C.
* src/backend/libpq/pqcomm.c (StreamConnection): Work around accept() bug.
* src/include/port/unixware.h: ...with help from here.
* src/backend/nodes/print.c (plannode_type): Remove some "break"s after
"return"s.
* src/backend/tcop/dest.c (DestToFunction): ditto.
* src/backend/nodes/readfuncs.c: Add proper prototypes.
* src/backend/utils/adt/numutils.c (pg_atoi): Cope specially with strtol()
setting EINVAL. This saves us from creating an extra set of regression test
output for the affected systems.
* src/include/storage/s_lock.h (tas): Correct prototype.
* src/interfaces/libpq/fe-connect.c (parseServiceInfo): Don't use variable
as dimension in array definition.
* src/makefiles/Makefile.unixware: Add support for GCC.
* src/template/unixware: same here
* src/test/regress/expected/abstime-solaris-1947.out: Adjust whitespace.
* src/test/regress/expected/horology-solaris-1947.out: Part of this file
was evidently missing.
* src/test/regress/pg_regress.sh: Fix shell. mkdir -p returns non-zero if
the directory exists.
* src/test/regress/resultmap: Add entries for Unixware.
source directory. This involves mostly makefiles using $(srcdir) when they
might have used ".". (Regression tests don't work with this, yet.)
Sort out usage of CPPFLAGS, CFLAGS (and CXXFLAGS). Add "override" keyword
in most places, to preserve necessary flags even when the user overrode the
flags.
anywhere else that Makefile.shlib needs to modify CFLAGS to produce
valid code for a shared library. I'm not real clear on *why* the use
of override causes make to ignore the later attempt to assign
CFLAGS +=
but it indubitably does --- at least on gmake 3.79.1. gmake bug?
I prepared and tested a patch vs. 7.0.2, and it works fine. I've added
another option which allows users to have their own service file in
~/.pg_service.conf, which might come handy sometimes.
Mario Weilguni
As a result, backend/libpq/pqcomm.c and interfaces/libpq/fe-connect.c
fail to compile.
The <netinet/tcp.h> header needs to be preceded by <netinet/in.h>, at
least on IRIX, Solaris and AIX. The simple configure test fails.
(That header on Linux is idempotent.)
The basic problem is that <netinet/tcp.h> is a BSD header. The
correct header for TCP internals such as TCP_NODELAY on a UNIX system
is <xti.h>. By UNIX I mean UNIX95 (aka XPG4v2 or SUSv1) or later.
The current UNIX standard (UNIX98 aka SUSv2) is available online at
<http://www.opengroup.org/onlinepubs/7908799/>.
The fix is to add header support for <xti.h> into configure.in and
config.h.in.
The 2 files which conditionally include <netinet/tcp.h> need also to
conditionally include <xti.h>.
Pete Forman
working on the VERY latest version of BeOS. I'm sure there will be
alot of comments, but then if there weren't I'd be disappointed!
Thanks for your continuing efforts to get this into your tree.
Haven't bothered with the new files as they haven't changed.
BTW Peter, the compiler is "broken" about the bool define and so on.
I'm filing a bug report to try and get it addressed. Hopefully then we
can tidy up the code a bit.
I await the replies with interest :)
David Reid
Update the installation instructions (formerly misnamed "FAQ"), add configure
checks for some headers rather than having users copy stubs manually (ugh!).
Use Autoconf check for exe extension. This also avoids inheriting the value
of $(X) from the environment.
DESTDIR=/else/where' and prepends the value of DESTDIR to the full
installation paths (e.g., /else/where/usr/local/pgsql/bin). This allows
users to install the package into a location different from the one that
was configured and hard-coded into various scripts, e.g., for creating
binary packages.
DESTDIR is in many cases preferrable over `make install
prefix=/else/where' because
a) `prefix' affects the path that is hard-coded into the files, which can
lead to a `make install prefix=xxx' (as done by the regression test
driver) corrupting the files in the source tree with wrong paths.
b) it doesn't work at all if a directory was overridden to not depend on
`prefix', e.g., --sysconfdir=/etc.
(Updating the regression test driver to use DESTDIR is a separate
undertaking.)
See also autoconf@gnu.org, From: Akim Demaille <akim@epita.fr>, Date: 08
Sep 2000 12:48:59 +0200, Message-ID:
<mv4em2vb1lw.fsf@nostromo.lrde.epita.fr>, Subject: Re: HTML format
documentation.