mirror of
https://git.postgresql.org/git/postgresql.git
synced 2024-10-02 08:11:17 +02:00
Add to mmap discussion.
This commit is contained in:
parent
29c18bca50
commit
6fdd71c133
@ -2014,3 +2014,395 @@ KwvG7YLsJ+xpsTUS67KD+4M=
|
|||||||
|
|
||||||
--HjNkcEWJ4DMx36DP--
|
--HjNkcEWJ4DMx36DP--
|
||||||
|
|
||||||
|
From pgsql-performance-owner+M1354=pgman=candle.pha.pa.us@postgresql.org Fri Mar 7 01:09:07 2003
|
||||||
|
Return-path: <pgsql-performance-owner+M1354=pgman=candle.pha.pa.us@postgresql.org>
|
||||||
|
Received: from relay2.pgsql.com (relay2.pgsql.com [64.49.215.143])
|
||||||
|
by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id h27693604295
|
||||||
|
for <pgman@candle.pha.pa.us>; Fri, 7 Mar 2003 01:09:05 -0500 (EST)
|
||||||
|
Received: from postgresql.org (postgresql.org [64.49.215.8])
|
||||||
|
by relay2.pgsql.com (Postfix) with ESMTP id 95CD2EDFD3B
|
||||||
|
for <pgman@candle.pha.pa.us>; Fri, 7 Mar 2003 01:09:03 -0500 (EST)
|
||||||
|
X-Original-To: pgsql-performance@postgresql.org
|
||||||
|
Received: from perrin.int.nxad.com (internal.ext.nxad.com [69.1.70.251])
|
||||||
|
by postgresql.org (Postfix) with ESMTP id F16034768E2
|
||||||
|
for <pgsql-performance@postgresql.org>; Fri, 7 Mar 2003 01:04:33 -0500 (EST)
|
||||||
|
Received: by perrin.int.nxad.com (Postfix, from userid 1001)
|
||||||
|
id 7969A21065; Thu, 6 Mar 2003 22:04:12 -0800 (PST)
|
||||||
|
Date: Thu, 6 Mar 2003 22:04:12 -0800
|
||||||
|
From: Sean Chittenden <sean@chittenden.org>
|
||||||
|
To: Neil Conway <neilc@samurai.com>
|
||||||
|
cc: Tom Lane <tgl@sss.pgh.pa.us>,
|
||||||
|
Christopher Kings-Lynne <chriskl@familyhealth.com.au>,
|
||||||
|
PostgreSQL Performance <pgsql-performance@postgresql.org>
|
||||||
|
Subject: Re: [PERFORM] [COMMITTERS] pgsql-server/ /configure /configure.in rc/incl ...
|
||||||
|
Message-ID: <20030307060412.GA19138@perrin.int.nxad.com>
|
||||||
|
References: <20030306031656.1876F4762E0@postgresql.org> <032f01c2e390$b1842b20$6500a8c0@fhp.internal> <11077.1046921667@sss.pgh.pa.us> <033f01c2e392$71476570$6500a8c0@fhp.internal> <12228.1046922471@sss.pgh.pa.us> <20030306094117.GA79234@perrin.int.nxad.com> <15071.1046964336@sss.pgh.pa.us> <20030307003640.GF79234@perrin.int.nxad.com> <1046998072.10527.67.camel@tokyo>
|
||||||
|
MIME-Version: 1.0
|
||||||
|
Content-Type: multipart/signed; micalg=pgp-sha1;
|
||||||
|
protocol="application/pgp-signature"; boundary="KsGdsel6WgEHnImy"
|
||||||
|
Content-Disposition: inline
|
||||||
|
In-Reply-To: <1046998072.10527.67.camel@tokyo>
|
||||||
|
User-Agent: Mutt/1.4i
|
||||||
|
X-PGP-Key: finger seanc@FreeBSD.org
|
||||||
|
X-PGP-Fingerprint: 3849 3760 1AFE 7B17 11A0 83A6 DD99 E31F BC84 B341
|
||||||
|
X-Web-Homepage: http://sean.chittenden.org/
|
||||||
|
Precedence: bulk
|
||||||
|
Sender: pgsql-performance-owner@postgresql.org
|
||||||
|
Status: OR
|
||||||
|
|
||||||
|
--KsGdsel6WgEHnImy
|
||||||
|
Content-Type: text/plain; charset=us-ascii
|
||||||
|
Content-Disposition: inline
|
||||||
|
Content-Transfer-Encoding: quoted-printable
|
||||||
|
|
||||||
|
> > I don't have my copy of Steven's handy (it's some 700mi away atm
|
||||||
|
> > otherwise I'd cite it), but if Tom or someone else has it handy, look
|
||||||
|
> > up the example re: the performance gain from read()'ing an mmap()'ed
|
||||||
|
> > file versus a non-mmap()'ed file. The difference is non-trivial and
|
||||||
|
> > _WELL_ worth the time given the speed increase.
|
||||||
|
>=20
|
||||||
|
> Can anyone confirm this? If so, one easy step we could take in this
|
||||||
|
> direction would be adapting COPY FROM to use mmap().
|
||||||
|
|
||||||
|
Weeee! Alright, so I got to have some fun writing out some simple
|
||||||
|
tests with mmap() and friends tonight. Are the results interesting?
|
||||||
|
Absolutely! Is this a simple benchmark? Yup. Do I think it
|
||||||
|
simulates PostgreSQL? Eh, not particularly. Does it demonstrate that
|
||||||
|
mmap() is a win and something worth implementing? I sure hope so. Is
|
||||||
|
this a test program to demonstrate the ideal use of mmap() in
|
||||||
|
PostgreSQL? No. Is it a place to start a factual discussion? I hope
|
||||||
|
so.
|
||||||
|
|
||||||
|
I have here four tests that are conditionalized by cpp.
|
||||||
|
|
||||||
|
# The first one uses read() and write() but with the buffer size set
|
||||||
|
# to the same size as the file.
|
||||||
|
gcc -O3 -finline-functions -fkeep-inline-functions -funroll-loops -o test-=
|
||||||
|
mmap test-mmap.c
|
||||||
|
/usr/bin/time ./test-mmap > /dev/null
|
||||||
|
Beginning tests with file: services
|
||||||
|
|
||||||
|
Page size: 4096
|
||||||
|
File read size is the same as the file size
|
||||||
|
Number of iterations: 100000
|
||||||
|
Start time: 1047013002.412516
|
||||||
|
Time: 82.88178
|
||||||
|
|
||||||
|
Completed tests
|
||||||
|
82.09 real 2.13 user 68.98 sys
|
||||||
|
|
||||||
|
# The second one uses read() and write() with the default buffer size:
|
||||||
|
# 65536
|
||||||
|
gcc -O3 -finline-functions -fkeep-inline-functions -funroll-loops -DDEFAUL=
|
||||||
|
T_READSIZE=3D1 -o test-mmap test-mmap.c
|
||||||
|
/usr/bin/time ./test-mmap > /dev/null
|
||||||
|
Beginning tests with file: services
|
||||||
|
|
||||||
|
Page size: 4096
|
||||||
|
File read size is default read size: 65536
|
||||||
|
Number of iterations: 100000
|
||||||
|
Start time: 1047013085.16204
|
||||||
|
Time: 18.155511
|
||||||
|
|
||||||
|
Completed tests
|
||||||
|
18.16 real 0.90 user 14.79 sys
|
||||||
|
# Please note this is significantly faster, but that's expected
|
||||||
|
|
||||||
|
# The third test uses mmap() + madvise() + write()
|
||||||
|
gcc -O3 -finline-functions -fkeep-inline-functions -funroll-loops -DDEFAUL=
|
||||||
|
T_READSIZE=3D1 -DDO_MMAP=3D1 -o test-mmap test-mmap.c
|
||||||
|
/usr/bin/time ./test-mmap > /dev/null
|
||||||
|
Beginning tests with file: services
|
||||||
|
|
||||||
|
Page size: 4096
|
||||||
|
File read size is the same as the file size
|
||||||
|
Number of iterations: 100000
|
||||||
|
Start time: 1047013103.859818
|
||||||
|
Time: 8.4294203644
|
||||||
|
|
||||||
|
Completed tests
|
||||||
|
7.24 real 0.41 user 5.92 sys
|
||||||
|
# Faster still, and twice as fast as the normal read() case
|
||||||
|
|
||||||
|
# The last test only calls mmap()'s once when the file is opened and
|
||||||
|
# only msync()'s, munmap()'s, close()'s the file once at exit.
|
||||||
|
gcc -O3 -finline-functions -fkeep-inline-functions -funroll-loops -DDEFAUL=
|
||||||
|
T_READSIZE=3D1 -DDO_MMAP=3D1 -DDO_MMAP_ONCE=3D1 -o test-mmap test-mmap.c
|
||||||
|
/usr/bin/time ./test-mmap > /dev/null
|
||||||
|
Beginning tests with file: services
|
||||||
|
|
||||||
|
Page size: 4096
|
||||||
|
File read size is the same as the file size
|
||||||
|
Number of iterations: 100000
|
||||||
|
Start time: 1047013111.623712
|
||||||
|
Time: 1.174076
|
||||||
|
|
||||||
|
Completed tests
|
||||||
|
1.18 real 0.09 user 0.92 sys
|
||||||
|
# Substantially faster
|
||||||
|
|
||||||
|
|
||||||
|
Obviously this isn't perfect, but reading and writing data is faster
|
||||||
|
(specifically moving pages through the VM/OS). Doing partial writes
|
||||||
|
from mmap()'ed data should be faster along with scanning through
|
||||||
|
mmap()'ed portions of - or completely mmap()'ed - files because the
|
||||||
|
pages are already loaded in the VM. PostgreSQL's LRU file descriptor
|
||||||
|
cache could easily be adjusted to add mmap()'ing of frequently
|
||||||
|
accessed files (specifically, system catalogs come to mind). It's not
|
||||||
|
hard to figure out how often particular files are accessed and to
|
||||||
|
either _avoid_ mmap()'ing a file that isn't accessed often, or to
|
||||||
|
mmap() files that _are_ accessed often. mmap() does have a cost, but
|
||||||
|
I'd wager that mmap()'ing the same file a second or third time from a
|
||||||
|
different process would be more efficient. The speedup of searching
|
||||||
|
through an mmap()'ed file may be worth it, however, to mmap() all
|
||||||
|
files if the system is under a tunable resource limit
|
||||||
|
(max_mmaped_bytes?).
|
||||||
|
|
||||||
|
If someone is so inclined or there's enough interest, I can reverse
|
||||||
|
this test case so that data is written to an mmap()'ed file, but the
|
||||||
|
same performance difference should hold true (assuming this isn't a
|
||||||
|
write to a tape drive ::grin::).
|
||||||
|
|
||||||
|
The URL for the program used to generate the above tests is at:
|
||||||
|
|
||||||
|
http://people.freebsd.org/~seanc/mmap_test/
|
||||||
|
|
||||||
|
|
||||||
|
Please ask if you have questions. -sc
|
||||||
|
|
||||||
|
--=20
|
||||||
|
Sean Chittenden
|
||||||
|
|
||||||
|
--KsGdsel6WgEHnImy
|
||||||
|
Content-Type: application/pgp-signature
|
||||||
|
Content-Disposition: inline
|
||||||
|
|
||||||
|
-----BEGIN PGP SIGNATURE-----
|
||||||
|
Comment: Sean Chittenden <sean@chittenden.org>
|
||||||
|
|
||||||
|
iD8DBQE+aDZc3ZnjH7yEs0ERAid6AJ9/TAYMUx2+ZcD2680OlKJBj5FzrACgquIG
|
||||||
|
PBNCzM0OegBXrPROJ/uIKDM=
|
||||||
|
=y7O6
|
||||||
|
-----END PGP SIGNATURE-----
|
||||||
|
|
||||||
|
--KsGdsel6WgEHnImy--
|
||||||
|
|
||||||
|
From pgsql-performance-owner+M1358=pgman=candle.pha.pa.us@postgresql.org Fri Mar 7 16:47:38 2003
|
||||||
|
Return-path: <pgsql-performance-owner+M1358=pgman=candle.pha.pa.us@postgresql.org>
|
||||||
|
Received: from relay2.pgsql.com (relay2.pgsql.com [64.49.215.143])
|
||||||
|
by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id h27LlX429809
|
||||||
|
for <pgman@candle.pha.pa.us>; Fri, 7 Mar 2003 16:47:35 -0500 (EST)
|
||||||
|
Received: from postgresql.org (postgresql.org [64.49.215.8])
|
||||||
|
by relay2.pgsql.com (Postfix) with ESMTP id D40CBEDFE05
|
||||||
|
for <pgman@candle.pha.pa.us>; Fri, 7 Mar 2003 16:47:32 -0500 (EST)
|
||||||
|
X-Original-To: pgsql-performance@postgresql.org
|
||||||
|
Received: from perrin.int.nxad.com (internal.ext.nxad.com [69.1.70.251])
|
||||||
|
by postgresql.org (Postfix) with ESMTP id 913B5474E44
|
||||||
|
for <pgsql-performance@postgresql.org>; Fri, 7 Mar 2003 16:46:50 -0500 (EST)
|
||||||
|
Received: by perrin.int.nxad.com (Postfix, from userid 1001)
|
||||||
|
id A55392105B; Fri, 7 Mar 2003 13:46:30 -0800 (PST)
|
||||||
|
Date: Fri, 7 Mar 2003 13:46:30 -0800
|
||||||
|
From: Sean Chittenden <sean@chittenden.org>
|
||||||
|
To: Tom Lane <tgl@sss.pgh.pa.us>
|
||||||
|
cc: Neil Conway <neilc@samurai.com>,
|
||||||
|
Christopher Kings-Lynne <chriskl@familyhealth.com.au>,
|
||||||
|
PostgreSQL Performance <pgsql-performance@postgresql.org>
|
||||||
|
Subject: Re: [PERFORM] [COMMITTERS] pgsql-server/ /configure /configure.in rc/incl ...
|
||||||
|
Message-ID: <20030307214630.GI79234@perrin.int.nxad.com>
|
||||||
|
References: <032f01c2e390$b1842b20$6500a8c0@fhp.internal> <11077.1046921667@sss.pgh.pa.us> <033f01c2e392$71476570$6500a8c0@fhp.internal> <12228.1046922471@sss.pgh.pa.us> <20030306094117.GA79234@perrin.int.nxad.com> <15071.1046964336@sss.pgh.pa.us> <20030307003640.GF79234@perrin.int.nxad.com> <1046998072.10527.67.camel@tokyo> <20030307060412.GA19138@perrin.int.nxad.com> <29933.1047047386@sss.pgh.pa.us>
|
||||||
|
MIME-Version: 1.0
|
||||||
|
Content-Type: multipart/signed; micalg=pgp-sha1;
|
||||||
|
protocol="application/pgp-signature"; boundary="TALVG7vV++YnpwZG"
|
||||||
|
Content-Disposition: inline
|
||||||
|
In-Reply-To: <29933.1047047386@sss.pgh.pa.us>
|
||||||
|
User-Agent: Mutt/1.4i
|
||||||
|
X-PGP-Key: finger seanc@FreeBSD.org
|
||||||
|
X-PGP-Fingerprint: 3849 3760 1AFE 7B17 11A0 83A6 DD99 E31F BC84 B341
|
||||||
|
X-Web-Homepage: http://sean.chittenden.org/
|
||||||
|
Precedence: bulk
|
||||||
|
Sender: pgsql-performance-owner@postgresql.org
|
||||||
|
Status: OR
|
||||||
|
|
||||||
|
--TALVG7vV++YnpwZG
|
||||||
|
Content-Type: text/plain; charset=us-ascii
|
||||||
|
Content-Disposition: inline
|
||||||
|
Content-Transfer-Encoding: quoted-printable
|
||||||
|
|
||||||
|
> > Absolutely! Is this a simple benchmark? Yup. Do I think it
|
||||||
|
> > simulates PostgreSQL? Eh, not particularly.
|
||||||
|
|
||||||
|
I think quite a few of these Q's would have been answered by reading
|
||||||
|
the code/Makefile....
|
||||||
|
|
||||||
|
> This would be on what OS?
|
||||||
|
|
||||||
|
FreeBSD, but it shouldn't matter. Any reasonably written VM should
|
||||||
|
have similar numbers (though BSD is generally regarded as having the
|
||||||
|
best VM, which, I think Linux poached not that long ago, iirc
|
||||||
|
::grimace::).
|
||||||
|
|
||||||
|
> What hardware?
|
||||||
|
|
||||||
|
My ultra-pathetic laptop with some fine - overly-noisy and can hardly
|
||||||
|
buildworld - IDE drives.
|
||||||
|
|
||||||
|
> What size test file?
|
||||||
|
|
||||||
|
In this case, only 72K. I've just updated the test program to use an
|
||||||
|
array of files though.
|
||||||
|
|
||||||
|
> Do the "iterations" mean so many reads of the entire file, or so
|
||||||
|
> many buffer-sized read requests?
|
||||||
|
|
||||||
|
In some cases, yes. With the file mmap()'ed, sorta. One of the test
|
||||||
|
cases (the one that did it in ~8s), mmap()'ed and munmap()'ed the file
|
||||||
|
every iteration and was twice as fast as the vanilla read() call.
|
||||||
|
|
||||||
|
> Did the mmap case actually *read* anything, or just map and unmap
|
||||||
|
> the file?
|
||||||
|
|
||||||
|
Nope, read it and wrote it out to stdout (which was redirected to
|
||||||
|
/dev/null).
|
||||||
|
|
||||||
|
> Also, what did you do to normalize for the effects of the test file
|
||||||
|
> being already in kernel disk cache after the first test?
|
||||||
|
|
||||||
|
That honestly doesn't matter too much since I wasn't testing the rate
|
||||||
|
of reading in files from my hard drive, only the OS's ability to
|
||||||
|
read/write pages of data around. In any case, I've updated my test
|
||||||
|
case to iterate through an array of files instead of just reading in a
|
||||||
|
copy of /etc/services. My laptop is generally a poor benchmark for
|
||||||
|
disk read performance given it takes 8hrs to buildworld, over 12hrs to
|
||||||
|
build mozilla, 18 for KDE, and about 48hrs for Open Office. :)
|
||||||
|
Someone with faster disks may want to try this and report back, but it
|
||||||
|
doesn't matter much in terms of relevancy for considering the benefits
|
||||||
|
of mmap(). The point is that there are calls that can be used that
|
||||||
|
substantially speed up read()'s and write()'s by allowing the VM to
|
||||||
|
align pages of data and give hints about its usage. For the sake of
|
||||||
|
argument re: the previously done tests, I'll reverse the order in
|
||||||
|
which I ran them and I bet dime to dollar that the times will be
|
||||||
|
identical.
|
||||||
|
|
||||||
|
% make =
|
||||||
|
~/open_source/mmap_test
|
||||||
|
cp -f /etc/services ./services
|
||||||
|
gcc -O3 -finline-functions -fkeep-inline-functions -funroll-loops -DDEFAUL=
|
||||||
|
T_READSIZE=3D1 -DDO_MMAP=3D1 -DDO_MMAP_ONCE=3D1 -o mmap-test mmap-test.c
|
||||||
|
/usr/bin/time ./mmap-test > /dev/null
|
||||||
|
Beginning tests with file: services
|
||||||
|
|
||||||
|
Page size: 4096
|
||||||
|
File read size is the same as the file size
|
||||||
|
Number of iterations: 100000
|
||||||
|
Start time: 1047064672.276544
|
||||||
|
Time: 1.281477
|
||||||
|
|
||||||
|
Completed tests
|
||||||
|
1.29 real 0.10 user 0.92 sys
|
||||||
|
gcc -O3 -finline-functions -fkeep-inline-functions -funroll-loops -DDEFAUL=
|
||||||
|
T_READSIZE=3D1 -DDO_MMAP=3D1 -o mmap-test mmap-test.c
|
||||||
|
/usr/bin/time ./mmap-test > /dev/null
|
||||||
|
Beginning tests with file: services
|
||||||
|
|
||||||
|
Page size: 4096
|
||||||
|
File read size is the same as the file size
|
||||||
|
Number of iterations: 100000
|
||||||
|
Start time: 1047064674.266191
|
||||||
|
Time: 7.486622
|
||||||
|
|
||||||
|
Completed tests
|
||||||
|
7.49 real 0.41 user 6.01 sys
|
||||||
|
gcc -O3 -finline-functions -fkeep-inline-functions -funroll-loops -DDEFAUL=
|
||||||
|
T_READSIZE=3D1 -o mmap-test mmap-test.c
|
||||||
|
/usr/bin/time ./mmap-test > /dev/null
|
||||||
|
Beginning tests with file: services
|
||||||
|
|
||||||
|
Page size: 4096
|
||||||
|
File read size is default read size: 65536
|
||||||
|
Number of iterations: 100000
|
||||||
|
Start time: 1047064682.288637
|
||||||
|
Time: 19.35214
|
||||||
|
|
||||||
|
Completed tests
|
||||||
|
19.04 real 0.88 user 15.43 sys
|
||||||
|
gcc -O3 -finline-functions -fkeep-inline-functions -funroll-loops -o mmap-=
|
||||||
|
test mmap-test.c
|
||||||
|
/usr/bin/time ./mmap-test > /dev/null
|
||||||
|
Beginning tests with file: services
|
||||||
|
|
||||||
|
Page size: 4096
|
||||||
|
File read size is the same as the file size
|
||||||
|
Number of iterations: 100000
|
||||||
|
Start time: 1047064701.867031
|
||||||
|
Time: 82.4294540875
|
||||||
|
|
||||||
|
Completed tests
|
||||||
|
81.57 real 2.10 user 69.55 sys
|
||||||
|
|
||||||
|
|
||||||
|
Here's the updated test that iterates through. Ooh! One better, the
|
||||||
|
files I've used are actual data files from ~pgsql. The new benchmark
|
||||||
|
iterates through the list of files and and calls bench() once for each
|
||||||
|
file and restarts at the first file after reaching the end of its
|
||||||
|
list (ARGV).
|
||||||
|
|
||||||
|
Whoa, if these tests are even close to real world, then we at the very
|
||||||
|
least should be mmap()'ing the file every time we read it (assuming
|
||||||
|
we're reading more than just a handful of bytes):
|
||||||
|
|
||||||
|
find /usr/local/pgsql/data -type f | /usr/bin/xargs /usr/bin/time ./mmap-te=
|
||||||
|
st > /dev/null
|
||||||
|
Page size: 4096
|
||||||
|
File read size is the same as the file size
|
||||||
|
Number of iterations: 100000
|
||||||
|
Start time: 1047071143.463360
|
||||||
|
Time: 12.109530
|
||||||
|
|
||||||
|
Completed tests
|
||||||
|
12.11 real 0.36 user 6.80 sys
|
||||||
|
|
||||||
|
find /usr/local/pgsql/data -type f | /usr/bin/xargs /usr/bin/time ./mmap-te=
|
||||||
|
st > /dev/null
|
||||||
|
Page size: 4096
|
||||||
|
File read size is default read size: 65536
|
||||||
|
Number of iterations: 100000
|
||||||
|
.... [been waiting here for >40min now....]
|
||||||
|
|
||||||
|
|
||||||
|
Ah well, if these tests finish this century, I'll post the results in
|
||||||
|
a bit, but it's pretty clearly a win. In terms of the data that I'm
|
||||||
|
copying, I'm copying ~700MB of data from my test DB on my laptop. I
|
||||||
|
only have 256MB of RAM so I can pretty much promise you that the data
|
||||||
|
isn't in my system buffers. If anyone else would like to run the
|
||||||
|
tests or look at the results, please check it out:
|
||||||
|
|
||||||
|
o1 and o2 should be the only targets used if FILES is bigger than the
|
||||||
|
RAM on the system. o3's by far and away the fastest, but only in rare
|
||||||
|
cases will a DBA have more RAM than data. But, as mentioned earlier,
|
||||||
|
the LRU cache could easily be modified to munmap() infrequently
|
||||||
|
accessed files to keep the size of mmap()'ed data down to a reasonable
|
||||||
|
level.
|
||||||
|
|
||||||
|
The updated test programs are at:
|
||||||
|
|
||||||
|
http://people.FreeBSD.org/~seanc/mmap_test/
|
||||||
|
|
||||||
|
-sc
|
||||||
|
|
||||||
|
--=20
|
||||||
|
Sean Chittenden
|
||||||
|
|
||||||
|
--TALVG7vV++YnpwZG
|
||||||
|
Content-Type: application/pgp-signature
|
||||||
|
Content-Disposition: inline
|
||||||
|
|
||||||
|
-----BEGIN PGP SIGNATURE-----
|
||||||
|
Comment: Sean Chittenden <sean@chittenden.org>
|
||||||
|
|
||||||
|
iD8DBQE+aRM23ZnjH7yEs0ERAoqhAKCFgmhpvNMqe9tucoFvK1H6J50z2QCeIZEI
|
||||||
|
mgBHwu/H1pe1sXIX9UG2V+I=
|
||||||
|
=cFRQ
|
||||||
|
-----END PGP SIGNATURE-----
|
||||||
|
|
||||||
|
--TALVG7vV++YnpwZG--
|
||||||
|
|
||||||
|
Loading…
Reference in New Issue
Block a user