1NAME
2 MLDBM::Sync - safe concurrent access to MLDBM databases
3
4SYNOPSIS
5 use MLDBM::Sync; # this gets the default, SDBM_File
6 use MLDBM qw(DB_File Storable); # use Storable for serializing
7 use MLDBM qw(MLDBM::Sync::SDBM_File); # use extended SDBM_File, handles values > 1024 bytes
8 use Fcntl qw(:DEFAULT); # import symbols O_CREAT & O_RDWR for use with DBMs
9
10 # NORMAL PROTECTED read/write with implicit locks per i/o request
11 my $sync_dbm_obj = tie %cache, 'MLDBM::Sync' [..other DBM args..] or die $!;
12 $cache{"AAAA"} = "BBBB";
13 my $value = $cache{"AAAA"};
14
15 # SERIALIZED PROTECTED read/write with explicit lock for both i/o requests
16 my $sync_dbm_obj = tie %cache, 'MLDBM::Sync', '/tmp/syncdbm', O_CREAT|O_RDWR, 0640;
17 $sync_dbm_obj->Lock;
18 $cache{"AAAA"} = "BBBB";
19 my $value = $cache{"AAAA"};
20 $sync_dbm_obj->UnLock;
21
22 # SERIALIZED PROTECTED READ access with explicit read lock for both reads
23 $sync_dbm_obj->ReadLock;
24 my @keys = keys %cache;
25 my $value = $cache{'AAAA'};
26 $sync_dbm_obj->UnLock;
27
28 # MEMORY CACHE LAYER with Tie::Cache
29 $sync_dbm_obj->SyncCacheSize('100K');
30
31 # KEY CHECKSUMS, for lookups on MD5 checksums on large keys
32 my $sync_dbm_obj = tie %cache, 'MLDBM::Sync', '/tmp/syncdbm', O_CREAT|O_RDWR, 0640;
33 $sync_dbm_obj->SyncKeysChecksum(1);
34 my $large_key = "KEY" x 10000;
35 $sync{$large_key} = "LARGE";
36 my $value = $sync{$large_key};
37
38DESCRIPTION
39 This module wraps around the MLDBM interface, by handling concurrent
40 access to MLDBM databases with file locking, and flushes i/o explicity
41 per lock/unlock. The new [Read]Lock()/UnLock() API can be used to
42 serialize requests logically and improve performance for bundled reads &
43 writes.
44
45 my $sync_dbm_obj = tie %cache, 'MLDBM::Sync', '/tmp/syncdbm', O_CREAT|O_RDWR, 0640;
46
47 # Write locked critical section
48 $sync_dbm_obj->Lock;
49 ... all accesses to DBM LOCK_EX protected, and go to same tied file handles
50 $cache{'KEY'} = 'VALUE';
51 $sync_dbm_obj->UnLock;
52
53 # Read locked critical section
54 $sync_dbm_obj->ReadLock;
55 ... all read accesses to DBM LOCK_SH protected, and go to same tied files
56 ... WARNING, cannot write to DBM in ReadLock() section, will die()
57 ... WARNING, my $v = $cache{'KEY'}{'SUBKEY'} will trigger a write so not safe
58 ... to use in ReadLock() section
59 my $value = $cache{'KEY'};
60 $sync_dbm_obj->UnLock;
61
62 # Normal access OK too, without explicity locking
63 $cache{'KEY'} = 'VALUE';
64 my $value = $cache{'KEY'};
65
66 MLDBM continues to serve as the underlying OO layer that serializes
67 complex data structures to be stored in the databases. See the MLDBM the
68 BUGS manpage section for important limitations.
69
70 MLDBM::Sync also provides built in RAM caching with Tie::Cache md5 key
71 checksum functionality.
72
73INSTALL
74 Like any other CPAN module, either use CPAN.pm, or perl -MCPAN "-e"
75 shell, or get the file MLDBM-Sync-x.xx.tar.gz, unzip, untar and:
76
77 perl Makefile.PL
78 make
79 make test
80 make install
81
82LOCKING
83 The MLDBM::Sync wrapper protects MLDBM databases by locking and
84 unlocking around read and write requests to the databases. Also
85 necessary is for each new lock to tie() to the database internally,
86 untie()ing when unlocking. This flushes any i/o for the dbm to the
87 operating system, and allows for concurrent read/write access to the
88 databases.
89
90 Without any extra effort from the developer, an existing MLDBM database
91 will benefit from MLDBM::sync.
92
93 my $dbm_obj = tie %dbm, ...;
94 $dbm{"key"} = "value";
95
96 As a write or STORE operation, the above will automatically cause the
97 following:
98
99 $dbm_obj->Lock; # also ties
100 $dbm{"key"} = "value";
101 $dbm_obj->UnLock; # also unties
102
103 Just so, a read or FETCH operation like:
104
105 my $value = $dbm{"key"};
106
107 will really trigger:
108
109 $dbm_obj->ReadLock; # also ties
110 my $value = $dbm{"key"};
111 $dbm_obj->Lock; # also unties
112
113 However, these lock operations are expensive because of the underlying
114 tie()/untie() that occurs for i/o flushing, so when bundling reads &
115 writes, a developer may explicitly use this API for greater performance:
116
117 # tie once to database, write 100 times
118 $dbm_obj->Lock;
119 for (1..100) {
120 $dbm{$_} = $_ * 100;
121 ...
122 }
123 $dbm_obj->UnLock;
124
125 # only tie once to database, and read 100 times
126 $dbm_obj->ReadLock;
127 for(1..100) {
128 my $value = $dbm{$_};
129 ...
130 }
131 $dbm_obj->UnLock;
132
133CACHING
134 I built MLDBM::Sync to serve as a fast and robust caching layer for use
135 in multi-process environments like mod_perl. In order to provide an
136 additional speed boost when caching static data, I have added an RAM
137 caching layer with Tie::Cache, which regulates the size of the memory
138 used with its MaxBytes setting.
139
140 To activate this caching, just:
141
142 my $dbm = tie %cache, 'MLDBM::Sync', '/tmp/syncdbm', O_CREAT|O_RDWR, 0640;
143 $dbm->SyncCacheSize(100000); # 100000 bytes max memory used
144 $dbm->SyncCacheSize('100K'); # 100 Kbytes max memory used
145 $dbm->SyncCacheSize('1M'); # 1 Megabyte max memory used
146
147 The ./bench/bench_sync.pl, run like "bench_sync.pl "-c"" will run the
148 tests with caching turned on creating a benchmark with 50% cache hits.
149
150 One run without caching was:
151
152 === INSERT OF 50 BYTE RECORDS ===
153 Time for 100 writes + 100 reads for SDBM_File 0.16 seconds 12288 bytes
154 Time for 100 writes + 100 reads for MLDBM::Sync::SDBM_File 0.17 seconds 12288 bytes
155 Time for 100 writes + 100 reads for GDBM_File 3.37 seconds 17980 bytes
156 Time for 100 writes + 100 reads for DB_File 4.45 seconds 20480 bytes
157
158 And with caching, with 50% cache hits:
159
160 === INSERT OF 50 BYTE RECORDS ===
161 Time for 100 writes + 100 reads for SDBM_File 0.11 seconds 12288 bytes
162 Time for 100 writes + 100 reads for MLDBM::Sync::SDBM_File 0.11 seconds 12288 bytes
163 Time for 100 writes + 100 reads for GDBM_File 2.49 seconds 17980 bytes
164 Time for 100 writes + 100 reads for DB_File 2.55 seconds 20480 bytes
165
166 Even for SDBM_File, this speedup is near 33%.
167
168KEYS CHECKSUM
169 A common operation on database lookups is checksumming the key, prior to
170 the lookup, because the key could be very large, and all one really
171 wants is the data it maps too. To enable this functionality
172 automatically with MLDBM::Sync, just:
173
174 my $sync_dbm_obj = tie %cache, 'MLDBM::Sync', '/tmp/syncdbm', O_CREAT|O_RDWR, 0640;
175 $sync_dbm_obj->SyncKeysChecksum(1);
176
177 !! WARNING: keys() & each() do not work on these databases
178 !! as of v.03, so the developer will not be fooled into thinking
179 !! the stored key values are meaningful to the calling application
180 !! and will die() if called.
181 !!
182 !! This behavior could be relaxed in the future.
183
184 An example of this might be to cache a XSLT conversion, which are
185 typically very expensive. You have the XML data and the XSLT data, so
186 all you do is:
187
188 # $xml_data, $xsl_data are strings
189 my $xslt_output;
190 unless ($xslt_output = $cache{$xml_data.'&&&&'.$xsl_data}) {
191 ... do XSLT conversion here for $xslt_output ...
192 $cache{$xml_data.'&&&&'.xsl_data} = $xslt_output;
193 }
194
195 What you save by doing this is having to create HUGE keys to lookup on,
196 which no DBM is likely to do efficiently. This is the same method that
197 File::Cache uses internally to hash its file lookups in its directories.
198
199New MLDBM::Sync::SDBM_File
200 SDBM_File, the default used for MLDBM and therefore MLDBM::Sync has a
201 limit of 1024 bytes for the size of a record.
202
203 SDBM_File is also an order of magnitude faster for small records to use
204 with MLDBM::Sync, than DB_File or GDBM_File, because the tie()/untie()
205 to the dbm is much faster. Therefore, bundled with MLDBM::Sync release
206 is a MLDBM::Sync::SDBM_File layer which works around this 1024 byte
207 limit. To use, just:
208
209 use MLDBM qw(MLDBM::Sync::SDBM_File);
210
211 It works by breaking up up the STORE() values into small 128 byte
212 segments, and spreading those segments across many records, creating a
213 virtual record layer. It also uses Compress::Zlib to compress STORED
214 data, reducing the number of these 128 byte records. In benchmarks, 128
215 byte record segments seemed to be a sweet spot for space/time
216 efficiency, as SDBM_File created very bloated *.pag files for 128+ byte
217 records.
218
219BENCHMARKS
220 In the distribution ./bench directory is a bench_sync.pl script that can
221 benchmark using the various DBMs with MLDBM::Sync.
222
223 The MLDBM::Sync::SDBM_File DBM is special because is uses SDBM_File for
224 fast small inserts, but slows down linearly with the size of the data
225 being inserted and read.
226
227 The results for a dual PIII-450 linux 2.4.7, with a ext3 file system
228 blocksize 4096 mounted async on a RAID-1 2xIDE 7200 RPM disk were as
229 follows:
230
231 === INSERT OF 50 BYTE RECORDS ===
232 Time for 100 writes + 100 reads for SDBM_File 0.16 seconds 12288 bytes
233 Time for 100 writes + 100 reads for MLDBM::Sync::SDBM_File 0.19 seconds 12288 bytes
234 Time for 100 writes + 100 reads for GDBM_File 1.09 seconds 18066 bytes
235 Time for 100 writes + 100 reads for DB_File 0.67 seconds 12288 bytes
236 Time for 100 writes + 100 reads for Tie::TextDir .04 0.31 seconds 13192 bytes
237
238 === INSERT OF 500 BYTE RECORDS ===
239 (skipping test for SDBM_File 100 byte limit)
240 Time for 100 writes + 100 reads for MLDBM::Sync::SDBM_File 0.52 seconds 110592 bytes
241 Time for 100 writes + 100 reads for GDBM_File 1.20 seconds 63472 bytes
242 Time for 100 writes + 100 reads for DB_File 0.66 seconds 86016 bytes
243 Time for 100 writes + 100 reads for Tie::TextDir .04 0.32 seconds 58192 bytes
244
245 === INSERT OF 5000 BYTE RECORDS ===
246 (skipping test for SDBM_File 100 byte limit)
247 Time for 100 writes + 100 reads for MLDBM::Sync::SDBM_File 1.41 seconds 1163264 bytes
248 Time for 100 writes + 100 reads for GDBM_File 1.38 seconds 832400 bytes
249 Time for 100 writes + 100 reads for DB_File 1.21 seconds 831488 bytes
250 Time for 100 writes + 100 reads for Tie::TextDir .04 0.58 seconds 508192 bytes
251
252 === INSERT OF 20000 BYTE RECORDS ===
253 (skipping test for SDBM_File 100 byte limit)
254 (skipping test for MLDBM::Sync db size > 1M)
255 Time for 100 writes + 100 reads for GDBM_File 2.23 seconds 2063912 bytes
256 Time for 100 writes + 100 reads for DB_File 1.89 seconds 2060288 bytes
257 Time for 100 writes + 100 reads for Tie::TextDir .04 1.26 seconds 2008192 bytes
258
259 === INSERT OF 50000 BYTE RECORDS ===
260 (skipping test for SDBM_File 100 byte limit)
261 (skipping test for MLDBM::Sync db size > 1M)
262 Time for 100 writes + 100 reads for GDBM_File 3.66 seconds 5337944 bytes
263 Time for 100 writes + 100 reads for DB_File 3.64 seconds 5337088 bytes
264 Time for 100 writes + 100 reads for Tie::TextDir .04 2.80 seconds 5008192 bytes
265
266AUTHORS
267 Copyright (c) 2001-2002 Joshua Chamas, Chamas Enterprises Inc. All
268 rights reserved. Sponsored by development on NodeWorks
269 http://www.nodeworks.com and Apache::ASP http://www.apache-asp.org
270
271 This program is free software; you can redistribute it and/or modify it
272 under the same terms as Perl itself.
273
274SEE ALSO
275 MLDBM(3), SDBM_File(3), DB_File(3), GDBM_File(3)
276