1<!doctype html public "-//W3C//DTD HTML 4.01 Transitional//EN" 2 "http://www.w3.org/TR/html4/loose.dtd"> 3 4<html> 5 6<head> 7 8<title>Postfix Built-in Content Inspection</title> 9 10<meta http-equiv="Content-Type" content="text/html; charset=us-ascii"> 11 12</head> 13 14<body> 15 16<h1><img src="postfix-logo.jpg" width="203" height="98" alt=""> 17Postfix Built-in Content Inspection</h1> 18 19<hr> 20 21<h2>Built-in content inspection introduction </h2> 22 23<p> Postfix supports a built-in filter mechanism that examines 24message header and message body content, one line at a time, before 25it is stored in the Postfix queue. The filter is usually implemented 26with POSIX or PCRE regular expressions, as described in the 27<a href="header_checks.5.html">header_checks(5)</a> manual page. </p> 28 29<p> The original purpose of the built-in filter is to stop an 30outbreak of specific email worms or viruses, and it does this job 31well. The filter has also helped to block bounced junk email, 32bounced email from worms or viruses, and notifications from virus 33detection systems. Information about this secondary application 34is given in the <a href="BACKSCATTER_README.html">BACKSCATTER_README</a> document. </p> 35 36<p> Because the built-in filter is optimized for stopping specific 37worms and virus outbreaks, it has <a href="#limitations">limitations</a> 38that make it NOT suitable for general junk email and virus detection. 39For that, you should use one of the external content inspection 40methods that are described in the <a href="FILTER_README.html">FILTER_README</a>, <a href="SMTPD_PROXY_README.html">SMTPD_PROXY_README</a> 41and <a href="MILTER_README.html">MILTER_README</a> documents. </p> 42 43<p> The following diagram gives an over-all picture of how Postfix 44built-in content inspection works: </p> 45 46<blockquote> 47 48<table> 49 50<tr> 51 52 <td colspan="4"> <td bgcolor="#f0f0ff" align="center" 53 valign="middle"> Postmaster<br> notifications </td> 54 55</tr> 56 57<tr> 58 59 <td colspan="4"> <td align="center"> <tt> |<br>v </tt></td> 60 61</tr> 62 63<tr> 64 65 <td bgcolor="#f0f0ff" align="center" valign="middle"> 66 Network or<br> local users </td> 67 68 <td align="center" valign="middle"> <tt> -> </tt> </td> 69 70 <td bgcolor="#f0f0ff" align="center" valign="middle"> 71 72 <b> Built-in<br> filter</b> </td> 73 74 <td align="center" valign="middle"> <tt> -> </tt> </td> 75 76 <td bgcolor="#f0f0ff" align="center" valign="middle"> 77 Postfix<br> queue </td> 78 79 <td align="center" valign="middle"> <tt> -> </tt> </td> 80 81 <td bgcolor="#f0f0ff" align="center" valign="middle"> 82 Delivery<br> agents </td> 83 84 <td align="center" valign="middle"> <tt> -> </tt> </td> 85 86 <td bgcolor="#f0f0ff" align="center" valign="middle"> 87 Network or<br> local mailbox </td> 88 89</tr> 90 91<tr> 92 93 <td colspan="4"> <td align="center"> ^<br> <tt> | </tt> </td> 94 <td> </td> <td align="center"> <tt> |<br>v </tt> </td> 95 96</tr> 97 98<tr> 99 100 <td colspan="4"> <td colspan="3" bgcolor="#f0f0ff" align="center" 101 valign="middle"> Undeliverable mail<br> Forwarded mail</td> 102 103</tr> 104 105</table> 106 107</blockquote> 108 109<p> The picture makes clear that the filter works while Postfix is 110receiving new mail. This means that Postfix can reject mail from 111the network without having to return undeliverable mail to the 112originator address (which is often spoofed anyway). However, this 113ability comes at a price: if mail inspection takes too much time, 114then the remote client will time out, and the client may send the 115same message repeatedly. </p> 116 117<p>Topics covered by this document: </p> 118 119<ul> 120 121<li><a href="#what">What mail is subjected to header/body checks </a> 122 123<li><a href="#limitations">Limitations of Postfix header/body checks </a> 124 125<li><a href="#daily">Preventing daily mail status reports from being blocked </a> 126 127<li><a href="#remote_only">Configuring header/body checks for mail from outside users only</a> 128 129<li><a href="#domain_except">Configuring header/body checks for mail to some domains only</a> 130 131</ul> 132 133<h2><a name="what">What mail is subjected to header/body checks </a></h2> 134 135<p> Postfix header/body checks are implemented by the <a href="cleanup.8.html">cleanup(8)</a> 136server before it injects mail into the <a href="QSHAPE_README.html#incoming_queue">incoming queue</a>. The diagram 137below zooms in on the <a href="cleanup.8.html">cleanup(8)</a> server, and shows that this server 138handles mail from many different sources. In order to keep the 139diagram readable, the sources of postmaster notifications are not 140shown, because they can be produced by many Postfix daemon processes. 141</p> 142 143<blockquote> 144 145<table> 146 147<tr> <td colspan="2"> </td> <td bgcolor="#f0f0ff" align="center" 148valign="middle"> <a href="bounce.8.html">bounce(8)</a><br> (undeliverable) </td> </tr> 149 150<tr> <td bgcolor="#f0f0ff" align="center" valign="middle"> <b> 151<a href="smtpd.8.html">smtpd(8)</a><br> (network)</b> </td> <td align="left" valign="bottom"> 152<tt> \ </tt> </td> <td align="center" valign="middle"> <tt> |<br>v 153</tt> </td> </tr> 154 155<tr> <td> </td> <td> </td> </tr> 156 157<tr> <td bgcolor="#f0f0ff" align="center" valign="middle"> <b> 158<a href="qmqpd.8.html">qmqpd(8)</a><br> (network)</b> </td> <td align="center" valign="middle"> 159<tt> -\<br>-/ </tt> </td> <td bgcolor="#f0f0ff" align="center" 160valign="middle"> <a href="cleanup.8.html">cleanup(8)</a> </td> <td align="center" valign="middle"> 161<tt> -> </tt> </td> <td bgcolor="#f0f0ff" align="center" 162valign="middle"> <a href="QSHAPE_README.html#incoming_queue"> 163incoming<br> queue </a> </td> </tr> 164 165<tr> <td bgcolor="#f0f0ff" align="center" valign="middle"> <b> 166<a href="pickup.8.html">pickup(8)</a><br> (local)</b> </td> <td align="left" valign="top"> <tt> 167/ </tt> </td> <td align="center" valign="middle"> ^<br> <tt> | 168</tt> </td> </tr> 169 170<tr> <td colspan="2"> </td> <td bgcolor="#f0f0ff" align="center" 171valign="middle"> <a href="local.8.html">local(8)</a><br> (forwarded) </td> </tr> 172 173</table> 174 175</blockquote> 176 177<p> For efficiency reasons, only mail that enters from outside of 178Postfix is inspected with header/body checks. It would be inefficient 179to filter already filtered mail again, and it would be undesirable 180to block postmaster notifications. The table below summarizes what 181mail is and is not subject to header/body checks. </p> 182 183<blockquote> 184 185<table border="1"> 186 187<tr> <th> Message type </th> <th> Source </th> <th> Header/body checks? </th> </tr> 188 189<tr> <td> Undeliverable mail </td> <td> <a href="bounce.8.html">bounce(8)</a> </td> <td> No </td> </tr> 190 191<tr> <td> Network mail </td> <td> <a href="smtpd.8.html">smtpd(8)</a> </td> <td> Configurable </td> </tr> 192 193<tr> <td> Network mail </td> <td> <a href="qmqpd.8.html">qmqpd(8)</a> </td> <td> Configurable </td> </tr> 194 195<tr> <td> Local submission </td> <td> <a href="pickup.8.html">pickup(8)</a> </td> <td> Configurable </td> </tr> 196 197<tr> <td> Local forwarding </td> <td> <a href="local.8.html">local(8)</a> </td> <td> No </td> </tr> 198 199<tr> <td> Postmaster notice </td> <td> many </td> <td> No </td> </tr> 200 201</table> 202 203</blockquote> 204 205<p> How does Postfix decide what mail needs to be filtered? It 206would be clumsy to make the decision in the <a href="cleanup.8.html">cleanup(8)</a> server, as 207this program receives mail from so many different sources. Instead, 208header/body checks are requested by the source. Examples of how 209to turn off header/body checks for mail received with <a href="smtpd.8.html">smtpd(8)</a>, 210<a href="qmqpd.8.html">qmqpd(8)</a> or <a href="pickup.8.html">pickup(8)</a> are given below under "<a 211href="#remote_only">Configuring header/body checks for mail from 212outside users only</a>" and "<a href="#domain_except">Configuring 213header/body checks for mail to some domains only</a>". </p> 214 215<h2><a name="limitations">Limitations of Postfix header/body checks </a></h2> 216 217<ul> 218 219<li> <p> Header/body checks do not decode message headers or message 220body content. For example, if text in the message body is BASE64 221encoded (<a href="http://tools.ietf.org/html/rfc2045">RFC 2045</a>) then your regular expressions will have to match 222the BASE64 encoded form. Likewise, message headers with encoded 223non-ASCII characters (<a href="http://tools.ietf.org/html/rfc2047">RFC 2047</a>) need to be matched in their encoded 224form. </p> 225 226<li> <p> Header/body checks cannot filter on a combination of 227message headers or body lines. Header/body checks examine content 228one message header at a time, or one message body line at a time, 229and cannot carry a decision over to the next message header or body 230line. </p> 231 232<li> <p> Header/body checks cannot depend on the recipient of a 233message. </p> 234 235<ul> 236 237<li> <p> One message can have multiple recipients, and all recipients 238of a message receive the same treatment. Workarounds have been 239proposed that involve selectively deferring some recipients of 240multi-recipient mail, but that results in poor SMTP performance 241and does not work for non-SMTP mail. </p> 242 243<li> <p> Some sources of mail send the headers and content ahead 244of the recipient information. It would be inefficient to buffer up 245an entire message before deciding if it needs to be filtered, and 246it would be clumsy to filter mail and to buffer up all the actions 247until it is known whether those actions need to be executed. </p> 248 249</ul> 250 251<li> <p> Despite warnings, some people try to use the built-in 252filter feature for general junk email and/or virus blocking, using 253hundreds or even thousands of regular expressions. This can result 254in catastrophic performance failure. The symptoms are as follows: 255</p> 256 257<ul> 258 259<li> <p> The <a href="cleanup.8.html">cleanup(8)</a> processes use up all available CPU time in 260order to process the regular expressions, and/or they use up all 261available memory so that the system begins to swap. This slows down 262all incoming mail deliveries. </p> 263 264<li> <p> As Postfix needs more and more time to receive an email 265message, the number of simultaneous SMTP sessions increases to the 266point that the SMTP server process limit is reached. </p> 267 268<li> <p> While all SMTP server processes are waiting for the 269<a href="cleanup.8.html">cleanup(8)</a> servers to finish, new SMTP clients have to wait until 270an SMTP server process becomes available. This causes mail deliveries 271to time out before they have even begun. </p> 272 273</ul> 274 275<p> The remedy for this type of performance problem is simple: 276don't use header/body checks for general junk email and/or virus 277blocking, and don't filter mail before it is queued. When performance 278is a concern, use an external content filter that runs after mail 279is queued, as described in the <a href="FILTER_README.html">FILTER_README</a> document. </p> 280 281</ul> 282 283<h2><a name="daily">Preventing daily mail status reports from being blocked </a></h2> 284 285<p>The following is quoted from Jim Seymour's Pflogsumm FAQ at 286<a href="http://jimsun.linxnet.com/downloads/pflogsumm-faq.txt">http://jimsun.linxnet.com/downloads/pflogsumm-faq.txt</a>. Pflogsumm 287is a program that analyzes Postfix logs, including the logging from 288rejected mail. If these logs contain text that was rejected by 289Postfix <a href="postconf.5.html#body_checks">body_checks</a> patterns, then the logging is also likely to 290be rejected by those same <a href="postconf.5.html#body_checks">body_checks</a> patterns. This problem does 291not exist with <a href="postconf.5.html#header_checks">header_checks</a> patterns, because those are not applied 292to the text that is part of the mail status report. </p> 293 294<blockquote> 295 296<p>You configure Postfix to do body checks, Postfix does its thing, 297Pflogsumm reports it and Postfix catches the same string in the 298Pflogsumm report. There are several solutions to this. </p> 299 300<p> Wolfgang Zeikat contributed this: </p> 301 302<blockquote> 303<pre> 304#!/usr/bin/perl 305use MIME::Lite; 306 307### Create a new message: 308$msg = MIME::Lite->new( 309 From => 'your@send.er', 310 To => 'your@recipie.nt', 311 # Cc => 'some@other.com, some@more.com', 312 Subject => 'pflogsumm', 313 Date => `date`, 314 Type => 'text/plain', 315 Encoding => 'base64', 316 Path => '/tmp/pflogg', 317); 318 319$msg->send; 320</pre> 321</blockquote> 322 323<p> Where "/tmp/pflogg" is the output of Pflogsumm. This puts Pflogsumm's 324output in a base64 MIME attachment. </p> 325 326</blockquote> 327 328<p> Note by Wietse: if you run this on a machine that is accessible 329by untrusted users, it is safer to store the Pflogsumm report in 330a directory that is not world writable. </p> 331 332<blockquote> 333 334<p> In a follow-up to a thread in the postfix-users mailing list, Ralf 335Hildebrandt noted: </p> 336 337<blockquote> <p> "mpack does the same thing." </p> </blockquote> 338 339</blockquote> 340 341<p> And it does. Which tool one should use is a matter of preference. 342</p> 343 344<p> Other solutions involve additional <a href="postconf.5.html#body_checks">body_checks</a> rules that make 345exceptions for daily mail status reports, but this is not recommended. 346Such rules slow down all mail and complicate Postfix maintenance. 347</p> 348 349<h2><a name="remote_only">Configuring header/body checks for mail from outside users only</a></h2> 350 351<p> The following information applies to Postfix 2.1 and later. 352Earlier 353Postfix versions do not support the <a href="postconf.5.html#receive_override_options">receive_override_options</a> feature. 354</p> 355 356<p> The easiest approach is to configure ONE Postfix instance with 357multiple SMTP server IP addresses in <a href="master.5.html">master.cf</a>: </p> 358 359<ul> 360 361<li> <p> Two SMTP server IP addresses for mail from inside users 362only, with header/body filtering turned off, and a local mail pickup 363service with header/body filtering turned off. </p> 364 365<pre> 366/etc/postfix.<a href="master.5.html">master.cf</a>: 367 # ================================================================== 368 # service type private unpriv chroot wakeup maxproc command 369 # (yes) (yes) (yes) (never) (100) 370 # ================================================================== 371 1.2.3.4:smtp inet n - n - - smtpd 372 -o <a href="postconf.5.html#receive_override_options">receive_override_options</a>=<a href="postconf.5.html#no_header_body_checks">no_header_body_checks</a> 373 127.0.0.1:smtp inet n - n - - smtpd 374 -o <a href="postconf.5.html#receive_override_options">receive_override_options</a>=<a href="postconf.5.html#no_header_body_checks">no_header_body_checks</a> 375 pickup fifo n - n 60 1 pickup 376 -o <a href="postconf.5.html#receive_override_options">receive_override_options</a>=<a href="postconf.5.html#no_header_body_checks">no_header_body_checks</a> 377</pre> 378 379<li> <p> Add some firewall rule to prevent access to 1.2.3.4:smtp 380from the outside world. </p> 381 382<li> <p> One SMTP server address for mail from outside users with 383header/body filtering turned on via <a href="postconf.5.html">main.cf</a>. </p> 384 385<pre> 386/etc/postfix.<a href="master.5.html">master.cf</a>: 387 # ================================================================= 388 # service type private unpriv chroot wakeup maxproc command 389 # (yes) (yes) (yes) (never) (100) 390 # ================================================================= 391 1.2.3.5:smtp inet n - n - - smtpd 392</pre> 393 394</ul> 395 396<h2><a name="domain_except">Configuring header/body checks for mail to some domains only</a></h2> 397 398<p> The following information applies to Postfix 2.1. Earlier 399Postfix versions do not support the <a href="postconf.5.html#receive_override_options">receive_override_options</a> feature. 400</p> 401 402<p> If you are MX service provider and want to apply disable 403head/body checks for some domains, you can configure ONE Postfix 404instance with multiple SMTP server IP addresses in <a href="master.5.html">master.cf</a>. Each 405address provides a different service. </p> 406 407<blockquote> 408 409<pre> 410/etc/postfix.<a href="master.5.html">master.cf</a>: 411 # ================================================================= 412 # service type private unpriv chroot wakeup maxproc command 413 # (yes) (yes) (yes) (never) (100) 414 # ================================================================= 415 # SMTP service for domains with header/body checks turned on. 416 1.2.3.4:smtp inet n - n - - smtpd 417 418 # SMTP service for domains with header/body checks turned off. 419 1.2.3.5:smtp inet n - n - - smtpd 420 -o <a href="postconf.5.html#receive_override_options">receive_override_options</a>=<a href="postconf.5.html#no_header_body_checks">no_header_body_checks</a> 421</pre> 422</blockquote> 423 424<p> Once this is set up you can configure MX records in the DNS 425that route each domain to the proper SMTP server instance. </p> 426 427</body> 428 429</html> 430