1=head1 NAME 2 3perlhack - How to hack at the Perl internals 4 5=head1 DESCRIPTION 6 7This document attempts to explain how Perl development takes place, 8and ends with some suggestions for people wanting to become bona fide 9porters. 10 11The perl5-porters mailing list is where the Perl standard distribution 12is maintained and developed. The list can get anywhere from 10 to 150 13messages a day, depending on the heatedness of the debate. Most days 14there are two or three patches, extensions, features, or bugs being 15discussed at a time. 16 17A searchable archive of the list is at: 18 19 http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/ 20 21The list is also archived under the usenet group name 22C<perl.porters-gw> at: 23 24 http://www.deja.com/ 25 26List subscribers (the porters themselves) come in several flavours. 27Some are quiet curious lurkers, who rarely pitch in and instead watch 28the ongoing development to ensure they're forewarned of new changes or 29features in Perl. Some are representatives of vendors, who are there 30to make sure that Perl continues to compile and work on their 31platforms. Some patch any reported bug that they know how to fix, 32some are actively patching their pet area (threads, Win32, the regexp 33engine), while others seem to do nothing but complain. In other 34words, it's your usual mix of technical people. 35 36Over this group of porters presides Larry Wall. He has the final word 37in what does and does not change in the Perl language. Various 38releases of Perl are shepherded by a ``pumpking'', a porter 39responsible for gathering patches, deciding on a patch-by-patch 40feature-by-feature basis what will and will not go into the release. 41For instance, Gurusamy Sarathy is the pumpking for the 5.6 release of 42Perl. 43 44In addition, various people are pumpkings for different things. For 45instance, Andy Dougherty and Jarkko Hietaniemi share the I<Configure> 46pumpkin, and Tom Christiansen is the documentation pumpking. 47 48Larry sees Perl development along the lines of the US government: 49there's the Legislature (the porters), the Executive branch (the 50pumpkings), and the Supreme Court (Larry). The legislature can 51discuss and submit patches to the executive branch all they like, but 52the executive branch is free to veto them. Rarely, the Supreme Court 53will side with the executive branch over the legislature, or the 54legislature over the executive branch. Mostly, however, the 55legislature and the executive branch are supposed to get along and 56work out their differences without impeachment or court cases. 57 58You might sometimes see reference to Rule 1 and Rule 2. Larry's power 59as Supreme Court is expressed in The Rules: 60 61=over 4 62 63=item 1 64 65Larry is always by definition right about how Perl should behave. 66This means he has final veto power on the core functionality. 67 68=item 2 69 70Larry is allowed to change his mind about any matter at a later date, 71regardless of whether he previously invoked Rule 1. 72 73=back 74 75Got that? Larry is always right, even when he was wrong. It's rare 76to see either Rule exercised, but they are often alluded to. 77 78New features and extensions to the language are contentious, because 79the criteria used by the pumpkings, Larry, and other porters to decide 80which features should be implemented and incorporated are not codified 81in a few small design goals as with some other languages. Instead, 82the heuristics are flexible and often difficult to fathom. Here is 83one person's list, roughly in decreasing order of importance, of 84heuristics that new features have to be weighed against: 85 86=over 4 87 88=item Does concept match the general goals of Perl? 89 90These haven't been written anywhere in stone, but one approximation 91is: 92 93 1. Keep it fast, simple, and useful. 94 2. Keep features/concepts as orthogonal as possible. 95 3. No arbitrary limits (platforms, data sizes, cultures). 96 4. Keep it open and exciting to use/patch/advocate Perl everywhere. 97 5. Either assimilate new technologies, or build bridges to them. 98 99=item Where is the implementation? 100 101All the talk in the world is useless without an implementation. In 102almost every case, the person or people who argue for a new feature 103will be expected to be the ones who implement it. Porters capable 104of coding new features have their own agendas, and are not available 105to implement your (possibly good) idea. 106 107=item Backwards compatibility 108 109It's a cardinal sin to break existing Perl programs. New warnings are 110contentious--some say that a program that emits warnings is not 111broken, while others say it is. Adding keywords has the potential to 112break programs, changing the meaning of existing token sequences or 113functions might break programs. 114 115=item Could it be a module instead? 116 117Perl 5 has extension mechanisms, modules and XS, specifically to avoid 118the need to keep changing the Perl interpreter. You can write modules 119that export functions, you can give those functions prototypes so they 120can be called like built-in functions, you can even write XS code to 121mess with the runtime data structures of the Perl interpreter if you 122want to implement really complicated things. If it can be done in a 123module instead of in the core, it's highly unlikely to be added. 124 125=item Is the feature generic enough? 126 127Is this something that only the submitter wants added to the language, 128or would it be broadly useful? Sometimes, instead of adding a feature 129with a tight focus, the porters might decide to wait until someone 130implements the more generalized feature. For instance, instead of 131implementing a ``delayed evaluation'' feature, the porters are waiting 132for a macro system that would permit delayed evaluation and much more. 133 134=item Does it potentially introduce new bugs? 135 136Radical rewrites of large chunks of the Perl interpreter have the 137potential to introduce new bugs. The smaller and more localized the 138change, the better. 139 140=item Does it preclude other desirable features? 141 142A patch is likely to be rejected if it closes off future avenues of 143development. For instance, a patch that placed a true and final 144interpretation on prototypes is likely to be rejected because there 145are still options for the future of prototypes that haven't been 146addressed. 147 148=item Is the implementation robust? 149 150Good patches (tight code, complete, correct) stand more chance of 151going in. Sloppy or incorrect patches might be placed on the back 152burner until the pumpking has time to fix, or might be discarded 153altogether without further notice. 154 155=item Is the implementation generic enough to be portable? 156 157The worst patches make use of a system-specific features. It's highly 158unlikely that nonportable additions to the Perl language will be 159accepted. 160 161=item Is there enough documentation? 162 163Patches without documentation are probably ill-thought out or 164incomplete. Nothing can be added without documentation, so submitting 165a patch for the appropriate manpages as well as the source code is 166always a good idea. If appropriate, patches should add to the test 167suite as well. 168 169=item Is there another way to do it? 170 171Larry said ``Although the Perl Slogan is I<There's More Than One Way 172to Do It>, I hesitate to make 10 ways to do something''. This is a 173tricky heuristic to navigate, though--one man's essential addition is 174another man's pointless cruft. 175 176=item Does it create too much work? 177 178Work for the pumpking, work for Perl programmers, work for module 179authors, ... Perl is supposed to be easy. 180 181=item Patches speak louder than words 182 183Working code is always preferred to pie-in-the-sky ideas. A patch to 184add a feature stands a much higher chance of making it to the language 185than does a random feature request, no matter how fervently argued the 186request might be. This ties into ``Will it be useful?'', as the fact 187that someone took the time to make the patch demonstrates a strong 188desire for the feature. 189 190=back 191 192If you're on the list, you might hear the word ``core'' bandied 193around. It refers to the standard distribution. ``Hacking on the 194core'' means you're changing the C source code to the Perl 195interpreter. ``A core module'' is one that ships with Perl. 196 197=head2 Keeping in sync 198 199The source code to the Perl interpreter, in its different versions, is 200kept in a repository managed by a revision control system (which is 201currently the Perforce program, see http://perforce.com/). The 202pumpkings and a few others have access to the repository to check in 203changes. Periodically the pumpking for the development version of Perl 204will release a new version, so the rest of the porters can see what's 205changed. The current state of the main trunk of repository, and patches 206that describe the individual changes that have happened since the last 207public release are available at this location: 208 209 ftp://ftp.linux.activestate.com/pub/staff/gsar/APC/ 210 211If you are a member of the perl5-porters mailing list, it is a good 212thing to keep in touch with the most recent changes. If not only to 213verify if what you would have posted as a bug report isn't already 214solved in the most recent available perl development branch, also 215known as perl-current, bleading edge perl, bleedperl or bleadperl. 216 217Needless to say, the source code in perl-current is usually in a perpetual 218state of evolution. You should expect it to be very buggy. Do B<not> use 219it for any purpose other than testing and development. 220 221Keeping in sync with the most recent branch can be done in several ways, 222but the most convenient and reliable way is using B<rsync>, available at 223ftp://rsync.samba.org/pub/rsync/ . (You can also get the most recent 224branch by FTP.) 225 226If you choose to keep in sync using rsync, there are two approaches 227to doing so: 228 229=over 4 230 231=item rsync'ing the source tree 232 233Presuming you are in the directory where your perl source resides 234and you have rsync installed and available, you can `upgrade' to 235the bleadperl using: 236 237 # rsync -avz rsync://ftp.linux.activestate.com/perl-current/ . 238 239This takes care of updating every single item in the source tree to 240the latest applied patch level, creating files that are new (to your 241distribution) and setting date/time stamps of existing files to 242reflect the bleadperl status. 243 244You can than check what patch was the latest that was applied by 245looking in the file B<.patch>, which will show the number of the 246latest patch. 247 248If you have more than one machine to keep in sync, and not all of 249them have access to the WAN (so you are not able to rsync all the 250source trees to the real source), there are some ways to get around 251this problem. 252 253=over 4 254 255=item Using rsync over the LAN 256 257Set up a local rsync server which makes the rsynced source tree 258available to the LAN and sync the other machines against this 259directory. 260 261From http://rsync.samba.org/README.html: 262 263 "Rsync uses rsh or ssh for communication. It does not need to be 264 setuid and requires no special privileges for installation. It 265 does not require a inetd entry or a deamon. You must, however, 266 have a working rsh or ssh system. Using ssh is recommended for 267 its security features." 268 269=item Using pushing over the NFS 270 271Having the other systems mounted over the NFS, you can take an 272active pushing approach by checking the just updated tree against 273the other not-yet synced trees. An example would be 274 275 #!/usr/bin/perl -w 276 277 use strict; 278 use File::Copy; 279 280 my %MF = map { 281 m/(\S+)/; 282 $1 => [ (stat $1)[2, 7, 9] ]; # mode, size, mtime 283 } `cat MANIFEST`; 284 285 my %remote = map { $_ => "/$_/pro/3gl/CPAN/perl-5.7.1" } qw(host1 host2); 286 287 foreach my $host (keys %remote) { 288 unless (-d $remote{$host}) { 289 print STDERR "Cannot Xsync for host $host\n"; 290 next; 291 } 292 foreach my $file (keys %MF) { 293 my $rfile = "$remote{$host}/$file"; 294 my ($mode, $size, $mtime) = (stat $rfile)[2, 7, 9]; 295 defined $size or ($mode, $size, $mtime) = (0, 0, 0); 296 $size == $MF{$file}[1] && $mtime == $MF{$file}[2] and next; 297 printf "%4s %-34s %8d %9d %8d %9d\n", 298 $host, $file, $MF{$file}[1], $MF{$file}[2], $size, $mtime; 299 unlink $rfile; 300 copy ($file, $rfile); 301 utime time, $MF{$file}[2], $rfile; 302 chmod $MF{$file}[0], $rfile; 303 } 304 } 305 306though this is not perfect. It could be improved with checking 307file checksums before updating. Not all NFS systems support 308reliable utime support (when used over the NFS). 309 310=back 311 312=item rsync'ing the patches 313 314The source tree is maintained by the pumpking who applies patches to 315the files in the tree. These patches are either created by the 316pumpking himself using C<diff -c> after updating the file manually or 317by applying patches sent in by posters on the perl5-porters list. 318These patches are also saved and rsync'able, so you can apply them 319yourself to the source files. 320 321Presuming you are in a directory where your patches reside, you can 322get them in sync with 323 324 # rsync -avz rsync://ftp.linux.activestate.com/perl-current-diffs/ . 325 326This makes sure the latest available patch is downloaded to your 327patch directory. 328 329It's then up to you to apply these patches, using something like 330 331 # last=`ls -rt1 *.gz | tail -1` 332 # rsync -avz rsync://ftp.linux.activestate.com/perl-current-diffs/ . 333 # find . -name '*.gz' -newer $last -exec gzcat {} \; >blead.patch 334 # cd ../perl-current 335 # patch -p1 -N <../perl-current-diffs/blead.patch 336 337or, since this is only a hint towards how it works, use CPAN-patchaperl 338from Andreas K�nig to have better control over the patching process. 339 340=back 341 342=head2 Why rsync the source tree 343 344=over 4 345 346=item It's easier 347 348Since you don't have to apply the patches yourself, you are sure all 349files in the source tree are in the right state. 350 351=item It's more recent 352 353According to Gurusamy Sarathy: 354 355 "... The rsync mirror is automatic and syncs with the repository 356 every five minutes. 357 358 "Updating the patch area still requires manual intervention 359 (with all the goofiness that implies, which you've noted) and 360 is typically on a daily cycle. Making this process automatic 361 is on my tuit list, but don't ask me when." 362 363=item It's more reliable 364 365Well, since the patches are updated by hand, I don't have to say any 366more ... (see Sarathy's remark). 367 368=back 369 370=head2 Why rsync the patches 371 372=over 4 373 374=item It's easier 375 376If you have more than one machine that you want to keep in track with 377bleadperl, it's easier to rsync the patches only once and then apply 378them to all the source trees on the different machines. 379 380In case you try to keep in pace on 5 different machines, for which 381only one of them has access to the WAN, rsync'ing all the source 382trees should than be done 5 times over the NFS. Having 383rsync'ed the patches only once, I can apply them to all the source 384trees automatically. Need you say more ;-) 385 386=item It's a good reference 387 388If you do not only like to have the most recent development branch, 389but also like to B<fix> bugs, or extend features, you want to dive 390into the sources. If you are a seasoned perl core diver, you don't 391need no manuals, tips, roadmaps, perlguts.pod or other aids to find 392your way around. But if you are a starter, the patches may help you 393in finding where you should start and how to change the bits that 394bug you. 395 396The file B<Changes> is updated on occasions the pumpking sees as his 397own little sync points. On those occasions, he releases a tar-ball of 398the current source tree (i.e. perl@7582.tar.gz), which will be an 399excellent point to start with when choosing to use the 'rsync the 400patches' scheme. Starting with perl@7582, which means a set of source 401files on which the latest applied patch is number 7582, you apply all 402succeeding patches available from then on (7583, 7584, ...). 403 404You can use the patches later as a kind of search archive. 405 406=over 4 407 408=item Finding a start point 409 410If you want to fix/change the behaviour of function/feature Foo, just 411scan the patches for patches that mention Foo either in the subject, 412the comments, or the body of the fix. A good chance the patch shows 413you the files that are affected by that patch which are very likely 414to be the starting point of your journey into the guts of perl. 415 416=item Finding how to fix a bug 417 418If you've found I<where> the function/feature Foo misbehaves, but you 419don't know how to fix it (but you do know the change you want to 420make), you can, again, peruse the patches for similar changes and 421look how others apply the fix. 422 423=item Finding the source of misbehaviour 424 425When you keep in sync with bleadperl, the pumpking would love to 426I<see> that the community efforts realy work. So after each of his 427sync points, you are to 'make test' to check if everything is still 428in working order. If it is, you do 'make ok', which will send an OK 429report to perlbug@perl.org. (If you do not have access to a mailer 430from the system you just finished successfully 'make test', you can 431do 'make okfile', which creates the file C<perl.ok>, which you can 432than take to your favourite mailer and mail yourself). 433 434But of course, as always, things will not allways lead to a success 435path, and one or more test do not pass the 'make test'. Before 436sending in a bug report (using 'make nok' or 'make nokfile'), check 437the mailing list if someone else has reported the bug already and if 438so, confirm it by replying to that message. If not, you might want to 439trace the source of that misbehaviour B<before> sending in the bug, 440which will help all the other porters in finding the solution. 441 442Here the saved patches come in very handy. You can check the list of 443patches to see which patch changed what file and what change caused 444the misbehaviour. If you note that in the bug report, it saves the 445one trying to solve it, looking for that point. 446 447=back 448 449If searching the patches is too bothersome, you might consider using 450perl's bugtron to find more information about discussions and 451ramblings on posted bugs. 452 453=back 454 455If you want to get the best of both worlds, rsync both the source 456tree for convenience, reliability and ease and rsync the patches 457for reference. 458 459=head2 Submitting patches 460 461Always submit patches to I<perl5-porters@perl.org>. This lets other 462porters review your patch, which catches a surprising number of errors 463in patches. Either use the diff program (available in source code 464form from I<ftp://ftp.gnu.org/pub/gnu/>), or use Johan Vromans' 465I<makepatch> (available from I<CPAN/authors/id/JV/>). Unified diffs 466are preferred, but context diffs are accepted. Do not send RCS-style 467diffs or diffs without context lines. More information is given in 468the I<Porting/patching.pod> file in the Perl source distribution. 469Please patch against the latest B<development> version (e.g., if 470you're fixing a bug in the 5.005 track, patch against the latest 4715.005_5x version). Only patches that survive the heat of the 472development branch get applied to maintenance versions. 473 474Your patch should update the documentation and test suite. 475 476To report a bug in Perl, use the program I<perlbug> which comes with 477Perl (if you can't get Perl to work, send mail to the address 478I<perlbug@perl.org> or I<perlbug@perl.com>). Reporting bugs through 479I<perlbug> feeds into the automated bug-tracking system, access to 480which is provided through the web at I<http://bugs.perl.org/>. It 481often pays to check the archives of the perl5-porters mailing list to 482see whether the bug you're reporting has been reported before, and if 483so whether it was considered a bug. See above for the location of 484the searchable archives. 485 486The CPAN testers (I<http://testers.cpan.org/>) are a group of 487volunteers who test CPAN modules on a variety of platforms. Perl Labs 488(I<http://labs.perl.org/>) automatically tests Perl source releases on 489platforms and gives feedback to the CPAN testers mailing list. Both 490efforts welcome volunteers. 491 492It's a good idea to read and lurk for a while before chipping in. 493That way you'll get to see the dynamic of the conversations, learn the 494personalities of the players, and hopefully be better prepared to make 495a useful contribution when do you speak up. 496 497If after all this you still think you want to join the perl5-porters 498mailing list, send mail to I<perl5-porters-subscribe@perl.org>. To 499unsubscribe, send mail to I<perl5-porters-unsubscribe@perl.org>. 500 501To hack on the Perl guts, you'll need to read the following things: 502 503=over 3 504 505=item L<perlguts> 506 507This is of paramount importance, since it's the documentation of what 508goes where in the Perl source. Read it over a couple of times and it 509might start to make sense - don't worry if it doesn't yet, because the 510best way to study it is to read it in conjunction with poking at Perl 511source, and we'll do that later on. 512 513You might also want to look at Gisle Aas's illustrated perlguts - 514there's no guarantee that this will be absolutely up-to-date with the 515latest documentation in the Perl core, but the fundamentals will be 516right. (http://gisle.aas.no/perl/illguts/) 517 518=item L<perlxstut> and L<perlxs> 519 520A working knowledge of XSUB programming is incredibly useful for core 521hacking; XSUBs use techniques drawn from the PP code, the portion of the 522guts that actually executes a Perl program. It's a lot gentler to learn 523those techniques from simple examples and explanation than from the core 524itself. 525 526=item L<perlapi> 527 528The documentation for the Perl API explains what some of the internal 529functions do, as well as the many macros used in the source. 530 531=item F<Porting/pumpkin.pod> 532 533This is a collection of words of wisdom for a Perl porter; some of it is 534only useful to the pumpkin holder, but most of it applies to anyone 535wanting to go about Perl development. 536 537=item The perl5-porters FAQ 538 539This is posted to perl5-porters at the beginning on every month, and 540should be available from http://perlhacker.org/p5p-faq; alternatively, 541you can get the FAQ emailed to you by sending mail to 542C<perl5-porters-faq@perl.org>. It contains hints on reading 543perl5-porters, information on how perl5-porters works and how Perl 544development in general works. 545 546=back 547 548=head2 Finding Your Way Around 549 550Perl maintenance can be split into a number of areas, and certain people 551(pumpkins) will have responsibility for each area. These areas sometimes 552correspond to files or directories in the source kit. Among the areas are: 553 554=over 3 555 556=item Core modules 557 558Modules shipped as part of the Perl core live in the F<lib/> and F<ext/> 559subdirectories: F<lib/> is for the pure-Perl modules, and F<ext/> 560contains the core XS modules. 561 562=item Documentation 563 564Documentation maintenance includes looking after everything in the 565F<pod/> directory, (as well as contributing new documentation) and 566the documentation to the modules in core. 567 568=item Configure 569 570The configure process is the way we make Perl portable across the 571myriad of operating systems it supports. Responsibility for the 572configure, build and installation process, as well as the overall 573portability of the core code rests with the configure pumpkin - others 574help out with individual operating systems. 575 576The files involved are the operating system directories, (F<win32/>, 577F<os2/>, F<vms/> and so on) the shell scripts which generate F<config.h> 578and F<Makefile>, as well as the metaconfig files which generate 579F<Configure>. (metaconfig isn't included in the core distribution.) 580 581=item Interpreter 582 583And of course, there's the core of the Perl interpreter itself. Let's 584have a look at that in a little more detail. 585 586=back 587 588Before we leave looking at the layout, though, don't forget that 589F<MANIFEST> contains not only the file names in the Perl distribution, 590but short descriptions of what's in them, too. For an overview of the 591important files, try this: 592 593 perl -lne 'print if /^[^\/]+\.[ch]\s+/' MANIFEST 594 595=head2 Elements of the interpreter 596 597The work of the interpreter has two main stages: compiling the code 598into the internal representation, or bytecode, and then executing it. 599L<perlguts/Compiled code> explains exactly how the compilation stage 600happens. 601 602Here is a short breakdown of perl's operation: 603 604=over 3 605 606=item Startup 607 608The action begins in F<perlmain.c>. (or F<miniperlmain.c> for miniperl) 609This is very high-level code, enough to fit on a single screen, and it 610resembles the code found in L<perlembed>; most of the real action takes 611place in F<perl.c> 612 613First, F<perlmain.c> allocates some memory and constructs a Perl 614interpreter: 615 616 1 PERL_SYS_INIT3(&argc,&argv,&env); 617 2 618 3 if (!PL_do_undump) { 619 4 my_perl = perl_alloc(); 620 5 if (!my_perl) 621 6 exit(1); 622 7 perl_construct(my_perl); 623 8 PL_perl_destruct_level = 0; 624 9 } 625 626Line 1 is a macro, and its definition is dependent on your operating 627system. Line 3 references C<PL_do_undump>, a global variable - all 628global variables in Perl start with C<PL_>. This tells you whether the 629current running program was created with the C<-u> flag to perl and then 630F<undump>, which means it's going to be false in any sane context. 631 632Line 4 calls a function in F<perl.c> to allocate memory for a Perl 633interpreter. It's quite a simple function, and the guts of it looks like 634this: 635 636 my_perl = (PerlInterpreter*)PerlMem_malloc(sizeof(PerlInterpreter)); 637 638Here you see an example of Perl's system abstraction, which we'll see 639later: C<PerlMem_malloc> is either your system's C<malloc>, or Perl's 640own C<malloc> as defined in F<malloc.c> if you selected that option at 641configure time. 642 643Next, in line 7, we construct the interpreter; this sets up all the 644special variables that Perl needs, the stacks, and so on. 645 646Now we pass Perl the command line options, and tell it to go: 647 648 exitstatus = perl_parse(my_perl, xs_init, argc, argv, (char **)NULL); 649 if (!exitstatus) { 650 exitstatus = perl_run(my_perl); 651 } 652 653 654C<perl_parse> is actually a wrapper around C<S_parse_body>, as defined 655in F<perl.c>, which processes the command line options, sets up any 656statically linked XS modules, opens the program and calls C<yyparse> to 657parse it. 658 659=item Parsing 660 661The aim of this stage is to take the Perl source, and turn it into an op 662tree. We'll see what one of those looks like later. Strictly speaking, 663there's three things going on here. 664 665C<yyparse>, the parser, lives in F<perly.c>, although you're better off 666reading the original YACC input in F<perly.y>. (Yes, Virginia, there 667B<is> a YACC grammar for Perl!) The job of the parser is to take your 668code and `understand' it, splitting it into sentences, deciding which 669operands go with which operators and so on. 670 671The parser is nobly assisted by the lexer, which chunks up your input 672into tokens, and decides what type of thing each token is: a variable 673name, an operator, a bareword, a subroutine, a core function, and so on. 674The main point of entry to the lexer is C<yylex>, and that and its 675associated routines can be found in F<toke.c>. Perl isn't much like 676other computer languages; it's highly context sensitive at times, it can 677be tricky to work out what sort of token something is, or where a token 678ends. As such, there's a lot of interplay between the tokeniser and the 679parser, which can get pretty frightening if you're not used to it. 680 681As the parser understands a Perl program, it builds up a tree of 682operations for the interpreter to perform during execution. The routines 683which construct and link together the various operations are to be found 684in F<op.c>, and will be examined later. 685 686=item Optimization 687 688Now the parsing stage is complete, and the finished tree represents 689the operations that the Perl interpreter needs to perform to execute our 690program. Next, Perl does a dry run over the tree looking for 691optimisations: constant expressions such as C<3 + 4> will be computed 692now, and the optimizer will also see if any multiple operations can be 693replaced with a single one. For instance, to fetch the variable C<$foo>, 694instead of grabbing the glob C<*foo> and looking at the scalar 695component, the optimizer fiddles the op tree to use a function which 696directly looks up the scalar in question. The main optimizer is C<peep> 697in F<op.c>, and many ops have their own optimizing functions. 698 699=item Running 700 701Now we're finally ready to go: we have compiled Perl byte code, and all 702that's left to do is run it. The actual execution is done by the 703C<runops_standard> function in F<run.c>; more specifically, it's done by 704these three innocent looking lines: 705 706 while ((PL_op = CALL_FPTR(PL_op->op_ppaddr)(aTHX))) { 707 PERL_ASYNC_CHECK(); 708 } 709 710You may be more comfortable with the Perl version of that: 711 712 PERL_ASYNC_CHECK() while $Perl::op = &{$Perl::op->{function}}; 713 714Well, maybe not. Anyway, each op contains a function pointer, which 715stipulates the function which will actually carry out the operation. 716This function will return the next op in the sequence - this allows for 717things like C<if> which choose the next op dynamically at run time. 718The C<PERL_ASYNC_CHECK> makes sure that things like signals interrupt 719execution if required. 720 721The actual functions called are known as PP code, and they're spread 722between four files: F<pp_hot.c> contains the `hot' code, which is most 723often used and highly optimized, F<pp_sys.c> contains all the 724system-specific functions, F<pp_ctl.c> contains the functions which 725implement control structures (C<if>, C<while> and the like) and F<pp.c> 726contains everything else. These are, if you like, the C code for Perl's 727built-in functions and operators. 728 729=back 730 731=head2 Internal Variable Types 732 733You should by now have had a look at L<perlguts>, which tells you about 734Perl's internal variable types: SVs, HVs, AVs and the rest. If not, do 735that now. 736 737These variables are used not only to represent Perl-space variables, but 738also any constants in the code, as well as some structures completely 739internal to Perl. The symbol table, for instance, is an ordinary Perl 740hash. Your code is represented by an SV as it's read into the parser; 741any program files you call are opened via ordinary Perl filehandles, and 742so on. 743 744The core L<Devel::Peek|Devel::Peek> module lets us examine SVs from a 745Perl program. Let's see, for instance, how Perl treats the constant 746C<"hello">. 747 748 % perl -MDevel::Peek -e 'Dump("hello")' 749 1 SV = PV(0xa041450) at 0xa04ecbc 750 2 REFCNT = 1 751 3 FLAGS = (POK,READONLY,pPOK) 752 4 PV = 0xa0484e0 "hello"\0 753 5 CUR = 5 754 6 LEN = 6 755 756Reading C<Devel::Peek> output takes a bit of practise, so let's go 757through it line by line. 758 759Line 1 tells us we're looking at an SV which lives at C<0xa04ecbc> in 760memory. SVs themselves are very simple structures, but they contain a 761pointer to a more complex structure. In this case, it's a PV, a 762structure which holds a string value, at location C<0xa041450>. Line 2 763is the reference count; there are no other references to this data, so 764it's 1. 765 766Line 3 are the flags for this SV - it's OK to use it as a PV, it's a 767read-only SV (because it's a constant) and the data is a PV internally. 768Next we've got the contents of the string, starting at location 769C<0xa0484e0>. 770 771Line 5 gives us the current length of the string - note that this does 772B<not> include the null terminator. Line 6 is not the length of the 773string, but the length of the currently allocated buffer; as the string 774grows, Perl automatically extends the available storage via a routine 775called C<SvGROW>. 776 777You can get at any of these quantities from C very easily; just add 778C<Sv> to the name of the field shown in the snippet, and you've got a 779macro which will return the value: C<SvCUR(sv)> returns the current 780length of the string, C<SvREFCOUNT(sv)> returns the reference count, 781C<SvPV(sv, len)> returns the string itself with its length, and so on. 782More macros to manipulate these properties can be found in L<perlguts>. 783 784Let's take an example of manipulating a PV, from C<sv_catpvn>, in F<sv.c> 785 786 1 void 787 2 Perl_sv_catpvn(pTHX_ register SV *sv, register const char *ptr, register STRLEN len) 788 3 { 789 4 STRLEN tlen; 790 5 char *junk; 791 792 6 junk = SvPV_force(sv, tlen); 793 7 SvGROW(sv, tlen + len + 1); 794 8 if (ptr == junk) 795 9 ptr = SvPVX(sv); 796 10 Move(ptr,SvPVX(sv)+tlen,len,char); 797 11 SvCUR(sv) += len; 798 12 *SvEND(sv) = '\0'; 799 13 (void)SvPOK_only_UTF8(sv); /* validate pointer */ 800 14 SvTAINT(sv); 801 15 } 802 803This is a function which adds a string, C<ptr>, of length C<len> onto 804the end of the PV stored in C<sv>. The first thing we do in line 6 is 805make sure that the SV B<has> a valid PV, by calling the C<SvPV_force> 806macro to force a PV. As a side effect, C<tlen> gets set to the current 807value of the PV, and the PV itself is returned to C<junk>. 808 809In line 7, we make sure that the SV will have enough room to accommodate 810the old string, the new string and the null terminator. If C<LEN> isn't 811big enough, C<SvGROW> will reallocate space for us. 812 813Now, if C<junk> is the same as the string we're trying to add, we can 814grab the string directly from the SV; C<SvPVX> is the address of the PV 815in the SV. 816 817Line 10 does the actual catenation: the C<Move> macro moves a chunk of 818memory around: we move the string C<ptr> to the end of the PV - that's 819the start of the PV plus its current length. We're moving C<len> bytes 820of type C<char>. After doing so, we need to tell Perl we've extended the 821string, by altering C<CUR> to reflect the new length. C<SvEND> is a 822macro which gives us the end of the string, so that needs to be a 823C<"\0">. 824 825Line 13 manipulates the flags; since we've changed the PV, any IV or NV 826values will no longer be valid: if we have C<$a=10; $a.="6";> we don't 827want to use the old IV of 10. C<SvPOK_only_utf8> is a special UTF8-aware 828version of C<SvPOK_only>, a macro which turns off the IOK and NOK flags 829and turns on POK. The final C<SvTAINT> is a macro which launders tainted 830data if taint mode is turned on. 831 832AVs and HVs are more complicated, but SVs are by far the most common 833variable type being thrown around. Having seen something of how we 834manipulate these, let's go on and look at how the op tree is 835constructed. 836 837=head2 Op Trees 838 839First, what is the op tree, anyway? The op tree is the parsed 840representation of your program, as we saw in our section on parsing, and 841it's the sequence of operations that Perl goes through to execute your 842program, as we saw in L</Running>. 843 844An op is a fundamental operation that Perl can perform: all the built-in 845functions and operators are ops, and there are a series of ops which 846deal with concepts the interpreter needs internally - entering and 847leaving a block, ending a statement, fetching a variable, and so on. 848 849The op tree is connected in two ways: you can imagine that there are two 850"routes" through it, two orders in which you can traverse the tree. 851First, parse order reflects how the parser understood the code, and 852secondly, execution order tells perl what order to perform the 853operations in. 854 855The easiest way to examine the op tree is to stop Perl after it has 856finished parsing, and get it to dump out the tree. This is exactly what 857the compiler backends L<B::Terse|B::Terse> and L<B::Debug|B::Debug> do. 858 859Let's have a look at how Perl sees C<$a = $b + $c>: 860 861 % perl -MO=Terse -e '$a=$b+$c' 862 1 LISTOP (0x8179888) leave 863 2 OP (0x81798b0) enter 864 3 COP (0x8179850) nextstate 865 4 BINOP (0x8179828) sassign 866 5 BINOP (0x8179800) add [1] 867 6 UNOP (0x81796e0) null [15] 868 7 SVOP (0x80fafe0) gvsv GV (0x80fa4cc) *b 869 8 UNOP (0x81797e0) null [15] 870 9 SVOP (0x8179700) gvsv GV (0x80efeb0) *c 871 10 UNOP (0x816b4f0) null [15] 872 11 SVOP (0x816dcf0) gvsv GV (0x80fa460) *a 873 874Let's start in the middle, at line 4. This is a BINOP, a binary 875operator, which is at location C<0x8179828>. The specific operator in 876question is C<sassign> - scalar assignment - and you can find the code 877which implements it in the function C<pp_sassign> in F<pp_hot.c>. As a 878binary operator, it has two children: the add operator, providing the 879result of C<$b+$c>, is uppermost on line 5, and the left hand side is on 880line 10. 881 882Line 10 is the null op: this does exactly nothing. What is that doing 883there? If you see the null op, it's a sign that something has been 884optimized away after parsing. As we mentioned in L</Optimization>, 885the optimization stage sometimes converts two operations into one, for 886example when fetching a scalar variable. When this happens, instead of 887rewriting the op tree and cleaning up the dangling pointers, it's easier 888just to replace the redundant operation with the null op. Originally, 889the tree would have looked like this: 890 891 10 SVOP (0x816b4f0) rv2sv [15] 892 11 SVOP (0x816dcf0) gv GV (0x80fa460) *a 893 894That is, fetch the C<a> entry from the main symbol table, and then look 895at the scalar component of it: C<gvsv> (C<pp_gvsv> into F<pp_hot.c>) 896happens to do both these things. 897 898The right hand side, starting at line 5 is similar to what we've just 899seen: we have the C<add> op (C<pp_add> also in F<pp_hot.c>) add together 900two C<gvsv>s. 901 902Now, what's this about? 903 904 1 LISTOP (0x8179888) leave 905 2 OP (0x81798b0) enter 906 3 COP (0x8179850) nextstate 907 908C<enter> and C<leave> are scoping ops, and their job is to perform any 909housekeeping every time you enter and leave a block: lexical variables 910are tidied up, unreferenced variables are destroyed, and so on. Every 911program will have those first three lines: C<leave> is a list, and its 912children are all the statements in the block. Statements are delimited 913by C<nextstate>, so a block is a collection of C<nextstate> ops, with 914the ops to be performed for each statement being the children of 915C<nextstate>. C<enter> is a single op which functions as a marker. 916 917That's how Perl parsed the program, from top to bottom: 918 919 Program 920 | 921 Statement 922 | 923 = 924 / \ 925 / \ 926 $a + 927 / \ 928 $b $c 929 930However, it's impossible to B<perform> the operations in this order: 931you have to find the values of C<$b> and C<$c> before you add them 932together, for instance. So, the other thread that runs through the op 933tree is the execution order: each op has a field C<op_next> which points 934to the next op to be run, so following these pointers tells us how perl 935executes the code. We can traverse the tree in this order using 936the C<exec> option to C<B::Terse>: 937 938 % perl -MO=Terse,exec -e '$a=$b+$c' 939 1 OP (0x8179928) enter 940 2 COP (0x81798c8) nextstate 941 3 SVOP (0x81796c8) gvsv GV (0x80fa4d4) *b 942 4 SVOP (0x8179798) gvsv GV (0x80efeb0) *c 943 5 BINOP (0x8179878) add [1] 944 6 SVOP (0x816dd38) gvsv GV (0x80fa468) *a 945 7 BINOP (0x81798a0) sassign 946 8 LISTOP (0x8179900) leave 947 948This probably makes more sense for a human: enter a block, start a 949statement. Get the values of C<$b> and C<$c>, and add them together. 950Find C<$a>, and assign one to the other. Then leave. 951 952The way Perl builds up these op trees in the parsing process can be 953unravelled by examining F<perly.y>, the YACC grammar. Let's take the 954piece we need to construct the tree for C<$a = $b + $c> 955 956 1 term : term ASSIGNOP term 957 2 { $$ = newASSIGNOP(OPf_STACKED, $1, $2, $3); } 958 3 | term ADDOP term 959 4 { $$ = newBINOP($2, 0, scalar($1), scalar($3)); } 960 961If you're not used to reading BNF grammars, this is how it works: You're 962fed certain things by the tokeniser, which generally end up in upper 963case. Here, C<ADDOP>, is provided when the tokeniser sees C<+> in your 964code. C<ASSIGNOP> is provided when C<=> is used for assigning. These are 965`terminal symbols', because you can't get any simpler than them. 966 967The grammar, lines one and three of the snippet above, tells you how to 968build up more complex forms. These complex forms, `non-terminal symbols' 969are generally placed in lower case. C<term> here is a non-terminal 970symbol, representing a single expression. 971 972The grammar gives you the following rule: you can make the thing on the 973left of the colon if you see all the things on the right in sequence. 974This is called a "reduction", and the aim of parsing is to completely 975reduce the input. There are several different ways you can perform a 976reduction, separated by vertical bars: so, C<term> followed by C<=> 977followed by C<term> makes a C<term>, and C<term> followed by C<+> 978followed by C<term> can also make a C<term>. 979 980So, if you see two terms with an C<=> or C<+>, between them, you can 981turn them into a single expression. When you do this, you execute the 982code in the block on the next line: if you see C<=>, you'll do the code 983in line 2. If you see C<+>, you'll do the code in line 4. It's this code 984which contributes to the op tree. 985 986 | term ADDOP term 987 { $$ = newBINOP($2, 0, scalar($1), scalar($3)); } 988 989What this does is creates a new binary op, and feeds it a number of 990variables. The variables refer to the tokens: C<$1> is the first token in 991the input, C<$2> the second, and so on - think regular expression 992backreferences. C<$$> is the op returned from this reduction. So, we 993call C<newBINOP> to create a new binary operator. The first parameter to 994C<newBINOP>, a function in F<op.c>, is the op type. It's an addition 995operator, so we want the type to be C<ADDOP>. We could specify this 996directly, but it's right there as the second token in the input, so we 997use C<$2>. The second parameter is the op's flags: 0 means `nothing 998special'. Then the things to add: the left and right hand side of our 999expression, in scalar context. 1000 1001=head2 Stacks 1002 1003When perl executes something like C<addop>, how does it pass on its 1004results to the next op? The answer is, through the use of stacks. Perl 1005has a number of stacks to store things it's currently working on, and 1006we'll look at the three most important ones here. 1007 1008=over 3 1009 1010=item Argument stack 1011 1012Arguments are passed to PP code and returned from PP code using the 1013argument stack, C<ST>. The typical way to handle arguments is to pop 1014them off the stack, deal with them how you wish, and then push the result 1015back onto the stack. This is how, for instance, the cosine operator 1016works: 1017 1018 NV value; 1019 value = POPn; 1020 value = Perl_cos(value); 1021 XPUSHn(value); 1022 1023We'll see a more tricky example of this when we consider Perl's macros 1024below. C<POPn> gives you the NV (floating point value) of the top SV on 1025the stack: the C<$x> in C<cos($x)>. Then we compute the cosine, and push 1026the result back as an NV. The C<X> in C<XPUSHn> means that the stack 1027should be extended if necessary - it can't be necessary here, because we 1028know there's room for one more item on the stack, since we've just 1029removed one! The C<XPUSH*> macros at least guarantee safety. 1030 1031Alternatively, you can fiddle with the stack directly: C<SP> gives you 1032the first element in your portion of the stack, and C<TOP*> gives you 1033the top SV/IV/NV/etc. on the stack. So, for instance, to do unary 1034negation of an integer: 1035 1036 SETi(-TOPi); 1037 1038Just set the integer value of the top stack entry to its negation. 1039 1040Argument stack manipulation in the core is exactly the same as it is in 1041XSUBs - see L<perlxstut>, L<perlxs> and L<perlguts> for a longer 1042description of the macros used in stack manipulation. 1043 1044=item Mark stack 1045 1046I say `your portion of the stack' above because PP code doesn't 1047necessarily get the whole stack to itself: if your function calls 1048another function, you'll only want to expose the arguments aimed for the 1049called function, and not (necessarily) let it get at your own data. The 1050way we do this is to have a `virtual' bottom-of-stack, exposed to each 1051function. The mark stack keeps bookmarks to locations in the argument 1052stack usable by each function. For instance, when dealing with a tied 1053variable, (internally, something with `P' magic) Perl has to call 1054methods for accesses to the tied variables. However, we need to separate 1055the arguments exposed to the method to the argument exposed to the 1056original function - the store or fetch or whatever it may be. Here's how 1057the tied C<push> is implemented; see C<av_push> in F<av.c>: 1058 1059 1 PUSHMARK(SP); 1060 2 EXTEND(SP,2); 1061 3 PUSHs(SvTIED_obj((SV*)av, mg)); 1062 4 PUSHs(val); 1063 5 PUTBACK; 1064 6 ENTER; 1065 7 call_method("PUSH", G_SCALAR|G_DISCARD); 1066 8 LEAVE; 1067 9 POPSTACK; 1068 1069The lines which concern the mark stack are the first, fifth and last 1070lines: they save away, restore and remove the current position of the 1071argument stack. 1072 1073Let's examine the whole implementation, for practice: 1074 1075 1 PUSHMARK(SP); 1076 1077Push the current state of the stack pointer onto the mark stack. This is 1078so that when we've finished adding items to the argument stack, Perl 1079knows how many things we've added recently. 1080 1081 2 EXTEND(SP,2); 1082 3 PUSHs(SvTIED_obj((SV*)av, mg)); 1083 4 PUSHs(val); 1084 1085We're going to add two more items onto the argument stack: when you have 1086a tied array, the C<PUSH> subroutine receives the object and the value 1087to be pushed, and that's exactly what we have here - the tied object, 1088retrieved with C<SvTIED_obj>, and the value, the SV C<val>. 1089 1090 5 PUTBACK; 1091 1092Next we tell Perl to make the change to the global stack pointer: C<dSP> 1093only gave us a local copy, not a reference to the global. 1094 1095 6 ENTER; 1096 7 call_method("PUSH", G_SCALAR|G_DISCARD); 1097 8 LEAVE; 1098 1099C<ENTER> and C<LEAVE> localise a block of code - they make sure that all 1100variables are tidied up, everything that has been localised gets 1101its previous value returned, and so on. Think of them as the C<{> and 1102C<}> of a Perl block. 1103 1104To actually do the magic method call, we have to call a subroutine in 1105Perl space: C<call_method> takes care of that, and it's described in 1106L<perlcall>. We call the C<PUSH> method in scalar context, and we're 1107going to discard its return value. 1108 1109 9 POPSTACK; 1110 1111Finally, we remove the value we placed on the mark stack, since we 1112don't need it any more. 1113 1114=item Save stack 1115 1116C doesn't have a concept of local scope, so perl provides one. We've 1117seen that C<ENTER> and C<LEAVE> are used as scoping braces; the save 1118stack implements the C equivalent of, for example: 1119 1120 { 1121 local $foo = 42; 1122 ... 1123 } 1124 1125See L<perlguts/Localising Changes> for how to use the save stack. 1126 1127=back 1128 1129=head2 Millions of Macros 1130 1131One thing you'll notice about the Perl source is that it's full of 1132macros. Some have called the pervasive use of macros the hardest thing 1133to understand, others find it adds to clarity. Let's take an example, 1134the code which implements the addition operator: 1135 1136 1 PP(pp_add) 1137 2 { 1138 3 dSP; dATARGET; tryAMAGICbin(add,opASSIGN); 1139 4 { 1140 5 dPOPTOPnnrl_ul; 1141 6 SETn( left + right ); 1142 7 RETURN; 1143 8 } 1144 9 } 1145 1146Every line here (apart from the braces, of course) contains a macro. The 1147first line sets up the function declaration as Perl expects for PP code; 1148line 3 sets up variable declarations for the argument stack and the 1149target, the return value of the operation. Finally, it tries to see if 1150the addition operation is overloaded; if so, the appropriate subroutine 1151is called. 1152 1153Line 5 is another variable declaration - all variable declarations start 1154with C<d> - which pops from the top of the argument stack two NVs (hence 1155C<nn>) and puts them into the variables C<right> and C<left>, hence the 1156C<rl>. These are the two operands to the addition operator. Next, we 1157call C<SETn> to set the NV of the return value to the result of adding 1158the two values. This done, we return - the C<RETURN> macro makes sure 1159that our return value is properly handled, and we pass the next operator 1160to run back to the main run loop. 1161 1162Most of these macros are explained in L<perlapi>, and some of the more 1163important ones are explained in L<perlxs> as well. Pay special attention 1164to L<perlguts/Background and PERL_IMPLICIT_CONTEXT> for information on 1165the C<[pad]THX_?> macros. 1166 1167 1168=head2 Poking at Perl 1169 1170To really poke around with Perl, you'll probably want to build Perl for 1171debugging, like this: 1172 1173 ./Configure -d -D optimize=-g 1174 make 1175 1176C<-g> is a flag to the C compiler to have it produce debugging 1177information which will allow us to step through a running program. 1178F<Configure> will also turn on the C<DEBUGGING> compilation symbol which 1179enables all the internal debugging code in Perl. There are a whole bunch 1180of things you can debug with this: L<perlrun> lists them all, and the 1181best way to find out about them is to play about with them. The most 1182useful options are probably 1183 1184 l Context (loop) stack processing 1185 t Trace execution 1186 o Method and overloading resolution 1187 c String/numeric conversions 1188 1189Some of the functionality of the debugging code can be achieved using XS 1190modules. 1191 1192 -Dr => use re 'debug' 1193 -Dx => use O 'Debug' 1194 1195=head2 Using a source-level debugger 1196 1197If the debugging output of C<-D> doesn't help you, it's time to step 1198through perl's execution with a source-level debugger. 1199 1200=over 3 1201 1202=item * 1203 1204We'll use C<gdb> for our examples here; the principles will apply to any 1205debugger, but check the manual of the one you're using. 1206 1207=back 1208 1209To fire up the debugger, type 1210 1211 gdb ./perl 1212 1213You'll want to do that in your Perl source tree so the debugger can read 1214the source code. You should see the copyright message, followed by the 1215prompt. 1216 1217 (gdb) 1218 1219C<help> will get you into the documentation, but here are the most 1220useful commands: 1221 1222=over 3 1223 1224=item run [args] 1225 1226Run the program with the given arguments. 1227 1228=item break function_name 1229 1230=item break source.c:xxx 1231 1232Tells the debugger that we'll want to pause execution when we reach 1233either the named function (but see L<perlguts/Internal Functions>!) or the given 1234line in the named source file. 1235 1236=item step 1237 1238Steps through the program a line at a time. 1239 1240=item next 1241 1242Steps through the program a line at a time, without descending into 1243functions. 1244 1245=item continue 1246 1247Run until the next breakpoint. 1248 1249=item finish 1250 1251Run until the end of the current function, then stop again. 1252 1253=item 'enter' 1254 1255Just pressing Enter will do the most recent operation again - it's a 1256blessing when stepping through miles of source code. 1257 1258=item print 1259 1260Execute the given C code and print its results. B<WARNING>: Perl makes 1261heavy use of macros, and F<gdb> is not aware of macros. You'll have to 1262substitute them yourself. So, for instance, you can't say 1263 1264 print SvPV_nolen(sv) 1265 1266but you have to say 1267 1268 print Perl_sv_2pv_nolen(sv) 1269 1270You may find it helpful to have a "macro dictionary", which you can 1271produce by saying C<cpp -dM perl.c | sort>. Even then, F<cpp> won't 1272recursively apply the macros for you. 1273 1274=back 1275 1276=head2 Dumping Perl Data Structures 1277 1278One way to get around this macro hell is to use the dumping functions in 1279F<dump.c>; these work a little like an internal 1280L<Devel::Peek|Devel::Peek>, but they also cover OPs and other structures 1281that you can't get at from Perl. Let's take an example. We'll use the 1282C<$a = $b + $c> we used before, but give it a bit of context: 1283C<$b = "6XXXX"; $c = 2.3;>. Where's a good place to stop and poke around? 1284 1285What about C<pp_add>, the function we examined earlier to implement the 1286C<+> operator: 1287 1288 (gdb) break Perl_pp_add 1289 Breakpoint 1 at 0x46249f: file pp_hot.c, line 309. 1290 1291Notice we use C<Perl_pp_add> and not C<pp_add> - see L<perlguts/Internal Functions>. 1292With the breakpoint in place, we can run our program: 1293 1294 (gdb) run -e '$b = "6XXXX"; $c = 2.3; $a = $b + $c' 1295 1296Lots of junk will go past as gdb reads in the relevant source files and 1297libraries, and then: 1298 1299 Breakpoint 1, Perl_pp_add () at pp_hot.c:309 1300 309 dSP; dATARGET; tryAMAGICbin(add,opASSIGN); 1301 (gdb) step 1302 311 dPOPTOPnnrl_ul; 1303 (gdb) 1304 1305We looked at this bit of code before, and we said that C<dPOPTOPnnrl_ul> 1306arranges for two C<NV>s to be placed into C<left> and C<right> - let's 1307slightly expand it: 1308 1309 #define dPOPTOPnnrl_ul NV right = POPn; \ 1310 SV *leftsv = TOPs; \ 1311 NV left = USE_LEFT(leftsv) ? SvNV(leftsv) : 0.0 1312 1313C<POPn> takes the SV from the top of the stack and obtains its NV either 1314directly (if C<SvNOK> is set) or by calling the C<sv_2nv> function. 1315C<TOPs> takes the next SV from the top of the stack - yes, C<POPn> uses 1316C<TOPs> - but doesn't remove it. We then use C<SvNV> to get the NV from 1317C<leftsv> in the same way as before - yes, C<POPn> uses C<SvNV>. 1318 1319Since we don't have an NV for C<$b>, we'll have to use C<sv_2nv> to 1320convert it. If we step again, we'll find ourselves there: 1321 1322 Perl_sv_2nv (sv=0xa0675d0) at sv.c:1669 1323 1669 if (!sv) 1324 (gdb) 1325 1326We can now use C<Perl_sv_dump> to investigate the SV: 1327 1328 SV = PV(0xa057cc0) at 0xa0675d0 1329 REFCNT = 1 1330 FLAGS = (POK,pPOK) 1331 PV = 0xa06a510 "6XXXX"\0 1332 CUR = 5 1333 LEN = 6 1334 $1 = void 1335 1336We know we're going to get C<6> from this, so let's finish the 1337subroutine: 1338 1339 (gdb) finish 1340 Run till exit from #0 Perl_sv_2nv (sv=0xa0675d0) at sv.c:1671 1341 0x462669 in Perl_pp_add () at pp_hot.c:311 1342 311 dPOPTOPnnrl_ul; 1343 1344We can also dump out this op: the current op is always stored in 1345C<PL_op>, and we can dump it with C<Perl_op_dump>. This'll give us 1346similar output to L<B::Debug|B::Debug>. 1347 1348 { 1349 13 TYPE = add ===> 14 1350 TARG = 1 1351 FLAGS = (SCALAR,KIDS) 1352 { 1353 TYPE = null ===> (12) 1354 (was rv2sv) 1355 FLAGS = (SCALAR,KIDS) 1356 { 1357 11 TYPE = gvsv ===> 12 1358 FLAGS = (SCALAR) 1359 GV = main::b 1360 } 1361 } 1362 1363< finish this later > 1364 1365=head2 Patching 1366 1367All right, we've now had a look at how to navigate the Perl sources and 1368some things you'll need to know when fiddling with them. Let's now get 1369on and create a simple patch. Here's something Larry suggested: if a 1370C<U> is the first active format during a C<pack>, (for example, 1371C<pack "U3C8", @stuff>) then the resulting string should be treated as 1372UTF8 encoded. 1373 1374How do we prepare to fix this up? First we locate the code in question - 1375the C<pack> happens at runtime, so it's going to be in one of the F<pp> 1376files. Sure enough, C<pp_pack> is in F<pp.c>. Since we're going to be 1377altering this file, let's copy it to F<pp.c~>. 1378 1379Now let's look over C<pp_pack>: we take a pattern into C<pat>, and then 1380loop over the pattern, taking each format character in turn into 1381C<datum_type>. Then for each possible format character, we swallow up 1382the other arguments in the pattern (a field width, an asterisk, and so 1383on) and convert the next chunk input into the specified format, adding 1384it onto the output SV C<cat>. 1385 1386How do we know if the C<U> is the first format in the C<pat>? Well, if 1387we have a pointer to the start of C<pat> then, if we see a C<U> we can 1388test whether we're still at the start of the string. So, here's where 1389C<pat> is set up: 1390 1391 STRLEN fromlen; 1392 register char *pat = SvPVx(*++MARK, fromlen); 1393 register char *patend = pat + fromlen; 1394 register I32 len; 1395 I32 datumtype; 1396 SV *fromstr; 1397 1398We'll have another string pointer in there: 1399 1400 STRLEN fromlen; 1401 register char *pat = SvPVx(*++MARK, fromlen); 1402 register char *patend = pat + fromlen; 1403 + char *patcopy; 1404 register I32 len; 1405 I32 datumtype; 1406 SV *fromstr; 1407 1408And just before we start the loop, we'll set C<patcopy> to be the start 1409of C<pat>: 1410 1411 items = SP - MARK; 1412 MARK++; 1413 sv_setpvn(cat, "", 0); 1414 + patcopy = pat; 1415 while (pat < patend) { 1416 1417Now if we see a C<U> which was at the start of the string, we turn on 1418the UTF8 flag for the output SV, C<cat>: 1419 1420 + if (datumtype == 'U' && pat==patcopy+1) 1421 + SvUTF8_on(cat); 1422 if (datumtype == '#') { 1423 while (pat < patend && *pat != '\n') 1424 pat++; 1425 1426Remember that it has to be C<patcopy+1> because the first character of 1427the string is the C<U> which has been swallowed into C<datumtype!> 1428 1429Oops, we forgot one thing: what if there are spaces at the start of the 1430pattern? C<pack(" U*", @stuff)> will have C<U> as the first active 1431character, even though it's not the first thing in the pattern. In this 1432case, we have to advance C<patcopy> along with C<pat> when we see spaces: 1433 1434 if (isSPACE(datumtype)) 1435 continue; 1436 1437needs to become 1438 1439 if (isSPACE(datumtype)) { 1440 patcopy++; 1441 continue; 1442 } 1443 1444OK. That's the C part done. Now we must do two additional things before 1445this patch is ready to go: we've changed the behaviour of Perl, and so 1446we must document that change. We must also provide some more regression 1447tests to make sure our patch works and doesn't create a bug somewhere 1448else along the line. 1449 1450The regression tests for each operator live in F<t/op/>, and so we make 1451a copy of F<t/op/pack.t> to F<t/op/pack.t~>. Now we can add our tests 1452to the end. First, we'll test that the C<U> does indeed create Unicode 1453strings: 1454 1455 print 'not ' unless "1.20.300.4000" eq sprintf "%vd", pack("U*",1,20,300,4000); 1456 print "ok $test\n"; $test++; 1457 1458Now we'll test that we got that space-at-the-beginning business right: 1459 1460 print 'not ' unless "1.20.300.4000" eq 1461 sprintf "%vd", pack(" U*",1,20,300,4000); 1462 print "ok $test\n"; $test++; 1463 1464And finally we'll test that we don't make Unicode strings if C<U> is B<not> 1465the first active format: 1466 1467 print 'not ' unless v1.20.300.4000 ne 1468 sprintf "%vd", pack("C0U*",1,20,300,4000); 1469 print "ok $test\n"; $test++; 1470 1471Mustn't forget to change the number of tests which appears at the top, or 1472else the automated tester will get confused: 1473 1474 -print "1..156\n"; 1475 +print "1..159\n"; 1476 1477We now compile up Perl, and run it through the test suite. Our new 1478tests pass, hooray! 1479 1480Finally, the documentation. The job is never done until the paperwork is 1481over, so let's describe the change we've just made. The relevant place 1482is F<pod/perlfunc.pod>; again, we make a copy, and then we'll insert 1483this text in the description of C<pack>: 1484 1485 =item * 1486 1487 If the pattern begins with a C<U>, the resulting string will be treated 1488 as Unicode-encoded. You can force UTF8 encoding on in a string with an 1489 initial C<U0>, and the bytes that follow will be interpreted as Unicode 1490 characters. If you don't want this to happen, you can begin your pattern 1491 with C<C0> (or anything else) to force Perl not to UTF8 encode your 1492 string, and then follow this with a C<U*> somewhere in your pattern. 1493 1494All done. Now let's create the patch. F<Porting/patching.pod> tells us 1495that if we're making major changes, we should copy the entire directory 1496to somewhere safe before we begin fiddling, and then do 1497 1498 diff -ruN old new > patch 1499 1500However, we know which files we've changed, and we can simply do this: 1501 1502 diff -u pp.c~ pp.c > patch 1503 diff -u t/op/pack.t~ t/op/pack.t >> patch 1504 diff -u pod/perlfunc.pod~ pod/perlfunc.pod >> patch 1505 1506We end up with a patch looking a little like this: 1507 1508 --- pp.c~ Fri Jun 02 04:34:10 2000 1509 +++ pp.c Fri Jun 16 11:37:25 2000 1510 @@ -4375,6 +4375,7 @@ 1511 register I32 items; 1512 STRLEN fromlen; 1513 register char *pat = SvPVx(*++MARK, fromlen); 1514 + char *patcopy; 1515 register char *patend = pat + fromlen; 1516 register I32 len; 1517 I32 datumtype; 1518 @@ -4405,6 +4406,7 @@ 1519 ... 1520 1521And finally, we submit it, with our rationale, to perl5-porters. Job 1522done! 1523 1524=head1 EXTERNAL TOOLS FOR DEBUGGING PERL 1525 1526Sometimes it helps to use external tools while debugging and 1527testing Perl. This section tries to guide you through using 1528some common testing and debugging tools with Perl. This is 1529meant as a guide to interfacing these tools with Perl, not 1530as any kind of guide to the use of the tools themselves. 1531 1532=head2 Rational Software's Purify 1533 1534Purify is a commercial tool that is helpful in identifying 1535memory overruns, wild pointers, memory leaks and other such 1536badness. Perl must be compiled in a specific way for 1537optimal testing with Purify. Purify is available under 1538Windows NT, Solaris, HP-UX, SGI, and Siemens Unix. 1539 1540The only currently known leaks happen when there are 1541compile-time errors within eval or require. (Fixing these 1542is non-trivial, unfortunately, but they must be fixed 1543eventually.) 1544 1545=head2 Purify on Unix 1546 1547On Unix, Purify creates a new Perl binary. To get the most 1548benefit out of Purify, you should create the perl to Purify 1549using: 1550 1551 sh Configure -Accflags=-DPURIFY -Doptimize='-g' \ 1552 -Uusemymalloc -Dusemultiplicity 1553 1554where these arguments mean: 1555 1556=over 4 1557 1558=item -Accflags=-DPURIFY 1559 1560Disables Perl's arena memory allocation functions, as well as 1561forcing use of memory allocation functions derived from the 1562system malloc. 1563 1564=item -Doptimize='-g' 1565 1566Adds debugging information so that you see the exact source 1567statements where the problem occurs. Without this flag, all 1568you will see is the source filename of where the error occurred. 1569 1570=item -Uusemymalloc 1571 1572Disable Perl's malloc so that Purify can more closely monitor 1573allocations and leaks. Using Perl's malloc will make Purify 1574report most leaks in the "potential" leaks category. 1575 1576=item -Dusemultiplicity 1577 1578Enabling the multiplicity option allows perl to clean up 1579thoroughly when the interpreter shuts down, which reduces the 1580number of bogus leak reports from Purify. 1581 1582=back 1583 1584Once you've compiled a perl suitable for Purify'ing, then you 1585can just: 1586 1587 make pureperl 1588 1589which creates a binary named 'pureperl' that has been Purify'ed. 1590This binary is used in place of the standard 'perl' binary 1591when you want to debug Perl memory problems. 1592 1593As an example, to show any memory leaks produced during the 1594standard Perl testset you would create and run the Purify'ed 1595perl as: 1596 1597 make pureperl 1598 cd t 1599 ../pureperl -I../lib harness 1600 1601which would run Perl on test.pl and report any memory problems. 1602 1603Purify outputs messages in "Viewer" windows by default. If 1604you don't have a windowing environment or if you simply 1605want the Purify output to unobtrusively go to a log file 1606instead of to the interactive window, use these following 1607options to output to the log file "perl.log": 1608 1609 setenv PURIFYOPTIONS "-chain-length=25 -windows=no \ 1610 -log-file=perl.log -append-logfile=yes" 1611 1612If you plan to use the "Viewer" windows, then you only need this option: 1613 1614 setenv PURIFYOPTIONS "-chain-length=25" 1615 1616=head2 Purify on NT 1617 1618Purify on Windows NT instruments the Perl binary 'perl.exe' 1619on the fly. There are several options in the makefile you 1620should change to get the most use out of Purify: 1621 1622=over 4 1623 1624=item DEFINES 1625 1626You should add -DPURIFY to the DEFINES line so the DEFINES 1627line looks something like: 1628 1629 DEFINES = -DWIN32 -D_CONSOLE -DNO_STRICT $(CRYPT_FLAG) -DPURIFY=1 1630 1631to disable Perl's arena memory allocation functions, as 1632well as to force use of memory allocation functions derived 1633from the system malloc. 1634 1635=item USE_MULTI = define 1636 1637Enabling the multiplicity option allows perl to clean up 1638thoroughly when the interpreter shuts down, which reduces the 1639number of bogus leak reports from Purify. 1640 1641=item #PERL_MALLOC = define 1642 1643Disable Perl's malloc so that Purify can more closely monitor 1644allocations and leaks. Using Perl's malloc will make Purify 1645report most leaks in the "potential" leaks category. 1646 1647=item CFG = Debug 1648 1649Adds debugging information so that you see the exact source 1650statements where the problem occurs. Without this flag, all 1651you will see is the source filename of where the error occurred. 1652 1653=back 1654 1655As an example, to show any memory leaks produced during the 1656standard Perl testset you would create and run Purify as: 1657 1658 cd win32 1659 make 1660 cd ../t 1661 purify ../perl -I../lib harness 1662 1663which would instrument Perl in memory, run Perl on test.pl, 1664then finally report any memory problems. 1665 1666=head2 CONCLUSION 1667 1668We've had a brief look around the Perl source, an overview of the stages 1669F<perl> goes through when it's running your code, and how to use a 1670debugger to poke at the Perl guts. We took a very simple problem and 1671demonstrated how to solve it fully - with documentation, regression 1672tests, and finally a patch for submission to p5p. Finally, we talked 1673about how to use external tools to debug and test Perl. 1674 1675I'd now suggest you read over those references again, and then, as soon 1676as possible, get your hands dirty. The best way to learn is by doing, 1677so: 1678 1679=over 3 1680 1681=item * 1682 1683Subscribe to perl5-porters, follow the patches and try and understand 1684them; don't be afraid to ask if there's a portion you're not clear on - 1685who knows, you may unearth a bug in the patch... 1686 1687=item * 1688 1689Keep up to date with the bleeding edge Perl distributions and get 1690familiar with the changes. Try and get an idea of what areas people are 1691working on and the changes they're making. 1692 1693=item * 1694 1695Do read the README associated with your operating system, e.g. README.aix 1696on the IBM AIX OS. Don't hesitate to supply patches to that README if 1697you find anything missing or changed over a new OS release. 1698 1699=item * 1700 1701Find an area of Perl that seems interesting to you, and see if you can 1702work out how it works. Scan through the source, and step over it in the 1703debugger. Play, poke, investigate, fiddle! You'll probably get to 1704understand not just your chosen area but a much wider range of F<perl>'s 1705activity as well, and probably sooner than you'd think. 1706 1707=back 1708 1709=over 3 1710 1711=item I<The Road goes ever on and on, down from the door where it began.> 1712 1713=back 1714 1715If you can do these things, you've started on the long road to Perl porting. 1716Thanks for wanting to help make Perl better - and happy hacking! 1717 1718=head1 AUTHOR 1719 1720This document was written by Nathan Torkington, and is maintained by 1721the perl5-porters mailing list. 1722 1723