1 SCCS Id: @(#)ovlmgr.doc 3.1 91/02/02 2 Copyright (c) 1989, 1990, 1991, 1992, 1993 Pierre G Martineau and 3 Stephen P Spackman. All Rights Reserved. 4 NetHack may be freely redistributed. See license for details. 5 ==================================== 6 Brief notes about ovlmgr.asm [v30a0] 7 ==================================== 8 (revised 1991february02) 9 10OVLMGR.ASM is a multiple-residency overlay manager for use with the 11Microsoft Overlay Linker. It is functionally compatible with the one 12in the MSC library _except_: 13 14- it usually accesses the disk less often and is a lot faster in some 15 applications. 16- it permits overlays to be stored in the .EXE file and/or in separate 17 .OVL files. 18- it has different tuning characteristics. 19- you must (of course) link OVLMGR.OBJ into the root overlay (that is, 20 outside any parentheses in the link command). 21 22See also the notes below. 23 24 As with other Microsoft-compatible overlay handlers you must 25be *very* careful never to call a function in an overlay through a 26pointer, unless the initiator of the call resides in the *same* 27physical overlay as the target (This is, of course, *not* the same 28thing as the called function being declared static, since the static 29declaration affects only the visibility of the name of the function, 30not the distribution of pointers to it.) (1). Furthermore, setjmp() 31and longjmp() are not supported. 32 33 Unlike the Microsoft system, most of the available memory is 34used to hold overlays. Care must be taken to ensure that enough space 35is reserved for the C heap. This can be accomplished through 36information stored in the .EXE file (currently the minalloc parameter, 37as described below). 38 39 Furthermore, expanded memory support (EMS) is now an integral 40part of the overlay manager. LIM EMS versions 3.2 and 4.0 are 41supported. Note that the page frame must be 4 pages long (64K bytes) to 42be able to operate correctly (most drivers allocate a 64K frame by 43default). The overlay manager will use as much EMS as is necessary in 4464K chunks, up to a limit of 16 chunks (1 Meg). Both hardware and 45software EMS drivers have been tested and found to be completely 46compatible. 47 48 Starting with version 30a0, overlays are not restricted to 49being stored in the main .EXE files (as they are with Microsoft's 50overlay manager). Using the utility EXESMURF arbitrary contiguous 51sequences of overlays can be unloaded into external overlay files. 52Although EXESMURF provides some flexibility in naming these files, 53OVLMGR presently only supports its default option, whereby the 54overlays of a programme PROGRAM.EXE must match the pattern 55PROGRAM?.OVL (if the basename had eight characters, as FILENAME.EXE, 56then the last character is replaced: FILENAM?.EXE) and reside in the 57*same* directory as the .EXE (not even a path search is performed). 58This mechanism permits large applications to be represented with small 59files, resulting in a slight performance improvement (due to less and 60shorter disk seeking) and easier transfer with floppy disks, at the 61cost of a heavier demand for file-handles. 62 63 ~ * ~ 64 65 OVLMGR.ASM currently has three assembly-time options, which are 66specified with the assembler's /D<symbol> option (or compatible). They 67are: 68 69 /DNOEMS Disable EMS support. 70 OVLMGR normally detects the presence of EMS memory 71 and makes use of it whenever it is present. This 72 flag instructs ovlmgr to ignore EMS and operate only 73 out of conventional memory. It should be used when 74 overlaying programmes which expect to use EMS 75 themselves. 76 77 /Di386 Use 80386-specific instruction sequences. 78 Use of this flag will make ovlmgr perform better on 79 machines with 80386 processors. However, the 80 resulting programme will not run at all on machines 81 with less capable CPUs. Use this option with 82 caution, especially in the case of distribution 83 code. 84 85 /DNOSPLIT Do not provide for external .OVL files. 86 If this flag is NOT set, OVLMGR will look for 87 overlays for the programme PROGRAM.EXE in all files 88 matching the pattern PROGRAM?.OVL, as well as in the 89 .EXE file itself. This arrangement may be slightly 90 faster and will result in more, smaller files, but 91 is obviously less robust, since mismatched .OVL 92 files can cause mayhem. .OVL files can be generated 93 with our EXESMURF .EXE file manipulation utility. 94 95 ~ * ~ 96 97 Although using the overlay manager is in essence much like using 98Microsoft's, they operate on a slightly different principle, and tuning 99for them is rather different. Technical part begins. 100 101 When overlay linking is requested (see your linker manual), the 102MS overlay linker changes all far calls into overlays from the (normal, 1038086) format: 104 105 offset contents 106 ------ -------- 107 :0000 CALL 108 :0001 target-offset 109 :0003 target-segment 110 111to this: 112 :0000 INT 113 :0001 int# target-mod# 114 :0003 target-offset 115 116(note that here we are looking at the actual layout of the machine 117code, not at the assembly code as such) and relocates the code parts 118of all the different overlays into the *same* physical area. The 119overlaid code is all actually placed at the end of the .EXE file, 120after the 'normal' executable image, along with all its administrative 121data (fixups etc.). 122 123 When this altered 'call' is executed, of course, the interrupt 124handler int# is invoked. Its job is to ensure that the target overlay 125module is in memory (reading it from the tail of the .EXE file if it 126isn't already loaded) and then transfer to the given offset within it, 127'faking up' the effect of the 'real' far call that would normally have 128occurred. Something similar must be done when the call returns, to 129ensure that the thing being returned *into* is still (or is once more) 130loaded. 131 132 The Microsoft linker, as we have said, relocates all the 133overlays to the same load address; and, in fact, it allocates am empty 134block of memory there that is at least as large as the largest 135overlay. Into this area all the overlays are loaded without further 136change; thus, there can only ever be one overlay in memory at one 137time. Transferring from one overlay to another causes one overlay to 138replace the other in the allocated overlay swap area. 139 140 Our overlay manager does not use the space allocated by the 141linker in the same way. Rather, it allocates almost all of the memory 142available from MS-DOS (including the original overlay area and any high 143DOS memory) as well as EMS memory if some is available and that option 144is being used. As overlays are needed, they are loaded wherever they 145will fit, and dynamically relocated to that address. Thus, many more 146than one overlay may be loaded at any given time, greatly increasing 147potential performance. Management of space is more or less according to 148an LRU policy - once all of memory is full, the least recently used 149overlay is selected as the most likely candidate for replacement. 150 151 The implications of this difference are as follows: while with 152the conventional (default) overlay manager, the best strategy is to 153group object modules together in an overlay whenever they are known to 154be used in rapid succession, to make each overlay as big as possible 155(all things being equal) in order to take advantage of all available 156memory, and to make as few overlays as possible (to reduce the amount of 157disk access), the best strategy with our overlay manager is almost the 158reverse. Having a lot of small overlays will increase the amount of 159useful stuff that can be resident in memory at the same time; all of 160memory will automatically be employed; and there is no advantage at all 161to uniformity of size (except perhaps in the unlikely case of *exact* 162uniformity!). 163 164 Although ovlmgr allocates all available memory while it is 165active, you will find that the DOS exec() call works normally. The 166memory that is allocated for administering the overlay system is freed 167before the exec call is made and reallocated afterwards (we trap the DOS 168function request vector to do this, which isn't very nice as a 169programming practise but makes the existence of the overlay manager far 170more transparent). There is, however, one circumstance under which this 171can be problematic: if you use the exec() call to load a TSR 172application, thereby causing memory that the overlay manager was using 173to become unavailable, you may make it impossible for the overlaid 174application to proceed. This is because code that is nominally 175'running' (i.e. is currently on the stack) cannot be relocated and must 176be reloaded at the *same address* that previously held it. If another 177process now owns that area of memory, there is nothing we can do. We 178believe that this should not be a serious concern in normal use. 179 180 ~ * ~ 181 182 Since all available memory is potentially used by ovlmgr, there 183is one additional concern in using it with C programmes: the allocation 184of sufficient space for the C heap (2). While previous versions of 185ovlmgr.asm required the change of an internal constant and re-assembly 186of ovlmgr to change the amount of space pre-allocated for this purpose, 187the current version uses the DOS minalloc parameter in the executable 188file to hold the size of the desired heap area. This parameter can be 189set at any time after the link process with either Microsoft's exemod 190utility or with the supplied utility, exesmurf. 191 192 ~ * ~ 193 194NOTA BENE: This is an early version of the overlay manager, but by now 195it should be fairly well debugged. If you are considering upgrading it 196please be aware that the following improvements are planned for the 197next version (though who knows when delivery will occur): 198 199 - compatible versions of setjmp() and longjmp() 200 - integral malloc() to eliminate the heap size guesswork 201 - support for swapped data areas (read-only and read/write) 202 - improved performance through dynamic link-loading (maybe) 203 - interlocking to permit floppy disk juggling use 204 - XMS support and improved EMS support 205 - support for divergent-functionality overlays (such as 206 hardware-specific modules) 207 - enabling the overlay locking code 208 - more flexibility in naming and locating external overlay files 209 - Major code revamping 210 211Swap On! 212 213------------------------------------------------------------------------ 214MESSAGES 215 216OVLMGR: EMS memory manager error. 217 218 An error occurred during an EMS access. Either the hardware has 219 reported a bug, the software driver has detected an anomaly or 220 the page frame is not 64K bytes in length. 221 222OVLMGR: Executable or overlay header missing or damaged. 223 224 The end of a file was reached unexpectedly during 225 initialisation, while trying to locate the overlays. This is a 226 very bad sign (though I am concerned that it might be triggered 227 spuriously by debug information or other non-executable tails on 228 files). 229 230OVLMGR: File I/O error. 231 232 An error occurred while trying to load an overlay. We don't 233 want this. 234 235OVLMGR: Inaccessible EXE file. Can't load overlays. 236 237 For some reason ovlmgr could not locate or read the original 238 .EXE file in which the overlays reside. This could be due to 239 your attempting to use a very old version of DOS, 240 an abject shortage of file handles, some strange event causing 241 the file to be deleted, a disk error, or the diskette that 242 contained the executable being removed. 243 244OVLMGR: Inaccessible OVL file. Can't load overlays. 245 246 An error was reported while attempting to open an .OVL file 247 which was expected (from its name) to contain external overlays. 248 The possible causes are similar to those of the previous 249 message. 250 251OVLMGR: Incomplete executable. OVL files missing? 252 253 OVLMGR was unable to locate all of its overlays for some reason. 254 This could be due to I/O errors on the disk drive, but is more 255 likely caused by an external .OVL file not being present in the 256 same directory as the .EXE. 257 258OVLMGR: Incorrect DOS version. Must be 3.00 or later. 259 260 The current version of ovlmgr does not support versions of DOS 261 prior to 3.0 because of the difficulty of locating the 262 executable file (and hence the overlays) at runtime. 263 264OVLMGR: Internal memory allocation failure. 265 266 Either an internal error has occurred in ovlmgr or the 267 application programme, or some event has caused memory that 268 ovlmgr believed it could count on becoming unavailable. A 269 typical example of the latter would be the result of 270 attempting to load a TSR while an overlaid application is 271 running. 272 273OVLMGR: Not enough free memory left to run this program. 274 275 Although DOS successfully loaded the programme, it proved 276 impossible to allocate enough additional contiguous memory to 277 load one or more of the overlays. Either reduce the 278 RAM-loading of the application by reducing the size of either 279 the root or the largest overlays, or increase the amount of 280 memory available by unloading TSRs and/or simplifying your 281 CONFIG.SYS. 282 283OVLMGR: Unable to resolve overlay file names. 284 285 Apparently the name reported to OVLMGR as being that of the 286 executable file is ill-formed, and it is thus not possible to 287 intuit what external overlay files would be called. It is 288 possible that this indicates that DOS has gone bonkers, but more 289 likely (I guess) that the .EXE was not invoked by DOS as we know 290 it. Either way, you have entered the Twilight Zone.... 291 292(xxxx:xxxx:xxxx:xxxx) 293 294 This is a diagnostic code composed of the following fields: 295 - error code 296 - version number 297 - available conventional memory 298 - EMS memory usage 299 Please note it in any bug reports or correspondence with the 300 development team. 301 302------------------------------------------------------------------------ 303KNOWN BUGS 304 305The present version cannot always be used as a direct replacement for 306Microsoft's overlay manager (even granted the documented differences) 307because the minimum size required for an overlaid programme to run is at 308least the size of the root plus TWICE the size of the largest overlay. 309If a programme has previously had its overlay structure tuned to take 310best advantage of Microsoft overlays, this may well cause a problem. 311The overlays themselves will need to be split up. 312 313When the MicroSoft linker discovers that an overlay as requested 314contains NO instructions at all (this can happen by mistake if you give 315a source file that winds up holding only data declarations its own 316overlay), it does not emit an overlay record for it at all - there is 317simply a gap in the overlay sequence in the file. The current version 318of OVLMGR detects this as an error, since it assumes that such a gap 319should have been filled by an external .OVL file. It is presently your 320responsibility to ensure that this does not occur. 321 322Files containing overlays are kept open all the time the application 323is running. Particularly if multiple external .OVL files are used, 324this can result in less file handles being available to the user 325programme than would otherwise be expected. 326 327ALL files that match the pattern for potential overlay files are 328opened, regardless of whether they actually contain overlays. 329 330The names of external overlay files have a very restricted form, and 331they must reside in the same directory with the .EXE. These 332limitations cause them to be useful for little else besides making 333distribution easier. 334 335Transfers between overlays are very slow in machine terms, even if both 336overlays happen to reside in memory at the time (still significantly 337faster than Microsoft's, though). This means that overlay patterns 338must be chosen on the basis of more than just logical dependency. 339 340Locking overlays into memory is not really implemented even though 341reading the source code might make you think it was. Actually, reading 342the source code itself isn't very well implemented right now. Comments 343and stuff would help. Yup, yup. 344 345Due to limitations in the LIM EMS standard (to 4.0), programmes that 346themselves use EMS memory cannot be overlaid with ovlmgr unless ovlmgr's 347own EMS support is disabled. This is accomplished by assembling with 348the /DNOEMS flag. 349 350------------------------------------------------------------------------ 351BUG ALERT 352 353To repeat a point made above, if you ever try to call a function in an 354overlay through a pointer, you *may* die with the Microsoft overlay 355manager. If you ever try to call a function in an overlay through a 356pointer, you *will* die with ours. Nothing in an overlay ever ends up 357in the same segment as the linker anticipated. You have been warned! 358 359------------------------------------------------------------------------ 360FOOTNOTES 361 362(1) This problem can be circumvented through the use of surrogate 363'trampoline' functions: functions that reside in the root overlay and 364simply pass right through to the 'real', overlaid, implementations. 365This can even be made transparent to the source code through the use 366of the C macro preprocessor, with a locution of the form 367 #define foo(x) foo_(x) 368visible everywhere except at the actual definition point of the 369trampoline. This method was implemented in NetHack 3.0, and remains today. 370 371(2) If you should get a message to the effect that NetHack can't 372allocate 28000 and some bytes when entering a maze level, that 373isn't our problem! In all probability you forgot to rebuild your 374special level files when you changed the compiler flags. We got 375that one, too, at one point. The same applies to similar messages when 376reading bones files or saved games: it is more likely that you forgot 377to discard them after recompiling your game than that the memory 378allowance is so greatly incorrect. 379 380---------------------------------------------------------------------- 381NOTICE 382 383OVLMGR.ASM is brought to you by Pierre Martineau and Stephen Spackman. 384It, and this document, are copyright. They are, however, provided as 385part of NetHack and may be freely distributed as described in the 386NetHack license. 387 388---------------------------------------------------------------------- 389Stephen P Spackman stephen@tira.uchicago.edu 390Pierre G Martineau pierre%ozrout.uucp@altitude.cam.org 391---------------------------------------------------------------------- 392 Copyright (c) 1989, 1990 Pierre G Martineau and Stephen P Spackman 393 All Rights Reserved. 394