1bochs 2.2.6: 2./configure --enable-smp --enable-disasm --enable-debugger --enable-all-optimizations --enable-4meg-pages --enable-global-pages --enable-pae --disable-reset-on-triple-fault 3bochs CVS after 2.2.6: 4./configure --enable-smp --enable-disasm --enable-debugger --enable-all-optimizations --enable-4meg-pages --enable-global-pages --enable-pae 5 6bootmain.c doesn't work right if the ELF sections aren't 7sector-aligned. so you can't use ld -N. and the sections may also need 8to be non-zero length, only really matters for tiny "kernels". 9 10kernel loaded at 1 megabyte. stack same place that bootasm.S left it. 11 12kinit() should find real mem size 13 and rescue useable memory below 1 meg 14 15no paging, no use of page table hardware, just segments 16 17no user area: no magic kernel stack mapping 18 so no copying of kernel stack during fork 19 though there is a kernel stack page for each process 20 21no kernel malloc(), just kalloc() for user core 22 23user pointers aren't valid in the kernel 24 25setting up first process 26 we do want a process zero, as template 27 but not runnable 28 just set up return-from-trap frame on new kernel stack 29 fake user program that calls exec 30 31map text read-only? 32shared text? 33 34what's on the stack during a trap or sys call? 35 PUSHA before scheduler switch? for callee-saved registers. 36 segment contents? 37 what does iret need to get out of the kernel? 38 how does INT know what kernel stack to use? 39 40are interrupts turned on in the kernel? probably. 41 42per-cpu curproc 43one tss per process, or one per cpu? 44one segment array per cpu, or per process? 45 46pass curproc explicitly, or implicit from cpu #? 47 e.g. argument to newproc()? 48 hmm, you need a global curproc[cpu] for trap() &c 49 50test stack expansion 51test running out of memory, process slots 52 53we can't really use a separate stack segment, since stack addresses 54need to work correctly as ordinary pointers. the same may be true of 55data vs text. how can we have a gap between data and stack, so that 56both can grow, without committing 4GB of physical memory? does this 57mean we need paging? 58 59what's the simplest way to add the paging we need? 60 one page table, re-write it each time we leave the kernel? 61 page table per process? 62 probably need to use 0-0xffffffff segments, so that 63 both data and stack pointers always work 64 so is it now worth it to make a process's phys mem contiguous? 65 or could use segment limits and 4 meg pages? 66 but limits would prevent using stack pointers as data pointers 67 how to write-protect text? not important? 68 69perhaps have fixed-size stack, put it in the data segment? 70 71oops, if kernel stack is in contiguous user phys mem, then moving 72users' memory (e.g. to expand it) will wreck any pointers into the 73kernel stack. 74 75do we need to set fs and gs? so user processes can't abuse them? 76 77setupsegs() may modify current segment table, is that legal? 78 79trap() ought to lgdt on return, since currently only done in swtch() 80 81protect hardware interrupt vectors from user INT instructions? 82 83test out-of-fd cases for creating pipe. 84test pipe reader closes then write 85test two readers, two writers. 86test children being inherited by grandparent &c 87 88some sleep()s should be interruptible by kill() 89 90cli/sti in acquire/release should nest! 91 in case you acquire two locks 92 93what would need fixing if we got rid of kernel_lock? 94 console output 95 proc_exit() needs lock on proc *array* to deallocate 96 kill() needs lock on proc *array* 97 allocator's free list 98 global fd table (really free-ness) 99 sys_close() on fd table 100 fork on proc list, also next pid 101 hold lock until public slots in proc struct initialized 102 103locks 104 init_lock 105 sequences CPU startup 106 proc_table_lock 107 also protects next_pid 108 per-fd lock *just* protects count read-modify-write 109 also maybe freeness? 110 memory allocator 111 printf 112 113wakeup needs proc_table_lock 114 so we need recursive locks? 115 or you must hold the lock to call wakeup? 116 117in general, the table locks protect both free-ness and 118 public variables of table elements 119 in many cases you can use table elements w/o a lock 120 e.g. if you are the process, or you are using an fd 121 122lock code shouldn't call cprintf... 123 124nasty hack to allow locks before first process, 125 and to allow them in interrupts when curproc may be zero 126 127race between release and sleep in sys_wait() 128race between sys_exit waking up parent and setting state=ZOMBIE 129race in pipe code when full/empty 130 131lock order 132 per-pipe lock 133 proc_table_lock fd_table_lock kalloc_lock 134 console_lock 135 136condition variable + mutex that protects it 137 proc * (for wait()), proc_table_lock 138 pipe structure, pipe lock 139 140systematic way to test sleep races? 141 print something at the start of sleep? 142 143do you have to be holding the mutex in order to call wakeup()? 144 145device interrupts don't clear FL_IF 146 so a recursive timer interrupt is possible 147 148what does inode->busy mean? 149 might be held across disk reads 150 no-one is allowed to do anything to the inode 151 protected by inode_table_lock 152inode->count counts in-memory pointers to the struct 153 prevents inode[] element from being re-used 154 protected by inode_table_lock 155 156blocks and inodes have ad-hoc sleep-locks 157 provide a single mechanism? 158 159need to lock bufs in bio between bread and brelse 160 161test 14-character file names 162and file arguments longer than 14 163and directories longer than one sector 164 165kalloc() can return 0; do callers handle this right? 166 167why directing interrupts to cpu 1 causes trouble 168 cpu 1 turns on interrupts with no tss! 169 and perhaps a stale gdt (from boot) 170 since it has never run a process, never called setupsegs() 171 but does cpu really need the tss? 172 not switching stacks 173 fake process per cpu, just for tss? 174 seems like a waste 175 move tss to cpu[]? 176 but tss points to per-process kernel stack 177 would also give us a gdt 178 OOPS that wasn't the problem 179 180wait for other cpu to finish starting before enabling interrupts? 181 some kind of crash in ide_init ioapic_enable cprintf 182move ide_init before mp_start? 183 didn't do any good 184 maybe cpu0 taking ide interrupt, cpu1 getting a nested lock error 185 186cprintfs are screwed up if locking is off 187 often loops forever 188 hah, just use lpt alone 189 190looks like cpu0 took the ide interrupt and was the last to hold 191the lock, but cpu1 thinks it is nested 192cpu0 is in load_icode / printf / cons_putc 193 probably b/c cpu1 cleared use_console_lock 194cpu1 is in scheduler() / printf / acquire 195 196 1: init timer 197 0: init timer 198 cpu 1 initial nlock 1 199 ne0s:t iidd el_occnkt rc 200 onsole cpu 1 old caller stack 1001A5 10071D 104DFF 1049FE 201 panic: acquire 202 ^CNext at t=33002418 203 (0) [0x00100091] 0008:0x00100091 (unk. ctxt): jmp .+0xfffffffe ; ebfe 204 (1) [0x00100332] 0008:0x00100332 (unk. ctxt): jmp .+0xfffffffe 205 206why is output interleaved even before panic? 207 208does release turn on interrupts even inside an interrupt handler? 209 210overflowing cpu[] stack? 211 probably not, change from 512 to 4096 didn't do anything 212 213 214 1: init timer 215 0: init timer 216 cnpeus te11 linnitki aclo nnoolleek cp1u 217 ss oarltd sccahleldeul esrt aocnk cpu 0111 Ej6 buf1 01A3140 C5118 218 0 219 la anic1::7 0a0c0 uuirr e 220 ^CNext at t=31691050 221 (0) [0x00100373] 0008:0x00100373 (unk. ctxt): jmp .+0xfffffffe ; ebfe 222 (1) [0x00100091] 0008:0x00100091 (unk. ctxt): jmp .+0xfffffffe ; ebfe 223 224cpu0: 225 2260: init timer 227nested lock console cpu 0 old caller stack 1001e6 101a34 1 0 228 (that's mpmain) 229panic: acquire 230 231cpu1: 232 2331: init timer 234cpu 1 initial nlock 1 235start scheduler on cpu 1 jmpbuf ... 236la 107000 lr ... 237 that is, nlock != 0 238 239maybe a race; acquire does 240 locked = 1 241 cpu = cpu() 242what if another acquire calls holding w/ locked = 1 but 243 before cpu is set? 244 245if I type a lot (kbd), i get a panic 246cpu1 in scheduler: panic "holding locks in scheduler" 247cpu0 also in the same panic! 248recursive interrupt? 249 FL_IF is probably set during interrupt... is that correct? 250again: 251 olding locks in scheduler 252 trap v 33 eip 100ED3 c (that is, interrupt while holding a lock) 253 100ed3 is in lapic_write 254again: 255 trap v 33 eip 102A3C cpu 1 nlock 1 (in acquire) 256 panic: interrupt while holding a lock 257again: 258 trap v 33 eip 102A3C cpu 1 nlock 1 259 panic: interrupt while holding a lock 260OR is it the cprintf("kbd overflow")? 261 no, get panic even w/o that cprintf 262OR a release() at interrupt time turns interrupts back on? 263 of course i don't think they were off... 264OK, fixing trap.c to make interrupts turn off FL_IF 265 that makes it take longer, but still panics 266 (maybe b/c release sets FL_IF) 267 268shouldn't something (PIC?) prevent recursive interrupts of same IRQ? 269 or should FL_IF be clear during all interrupts? 270 271maybe acquire should remember old FL_IF value, release should restore 272 if acquire did cli() 273 274DUH the increment of nlock in acquire() happens before the cli! 275 so the panic is probably not a real problem 276 test nlock, cli(), then increment? 277 278BUT now userfs doesn't do the final cat README 279 280AND w/ cprintf("kbd overflow"), panic holding locks in scheduler 281 maybe also simulataneous panic("interrupt while holding a lock") 282 283again (holding down x key): 284 kbd overflow 285 kbd oaaniicloowh 286 olding locks in scheduler 287 trap v 33 eip 100F5F c^CNext at t=32166285 288 (0) [0x0010033e] 0008:0010033e (unk. ctxt): jmp .+0xfffffffe (0x0010033e) ; ebfe 289 (1) [0x0010005c] 0008:0010005c (unk. ctxt): jmp .+0xfffffffe (0x0010005c) ; ebfe 290cpu0 paniced due to holding locks in scheduler 291cpu1 got panic("interrupt while holding a lock") 292 again in lapic_write. 293 while re-enabling an IRQ? 294 295again: 296cpu 0 panic("holding locks in scheduler") 297 but didn't trigger related panics earlier in scheduler or sched() 298 of course the panic is right after release() and thus sti() 299 so we may be seeing an interrupt that left locks held 300cpu 1 unknown panic 301why does it happen to both cpus at the same time? 302 303again: 304cpu 0 panic("holding locks in scheduler") 305 but trap() didn't see any held locks on return 306cpu 1 no apparent panic 307 308again: 309cpu 0 panic: holding too many locks in scheduler 310cpu 1 panic: kbd_intr returned while holding a lock 311 312again: 313cpu 0 panic: holding too man 314 la 10d70c lr 10027b 315 those don't seem to be locks... 316 only place non-constant lock is used is sleep()'s 2nd arg 317 maybe register not preserved across context switch? 318 it's in %esi... 319 sched() doesn't touch %esi 320 %esi is evidently callee-saved 321 something to do with interrupts? since ordinarily it works 322cpu 1 panic: kbd_int returned while holding a lock 323 la 107340 lr 107300 324 console_lock and kbd_lock 325 326maybe console_lock is often not released due to change 327 in use_console_lock (panic on other cpu) 328 329again: 330cpu 0: panic: h... 331 la 10D78C lr 102CA0 332cpu 1: panic: acquire FL_IF (later than cpu 0) 333 334but if sleep() were acquiring random locks, we'd see panics 335in release, after sleep() returned. 336actually when system is idle, maybe no-one sleeps at all. 337 just scheduler() and interrupts 338 339questions: 340 does userfs use pipes? or fork? 341 no 342 does anything bad happen if process 1 exits? eg exit() in cat.c 343 looks ok 344 are there really no processes left? 345 lock_init() so we can have a magic number? 346 347HMM maybe the variables at the end of struct cpu are being overwritten 348 nlocks, lastacquire, lastrelease 349 by cpu->stack? 350 adding junk buffers maybe causes crash to take longer... 351 when do we run on cpu stack? 352 just in scheduler()? 353 and interrupts from scheduler() 354 355OH! recursive interrupts will use up any amount of cpu[].stack! 356 underflow and wrecks *previous* cpu's struct 357 358better buffer cache replacement 359read/write of open file that's been unlinked 360