1My disk won't read properly! 2============================ 3 4So you're trying to read a disk and it's not working. Well, you're not the 5only one. Floppy disks are unreliable and awkward things. FluxEngine can 6help, but the tools aren't very user friendly. 7 8The good news is that as of 2019-04-30 the FluxEngine decoder algorithm has 9been written to be largely fire-and-forget and is mostly self-adjusting. 10However, there are still some things that can be tuned to produce better 11reads. 12 13Also, it's important to remember that some drives are problematic --- in 14particular, some 3.5" floppy drives are designed to work with just IBM scheme 15disks with a certain set of pulse intervals. There's a tool to let you diagnose 16this; see [the drive response](driveresponse.md) page. 17 18The sector map 19-------------- 20 21Every time FluxEngine does a read, it produces a dump like this: 22 23``` 24H.SS Tracks ---> 250. 0 XXXBXX...X.XXXX......?...........X...X.........X................................ 260. 1 ..X..X.X..XB.B.B........X...........X.......X................................... 270. 2 X.XXXX.XX....XB.................X.X..X..X......X................................ 280. 3 X.X..XXXX..?XXXX..................XX.X.......................................... 290. 4 X.X..X....X.X.XX....?....?........XXXX..X.....X................................. 300. 5 XXXX...?..X.XBX...?......C......C?.X.X...?....X..........X...................... 310. 6 XXXB.XX.XX???XXX...............CX.XXXX........X................................. 320. 7 XX..XX.XC..?.X......B.......X...X..XX...C....................................... 330. 8 X?.B...XXX.?..XX........X........XCXXX..X..X..X................................. 340. 9 BB.XX.X.X.X...BX.........C.......XXX...........X.....X.......................... 350.10 BX.XX.XX.X..XX.B...X.............XXX........................................C... 360.11 .C.X.C..BXXBXBX?X................XX..X......X................................... 370.12 BX.XXX....BX..X......C....X......XXX.......XX..........................XXXXXXXXX 380.13 X..BXX..X?.XX.X....X..............XXXX.X....X...............XXXXXXXXXXXXXXXXXXXX 390.14 X...XXB..X.X..X....X...X..C........X?...........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 400.15 X.BX.XX.X.XXX.X...........X.....X..X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 410.16 XBXX...XX.X.X.XX........B..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 420.17 XXB..X.B....XX..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 430.18 XXCXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 44Good sectors: 978/1520 (64%) 45Missing sectors: 503/1520 (33%) 46Bad sectors: 39/1520 (2%) 4780 tracks, 1 heads, 19 sectors, 512 bytes per sector, 760 kB total 48``` 49 50This is the **sector map**, and is showing me the status of every sector it 51found on the disk. (Tracks on the X-axis, sectors on the Y-axis.) This is a 52very bad read from a [Victor 9000](disk-victor9k.md) disk; good reads 53shouldn't look like this. A dot represents a good sector. A B is one where 54the CRC check failed; an X is one which couldn't be found at all; a ? is a 55sector which was found but contained no data. 56 57At the very bottom there's a summary: 64% good sectors. Let me try and improve 58that. 59 60(You may notice the wedge of Xs in the bottom right. This is because the 61Victor 9000 uses a varying number of sectors per track; the short tracks in 62the middle of the disk store less. So, it's perfectly normal that those 63sectors are missing. This will affect the 'good sectors' score, so it's 64normal not to have 100% on this type of disk.) 65 66Clock errors 67------------ 68 69When FluxEngine sees a track, it attempts to automatically guess the clock 70rate of the data in the track. It does this by looking for the magic bit 71pattern at the beginning of each sector record and measuring how long it 72takes. This is shown in the tracing FluxEngine produces as it runs. For 73example: 74 75``` 76 70.0: 868 ms in 427936 bytes 77 138 records, 69 sectors; 2.13us clock; 78 logical track 70.0; 6656 bytes decoded. 79 71.0: 890 ms in 387904 bytes 80 130 records, 65 sectors; 2.32us clock; 81 logical track 71.0; 6144 bytes decoded. 82``` 83 84Bits are then found by measuring the interval between pulses on the disk and 85comparing to this clock rate. 86 87However, floppy disk drives are extremely analogue devices and not necessarily calibrated very well, and the disk may be warped, or the rubber band which makes the drive work may have lost its bandiness, and so the bits are not necessarily precisely aligned. Because of this, FluxEngine can tolerate a certain amount of error. This is controlled by the `--bit-error-threshold` parameter. Varying this can have magical effects. For example, adding `--bit-error-threshold=0.4` turns the decode into this: 88 89``` 90H.SS Tracks ---> 910. 0 ...B............................................................................ 920. 1 ...........B...B................................................................ 930. 2 ..............B................................................................. 940. 3 ................................................................................ 950. 4 ................................................................................ 960. 5 ...B............................................................................ 970. 6 ...B.....B...................................................................... 980. 7 ...B...........B....B........................................................... 990. 8 ................................................................................ 1000. 9 .B............B................................................................. 1010.10 .B............BB................................................................ 1020.11 B............B.................................................................. 1030.12 ...B......B............................................................XXXXXXXXX 1040.13 ............B...............................................XXXXXXXXXXXXXXXXXXXX 1050.14 ...B......B.....................................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 1060.15 ..B.....BB..B.........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 1070.16 ..B.....B.B.............B..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 1080.17 ..B....B........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 1090.18 .B..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 110Good sectors: 1191/1520 (78%) 111Missing sectors: 296/1520 (19%) 112Bad sectors: 33/1520 (2%) 11380 tracks, 1 heads, 19 sectors, 512 bytes per sector, 760 kB total 114``` 115 116A drastic difference! 117 118The value of the parameter is the fraction of a clock of error to accept. The 119value typically varies from 0.0 to 0.5; the default is 0.2. Larger values 120make FluxEngine more tolerant, so trying 0.4 is the first thing to do when 121faced with a dubious disk. However, in some situations, increasing the value 122can actually _increase_ the error rate --- which is why 0.4 isn't the default 123--- so you'll need to experiment. 124 125That's the most common tuning parameter, but others are available: 126 127`--pulse-debounce-threshold` controls whether FluxEngine ignores pairs of pulses in rapid succession. This is common on some disks (I've observed them on Brother word processor disks). 128 129`--clock-interval-bias` adds a constant bias to the intervals between pulses 130before doing decodes. This is very occasionally necessary to get clean reads 131--- for example, if the machine which wrote the disk always writes pulses 132late. If you try this, use very small numbers (e.g. 0.02). Negative values 133are allowed. 134 135Both these parameters take a fraction of a clock as a parameter, and you'll 136probably never need to touch them. 137 138Clock detection 139--------------- 140 141A very useful tool for examining problematic disks is `fluxengine inspect`. 142This will let you examine the raw flux on a disk (or flux file). It'll also 143guess the clock rate on the disk for you, using a simple statistical analysis 144of the pulse intervals on the disk. (Note that the tool only works on one 145track at a time.) 146 147``` 148$ fluxengine inspect -s good.flux:t=0:s=0 149Clock detection histogram: 1503.58 737 ▉ 1513.67 3838 ████▊ 1523.75 13078 ████████████████▌ 1533.83 31702 ████████████████████████████████████████ 1543.92 26682 █████████████████████████████████▋ 1554.00 22282 ████████████████████████████ 1564.08 12222 ███████████████▍ 1574.17 4731 █████▉ 1584.25 1001 █▎ 159... 1607.33 236 ▎ 1617.42 1800 ██▎ 1627.50 5878 ███████▍ 1637.58 10745 █████████████▌ 1647.67 12442 ███████████████▋ 1657.75 10144 ████████████▊ 1667.83 6698 ████████▍ 1677.92 2779 ███▌ 1688.00 740 ▉ 169... 17011.17 159 ▏ 17111.25 624 ▊ 17211.33 1723 ██▏ 17311.42 3268 ████ 17411.50 4608 █████▊ 17511.58 4643 █████▊ 17611.67 3507 ████▍ 17711.75 2178 ██▋ 17811.83 868 █ 17911.92 258 ▎ 180... 181Noise floor: 317 182Signal level: 3170 183Peak start: 42 (3.50 us) 184Peak end: 52 (4.33 us) 185Median: 47 (3.92 us) 1863.92us clock detected. 187``` 188 189That's _not_ the histogram from the Victor disk; that's an Apple II disk, and 190shows three nice clear spikes (which is very characteristic of the GCR 191encoding which the Apple II used). The numbers at the bottom show that a peak 192has been detected between 3.50us and 4.33us, and the median is 3.92us, which 193corresponds more-or-less with the top of the peak; that's the clock rate 194FluxEngine can use. 195 196So, what does my Victor 9000 histogram look like? Let's look at the 197histogram for a single track: 198 199``` 200$ fluxengine inspect -s dubious.flux:t=0:s=0 201Clock detection histogram: 2021.33 1904 █▉ 2031.42 21669 ██████████████████████▌ 2041.50 2440 ██▌ 2051.58 469 ▍ 2061.67 7261 ███████▌ 2071.75 6808 ███████ 2081.83 3088 ███▏ 2091.92 2836 ██▉ 2102.00 8897 █████████▎ 2112.08 6200 ██████▍ 212... 2132.25 531 ▌ 2142.33 2802 ██▉ 2152.42 2136 ██▏ 2162.50 1886 █▉ 2172.58 10110 ██████████▌ 2182.67 8283 ████████▌ 2192.75 7779 ████████ 2202.83 2680 ██▊ 2212.92 13908 ██████████████▍ 2223.00 38431 ████████████████████████████████████████ 2233.08 35708 █████████████████████████████████████▏ 2243.17 5361 █████▌ 225... 2263.75 294 ▎ 2273.83 389 ▍ 2283.92 1224 █▎ 2294.00 3067 ███▏ 2304.08 4092 ████▎ 2314.17 6916 ███████▏ 2324.25 25639 ██████████████████████████▋ 2334.33 31407 ████████████████████████████████▋ 2344.42 10209 ██████████▋ 2354.50 1159 █▏ 236... 237Noise floor: 384 238Signal level: 1921 239Peak start: 15 (1.25 us) 240Peak end: 26 (2.17 us) 241Median: 20 (1.67 us) 2421.67 us clock detected. 243``` 244 245That's... not good. The disk is very noisy, and the intervals between pulses 246are horribly distributed. The detected clock from the decode is 1.45us, which 247does correspond more-or-less to a peak. You'll also notice that the 248double-clock interval is at 3.00us, which is _not_ twice 1.45us. The guessed 249clock by `fluxengine inspect` is 1.67us, which is clearly wrong. 250 251This demonstrates that the statistical clock guessing isn't brilliant, which 252is why I've just rewritten the decoder not to use it; nevertheless, it's a 253useful tool for examining disks. 254 255`fluxengine inspect` will also dump the raw flux data in various formats, but 256that's mostly only useful to me. Try `--dump-bits` to see the raw bit pattern 257on the disk (using the guessed clock, or `--manual-clock-rate-us` to set it 258yourself); `--dump-flux` will show you discrete pulses and the intervals 259between them. 260 261The tool's subject to change without notice as I tinker with it. 262