1New features with AN-2019-01-22: 2 3This is the first localization step for the schily source consolidation. Many 4programs now (hopefully) call gettext() for all strings that need localization. 5 6- The next step will include dgettext() calls for the libraries and the 7 missing programs 8 9- The following step will include the extracted strings 10 11- The last step will include German translations and install support 12 for the resulting binary message object files. 13 14----------> Please test and report compilation problems! <--------- 15 16***** NOTE: As mentioned since 2004, frontends to the tools should ***** 17***** call all programs in the "C" locale ***** 18***** by e.g. calling: LC_ALL=C cdrecord .... ***** 19***** unless these frontends support localized strings ***** 20***** used by the cdrtools with NLS support. ***** 21 22 *** WARNING *** 23 *** Need new smake *** 24 25 *** Due to the fact that schily-tools 2014-04-03 introduced to use new macro 26 *** expansions and a related bug fix in smake, you need a new smake 27 *** to compile this source. If your smake is too old and aborts, ensure this 28 *** by calling: 29 30 cd ./psmake 31 ./MAKE-all 32 cd .. 33 psmake/smake 34 psmake/smake install 35 36 The new smake version mentioned above is smake-1.2.4 37 The recent smake version is smake-1.3 38 39 *** Due to the fact that schily-tools 2014-01-26 introduced 40 *** optimizations for the Schily version of SunPro Make, you 41 *** need at least the dmake version from 2018/01/11 with support 42 *** for the "export" directive to compile with this makefile system. 43 44 45 WARNING: the new version of the isoinfo program makes use of the 46 *at() series of functions that have been introduced by Sun 47 in August 2001 and added to POSIX.1-2008. For older platforms, 48 libschily now includes emulations for these functions but 49 these emulations have not yet been tested thoroughly. 50 Please report problems! 51 52- Schily.Copyright: added new Copyright dates 53 54- include/schily/values.h: new include file. 55 56- include/schily/dbgmalloc.h: added more prototypes for libdbgmalloc. 57 58- libdbgmalloc & bsh: alloc.c now also prints the caller information 59 in the function aprintlist() 60 61- libstrar::unicode.c: Code has been aligned again with the same module 62 from star. 63 64- libstrar & star unicode.c: iconv() may return > 0 if there are 65 characters that could not be converted into an identical meaning. 66 We therefore now check for ret != 0 instead of ret == -1. 67 68 Thanks to Christian Weisgerber for reporting 69 70- libschily::match.c: Added a missing /* FALLTHROUGH */ comment. 71 72 Thanks to Pavel Raiskup for poiting to a related Coverity message. 73 74- libschily::fstream.c: Added a /* NOTREACHED */ comment to make coverity 75 quiet. 76 77 Thanks to Pavel Raiskup for poiting to a related Coverity message. 78 79- libschily::fnmatch.c: added a forgotten fix (from last January) for 80 a problem with missing brackets with if/then/else related to 81 [:upper:] and [:lower:] 82 83- libschily::fnmatch.c: added a forgotten _partial_ enhancement from 84 last January for [= ... =] and [. ... .] support. 85 86 Warning: this is not yet fully functional since the POSIX standard 87 does not include an interface to get this functionality from the 88 i18n part of libc. 89 90- libfind::find.c: Added a missing /* FALLTHROUGH */ comment. 91 92 Thanks to Pavel Raiskup for poiting to a related Coverity message. 93 94- libshedit & bsh: Added a missing /* FALLTHROUGH */ comment. 95 96 Thanks to Pavel Raiskup for poiting to a related Coverity message. 97 98- calltree: Added a missing /* FALLTHROUGH */ comment. 99 100 Thanks to Pavel Raiskup for poiting to a related Coverity message. 101 102- calltree: made the lexer more robust against EOF in the middle of 103 a token. 104 105 Thanks to Heiko Ei�feldt for reporting. 106 107- cdda2wav: replaced some strcat() calls by strlcat() calls. 108 109 Thanks to Pavel Raiskup for poiting to a related Coverity message. 110 111- cdrecord: Added a missing /* FALLTHROUGH */ comment. 112 113 Thanks to Pavel Raiskup for poiting to a related Coverity message. 114 115- cdrecord: Copyright bumped to 2019 116 117- cpp: cpypre.c the fallback file for platforms without "yacc" has been 118 modified to include schily/values.h instead of values.h. This is needed 119 for platforms without yacc and without values.h (e.g. Cygwin) 120 121 Thanks to Heiko Ei�feldt for reporting. 122 123- cdda2wav & ${SRCROOT}/Mcsw & ${SRCROOT}/Mocsw is now using 124 "$(MAKE)" instead of $(MAKE) to permit spaces in the path. 125 126 Thanks to Heiko Ei�feldt for reporting. 127 128- mkisofs: Added a missing /* FALLTHROUGH */ comment. 129 130 Thanks to Pavel Raiskup for poiting to a related Coverity message. 131 132- mkisofs: the Message "'iconv -l' lists more available names.\n" is now 133 printed even when there is no locale support in the mkisofs specific 134 directory: $INS_BASE/share/lib/siconv as on platforms with iconv(1), 135 the iconv(1) specific translations are still possible. 136 137 Thanks to Jan Engelhardt for reporting. 138 139- p: Added a missing /* FALLTHROUGH */ comment. 140 141 Thanks to Pavel Raiskup for poiting to a related Coverity message. 142 143- printf: Added a missing /* FALLTHROUGH */ comment. 144 145 Thanks to Pavel Raiskup for poiting to a related Coverity message. 146 147- star: added support for auto detection of "zstd" compressed archives. 148 149- star: added a new option -zstd to support compression and uncompression 150 using the program "zstd". 151 152- star: Recently, star did hang in the FIFO code on Solaris. This did 153 not happen on Solaris over 20 years before... 154 155 On Linux - on fast multi CPU machines - the probability that a 156 child process from fork() starts up before the parent is 1000x higher 157 than on Solaris, where 10 million tries were needed to reproduce the 158 same problem. 159 160 As a result, the FIFO in star on Linux could in rare cases (1 of 161 ~ 10000 tries) even finish the 1st read() from the input file before 162 the "tar"-process starts with e.g. command lines like "star -tv" or 163 "star -x". Since star introduced auto-byte-order detection and handling 164 in 1985, star needs a special start up sequence to do that. 165 166 Star introduced the FIFO in the late 1980s and the machines from that 167 time did always restart the parent before the fork()ed child starts. 168 The new OS behavior thus caused a situation that was not forseeable 169 when the FIFO has been designed. This new OS behavior caused a 170 deadlock in aprox. 1 of 10000 star calls on Linux and 1 of 10000000 171 star calls on Solaris. 172 173 Star now waits when entering the FIFO fill-process until the 174 FIFO get-process did start up before trying to wake up a waiting 175 get process. 176 177- star: On Linux, in 1 of 1.5 million tries, star did die from SIGPIPE. 178 Note that this did never happen on Solaris. 179 180 Star now ignores SIGPIPE and it seems that this fixed the problem 181 since it did not happen again after that change with even 100 million 182 tries. 183 184- star: The debug printing for the FIFO has been enhanced to print more 185 information from the FIFO control structure to make it easier to debug 186 problems like the ones mentioned above. 187 188- star: There seems to be a problem in pipe handling in the Linux kernel. 189 It seems that in rare cases, the read(2) on a pipe returns 0 even though 190 the write side did write(2) one byte to the pipe just before calling 191 exit(). Unfortunately, this problem is hard to debug as it happens only 192 once every ~30 million tries. Our workaround is to behave as if the 193 expected byte could be read and star currently prints something like: 194 195 star: Erfolg. Sync pipe read error pid 8141 ret 0 196 star: Erfolg. Ib 0 Ob 1 e 0 p 1 g 0 chan 5. 197 star: Erfolg. Trying to work around Kernel Pipe botch. 198 199 before it continues. Since the star exit code in such a case is 0, 200 we assume that this is a correct workaround and this case thus may 201 be made completely silent in the future. 202 203- star: an even less frequent FIFO problem (occurs once every 50 million 204 tries on fast multi CPU machines) has been identified. Star reports a 205 hard EOF on input even though the complete file with logical EOF has 206 been read and there is still input to process. In order to debug this 207 problem a debug message has been added to the code. 208 209 With this debug message, it turned out, that this problem happened 210 because a context switch occurred in the FIFO read process after it did 211 see an empty FIFO and later, after the process was resumed, the 212 following check for the FIFO_MEOF flag did see EOF. We now first check 213 for the FIFO_MEOF flag and later for the amount of data inside as the 214 FIFO as FIFO_MEOF is set after the FIFO content has been updated and 215 thus a context switch is no longer able to cause a wrong assumption 216 about the content of the FIFO. 217 218 If you still see this, please send a report. 219 220- star: added support to print debug malloc statistics to better debug 221 memory problems in star. 222 223- star: pathname.c:: free_pspace() now only frees the path buffer if it 224 is != NULL 225 226- star: fixed a bug in the file create.c that caused star to incorrectly 227 grow the path buffer by 2 bytes for every archived file. This caused 228 star to constantly grow if a larger amount of files are archived and 229 eat up all memory available to 32 bit processes if the archived 230 filesystem is larger than approx. 1 TB. 231 232- star: If the path name now cannot be handled because of low memory, 233 we print a warning that includes the text "out of memory". 234 235- star: Now checking whether open of /dev/null failed while running a 236 compress pipe. This avoids a core dump on defective OS installations. 237 238 Thanks to Pavel Raiskup for poiting to a related Coverity message. 239 240- star: props.c: Added a missing /* FALLTHROUGH */ comment. 241 242 Thanks to Pavel Raiskup for poiting to a related Coverity message. 243 244- star: create.c: Add more comment for the CPIO CRC format handler to 245 explain why the last instance if a series of hard links for a file 246 needs to archive the data. 247 248- star: diff.c: added a filling fillbytes(&finfo, ...) to make sure that 249 ACL pointers are initialized. 250 251 Thanks to Pavel Raiskup for poiting to a related Coverity message. 252 253- star: Several /* NOTREACHED */ comments have been added to tell 254 programs like coverity that after a NULL pointer check, there is no 255 continuation of the program 256 257 Thanks to Pavel Raiskup for poiting to a related Coverity message. 258 259- star: extract.c: A if (path->ps_path == '\0') has been corrected to 260 if (path->ps_path[0] == '\0') after a mktemp() call. This was a typo 261 introduced with the new support for extremely long path names. 262 263 Thanks to Pavel Raiskup for poiting to a related Coverity message. 264 265- star: extract.c An initalization for a struct pathstore has been 266 moved to the front to verify that path.ps_path is always initialized. 267 268 Thanks to Pavel Raiskup for poiting to a related Coverity message. 269 270- star: header.c: isgnumagic(&ptb->dbuf.t_vers) has been changed to 271 isgnumagic(ptb->ustar_dbuf.t_magic) as it is a "ustar" structure 272 that is going to be checked. 273 274 Thanks to Pavel Raiskup for poiting to a related Coverity message. 275 276- star: some Cstyle changes 277 278- ved: Added a missing /* FALLTHROUGH */ comment. 279 280 Thanks to Pavel Raiskup for poiting to a related Coverity message. 281 282- bsh / Bourne Shell / star: the function hop_dirs() no longer checks 283 for p2 != NULL before calling *p2 = '/' as p2 has been granted 284 to be != NULL from a break with strchr(p, '/') == NULL 285 286 Thanks to Pavel Raiskup for poiting to a related Coverity message. 287 288- Bourne Shell: Added a missing /* FALLTHROUGH */ comment. 289 290 Thanks to Pavel Raiskup for poiting to a related Coverity message. 291 292- Bourne Shell: added a range check for $OPTIND to tge getopts(1) 293 implementation 294 295 Thanks to Pavel Raiskup for poiting to a related Coverity message. 296 297- Bourne Shell: Removed a nonsense variable in expand() that caused 298 Coverity not to understand that a directory was correctly closed() 299 300 Thanks to Pavel Raiskup for poiting to a related Coverity message. 301 302- Bourne Shell: Added a paranoia comparison to make Coverity quiet. 303 304 Thanks to Pavel Raiskup for poiting to a related Coverity message. 305 306- Bourne Shell: avoid to call catpath() with a NULL pointer for path 307 308 Thanks to Pavel Raiskup for poiting to a related Coverity message. 309 310- Bourne Shell: check the write() return code in io.c 311 312 Thanks to Pavel Raiskup for poiting to a related Coverity message. 313 314- Bourne Shell: Added a paranopia check for "test -o" to make Coperity 315 quiet. 316 317 Thanks to Pavel Raiskup for poiting to a related Coverity message. 318 319- Bourne Shell: Added (void) before fcntl() in hope to make Coverity 320 quiet. 321 322 Thanks to Pavel Raiskup for poiting to a related Coverity message. 323 324- Bourne Shell: Removed dead code from readwc() 325 326 Thanks to Pavel Raiskup for poiting to a related Coverity message. 327 328- Bourne Shell: Cstyle changes to xec.c 329 330- Bourne Shell: "builtin -d ..." did access free()d memory. 331 332 Thanks to Pavel Raiskup for poiting to a related Coverity message. 333 334- Bourne Shell: expand.c: added a check for fd == -1 to avoid calling 335 openat() with that fd. 336 337 Thanks to Pavel Raiskup for poiting to a related Coverity message. 338 339- Bourne Shell: func.c: added a check for fd == -1 to avoid calling 340 read()/close() with that fd. 341 342 Thanks to Pavel Raiskup for poiting to a related Coverity message. 343 344- Bourne Shell: jobs.c: enlarged a buffer to be of same size as 345 numbuf[] to avoid a potential buffer overflow. 346 347 Thanks to Pavel Raiskup for poiting to a related Coverity message. 348 349- Bourne Shell: print.c::prt_cntl() had a very old (AT&T) bug with 350 printing byte sequences that get an error with mbtowc() and it 351 did not print byte sequences correctly that refer to legal but 352 "nonprintable" multi byte characters. 353 354 Thanks to Pavel Raiskup for poiting to a related Coverity message. 355 356- Bourne Shell: xec.c: the reserved word "time" could cause to 357 access uninitialized memory if the string in $TIMEFORMAT contains 358 the format %J 359 360 Thanks to Pavel Raiskup for poiting to a related Coverity message. 361 362- Bourne Shell: xec.c: added a paranoia check on whether 363 findnam("funcname") returns NULL even though the existence of a 364 function with that name already has been verified via the hash service. 365 366 Thanks to Pavel Raiskup for poiting to a related Coverity message. 367 368- SunPro Make: Added a missing /* FALLTHROUGH */ comment. 369 370 Thanks to Pavel Raiskup for poiting to a related Coverity message. 371 372- SCCS: libcomobj: dodelt.c & idsubst.c: added a forgotten 373 /* FALLTHRU */ 374 375- SCCS: admin.c: added some /* NOTREACHED */ comments to tell Coverity 376 that fatal() does not return. 377 378 Thanks to Pavel Raiskup for poiting to a related Coverity message. 379 380- SCCS: admin.c: The function adjust() did call strlen() on a non-null 381 terminated string. 382 383 Thanks to Pavel Raiskup for poiting to a related Coverity message. 384 385- SCCS: admin.c: The function adjust() returned a local string array. 386 387 Thanks to Pavel Raiskup for poiting to a related Coverity message. 388 389- SCCS: bdiff.c: added some /* NOTREACHED */ comments to tell Coverity 390 that fatal() does not return. 391 392 Thanks to Pavel Raiskup for poiting to a related Coverity message. 393 394- SCCS: prs.c: The getopt() string did not contain 'q' for the expected 395 support to the -q option. 396 397- SCCS: prs.c The code for handling the -d option was indented but curly 398 brackets have been missing. 399 400 Thanks to Pavel Raiskup for poiting to a related Coverity message. 401 402- SCCS: unget.c: The getopt() string did not contain 'q' for the expected 403 support to the -q option. 404 405- SCCS: sccs.c Cstyle changes 406 407- SCCS: sccs.c Added support for new separate macro string defintions 408 that apply to the "new mode". 409 410- SCCS: sccs.c New code allows to specify -N options specific to 411 macro stgring defitions. 412 413- SCCS: sccs.c New environment "SCCS_NOEXEC" allows to disable command 414 execution for debugging purposes. 415 416- SCCS: New version timestamp for SCCS commands lists 2019 417 418- ved: Added a missing /* FALLTHROUGH */ comment. 419 420 Thanks to Pavel Raiskup for poiting to a related Coverity message. 421 422 423- SCCS TODO: 424 425 - Convert sccs.c to use -NSCCS in the back end programs instead 426 of converting g-file names from the command line into s.file 427 names in the frontend in order to forward s.file names to the 428 backend programs. Doing it this way allows us to have the SCCS 429 history files "off tree" in a separate directory tree starting 430 with ".sccs/" at the "project root" directory, as the related 431 code to convert path names is in libcomobj::bulkprepare(). 432 433 - Add code to admin(1) and delta(1) to retrieve the list of 434 new or modified files in order to have all important code 435 for a "sccs commit" in a single program that does not need to 436 deal with ARG_MAX limitations. 437 438 - Add code to admin(1), delta(1) and get(1) to maintain/understand 439 the changeset file. 440 441 - Add code to libcomobj to understand the changeset file, 442 which is needed in order to e.g. know the file names and file 443 specific SIDs/state that correspond to a project global SID. 444 445 - Add code to all programs to maintain a lock (for the file 446 $PROJECTHOME/.sccs/SCCS/s.changeset, that would be in the file 447 $PROJECTHOME/.sccs/SCCS/z.changeset) to maintain a project 448 global lock. 449 450 - Find/verify a complete transactional model that allows to repair 451 complex changes to the set of files for a project that have 452 been aborted in the middle. The current idea is to create the 453 file $PROJECTHOME/.sccs/changeset with the deltas to the 454 changeset during a complex update operation. 455 456 - Find a way to decide when to use SCCS v6 type history files. 457 For the project mode, this is needed in order to prevent 458 historic SCCS implementations to believe they could modify 459 files without knowing about project global locks. 460 461 462 463 464 465- Bourne Shell Missing features for POSIX compliance: 466 467 - Support for $'...' quoting (this is not needed for the current 468 version of POSIX but for the next POSIX 469 version that will be named SUSv8). 470 The development of SUSv8 will start in 471 late 2016. 472 473 We are now expecting the Bourne Shell to be fully POSIX compliant. 474 475- Bourne Shell further TODO list: 476 477 - Finish loadable builtin support. 478 479 - POSIX does not allow us to implement ". -h", so we will 480 add a "source" builtin to be able to implement "source -h" 481 482- The following builtins (that are available in bsh) are still missing in 483 the Bourne Shell: 484 485 err echo with output going to stderr 486 glob echo with '\0' instead of ' ' between args 487 env a builtin version of /usr/bin/env 488 489 The following bsh intrinsics are still missing in the Bourne Shell: 490 491 - the restricted bsh has restriction features that 492 are missing in the Bourne shell. 493 494 - source -h read file into history but do not execute 495 496 and probably more features not yet identified to be bsh unique. 497 498 499 500Author: 501 502Joerg Schilling 503D-13353 Berlin 504Germany 505 506Email: joerg@schily.net, joerg.schilling@fokus.fraunhofer.de 507 508Please mail bugs and suggestions to me. 509