xref: /dports/lang/guile/guile-3.0.7/doc/ref/guile.info-4

This is guile.info, produced by makeinfo version 6.7 from guile.texi.

This manual documents Guile version 3.0.7.

   Copyright (C) 1996-1997, 2000-2005, 2009-2021 Free Software
Foundation, Inc.

   Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3 or
any later version published by the Free Software Foundation; with no
Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.  A
copy of the license is included in the section entitled “GNU Free
Documentation License.”
INFO-DIR-SECTION The Algorithmic Language Scheme
START-INFO-DIR-ENTRY
* Guile Reference: (guile).     The Guile reference manual.
END-INFO-DIR-ENTRY


File: guile.info,  Node: Random Access,  Next: Line/Delimited,  Prev: Buffering,  Up: Input and Output

6.12.7 Random Access
--------------------

 -- Scheme Procedure: seek fd_port offset whence
 -- C Function: scm_seek (fd_port, offset, whence)
     Sets the current position of FD_PORT to the integer OFFSET.  For a
     file port, OFFSET is expressed as a number of bytes; for other
     types of ports, such as string ports, OFFSET is an abstract
     representation of the position within the port’s data, not
     necessarily expressed as a number of bytes.  OFFSET is interpreted
     according to the value of WHENCE.

     One of the following variables should be supplied for WHENCE:
      -- Variable: SEEK_SET
          Seek from the beginning of the file.
      -- Variable: SEEK_CUR
          Seek from the current position.
      -- Variable: SEEK_END
          Seek from the end of the file.
     If FD_PORT is a file descriptor, the underlying system call is
     ‘lseek’.  PORT may be a string port.

     The value returned is the new position in FD_PORT.  This means that
     the current position of a port can be obtained using:
          (seek port 0 SEEK_CUR)

 -- Scheme Procedure: ftell fd_port
 -- C Function: scm_ftell (fd_port)
     Return an integer representing the current position of FD_PORT,
     measured from the beginning.  Equivalent to:

          (seek port 0 SEEK_CUR)

 -- Scheme Procedure: truncate-file file [length]
 -- C Function: scm_truncate_file (file, length)
     Truncate FILE to LENGTH bytes.  FILE can be a filename string, a
     port object, or an integer file descriptor.  The return value is
     unspecified.

     For a port or file descriptor LENGTH can be omitted, in which case
     the file is truncated at the current position (per ‘ftell’ above).

     On most systems a file can be extended by giving a length greater
     than the current size, but this is not mandatory in the POSIX
     standard.


File: guile.info,  Node: Line/Delimited,  Next: Default Ports,  Prev: Random Access,  Up: Input and Output

6.12.8 Line Oriented and Delimited Text
---------------------------------------

The delimited-I/O module can be accessed with:

     (use-modules (ice-9 rdelim))

   It can be used to read or write lines of text, or read text delimited
by a specified set of characters.

 -- Scheme Procedure: read-line [port] [handle-delim]
     Return a line of text from PORT if specified, otherwise from the
     value returned by ‘(current-input-port)’.  Under Unix, a line of
     text is terminated by the first end-of-line character or by
     end-of-file.

     If HANDLE-DELIM is specified, it should be one of the following
     symbols:
     ‘trim’
          Discard the terminating delimiter.  This is the default, but
          it will be impossible to tell whether the read terminated with
          a delimiter or end-of-file.
     ‘concat’
          Append the terminating delimiter (if any) to the returned
          string.
     ‘peek’
          Push the terminating delimiter (if any) back on to the port.
     ‘split’
          Return a pair containing the string read from the port and the
          terminating delimiter or end-of-file object.

 -- Scheme Procedure: read-line! buf [port]
     Read a line of text into the supplied string BUF and return the
     number of characters added to BUF.  If BUF is filled, then ‘#f’ is
     returned.  Read from PORT if specified, otherwise from the value
     returned by ‘(current-input-port)’.

 -- Scheme Procedure: read-delimited delims [port] [handle-delim]
     Read text until one of the characters in the string DELIMS is found
     or end-of-file is reached.  Read from PORT if supplied, otherwise
     from the value returned by ‘(current-input-port)’.  HANDLE-DELIM
     takes the same values as described for ‘read-line’.

 -- Scheme Procedure: read-delimited! delims buf [port] [handle-delim]
          [start] [end]
     Read text into the supplied string BUF.

     If a delimiter was found, return the number of characters written,
     except if HANDLE-DELIM is ‘split’, in which case the return value
     is a pair, as noted above.

     As a special case, if PORT was already at end-of-stream, the EOF
     object is returned.  Also, if no characters were written because
     the buffer was full, ‘#f’ is returned.

     It’s something of a wacky interface, to be honest.

 -- Scheme Procedure: %read-delimited! delims str gobble [port [start
          [end]]]
 -- C Function: scm_read_delimited_x (delims, str, gobble, port, start,
          end)
     Read characters from PORT into STR until one of the characters in
     the DELIMS string is encountered.  If GOBBLE is true, discard the
     delimiter character; otherwise, leave it in the input stream for
     the next read.  If PORT is not specified, use the value of
     ‘(current-input-port)’.  If START or END are specified, store data
     only into the substring of STR bounded by START and END (which
     default to the beginning and end of the string, respectively).

     Return a pair consisting of the delimiter that terminated the
     string and the number of characters read.  If reading stopped at
     the end of file, the delimiter returned is the EOF-OBJECT; if the
     string was filled without encountering a delimiter, this value is
     ‘#f’.

 -- Scheme Procedure: %read-line [port]
 -- C Function: scm_read_line (port)
     Read a newline-terminated line from PORT, allocating storage as
     necessary.  The newline terminator (if any) is removed from the
     string, and a pair consisting of the line and its delimiter is
     returned.  The delimiter may be either a newline or the EOF-OBJECT;
     if ‘%read-line’ is called at the end of file, it returns the pair
     ‘(#<eof> . #<eof>)’.

 -- Scheme Procedure: write-line obj [port]
 -- C Function: scm_write_line (obj, port)
     Display OBJ and a newline character to PORT.  If PORT is not
     specified, ‘(current-output-port)’ is used.  This procedure is
     equivalent to:
          (display obj [port])
          (newline [port])


File: guile.info,  Node: Default Ports,  Next: Port Types,  Prev: Line/Delimited,  Up: Input and Output

6.12.9 Default Ports for Input, Output and Errors
-------------------------------------------------

 -- Scheme Procedure: current-input-port
 -- C Function: scm_current_input_port ()
     Return the current input port.  This is the default port used by
     many input procedures.

     Initially this is the “standard input” in Unix and C terminology.
     When the standard input is a tty the port is unbuffered, otherwise
     it’s fully buffered.

     Unbuffered input is good if an application runs an interactive
     subprocess, since any type-ahead input won’t go into Guile’s buffer
     and be unavailable to the subprocess.

     Note that Guile buffering is completely separate from the tty “line
     discipline”.  In the usual cooked mode on a tty Guile only sees a
     line of input once the user presses <Return>.

 -- Scheme Procedure: current-output-port
 -- C Function: scm_current_output_port ()
     Return the current output port.  This is the default port used by
     many output procedures.

     Initially this is the “standard output” in Unix and C terminology.
     When the standard output is a tty this port is unbuffered,
     otherwise it’s fully buffered.

     Unbuffered output to a tty is good for ensuring progress output or
     a prompt is seen.  But an application which always prints whole
     lines could change to line buffered, or an application with a lot
     of output could go fully buffered and perhaps make explicit
     ‘force-output’ calls (*note Buffering::) at selected points.

 -- Scheme Procedure: current-error-port
 -- C Function: scm_current_error_port ()
     Return the port to which errors and warnings should be sent.

     Initially this is the “standard error” in Unix and C terminology.
     When the standard error is a tty this port is unbuffered, otherwise
     it’s fully buffered.

 -- Scheme Procedure: set-current-input-port port
 -- Scheme Procedure: set-current-output-port port
 -- Scheme Procedure: set-current-error-port port
 -- C Function: scm_set_current_input_port (port)
 -- C Function: scm_set_current_output_port (port)
 -- C Function: scm_set_current_error_port (port)
     Change the ports returned by ‘current-input-port’,
     ‘current-output-port’ and ‘current-error-port’, respectively, so
     that they use the supplied PORT for input or output.

 -- Scheme Procedure: with-input-from-port port thunk
 -- Scheme Procedure: with-output-to-port port thunk
 -- Scheme Procedure: with-error-to-port port thunk
     Call THUNK in a dynamic environment in which ‘current-input-port’,
     ‘current-output-port’ or ‘current-error-port’ is rebound to the
     given PORT.

 -- C Function: void scm_dynwind_current_input_port (SCM port)
 -- C Function: void scm_dynwind_current_output_port (SCM port)
 -- C Function: void scm_dynwind_current_error_port (SCM port)
     These functions must be used inside a pair of calls to
     ‘scm_dynwind_begin’ and ‘scm_dynwind_end’ (*note Dynamic Wind::).
     During the dynwind context, the indicated port is set to PORT.

     More precisely, the current port is swapped with a ‘backup’ value
     whenever the dynwind context is entered or left.  The backup value
     is initialized with the PORT argument.


File: guile.info,  Node: Port Types,  Next: Venerable Port Interfaces,  Prev: Default Ports,  Up: Input and Output

6.12.10 Types of Port
---------------------

* Menu:

* File Ports:: Ports on an operating system file.
* Bytevector Ports:: Ports on a bytevector.
* String Ports:: Ports on a Scheme string.
* Custom Ports:: Ports whose implementation you control.
* Soft Ports:: An older version of custom ports.
* Void Ports:: Ports on nothing at all.


File: guile.info,  Node: File Ports,  Next: Bytevector Ports,  Up: Port Types

6.12.10.1 File Ports
....................

The following procedures are used to open file ports.  See also *note
open: Ports and File Descriptors, for an interface to the Unix ‘open’
system call.

   All file access uses the “LFS” large file support functions when
available, so files bigger than 2 Gbytes (2^31 bytes) can be read and
written on a 32-bit system.

   Most systems have limits on how many files can be open, so it’s
strongly recommended that file ports be closed explicitly when no longer
required (*note Ports::).

 -- Scheme Procedure: open-file filename mode [#:guess-encoding=#f]
          [#:encoding=#f]
 -- C Function: scm_open_file_with_encoding (filename, mode,
          guess_encoding, encoding)
 -- C Function: scm_open_file (filename, mode)
     Open the file whose name is FILENAME, and return a port
     representing that file.  The attributes of the port are determined
     by the MODE string.  The way in which this is interpreted is
     similar to C stdio.  The first character must be one of the
     following:

     ‘r’
          Open an existing file for input.
     ‘w’
          Open a file for output, creating it if it doesn’t already
          exist or removing its contents if it does.
     ‘a’
          Open a file for output, creating it if it doesn’t already
          exist.  All writes to the port will go to the end of the file.
          The "append mode" can be turned off while the port is in use
          *note fcntl: Ports and File Descriptors.

     The following additional characters can be appended:

     ‘+’
          Open the port for both input and output.  E.g., ‘r+’: open an
          existing file for both input and output.
     ‘0’
          Create an "unbuffered" port.  In this case input and output
          operations are passed directly to the underlying port
          implementation without additional buffering.  This is likely
          to slow down I/O operations.  The buffering mode can be
          changed while a port is in use (*note Buffering::).
     ‘l’
          Add line-buffering to the port.  The port output buffer will
          be automatically flushed whenever a newline character is
          written.
     ‘b’
          Use binary mode, ensuring that each byte in the file will be
          read as one Scheme character.

          To provide this property, the file will be opened with the
          8-bit character encoding "ISO-8859-1", ignoring the default
          port encoding.  *Note Ports::, for more information on port
          encodings.

          Note that while it is possible to read and write binary data
          as characters or strings, it is usually better to treat bytes
          as octets, and byte sequences as bytevectors.  *Note Binary
          I/O::, for more.

          This option had another historical meaning, for DOS
          compatibility: in the default (textual) mode, DOS reads a
          CR-LF sequence as one LF byte.  The ‘b’ flag prevents this
          from happening, adding ‘O_BINARY’ to the underlying ‘open’
          call.  Still, the flag is generally useful because of its port
          encoding ramifications.

     Unless binary mode is requested, the character encoding of the new
     port is determined as follows: First, if GUESS-ENCODING is true,
     the ‘file-encoding’ procedure is used to guess the encoding of the
     file (*note Character Encoding of Source Files::).  If
     GUESS-ENCODING is false or if ‘file-encoding’ fails, ENCODING is
     used unless it is also false.  As a last resort, the default port
     encoding is used.  *Note Ports::, for more information on port
     encodings.  It is an error to pass a non-false GUESS-ENCODING or
     ENCODING if binary mode is requested.

     If a file cannot be opened with the access requested, ‘open-file’
     throws an exception.

 -- Scheme Procedure: open-input-file filename [#:guess-encoding=#f]
          [#:encoding=#f] [#:binary=#f]

     Open FILENAME for input.  If BINARY is true, open the port in
     binary mode, otherwise use text mode.  ENCODING and GUESS-ENCODING
     determine the character encoding as described above for
     ‘open-file’.  Equivalent to
          (open-file FILENAME
                     (if BINARY "rb" "r")
                     #:guess-encoding GUESS-ENCODING
                     #:encoding ENCODING)

 -- Scheme Procedure: open-output-file filename [#:encoding=#f]
          [#:binary=#f]

     Open FILENAME for output.  If BINARY is true, open the port in
     binary mode, otherwise use text mode.  ENCODING specifies the
     character encoding as described above for ‘open-file’.  Equivalent
     to
          (open-file FILENAME
                     (if BINARY "wb" "w")
                     #:encoding ENCODING)

 -- Scheme Procedure: call-with-input-file filename proc
          [#:guess-encoding=#f] [#:encoding=#f] [#:binary=#f]
 -- Scheme Procedure: call-with-output-file filename proc
          [#:encoding=#f] [#:binary=#f]
     Open FILENAME for input or output, and call ‘(PROC port)’ with the
     resulting port.  Return the value returned by PROC.  FILENAME is
     opened as per ‘open-input-file’ or ‘open-output-file’ respectively,
     and an error is signaled if it cannot be opened.

     When PROC returns, the port is closed.  If PROC does not return
     (e.g. if it throws an error), then the port might not be closed
     automatically, though it will be garbage collected in the usual way
     if not otherwise referenced.

 -- Scheme Procedure: with-input-from-file filename thunk
          [#:guess-encoding=#f] [#:encoding=#f] [#:binary=#f]
 -- Scheme Procedure: with-output-to-file filename thunk [#:encoding=#f]
          [#:binary=#f]
 -- Scheme Procedure: with-error-to-file filename thunk [#:encoding=#f]
          [#:binary=#f]
     Open FILENAME and call ‘(THUNK)’ with the new port setup as
     respectively the ‘current-input-port’, ‘current-output-port’, or
     ‘current-error-port’.  Return the value returned by THUNK.
     FILENAME is opened as per ‘open-input-file’ or ‘open-output-file’
     respectively, and an error is signaled if it cannot be opened.

     When THUNK returns, the port is closed and the previous setting of
     the respective current port is restored.

     The current port setting is managed with ‘dynamic-wind’, so the
     previous value is restored no matter how THUNK exits (eg. an
     exception), and if THUNK is re-entered (via a captured
     continuation) then it’s set again to the FILENAME port.

     The port is closed when THUNK returns normally, but not when exited
     via an exception or new continuation.  This ensures it’s still
     ready for use if THUNK is re-entered by a captured continuation.
     Of course the port is always garbage collected and closed in the
     usual way when no longer referenced anywhere.

 -- Scheme Procedure: port-mode port
 -- C Function: scm_port_mode (port)
     Return the port modes associated with the open port PORT.  These
     will not necessarily be identical to the modes used when the port
     was opened, since modes such as "append" which are used only during
     port creation are not retained.

 -- Scheme Procedure: port-filename port
 -- C Function: scm_port_filename (port)
     Return the filename associated with PORT, or ‘#f’ if no filename is
     associated with the port.

     PORT must be open; ‘port-filename’ cannot be used once the port is
     closed.

 -- Scheme Procedure: set-port-filename! port filename
 -- C Function: scm_set_port_filename_x (port, filename)
     Change the filename associated with PORT, using the current input
     port if none is specified.  Note that this does not change the
     port’s source of data, but only the value that is returned by
     ‘port-filename’ and reported in diagnostic output.

 -- Scheme Procedure: file-port? obj
 -- C Function: scm_file_port_p (obj)
     Determine whether OBJ is a port that is related to a file.


File: guile.info,  Node: Bytevector Ports,  Next: String Ports,  Prev: File Ports,  Up: Port Types

6.12.10.2 Bytevector Ports
..........................

 -- Scheme Procedure: open-bytevector-input-port bv [transcoder]
 -- C Function: scm_open_bytevector_input_port (bv, transcoder)
     Return an input port whose contents are drawn from bytevector BV
     (*note Bytevectors::).

     The TRANSCODER argument is currently not supported.

 -- Scheme Procedure: open-bytevector-output-port [transcoder]
 -- C Function: scm_open_bytevector_output_port (transcoder)
     Return two values: a binary output port and a procedure.  The
     latter should be called with zero arguments to obtain a bytevector
     containing the data accumulated by the port, as illustrated below.

          (call-with-values
            (lambda ()
              (open-bytevector-output-port))
            (lambda (port get-bytevector)
              (display "hello" port)
              (get-bytevector)))

          ⇒ #vu8(104 101 108 108 111)

     The TRANSCODER argument is currently not supported.

 -- Scheme Procedure: call-with-output-bytevector proc
     Call the one-argument procedure PROC with a newly created
     bytevector output port.  When the function returns, the bytevector
     composed of the characters written into the port is returned.  PROC
     should not close the port.

 -- Scheme Procedure: call-with-input-bytevector bytevector proc
     Call the one-argument procedure PROC with a newly created input
     port from which BYTEVECTOR’s contents may be read.  The values
     yielded by the PROC is returned.


File: guile.info,  Node: String Ports,  Next: Custom Ports,  Prev: Bytevector Ports,  Up: Port Types

6.12.10.3 String Ports
......................

 -- Scheme Procedure: call-with-output-string proc
 -- C Function: scm_call_with_output_string (proc)
     Calls the one-argument procedure PROC with a newly created output
     port.  When the function returns, the string composed of the
     characters written into the port is returned.  PROC should not
     close the port.

 -- Scheme Procedure: call-with-input-string string proc
 -- C Function: scm_call_with_input_string (string, proc)
     Calls the one-argument procedure PROC with a newly created input
     port from which STRING’s contents may be read.  The value yielded
     by the PROC is returned.

 -- Scheme Procedure: with-output-to-string thunk
     Calls the zero-argument procedure THUNK with the current output
     port set temporarily to a new string port.  It returns a string
     composed of the characters written to the current output.

 -- Scheme Procedure: with-input-from-string string thunk
     Calls the zero-argument procedure THUNK with the current input port
     set temporarily to a string port opened on the specified STRING.
     The value yielded by THUNK is returned.

 -- Scheme Procedure: open-input-string str
 -- C Function: scm_open_input_string (str)
     Take a string and return an input port that delivers characters
     from the string.  The port can be closed by ‘close-input-port’,
     though its storage will be reclaimed by the garbage collector if it
     becomes inaccessible.

 -- Scheme Procedure: open-output-string
 -- C Function: scm_open_output_string ()
     Return an output port that will accumulate characters for retrieval
     by ‘get-output-string’.  The port can be closed by the procedure
     ‘close-output-port’, though its storage will be reclaimed by the
     garbage collector if it becomes inaccessible.

 -- Scheme Procedure: get-output-string port
 -- C Function: scm_get_output_string (port)
     Given an output port created by ‘open-output-string’, return a
     string consisting of the characters that have been output to the
     port so far.

     ‘get-output-string’ must be used before closing PORT, once closed
     the string cannot be obtained.

   With string ports, the port-encoding is treated differently than
other types of ports.  When string ports are created, they do not
inherit a character encoding from the current locale.  They are given a
default locale that allows them to handle all valid string characters.
Typically one should not modify a string port’s character encoding away
from its default.  *Note Encoding::.


File: guile.info,  Node: Custom Ports,  Next: Soft Ports,  Prev: String Ports,  Up: Port Types

6.12.10.4 Custom Ports
......................

Custom ports allow the user to provide input and handle output via
user-supplied procedures.  Guile currently only provides custom binary
ports, not textual ports; for custom textual ports, *Note Soft Ports::.
We should add the R6RS custom textual port interfaces though.
Contributions are appreciated.

 -- Scheme Procedure: make-custom-binary-input-port id read!
          get-position set-position! close
     Return a new custom binary input port(1) named ID (a string) whose
     input is drained by invoking READ! and passing it a bytevector, an
     index where bytes should be written, and the number of bytes to
     read.  The ‘read!’ procedure must return an integer indicating the
     number of bytes read, or ‘0’ to indicate the end-of-file.

     Optionally, if GET-POSITION is not ‘#f’, it must be a thunk that
     will be called when ‘port-position’ is invoked on the custom binary
     port and should return an integer indicating the position within
     the underlying data stream; if GET-POSITION was not supplied, the
     returned port does not support ‘port-position’.

     Likewise, if SET-POSITION! is not ‘#f’, it should be a one-argument
     procedure.  When ‘set-port-position!’ is invoked on the custom
     binary input port, SET-POSITION! is passed an integer indicating
     the position of the next byte is to read.

     Finally, if CLOSE is not ‘#f’, it must be a thunk.  It is invoked
     when the custom binary input port is closed.

     The returned port is fully buffered by default, but its buffering
     mode can be changed using ‘setvbuf’ (*note Buffering::).

     Using a custom binary input port, the ‘open-bytevector-input-port’
     procedure (*note Bytevector Ports::) could be implemented as
     follows:

          (define (open-bytevector-input-port source)
            (define position 0)
            (define length (bytevector-length source))

            (define (read! bv start count)
              (let ((count (min count (- length position))))
                (bytevector-copy! source position
                                  bv start count)
                (set! position (+ position count))
                count))

            (define (get-position) position)

            (define (set-position! new-position)
              (set! position new-position))

            (make-custom-binary-input-port "the port" read!
                                            get-position set-position!
                                            #f))

          (read (open-bytevector-input-port (string->utf8 "hello")))
          ⇒ hello

 -- Scheme Procedure: make-custom-binary-output-port id write!
          get-position set-position! close
     Return a new custom binary output port named ID (a string) whose
     output is sunk by invoking WRITE! and passing it a bytevector, an
     index where bytes should be read from this bytevector, and the
     number of bytes to be “written”.  The ‘write!’ procedure must
     return an integer indicating the number of bytes actually written;
     when it is passed ‘0’ as the number of bytes to write, it should
     behave as though an end-of-file was sent to the byte sink.

     The other arguments are as for ‘make-custom-binary-input-port’.

 -- Scheme Procedure: make-custom-binary-input/output-port id read!
          write! get-position set-position! close
     Return a new custom binary input/output port named ID (a string).
     The various arguments are the same as for The other arguments are
     as for ‘make-custom-binary-input-port’ and
     ‘make-custom-binary-output-port’.  If buffering is enabled on the
     port, as is the case by default, input will be buffered in both
     directions; *Note Buffering::.  If the SET-POSITION! function is
     provided and not ‘#f’, then the port will also be marked as
     random-access, causing the buffer to be flushed between reads and
     writes.

   ---------- Footnotes ----------

   (1) This is similar in spirit to Guile’s “soft ports” (*note Soft
Ports::).


File: guile.info,  Node: Soft Ports,  Next: Void Ports,  Prev: Custom Ports,  Up: Port Types

6.12.10.5 Soft Ports
....................

A “soft port” is a port based on a vector of procedures capable of
accepting or delivering characters.  It allows emulation of I/O ports.

 -- Scheme Procedure: make-soft-port pv modes
     Return a port capable of receiving or delivering characters as
     specified by the MODES string (*note open-file: File Ports.).  PV
     must be a vector of length 5 or 6.  Its components are as follows:

       0. procedure accepting one character for output
       1. procedure accepting a string for output
       2. thunk for flushing output
       3. thunk for getting one character
       4. thunk for closing port (not by garbage collection)
       5. (if present and not ‘#f’) thunk for computing the number of
          characters that can be read from the port without blocking.

     For an output-only port only elements 0, 1, 2, and 4 need be
     procedures.  For an input-only port only elements 3 and 4 need be
     procedures.  Thunks 2 and 4 can instead be ‘#f’ if there is no
     useful operation for them to perform.

     If thunk 3 returns ‘#f’ or an ‘eof-object’ (*note eof-object?:
     (r5rs)Input.) it indicates that the port has reached end-of-file.
     For example:

          (define stdout (current-output-port))
          (define p (make-soft-port
                     (vector
                      (lambda (c) (write c stdout))
                      (lambda (s) (display s stdout))
                      (lambda () (display "." stdout))
                      (lambda () (char-upcase (read-char)))
                      (lambda () (display "@" stdout)))
                     "rw"))

          (write p p) ⇒ #<input-output: soft 8081e20>


File: guile.info,  Node: Void Ports,  Prev: Soft Ports,  Up: Port Types

6.12.10.6 Void Ports
....................

This kind of port causes any data to be discarded when written to, and
always returns the end-of-file object when read from.

 -- Scheme Procedure: %make-void-port mode
 -- C Function: scm_sys_make_void_port (mode)
     Create and return a new void port.  A void port acts like
     ‘/dev/null’.  The MODE argument specifies the input/output modes
     for this port: see the documentation for ‘open-file’ in *note File
     Ports::.


File: guile.info,  Node: Venerable Port Interfaces,  Next: Using Ports from C,  Prev: Port Types,  Up: Input and Output

6.12.11 Venerable Port Interfaces
---------------------------------

Over the 25 years or so that Guile has been around, its port system has
evolved, adding many useful features.  At the same time there have been
four major Scheme standards released in those 25 years, which also
evolve the common Scheme understanding of what a port interface should
be.  Alas, it would be too much to ask for all of these evolutionary
branches to be consistent.  Some of Guile’s original interfaces don’t
mesh with the later Scheme standards, and yet Guile can’t just drop old
interfaces.  Sadly as well, the R6RS and R7RS standards both part from a
base of R5RS, but end up in different and somewhat incompatible designs.

   Guile’s approach is to pick a set of port primitives that make sense
together.  We document that set of primitives, design our internal
interfaces around them, and recommend them to users.  As the R6RS I/O
system is the most capable standard that Scheme has yet produced in this
domain, we mostly recommend that; ‘(ice-9 binary-ports)’ and ‘(ice-9
textual-ports)’ are wholly modelled on ‘(rnrs io ports)’.  Guile does
not wholly copy R6RS, however; *Note R6RS Incompatibilities::.

   At the same time, we have many venerable port interfaces, lore handed
down to us from our hacker ancestors.  Most of these interfaces even
predate the expectation that Scheme should have modules, so they are
present in the default environment.  In Guile we support them as well
and we have no plans to remove them, but again we don’t recommend them
for new users.

 -- Scheme Procedure: char-ready? [port]
     Return ‘#t’ if a character is ready on input PORT and return ‘#f’
     otherwise.  If ‘char-ready?’ returns ‘#t’ then the next ‘read-char’
     operation on PORT is guaranteed not to hang.  If PORT is a file
     port at end of file then ‘char-ready?’ returns ‘#t’.

     ‘char-ready?’ exists to make it possible for a program to accept
     characters from interactive ports without getting stuck waiting for
     input.  Any input editors associated with such ports must make sure
     that characters whose existence has been asserted by ‘char-ready?’
     cannot be rubbed out.  If ‘char-ready?’ were to return ‘#f’ at end
     of file, a port at end of file would be indistinguishable from an
     interactive port that has no ready characters.

     Note that ‘char-ready?’ only works reliably for terminals and
     sockets with one-byte encodings.  Under the hood it will return
     ‘#t’ if the port has any input buffered, or if the file descriptor
     that backs the port polls as readable, indicating that Guile can
     fetch more bytes from the kernel.  However being able to fetch one
     byte doesn’t mean that a full character is available; *Note
     Encoding::.  Also, on many systems it’s possible for a file
     descriptor to poll as readable, but then block when it comes time
     to read bytes.  Note also that on Linux kernels, all file ports
     backed by files always poll as readable.  For non-file ports, this
     procedure always returns ‘#t’, except for soft ports, which have a
     ‘char-ready?’ handler.  *Note Soft Ports::.

     In short, this is a legacy procedure whose semantics are hard to
     provide.  However it is a useful check to see if any input is
     buffered.  *Note Non-Blocking I/O::.

 -- Scheme Procedure: read-char [port]
     The same as ‘get-char’, except that PORT defaults to the current
     input port.  *Note Textual I/O::.

 -- Scheme Procedure: peek-char [port]
     The same as ‘lookahead-char’, except that PORT defaults to the
     current input port.  *Note Textual I/O::.

 -- Scheme Procedure: unread-char cobj [port]
     The same as ‘unget-char’, except that PORT defaults to the current
     input port, and the arguments are swapped.  *Note Textual I/O::.

 -- Scheme Procedure: unread-string str port
 -- C Function: scm_unread_string (str, port)
     The same as ‘unget-string’, except that PORT defaults to the
     current input port, and the arguments are swapped.  *Note Textual
     I/O::.

 -- Scheme Procedure: newline [port]
     Send a newline to PORT.  If PORT is omitted, send to the current
     output port.  Equivalent to ‘(put-char port #\newline)’.

 -- Scheme Procedure: write-char chr [port]
     The same as ‘put-char’, except that PORT defaults to the current
     input port, and the arguments are swapped.  *Note Textual I/O::.


File: guile.info,  Node: Using Ports from C,  Next: I/O Extensions,  Prev: Venerable Port Interfaces,  Up: Input and Output

6.12.12 Using Ports from C
--------------------------

Guile’s C interfaces provides some niceties for sending and receiving
bytes and characters in a way that works better with C.

 -- C Function: size_t scm_c_read (SCM port, void *buffer, size_t size)
     Read up to SIZE bytes from PORT and store them in BUFFER.  The
     return value is the number of bytes actually read, which can be
     less than SIZE if end-of-file has been reached.

     Note that as this is a binary input procedure, this function does
     not update ‘port-line’ and ‘port-column’ (*note Textual I/O::).

 -- C Function: void scm_c_write (SCM port, const void *buffer, size_t
          size)
     Write SIZE bytes at BUFFER to PORT.

     Note that as this is a binary output procedure, this function does
     not update ‘port-line’ and ‘port-column’ (*note Textual I/O::).

 -- C Function: size_t scm_c_read_bytes (SCM port, SCM bv, size_t start,
          size_t count)
 -- C Function: void scm_c_write_bytes (SCM port, SCM bv, size_t start,
          size_t count)
     Like ‘scm_c_read’ and ‘scm_c_write’, but reading into or writing
     from the bytevector BV.  COUNT indicates the byte index at which to
     start in the bytevector, and the read or write will continue for
     COUNT bytes.

 -- C Function: void scm_unget_bytes (const unsigned char *buf, size_t
          len, SCM port)
 -- C Function: void scm_unget_byte (int c, SCM port)
 -- C Function: void scm_ungetc (scm_t_wchar c, SCM port)
     Like ‘unget-bytevector’, ‘unget-byte’, and ‘unget-char’,
     respectively.  *Note Textual I/O::.

 -- C Function: void scm_c_put_latin1_chars (SCM port, const scm_t_uint8
          *buf, size_t len)
 -- C Function: void scm_c_put_utf32_chars (SCM port, const scm_t_uint32
          *buf, size_t len);
     Write a string to PORT.  In the first case, the ‘scm_t_uint8*’
     buffer is a string in the latin-1 encoding.  In the second, the
     ‘scm_t_uint32*’ buffer is a string in the UTF-32 encoding.  These
     routines will update the port’s line and column.


File: guile.info,  Node: I/O Extensions,  Next: Non-Blocking I/O,  Prev: Using Ports from C,  Up: Input and Output

6.12.13 Implementing New Port Types in C
----------------------------------------

This section describes how to implement a new port type in C. Although
ports support many operations, as a data structure they present an
opaque interface to the user.  To the port implementor, you have two
pieces of information to work with: the port type, and the port’s
“stream”.  The port type is an opaque pointer allocated when defining
your port type.  It is your key into the port API, and it helps you
identify which ports are actually yours.  The “stream” is a pointer you
control, and which you set when you create a port.  Get a stream from a
port using the ‘SCM_STREAM’ macro.  Note that your port methods are only
ever called with ports of your type.

   A port type is created by calling ‘scm_make_port_type’.  Once you
have your port type, you can create ports with ‘scm_c_make_port’, or
‘scm_c_make_port_with_encoding’.

 -- Function: scm_t_port_type* scm_make_port_type (char *name, size_t
          (*read) (SCM port, SCM dst, size_t start, size_t count),
          size_t (*write) (SCM port, SCM src, size_t start, size_t
          count))
     Define a new port type.  The NAME, READ and WRITE parameters are
     initial values for those port type fields, as described below.  The
     other fields are initialized with default values and can be changed
     later.

 -- Function: SCM scm_c_make_port_with_encoding (scm_t_port_type *type,
          unsigned long mode_bits, SCM encoding, SCM
          conversion_strategy, scm_t_bits stream)
 -- Function: SCM scm_c_make_port (scm_t_port_type *type, unsigned long
          mode_bits, scm_t_bits stream)
     Make a port with the given TYPE.  The STREAM indicates the private
     data associated with the port, which your port implementation may
     later retrieve with ‘SCM_STREAM’.  The mode bits should include one
     or more of the flags ‘SCM_RDNG’ or ‘SCM_WRTNG’, indicating that the
     port is an input and/or an output port, respectively.  The mode
     bits may also include ‘SCM_BUF0’ or ‘SCM_BUFLINE’, indicating that
     the port should be unbuffered or line-buffered, respectively.  The
     default is that the port will be block-buffered.  *Note
     Buffering::.

     As you would imagine, ENCODING and CONVERSION_STRATEGY specify the
     port’s initial textual encoding and conversion strategy.  Both are
     symbols.  ‘scm_c_make_port’ is the same as
     ‘scm_c_make_port_with_encoding’, except it uses the default port
     encoding and conversion strategy.

   The port type has a number of associate procedures and properties
which collectively implement the port’s behavior.  Creating a new port
type mostly involves writing these procedures.

‘name’
     A pointer to a NUL terminated string: the name of the port type.
     This property is initialized via the first argument to
     ‘scm_make_port_type’.

‘read’
     A port’s ‘read’ implementation fills read buffers.  It should copy
     bytes to the supplied bytevector ‘dst’, starting at offset ‘start’
     and continuing for ‘count’ bytes, returning the number of bytes
     read.

‘write’
     A port’s ‘write’ implementation flushes write buffers to the
     mutable store.  It should write out bytes from the supplied
     bytevector ‘src’, starting at offset ‘start’ and continuing for
     ‘count’ bytes, and return the number of bytes that were written.

‘read_wait_fd’
‘write_wait_fd’
     If a port’s ‘read’ or ‘write’ function returns ‘(size_t) -1’, that
     indicates that reading or writing would block.  In that case to
     preserve the illusion of a blocking read or write operation,
     Guile’s C port run-time will ‘poll’ on the file descriptor returned
     by either the port’s ‘read_wait_fd’ or ‘write_wait_fd’ function.
     Set using

      -- Function: void scm_set_port_read_wait_fd (scm_t_port_type
               *type, int (*wait_fd) (SCM port))
      -- Function: void scm_set_port_write_wait_fd (scm_t_port_type
               *type, int (*wait_fd) (SCM port))

     Only a port type which implements the ‘read_wait_fd’ or
     ‘write_wait_fd’ port methods can usefully return ‘(size_t) -1’ from
     a read or write function.  *Note Non-Blocking I/O::, for more on
     non-blocking I/O in Guile.

‘print’
     Called when ‘write’ is called on the port, to print a port
     description.  For example, for a file port it may produce something
     like: ‘#<input: /etc/passwd 3>’.  Set using

      -- Function: void scm_set_port_print (scm_t_port_type *type, int
               (*print) (SCM port, SCM dest_port, scm_print_state
               *pstate))
          The first argument PORT is the port being printed, the second
          argument DEST_PORT is where its description should go.

‘close’
     Called when the port is closed.  It should free any resources used
     by the port.  Set using

      -- Function: void scm_set_port_close (scm_t_port_type *type, void
               (*close) (SCM port))

     By default, ports that are garbage collected just go away without
     closing.  If your port type needs to release some external resource
     like a file descriptor, or needs to make sure that its internal
     buffers are flushed even if the port is collected while it was
     open, then mark the port type as needing a close on GC.

      -- Function: void scm_set_port_needs_close_on_gc (scm_t_port_type
               *type, int needs_close_p)

‘seek’
     Set the current position of the port.  Guile will flush read and/or
     write buffers before seeking, as appropriate.

      -- Function: void scm_set_port_seek (scm_t_port_type *type,
               scm_t_off (*seek) (SCM port, scm_t_off offset, int
               whence))

‘truncate’
     Truncate the port data to be specified length.  Guile will flush
     buffers before hand, as appropriate.  Set using

      -- Function: void scm_set_port_truncate (scm_t_port_type *type,
               void (*truncate) (SCM port, scm_t_off length))

‘random_access_p’
     Determine whether this port is a random-access port.

     Seeking on a random-access port with buffered input, or switching
     to writing after reading, will cause the buffered input to be
     discarded and Guile will seek the port back the buffered number of
     bytes.  Likewise seeking on a random-access port with buffered
     output, or switching to reading after writing, will flush pending
     bytes with a call to the ‘write’ procedure.  *Note Buffering::.

     Indicate to Guile that your port needs this behavior by returning a
     nonzero value from your ‘random_access_p’ function.  The default
     implementation of this function returns nonzero if the port type
     supplies a seek implementation.

      -- Function: void scm_set_port_random_access_p (scm_t_port_type
               *type, int (*random_access_p) (SCM port));

‘get_natural_buffer_sizes’
     Guile will internally attach buffers to ports.  An input port
     always has a read buffer and an output port always has a write
     buffer.  *Note Buffering::.  A port buffer consists of a
     bytevector, along with some cursors into that bytevector denoting
     where to get and put data.

     Port implementations generally don’t have to be concerned with
     buffering: a port type’s ‘read’ or ‘write’ function will receive
     the buffer’s bytevector as an argument, along with an offset and a
     length into that bytevector, and should then either fill or empty
     that bytevector.  However in some cases, port implementations may
     be able to provide an appropriate default buffer size to Guile.

      -- Function: void scm_set_port_get_natural_buffer_sizes
               (scm_t_port_type *type, void (*get_natural_buffer_sizes)
               (SCM, size_t *read_buf_size, size_t *write_buf_size))
          Fill in READ_BUF_SIZE and WRITE_BUF_SIZE with an appropriate
          buffer size for this port, if one is known.

     File ports implement a ‘get_natural_buffer_sizes’ to let the
     operating system inform Guile about the appropriate buffer sizes
     for the particular file opened by the port.

   Note that calls to all of these methods can proceed in parallel and
concurrently and from any thread up until the point that the port is
closed.  The call to ‘close’ will happen when no other method is
running, and no method will be called after the ‘close’ method is
called.  If your port implementation needs mutual exclusion to prevent
concurrency, it is responsible for locking appropriately.


File: guile.info,  Node: Non-Blocking I/O,  Next: BOM Handling,  Prev: I/O Extensions,  Up: Input and Output

6.12.14 Non-Blocking I/O
------------------------

Most ports in Guile are “blocking”: when you try to read a character
from a port, Guile will block on the read until a character is ready, or
end-of-stream is detected.  Likewise whenever Guile goes to write
(possibly buffered) data to an output port, Guile will block until all
the data is written.

   Interacting with ports in blocking mode is very convenient: you can
write straightforward, sequential algorithms whose code flow reflects
the flow of data.  However, blocking I/O has two main limitations.

   The first is that it’s easy to get into a situation where code is
waiting on data.  Time spent waiting on data when code could be doing
something else is wasteful and prevents your program from reaching its
peak throughput.  If you implement a web server that sequentially
handles requests from clients, it’s very easy for the server to end up
waiting on a client to finish its HTTP request, or waiting on it to
consume the response.  The end result is that you are able to serve
fewer requests per second than you’d like to serve.

   The second limitation is related: a blocking parser over
user-controlled input is a denial-of-service vulnerability.  Indeed the
so-called “slow loris” attack of the early 2010s was just that: an
attack on common web servers that drip-fed HTTP requests, one character
at a time.  All it took was a handful of slow loris connections to
occupy an entire web server.

   In Guile we would like to preserve the ability to write
straightforward blocking networking processes of all kinds, but under
the hood to allow those processes to suspend their requests if they
would block.

   To do this, the first piece is to allow Guile ports to declare
themselves as being nonblocking.  This is currently supported only for
file ports, which also includes sockets, terminals, or any other port
that is backed by a file descriptor.  To do that, we use an arcane UNIX
incantation:

     (let ((flags (fcntl socket F_GETFL)))
       (fcntl socket F_SETFL (logior O_NONBLOCK flags)))

   Now the file descriptor is open in non-blocking mode.  If Guile tries
to read or write from this file and the read or write returns a result
indicating that more data can only be had by doing a blocking read or
write, Guile will block by polling on the socket’s ‘read-wait-fd’ or
‘write-wait-fd’, to preserve the illusion of a blocking read or write.
*Note I/O Extensions:: for more on those internal interfaces.

   So far we have just reproduced the status quo: the file descriptor is
non-blocking, but the operations on the port do block.  To go farther,
it would be nice if we could suspend the “thread” using delimited
continuations, and only resume the thread once the file descriptor is
readable or writable.  (*Note Prompts::).

   But here we run into a difficulty.  The ports code is implemented in
C, which means that although we can suspend the computation to some
outer prompt, we can’t resume it because Guile can’t resume delimited
continuations that capture the C stack.

   To overcome this difficulty we have created a compatible but entirely
parallel implementation of port operations.  To use this implementation,
do the following:

     (use-modules (ice-9 suspendable-ports))
     (install-suspendable-ports!)

   This will replace the core I/O primitives like ‘get-char’ and
‘put-bytevector’ with new versions that are exactly the same as the ones
in the standard library, but with two differences.  One is that when a
read or a write would block, the suspendable port operations call out
the value of the ‘current-read-waiter’ or ‘current-write-waiter’
parameter, as appropriate.  *Note Parameters::.  The default read and
write waiters do the same thing that the C read and write waiters do,
which is to poll.  User code can parameterize the waiters, though,
enabling the computation to suspend and allow the program to process
other I/O operations.  Because the new suspendable ports implementation
is written in Scheme, that suspended computation can resume again later
when it is able to make progress.  Success!

   The other main difference is that because the new ports
implementation is written in Scheme, it is slower than C, currently by a
factor of 3 or 4, though it depends on many factors.  For this reason we
have to keep the C implementations as the default ones.  One day when
Guile’s compiler is better, we can close this gap and have only one port
operation implementation again.

   Note that Guile does not currently include an implementation of the
facility to suspend the current thread and schedule other threads in the
meantime.  Before adding such a thing, we want to make sure that we’re
providing the right primitives that can be used to build schedulers and
other user-space concurrency patterns, and that the patterns that we
settle on are the right patterns.  In the meantime, have a look at 8sync
(<https://gnu.org/software/8sync>) for a prototype of an asynchronous
I/O and concurrency facility.

 -- Scheme Procedure: install-suspendable-ports!
     Replace the core ports implementation with suspendable ports, as
     described above.  This will mutate the values of the bindings like
     ‘get-char’, ‘put-u8’, and so on in place.

 -- Scheme Procedure: uninstall-suspendable-ports!
     Restore the original core ports implementation, un-doing the effect
     of ‘install-suspendable-ports!’.

 -- Scheme Parameter: current-read-waiter
 -- Scheme Parameter: current-write-waiter
     Parameters whose values are procedures of one argument, called when
     a suspendable port operation would block on a port while reading or
     writing, respectively.  The default values of these parameters do a
     blocking ‘poll’ on the port’s file descriptor.  The procedures are
     passed the port in question as their one argument.


File: guile.info,  Node: BOM Handling,  Prev: Non-Blocking I/O,  Up: Input and Output

6.12.15 Handling of Unicode Byte Order Marks
--------------------------------------------

This section documents the finer points of Guile’s handling of Unicode
byte order marks (BOMs).  A byte order mark (U+FEFF) is typically found
at the start of a UTF-16 or UTF-32 stream, to allow readers to reliably
determine the byte order.  Occasionally, a BOM is found at the start of
a UTF-8 stream, but this is much less common and not generally
recommended.

   Guile attempts to handle BOMs automatically, and in accordance with
the recommendations of the Unicode Standard, when the port encoding is
set to ‘UTF-8’, ‘UTF-16’, or ‘UTF-32’.  In brief, Guile automatically
writes a BOM at the start of a UTF-16 or UTF-32 stream, and
automatically consumes one from the start of a UTF-8, UTF-16, or UTF-32
stream.

   As specified in the Unicode Standard, a BOM is only handled specially
at the start of a stream, and only if the port encoding is set to
‘UTF-8’, ‘UTF-16’ or ‘UTF-32’.  If the port encoding is set to
‘UTF-16BE’, ‘UTF-16LE’, ‘UTF-32BE’, or ‘UTF-32LE’, then BOMs are _not_
handled specially, and none of the special handling described in this
section applies.

   • To ensure that Guile will properly detect the byte order of a
     UTF-16 or UTF-32 stream, you must perform a textual read before any
     writes, seeks, or binary I/O. Guile will not attempt to read a BOM
     unless a read is explicitly requested at the start of the stream.

   • If a textual write is performed before the first read, then an
     arbitrary byte order will be chosen.  Currently, big endian is the
     default on all platforms, but that may change in the future.  If
     you wish to explicitly control the byte order of an output stream,
     set the port encoding to ‘UTF-16BE’, ‘UTF-16LE’, ‘UTF-32BE’, or
     ‘UTF-32LE’, and explicitly write a BOM (‘#\xFEFF’) if desired.

   • If ‘set-port-encoding!’ is called in the middle of a stream, Guile
     treats this as a new logical “start of stream” for purposes of BOM
     handling, and will forget about any BOMs that had previously been
     seen.  Therefore, it may choose a different byte order than had
     been used previously.  This is intended to support multiple logical
     text streams embedded within a larger binary stream.

   • Binary I/O operations are not guaranteed to update Guile’s notion
     of whether the port is at the “start of the stream”, nor are they
     guaranteed to produce or consume BOMs.

   • For ports that support seeking (e.g.  normal files), the input and
     output streams are considered linked: if the user reads first, then
     a BOM will be consumed (if appropriate), but later writes will
     _not_ produce a BOM. Similarly, if the user writes first, then
     later reads will _not_ consume a BOM.

   • For ports that are not random access (e.g.  pipes, sockets, and
     terminals), the input and output streams are considered
     _independent_ for purposes of BOM handling: the first read will
     consume a BOM (if appropriate), and the first write will _also_
     produce a BOM (if appropriate).  However, the input and output
     streams will always use the same byte order.

   • Seeks to the beginning of a file will set the “start of stream”
     flags.  Therefore, a subsequent textual read or write will consume
     or produce a BOM. However, unlike ‘set-port-encoding!’, if a byte
     order had already been chosen for the port, it will remain in
     effect after a seek, and cannot be changed by the presence of a
     BOM. Seeks anywhere other than the beginning of a file clear the
     “start of stream” flags.


File: guile.info,  Node: Regular Expressions,  Next: LALR(1) Parsing,  Prev: Input and Output,  Up: API Reference

6.13 Regular Expressions
========================

A “regular expression” (or “regexp”) is a pattern that describes a whole
class of strings.  A full description of regular expressions and their
syntax is beyond the scope of this manual.

   If your system does not include a POSIX regular expression library,
and you have not linked Guile with a third-party regexp library such as
Rx, these functions will not be available.  You can tell whether your
Guile installation includes regular expression support by checking
whether ‘(provided? 'regex)’ returns true.

   The following regexp and string matching features are provided by the
‘(ice-9 regex)’ module.  Before using the described functions, you
should load this module by executing ‘(use-modules (ice-9 regex))’.

* Menu:

* Regexp Functions::            Functions that create and match regexps.
* Match Structures::            Finding what was matched by a regexp.
* Backslash Escapes::           Removing the special meaning of regexp
                                meta-characters.


File: guile.info,  Node: Regexp Functions,  Next: Match Structures,  Up: Regular Expressions

6.13.1 Regexp Functions
-----------------------

By default, Guile supports POSIX extended regular expressions.  That
means that the characters ‘(’, ‘)’, ‘+’ and ‘?’ are special, and must be
escaped if you wish to match the literal characters and there is no
support for “non-greedy” variants of ‘*’, ‘+’ or ‘?’.

   This regular expression interface was modeled after that implemented
by SCSH, the Scheme Shell.  It is intended to be upwardly compatible
with SCSH regular expressions.

   Zero bytes (‘#\nul’) cannot be used in regex patterns or input
strings, since the underlying C functions treat that as the end of
string.  If there’s a zero byte an error is thrown.

   Internally, patterns and input strings are converted to the current
locale’s encoding, and then passed to the C library’s regular expression
routines (*note (libc)Regular Expressions::).  The returned match
structures always point to characters in the strings, not to individual
bytes, even in the case of multi-byte encodings.

 -- Scheme Procedure: string-match pattern str [start]
     Compile the string PATTERN into a regular expression and compare it
     with STR.  The optional numeric argument START specifies the
     position of STR at which to begin matching.

     ‘string-match’ returns a “match structure” which describes what, if
     anything, was matched by the regular expression.  *Note Match
     Structures::.  If STR does not match PATTERN at all, ‘string-match’
     returns ‘#f’.

   Two examples of a match follow.  In the first example, the pattern
matches the four digits in the match string.  In the second, the pattern
matches nothing.

     (string-match "[0-9][0-9][0-9][0-9]" "blah2002")
     ⇒ #("blah2002" (4 . 8))

     (string-match "[A-Za-z]" "123456")
     ⇒ #f

   Each time ‘string-match’ is called, it must compile its PATTERN
argument into a regular expression structure.  This operation is
expensive, which makes ‘string-match’ inefficient if the same regular
expression is used several times (for example, in a loop).  For better
performance, you can compile a regular expression in advance and then
match strings against the compiled regexp.

 -- Scheme Procedure: make-regexp pat flag...
 -- C Function: scm_make_regexp (pat, flaglst)
     Compile the regular expression described by PAT, and return the
     compiled regexp structure.  If PAT does not describe a legal
     regular expression, ‘make-regexp’ throws a
     ‘regular-expression-syntax’ error.

     The FLAG arguments change the behavior of the compiled regular
     expression.  The following values may be supplied:

      -- Variable: regexp/icase
          Consider uppercase and lowercase letters to be the same when
          matching.

      -- Variable: regexp/newline
          If a newline appears in the target string, then permit the ‘^’
          and ‘$’ operators to match immediately after or immediately
          before the newline, respectively.  Also, the ‘.’ and ‘[^...]’
          operators will never match a newline character.  The intent of
          this flag is to treat the target string as a buffer containing
          many lines of text, and the regular expression as a pattern
          that may match a single one of those lines.

      -- Variable: regexp/basic
          Compile a basic (“obsolete”) regexp instead of the extended
          (“modern”) regexps that are the default.  Basic regexps do not
          consider ‘|’, ‘+’ or ‘?’ to be special characters, and require
          the ‘{...}’ and ‘(...)’ metacharacters to be backslash-escaped
          (*note Backslash Escapes::).  There are several other
          differences between basic and extended regular expressions,
          but these are the most significant.

      -- Variable: regexp/extended
          Compile an extended regular expression rather than a basic
          regexp.  This is the default behavior; this flag will not
          usually be needed.  If a call to ‘make-regexp’ includes both
          ‘regexp/basic’ and ‘regexp/extended’ flags, the one which
          comes last will override the earlier one.

 -- Scheme Procedure: regexp-exec rx str [start [flags]]
 -- C Function: scm_regexp_exec (rx, str, start, flags)
     Match the compiled regular expression RX against ‘str’.  If the
     optional integer START argument is provided, begin matching from
     that position in the string.  Return a match structure describing
     the results of the match, or ‘#f’ if no match could be found.

     The FLAGS argument changes the matching behavior.  The following
     flag values may be supplied, use ‘logior’ (*note Bitwise
     Operations::) to combine them,

      -- Variable: regexp/notbol
          Consider that the START offset into STR is not the beginning
          of a line and should not match operator ‘^’.

          If RX was created with the ‘regexp/newline’ option above, ‘^’
          will still match after a newline in STR.

      -- Variable: regexp/noteol
          Consider that the end of STR is not the end of a line and
          should not match operator ‘$’.

          If RX was created with the ‘regexp/newline’ option above, ‘$’
          will still match before a newline in STR.

     ;; Regexp to match uppercase letters
     (define r (make-regexp "[A-Z]*"))

     ;; Regexp to match letters, ignoring case
     (define ri (make-regexp "[A-Z]*" regexp/icase))

     ;; Search for bob using regexp r
     (match:substring (regexp-exec r "bob"))
     ⇒ ""                  ; no match

     ;; Search for bob using regexp ri
     (match:substring (regexp-exec ri "Bob"))
     ⇒ "Bob"               ; matched case insensitive

 -- Scheme Procedure: regexp? obj
 -- C Function: scm_regexp_p (obj)
     Return ‘#t’ if OBJ is a compiled regular expression, or ‘#f’
     otherwise.


 -- Scheme Procedure: list-matches regexp str [flags]
     Return a list of match structures which are the non-overlapping
     matches of REGEXP in STR.  REGEXP can be either a pattern string or
     a compiled regexp.  The FLAGS argument is as per ‘regexp-exec’
     above.

          (map match:substring (list-matches "[a-z]+" "abc 42 def 78"))
          ⇒ ("abc" "def")

 -- Scheme Procedure: fold-matches regexp str init proc [flags]
     Apply PROC to the non-overlapping matches of REGEXP in STR, to
     build a result.  REGEXP can be either a pattern string or a
     compiled regexp.  The FLAGS argument is as per ‘regexp-exec’ above.

     PROC is called as ‘(PROC match prev)’ where MATCH is a match
     structure and PREV is the previous return from PROC.  For the first
     call PREV is the given INIT parameter.  ‘fold-matches’ returns the
     final value from PROC.

     For example to count matches,

          (fold-matches "[a-z][0-9]" "abc x1 def y2" 0
                        (lambda (match count)
                          (1+ count)))
          ⇒ 2


   Regular expressions are commonly used to find patterns in one string
and replace them with the contents of another string.  The following
functions are convenient ways to do this.

 -- Scheme Procedure: regexp-substitute port match item ...
     Write to PORT selected parts of the match structure MATCH.  Or if
     PORT is ‘#f’ then form a string from those parts and return that.

     Each ITEM specifies a part to be written, and may be one of the
     following,

        • A string.  String arguments are written out verbatim.

        • An integer.  The submatch with that number is written
          (‘match:substring’).  Zero is the entire match.

        • The symbol ‘pre’.  The portion of the matched string preceding
          the regexp match is written (‘match:prefix’).

        • The symbol ‘post’.  The portion of the matched string
          following the regexp match is written (‘match:suffix’).

     For example, changing a match and retaining the text before and
     after,

          (regexp-substitute #f (string-match "[0-9]+" "number 25 is good")
                             'pre "37" 'post)
          ⇒ "number 37 is good"

     Or matching a YYYYMMDD format date such as ‘20020828’ and
     re-ordering and hyphenating the fields.

          (define date-regex
             "([0-9][0-9][0-9][0-9])([0-9][0-9])([0-9][0-9])")
          (define s "Date 20020429 12am.")
          (regexp-substitute #f (string-match date-regex s)
                             'pre 2 "-" 3 "-" 1 'post " (" 0 ")")
          ⇒ "Date 04-29-2002 12am. (20020429)"

 -- Scheme Procedure: regexp-substitute/global port regexp target
          item...
     Write to PORT selected parts of matches of REGEXP in TARGET.  If
     PORT is ‘#f’ then form a string from those parts and return that.
     REGEXP can be a string or a compiled regex.

     This is similar to ‘regexp-substitute’, but allows global
     substitutions on TARGET.  Each ITEM behaves as per
     ‘regexp-substitute’, with the following differences,

        • A function.  Called as ‘(ITEM match)’ with the match structure
          for the REGEXP match, it should return a string to be written
          to PORT.

        • The symbol ‘post’.  This doesn’t output anything, but instead
          causes ‘regexp-substitute/global’ to recurse on the unmatched
          portion of TARGET.

          This _must_ be supplied to perform a global search and replace
          on TARGET; without it ‘regexp-substitute/global’ returns after
          a single match and output.

     For example, to collapse runs of tabs and spaces to a single hyphen
     each,

          (regexp-substitute/global #f "[ \t]+"  "this   is   the text"
                                    'pre "-" 'post)
          ⇒ "this-is-the-text"

     Or using a function to reverse the letters in each word,

          (regexp-substitute/global #f "[a-z]+"  "to do and not-do"
            'pre (lambda (m) (string-reverse (match:substring m))) 'post)
          ⇒ "ot od dna ton-od"

     Without the ‘post’ symbol, just one regexp match is made.  For
     example the following is the date example from ‘regexp-substitute’
     above, without the need for the separate ‘string-match’ call.

          (define date-regex
             "([0-9][0-9][0-9][0-9])([0-9][0-9])([0-9][0-9])")
          (define s "Date 20020429 12am.")
          (regexp-substitute/global #f date-regex s
                                    'pre 2 "-" 3 "-" 1 'post " (" 0 ")")

          ⇒ "Date 04-29-2002 12am. (20020429)"


File: guile.info,  Node: Match Structures,  Next: Backslash Escapes,  Prev: Regexp Functions,  Up: Regular Expressions

6.13.2 Match Structures
-----------------------

A “match structure” is the object returned by ‘string-match’ and
‘regexp-exec’.  It describes which portion of a string, if any, matched
the given regular expression.  Match structures include: a reference to
the string that was checked for matches; the starting and ending
positions of the regexp match; and, if the regexp included any
parenthesized subexpressions, the starting and ending positions of each
submatch.

   In each of the regexp match functions described below, the ‘match’
argument must be a match structure returned by a previous call to
‘string-match’ or ‘regexp-exec’.  Most of these functions return some
information about the original target string that was matched against a
regular expression; we will call that string TARGET for easy reference.

 -- Scheme Procedure: regexp-match? obj
     Return ‘#t’ if OBJ is a match structure returned by a previous call
     to ‘regexp-exec’, or ‘#f’ otherwise.

 -- Scheme Procedure: match:substring match [n]
     Return the portion of TARGET matched by subexpression number N.
     Submatch 0 (the default) represents the entire regexp match.  If
     the regular expression as a whole matched, but the subexpression
     number N did not match, return ‘#f’.

     (define s (string-match "[0-9][0-9][0-9][0-9]" "blah2002foo"))
     (match:substring s)
     ⇒ "2002"

     ;; match starting at offset 6 in the string
     (match:substring
       (string-match "[0-9][0-9][0-9][0-9]" "blah987654" 6))
     ⇒ "7654"

 -- Scheme Procedure: match:start match [n]
     Return the starting position of submatch number N.

   In the following example, the result is 4, since the match starts at
character index 4:

     (define s (string-match "[0-9][0-9][0-9][0-9]" "blah2002foo"))
     (match:start s)
     ⇒ 4

 -- Scheme Procedure: match:end match [n]
     Return the ending position of submatch number N.

   In the following example, the result is 8, since the match runs
between characters 4 and 8 (i.e. the “2002”).

     (define s (string-match "[0-9][0-9][0-9][0-9]" "blah2002foo"))
     (match:end s)
     ⇒ 8

 -- Scheme Procedure: match:prefix match
     Return the unmatched portion of TARGET preceding the regexp match.

          (define s (string-match "[0-9][0-9][0-9][0-9]" "blah2002foo"))
          (match:prefix s)
          ⇒ "blah"

 -- Scheme Procedure: match:suffix match
     Return the unmatched portion of TARGET following the regexp match.

     (define s (string-match "[0-9][0-9][0-9][0-9]" "blah2002foo"))
     (match:suffix s)
     ⇒ "foo"

 -- Scheme Procedure: match:count match
     Return the number of parenthesized subexpressions from MATCH.  Note
     that the entire regular expression match itself counts as a
     subexpression, and failed submatches are included in the count.

 -- Scheme Procedure: match:string match
     Return the original TARGET string.

     (define s (string-match "[0-9][0-9][0-9][0-9]" "blah2002foo"))
     (match:string s)
     ⇒ "blah2002foo"


File: guile.info,  Node: Backslash Escapes,  Prev: Match Structures,  Up: Regular Expressions

6.13.3 Backslash Escapes
------------------------

Sometimes you will want a regexp to match characters like ‘*’ or ‘$’
exactly.  For example, to check whether a particular string represents a
menu entry from an Info node, it would be useful to match it against a
regexp like ‘^* [^:]*::’.  However, this won’t work; because the
asterisk is a metacharacter, it won’t match the ‘*’ at the beginning of
the string.  In this case, we want to make the first asterisk un-magic.

   You can do this by preceding the metacharacter with a backslash
character ‘\’.  (This is also called “quoting” the metacharacter, and is
known as a “backslash escape”.)  When Guile sees a backslash in a
regular expression, it considers the following glyph to be an ordinary
character, no matter what special meaning it would ordinarily have.
Therefore, we can make the above example work by changing the regexp to
‘^\* [^:]*::’.  The ‘\*’ sequence tells the regular expression engine to
match only a single asterisk in the target string.

   Since the backslash is itself a metacharacter, you may force a regexp
to match a backslash in the target string by preceding the backslash
with itself.  For example, to find variable references in a TeX program,
you might want to find occurrences of the string ‘\let\’ followed by any
number of alphabetic characters.  The regular expression
‘\\let\\[A-Za-z]*’ would do this: the double backslashes in the regexp
each match a single backslash in the target string.

 -- Scheme Procedure: regexp-quote str
     Quote each special character found in STR with a backslash, and
     return the resulting string.

   *Very important:* Using backslash escapes in Guile source code (as in
Emacs Lisp or C) can be tricky, because the backslash character has
special meaning for the Guile reader.  For example, if Guile encounters
the character sequence ‘\n’ in the middle of a string while processing
Scheme code, it replaces those characters with a newline character.
Similarly, the character sequence ‘\t’ is replaced by a horizontal tab.
Several of these “escape sequences” are processed by the Guile reader
before your code is executed.  Unrecognized escape sequences are
ignored: if the characters ‘\*’ appear in a string, they will be
translated to the single character ‘*’.

   This translation is obviously undesirable for regular expressions,
since we want to be able to include backslashes in a string in order to
escape regexp metacharacters.  Therefore, to make sure that a backslash
is preserved in a string in your Guile program, you must use _two_
consecutive backslashes:

     (define Info-menu-entry-pattern (make-regexp "^\\* [^:]*"))

   The string in this example is preprocessed by the Guile reader before
any code is executed.  The resulting argument to ‘make-regexp’ is the
string ‘^\* [^:]*’, which is what we really want.

   This also means that in order to write a regular expression that
matches a single backslash character, the regular expression string in
the source code must include _four_ backslashes.  Each consecutive pair
of backslashes gets translated by the Guile reader to a single
backslash, and the resulting double-backslash is interpreted by the
regexp engine as matching a single backslash character.  Hence:

     (define tex-variable-pattern (make-regexp "\\\\let\\\\=[A-Za-z]*"))

   The reason for the unwieldiness of this syntax is historical.  Both
regular expression pattern matchers and Unix string processing systems
have traditionally used backslashes with the special meanings described
above.  The POSIX regular expression specification and ANSI C standard
both require these semantics.  Attempting to abandon either convention
would cause other kinds of compatibility problems, possibly more severe
ones.  Therefore, without extending the Scheme reader to support strings
with different quoting conventions (an ungainly and confusing extension
when implemented in other languages), we must adhere to this cumbersome
escape syntax.


File: guile.info,  Node: LALR(1) Parsing,  Next: PEG Parsing,  Prev: Regular Expressions,  Up: API Reference

6.14 LALR(1) Parsing
====================

The ‘(system base lalr)’ module provides the ‘lalr-scm’ LALR(1) parser
generator by Dominique Boucher (https://github.com/schemeway/lalr-scm/).
‘lalr-scm’ uses the same algorithm as GNU Bison (*note Introduction to
Bison: (bison)Introduction.).  Parsers are defined using the
‘lalr-parser’ macro.

 -- Scheme Syntax: lalr-parser [OPTIONS] TOKENS RULES...
     Generate an LALR(1) syntax analyzer.  TOKENS is a list of symbols
     representing the terminal symbols of the grammar.  RULES are the
     grammar production rules.

     Each rule has the form ‘(NON-TERMINAL (RHS ...) : ACTION ...)’,
     where NON-TERMINAL is the name of the rule, RHS are the right-hand
     sides, i.e., the production rule, and ACTION is a semantic action
     associated with the rule.

     The generated parser is a two-argument procedure that takes a
     “tokenizer” and a “syntax error procedure”.  The tokenizer should
     be a thunk that returns lexical tokens as produced by
     ‘make-lexical-token’.  The syntax error procedure may be called
     with at least an error message (a string), and optionally the
     lexical token that caused the error.

   Please refer to the ‘lalr-scm’ documentation for details.


File: guile.info,  Node: PEG Parsing,  Next: Read/Load/Eval/Compile,  Prev: LALR(1) Parsing,  Up: API Reference

6.15 PEG Parsing
================

Parsing Expression Grammars (PEGs) are a way of specifying formal
languages for text processing.  They can be used either for matching
(like regular expressions) or for building recursive descent parsers
(like lex/yacc).  Guile uses a superset of PEG syntax that allows more
control over what information is preserved during parsing.

   Wikipedia has a clear and concise introduction to PEGs if you want to
familiarize yourself with the syntax:
<http://en.wikipedia.org/wiki/Parsing_expression_grammar>.

   The ‘(ice-9 peg)’ module works by compiling PEGs down to lambda
expressions.  These can either be stored in variables at compile-time by
the define macros (‘define-peg-pattern’ and
‘define-peg-string-patterns’) or calculated explicitly at runtime with
the compile functions (‘compile-peg-pattern’ and ‘peg-string-compile’).

   They can then be used for either parsing (‘match-pattern’) or
searching (‘search-for-pattern’).  For convenience, ‘search-for-pattern’
also takes pattern literals in case you want to inline a simple search
(people often use regular expressions this way).

   The rest of this documentation consists of a syntax reference, an API
reference, and a tutorial.

* Menu:

* PEG Syntax Reference::
* PEG API Reference::
* PEG Tutorial::
* PEG Internals::


File: guile.info,  Node: PEG Syntax Reference,  Next: PEG API Reference,  Up: PEG Parsing

6.15.1 PEG Syntax Reference
---------------------------

Normal PEG Syntax:
..................

 -- PEG Pattern: sequence a b
     Parses A.  If this succeeds, continues to parse B from the end of
     the text parsed as A.  Succeeds if both A and B succeed.

     ‘"a b"’

     ‘(and a b)’

 -- PEG Pattern: ordered choice a b
     Parses A.  If this fails, backtracks and parses B.  Succeeds if
     either A or B succeeds.

     ‘"a/b"’

     ‘(or a b)’

 -- PEG Pattern: zero or more a
     Parses A as many times in a row as it can, starting each A at the
     end of the text parsed by the previous A.  Always succeeds.

     ‘"a*"’

     ‘(* a)’

 -- PEG Pattern: one or more a
     Parses A as many times in a row as it can, starting each A at the
     end of the text parsed by the previous A.  Succeeds if at least one
     A was parsed.

     ‘"a+"’

     ‘(+ a)’

 -- PEG Pattern: optional a
     Tries to parse A.  Succeeds if A succeeds.

     ‘"a?"’

     ‘(? a)’

 -- PEG Pattern: followed by a
     Makes sure it is possible to parse A, but does not actually parse
     it.  Succeeds if A would succeed.

     ‘"&a"’

     ‘(followed-by a)’

 -- PEG Pattern: not followed by a
     Makes sure it is impossible to parse A, but does not actually parse
     it.  Succeeds if A would fail.

     ‘"!a"’

     ‘(not-followed-by a)’

 -- PEG Pattern: string literal ``abc''
     Parses the string "ABC".  Succeeds if that parsing succeeds.

     ‘"'abc'"’

     ‘"abc"’

 -- PEG Pattern: any character
     Parses any single character.  Succeeds unless there is no more text
     to be parsed.

     ‘"."’

     ‘peg-any’

 -- PEG Pattern: character class a b
     Alternative syntax for “Ordered Choice A B” if A and B are
     characters.

     ‘"[ab]"’

     ‘(or "a" "b")’

 -- PEG Pattern: range of characters a z
     Parses any character falling between A and Z.

     ‘"[a-z]"’

     ‘(range #\a #\z)’

   Example:

     "(a !b / c &d*) 'e'+"

   Would be:

     (and
      (or
       (and a (not-followed-by b))
       (and c (followed-by (* d))))
      (+ "e"))

Extended Syntax
...............

There is some extra syntax for S-expressions.

 -- PEG Pattern: ignore a
     Ignore the text matching A

 -- PEG Pattern: capture a
     Capture the text matching A.

 -- PEG Pattern: peg a
     Embed the PEG pattern A using string syntax.

   Example:

     "!a / 'b'"

   Is equivalent to

     (or (peg "!a") "b")

   and

     (or (not-followed-by a) "b")


File: guile.info,  Node: PEG API Reference,  Next: PEG Tutorial,  Prev: PEG Syntax Reference,  Up: PEG Parsing

6.15.2 PEG API Reference
------------------------

Define Macros
.............

The most straightforward way to define a PEG is by using one of the
define macros (both of these macroexpand into ‘define’ expressions).
These macros bind parsing functions to variables.  These parsing
functions may be invoked by ‘match-pattern’ or ‘search-for-pattern’,
which return a PEG match record.  Raw data can be retrieved from this
record with the PEG match deconstructor functions.  More complicated
(and perhaps enlightening) examples can be found in the tutorial.

 -- Scheme Macro: define-peg-string-patterns peg-string
     Defines all the nonterminals in the PEG PEG-STRING.  More
     precisely, ‘define-peg-string-patterns’ takes a superset of PEGs.
     A normal PEG has a ‘<-’ between the nonterminal and the pattern.
     ‘define-peg-string-patterns’ uses this symbol to determine what
     information it should propagate up the parse tree.  The normal ‘<-’
     propagates the matched text up the parse tree, ‘<--’ propagates the
     matched text up the parse tree tagged with the name of the
     nonterminal, and ‘<’ discards that matched text and propagates
     nothing up the parse tree.  Also, nonterminals may consist of any
     alphanumeric character or a “-” character (in normal PEGs
     nonterminals can only be alphabetic).

     For example, if we:
          (define-peg-string-patterns
            "as <- 'a'+
          bs <- 'b'+
          as-or-bs <- as/bs")
          (define-peg-string-patterns
            "as-tag <-- 'a'+
          bs-tag <-- 'b'+
          as-or-bs-tag <-- as-tag/bs-tag")
     Then:
          (match-pattern as-or-bs "aabbcc") ⇒
          #<peg start: 0 end: 2 string: aabbcc tree: aa>
          (match-pattern as-or-bs-tag "aabbcc") ⇒
          #<peg start: 0 end: 2 string: aabbcc tree: (as-or-bs-tag (as-tag aa))>

     Note that in doing this, we have bound 6 variables at the toplevel
     (AS, BS, AS-OR-BS, AS-TAG, BS-TAG, and AS-OR-BS-TAG).

 -- Scheme Macro: define-peg-pattern name capture-type peg-sexp
     Defines a single nonterminal NAME.  CAPTURE-TYPE determines how
     much information is passed up the parse tree.  PEG-SEXP is a PEG in
     S-expression form.

     Possible values for capture-type:

     ‘all’
          passes the matched text up the parse tree tagged with the name
          of the nonterminal.
     ‘body’
          passes the matched text up the parse tree.
     ‘none’
          passes nothing up the parse tree.

     For Example, if we:
          (define-peg-pattern as body (+ "a"))
          (define-peg-pattern bs body (+ "b"))
          (define-peg-pattern as-or-bs body (or as bs))
          (define-peg-pattern as-tag all (+ "a"))
          (define-peg-pattern bs-tag all (+ "b"))
          (define-peg-pattern as-or-bs-tag all (or as-tag bs-tag))
     Then:
          (match-pattern as-or-bs "aabbcc") ⇒
          #<peg start: 0 end: 2 string: aabbcc tree: aa>
          (match-pattern as-or-bs-tag "aabbcc") ⇒
          #<peg start: 0 end: 2 string: aabbcc tree: (as-or-bs-tag (as-tag aa))>

     Note that in doing this, we have bound 6 variables at the toplevel
     (AS, BS, AS-OR-BS, AS-TAG, BS-TAG, and AS-OR-BS-TAG).

Compile Functions
.................

It is sometimes useful to be able to compile anonymous PEG patterns at
runtime.  These functions let you do that using either syntax.

 -- Scheme Procedure: peg-string-compile peg-string capture-type
     Compiles the PEG pattern in PEG-STRING propagating according to
     CAPTURE-TYPE (capture-type can be any of the values from
     ‘define-peg-pattern’).

 -- Scheme Procedure: compile-peg-pattern peg-sexp capture-type
     Compiles the PEG pattern in PEG-SEXP propagating according to
     CAPTURE-TYPE (capture-type can be any of the values from
     ‘define-peg-pattern’).

   The functions return syntax objects, which can be useful if you want
to use them in macros.  If all you want is to define a new nonterminal,
you can do the following:

     (define exp '(+ "a"))
     (define as (compile (compile-peg-pattern exp 'body)))

   You can use this nonterminal with all of the regular PEG functions:

     (match-pattern as "aaaaa") ⇒
     #<peg start: 0 end: 5 string: bbbbb tree: bbbbb>

Parsing & Matching Functions
............................

For our purposes, “parsing” means parsing a string into a tree starting
from the first character, while “matching” means searching through the
string for a substring.  In practice, the only difference between the
two functions is that ‘match-pattern’ gives up if it can’t find a valid
substring starting at index 0 and ‘search-for-pattern’ keeps looking.
They are both equally capable of “parsing” and “matching” given those
constraints.

 -- Scheme Procedure: match-pattern nonterm string
     Parses STRING using the PEG stored in NONTERM.  If no match was
     found, ‘match-pattern’ returns false.  If a match was found, a PEG
     match record is returned.

     The ‘capture-type’ argument to ‘define-peg-pattern’ allows you to
     choose what information to hold on to while parsing.  The options
     are:

     ‘all’
          tag the matched text with the nonterminal
     ‘body’
          just the matched text
     ‘none’
          nothing

          (define-peg-pattern as all (+ "a"))
          (match-pattern as "aabbcc") ⇒
          #<peg start: 0 end: 2 string: aabbcc tree: (as aa)>

          (define-peg-pattern as body (+ "a"))
          (match-pattern as "aabbcc") ⇒
          #<peg start: 0 end: 2 string: aabbcc tree: aa>

          (define-peg-pattern as none (+ "a"))
          (match-pattern as "aabbcc") ⇒
          #<peg start: 0 end: 2 string: aabbcc tree: ()>

          (define-peg-pattern bs body (+ "b"))
          (match-pattern bs "aabbcc") ⇒
          #f

 -- Scheme Macro: search-for-pattern nonterm-or-peg string
     Searches through STRING looking for a matching subexpression.
     NONTERM-OR-PEG can either be a nonterminal or a literal PEG
     pattern.  When a literal PEG pattern is provided,
     ‘search-for-pattern’ works very similarly to the regular expression
     searches many hackers are used to.  If no match was found,
     ‘search-for-pattern’ returns false.  If a match was found, a PEG
     match record is returned.

          (define-peg-pattern as body (+ "a"))
          (search-for-pattern as "aabbcc") ⇒
          #<peg start: 0 end: 2 string: aabbcc tree: aa>
          (search-for-pattern (+ "a") "aabbcc") ⇒
          #<peg start: 0 end: 2 string: aabbcc tree: aa>
          (search-for-pattern "'a'+" "aabbcc") ⇒
          #<peg start: 0 end: 2 string: aabbcc tree: aa>

          (define-peg-pattern as all (+ "a"))
          (search-for-pattern as "aabbcc") ⇒
          #<peg start: 0 end: 2 string: aabbcc tree: (as aa)>

          (define-peg-pattern bs body (+ "b"))
          (search-for-pattern bs "aabbcc") ⇒
          #<peg start: 2 end: 4 string: aabbcc tree: bb>
          (search-for-pattern (+ "b") "aabbcc") ⇒
          #<peg start: 2 end: 4 string: aabbcc tree: bb>
          (search-for-pattern "'b'+" "aabbcc") ⇒
          #<peg start: 2 end: 4 string: aabbcc tree: bb>

          (define-peg-pattern zs body (+ "z"))
          (search-for-pattern zs "aabbcc") ⇒
          #f
          (search-for-pattern (+ "z") "aabbcc") ⇒
          #f
          (search-for-pattern "'z'+" "aabbcc") ⇒
          #f

PEG Match Records
.................

The ‘match-pattern’ and ‘search-for-pattern’ functions both return PEG
match records.  Actual information can be extracted from these with the
following functions.

 -- Scheme Procedure: peg:string match-record
     Returns the original string that was parsed in the creation of
     ‘match-record’.

 -- Scheme Procedure: peg:start match-record
     Returns the index of the first parsed character in the original
     string (from ‘peg:string’).  If this is the same as ‘peg:end’,
     nothing was parsed.

 -- Scheme Procedure: peg:end match-record
     Returns one more than the index of the last parsed character in the
     original string (from ‘peg:string’).  If this is the same as
     ‘peg:start’, nothing was parsed.

 -- Scheme Procedure: peg:substring match-record
     Returns the substring parsed by ‘match-record’.  This is equivalent
     to ‘(substring (peg:string match-record) (peg:start match-record)
     (peg:end match-record))’.

 -- Scheme Procedure: peg:tree match-record
     Returns the tree parsed by ‘match-record’.

 -- Scheme Procedure: peg-record? match-record
     Returns true if ‘match-record’ is a PEG match record, or false
     otherwise.

   Example:
     (define-peg-pattern bs all (peg "'b'+"))

     (search-for-pattern bs "aabbcc") ⇒
     #<peg start: 2 end: 4 string: aabbcc tree: (bs bb)>

     (let ((pm (search-for-pattern bs "aabbcc")))
        `((string ,(peg:string pm))
          (start ,(peg:start pm))
          (end ,(peg:end pm))
          (substring ,(peg:substring pm))
          (tree ,(peg:tree pm))
          (record? ,(peg-record? pm)))) ⇒
     ((string "aabbcc")
      (start 2)
      (end 4)
      (substring "bb")
      (tree (bs "bb"))
      (record? #t))

Miscellaneous
.............

 -- Scheme Procedure: context-flatten tst lst
     Takes a predicate TST and a list LST.  Flattens LST until all
     elements are either atoms or satisfy TST.  If LST itself satisfies
     TST, ‘(list lst)’ is returned (this is a flat list whose only
     element satisfies TST).

          (context-flatten (lambda (x) (and (number? (car x)) (= (car x) 1))) '(2 2 (1 1 (2 2)) (2 2 (1 1)))) ⇒
          (2 2 (1 1 (2 2)) 2 2 (1 1))
          (context-flatten (lambda (x) (and (number? (car x)) (= (car x) 1))) '(1 1 (1 1 (2 2)) (2 2 (1 1)))) ⇒
          ((1 1 (1 1 (2 2)) (2 2 (1 1))))

     If you’re wondering why this is here, take a look at the tutorial.

 -- Scheme Procedure: keyword-flatten terms lst
     A less general form of ‘context-flatten’.  Takes a list of terminal
     atoms ‘terms’ and flattens LST until all elements are either atoms,
     or lists which have an atom from ‘terms’ as their first element.
          (keyword-flatten '(a b) '(c a b (a c) (b c) (c (b a) (c a)))) ⇒
          (c a b (a c) (b c) c (b a) c a)

     If you’re wondering why this is here, take a look at the tutorial.


File: guile.info,  Node: PEG Tutorial,  Next: PEG Internals,  Prev: PEG API Reference,  Up: PEG Parsing

6.15.3 PEG Tutorial
-------------------

Parsing /etc/passwd
...................

This example will show how to parse /etc/passwd using PEGs.

   First we define an example /etc/passwd file:

     (define *etc-passwd*
       "root:x:0:0:root:/root:/bin/bash
     daemon:x:1:1:daemon:/usr/sbin:/bin/sh
     bin:x:2:2:bin:/bin:/bin/sh
     sys:x:3:3:sys:/dev:/bin/sh
     nobody:x:65534:65534:nobody:/nonexistent:/bin/sh
     messagebus:x:103:107::/var/run/dbus:/bin/false
     ")

   As a first pass at this, we might want to have all the entries in
/etc/passwd in a list.

   Doing this with string-based PEG syntax would look like this:
     (define-peg-string-patterns
       "passwd <- entry* !.
     entry <-- (! NL .)* NL*
     NL < '\n'")

   A ‘passwd’ file is 0 or more entries (‘entry*’) until the end of the
file (‘!.’ (‘.’ is any character, so ‘!.’ means “not anything”)).  We
want to capture the data in the nonterminal ‘passwd’, but not tag it
with the name, so we use ‘<-’.

   An entry is a series of 0 or more characters that aren’t newlines
(‘(! NL .)*’) followed by 0 or more newlines (‘NL*’).  We want to tag
all the entries with ‘entry’, so we use ‘<--’.

   A newline is just a literal newline (‘'\n'’).  We don’t want a bunch
of newlines cluttering up the output, so we use ‘<’ to throw away the
captured data.

   Here is the same PEG defined using S-expressions:
     (define-peg-pattern passwd body (and (* entry) (not-followed-by peg-any)))
     (define-peg-pattern entry all (and (* (and (not-followed-by NL) peg-any))
     			       (* NL)))
     (define-peg-pattern NL none "\n")

   Obviously this is much more verbose.  On the other hand, it’s more
explicit, and thus easier to build automatically.  However, there are
some tricks that make S-expressions easier to use in some cases.  One is
the ‘ignore’ keyword; the string syntax has no way to say “throw away
this text” except breaking it out into a separate nonterminal.  For
instance, to throw away the newlines we had to define ‘NL’.  In the
S-expression syntax, we could have simply written ‘(ignore "\n")’.
Also, for the cases where string syntax is really much cleaner, the
‘peg’ keyword can be used to embed string syntax in S-expression syntax.
For instance, we could have written:

     (define-peg-pattern passwd body (peg "entry* !."))

   However we define it, parsing ‘*etc-passwd*’ with the ‘passwd’
nonterminal yields the same results:

     (peg:tree (match-pattern passwd *etc-passwd*)) ⇒
     ((entry "root:x:0:0:root:/root:/bin/bash")
      (entry "daemon:x:1:1:daemon:/usr/sbin:/bin/sh")
      (entry "bin:x:2:2:bin:/bin:/bin/sh")
      (entry "sys:x:3:3:sys:/dev:/bin/sh")
      (entry "nobody:x:65534:65534:nobody:/nonexistent:/bin/sh")
      (entry "messagebus:x:103:107::/var/run/dbus:/bin/false"))

   However, here is something to be wary of:

     (peg:tree (match-pattern passwd "one entry")) ⇒
     (entry "one entry")

   By default, the parse trees generated by PEGs are compressed as much
as possible without losing information.  It may not look like this is
what you want at first, but uncompressed parse trees are an enormous
headache (there’s no easy way to predict how deep particular lists will
nest, there are empty lists littered everywhere, etc.  etc.).  One
side-effect of this, however, is that sometimes the compressor is too
aggressive.  No information is discarded when ‘((entry "one entry"))’ is
compressed to ‘(entry "one entry")’, but in this particular case it
probably isn’t what we want.

   There are two functions for easily dealing with this:
‘keyword-flatten’ and ‘context-flatten’.  The ‘keyword-flatten’ function
takes a list of keywords and a list to flatten, then tries to coerce the
list such that the first element of all sublists is one of the keywords.
The ‘context-flatten’ function is similar, but instead of a list of
keywords it takes a predicate that should indicate whether a given
sublist is good enough (refer to the API reference for more details).

   What we want here is ‘keyword-flatten’.
     (keyword-flatten '(entry) (peg:tree (match-pattern passwd *etc-passwd*))) ⇒
     ((entry "root:x:0:0:root:/root:/bin/bash")
      (entry "daemon:x:1:1:daemon:/usr/sbin:/bin/sh")
      (entry "bin:x:2:2:bin:/bin:/bin/sh")
      (entry "sys:x:3:3:sys:/dev:/bin/sh")
      (entry "nobody:x:65534:65534:nobody:/nonexistent:/bin/sh")
      (entry "messagebus:x:103:107::/var/run/dbus:/bin/false"))
     (keyword-flatten '(entry) (peg:tree (match-pattern passwd "one entry"))) ⇒
     ((entry "one entry"))

   Of course, this is a somewhat contrived example.  In practice we
would probably just tag the ‘passwd’ nonterminal to remove the ambiguity
(using either the ‘all’ keyword for S-expressions or the ‘<--’ symbol
for strings)..

     (define-peg-pattern tag-passwd all (peg "entry* !."))
     (peg:tree (match-pattern tag-passwd *etc-passwd*)) ⇒
     (tag-passwd
       (entry "root:x:0:0:root:/root:/bin/bash")
       (entry "daemon:x:1:1:daemon:/usr/sbin:/bin/sh")
       (entry "bin:x:2:2:bin:/bin:/bin/sh")
       (entry "sys:x:3:3:sys:/dev:/bin/sh")
       (entry "nobody:x:65534:65534:nobody:/nonexistent:/bin/sh")
       (entry "messagebus:x:103:107::/var/run/dbus:/bin/false"))
     (peg:tree (match-pattern tag-passwd "one entry"))
     (tag-passwd
       (entry "one entry"))

   If you’re ever uncertain about the potential results of parsing
something, remember the two absolute rules:
  1. No parsing information will ever be discarded.
  2. There will never be any lists with fewer than 2 elements.

   For the purposes of (1), "parsing information" means things tagged
with the ‘any’ keyword or the ‘<--’ symbol.  Plain strings will be
concatenated.

   Let’s extend this example a bit more and actually pull some useful
information out of the passwd file:

     (define-peg-string-patterns
       "passwd <-- entry* !.
     entry <-- login C pass C uid C gid C nameORcomment C homedir C shell NL*
     login <-- text
     pass <-- text
     uid <-- [0-9]*
     gid <-- [0-9]*
     nameORcomment <-- text
     homedir <-- path
     shell <-- path
     path <-- (SLASH pathELEMENT)*
     pathELEMENT <-- (!NL !C  !'/' .)*
     text <- (!NL !C  .)*
     C < ':'
     NL < '\n'
     SLASH < '/'")

   This produces rather pretty parse trees:
     (passwd
       (entry (login "root")
              (pass "x")
              (uid "0")
              (gid "0")
              (nameORcomment "root")
              (homedir (path (pathELEMENT "root")))
              (shell (path (pathELEMENT "bin") (pathELEMENT "bash"))))
       (entry (login "daemon")
              (pass "x")
              (uid "1")
              (gid "1")
              (nameORcomment "daemon")
              (homedir
                (path (pathELEMENT "usr") (pathELEMENT "sbin")))
              (shell (path (pathELEMENT "bin") (pathELEMENT "sh"))))
       (entry (login "bin")
              (pass "x")
              (uid "2")
              (gid "2")
              (nameORcomment "bin")
              (homedir (path (pathELEMENT "bin")))
              (shell (path (pathELEMENT "bin") (pathELEMENT "sh"))))
       (entry (login "sys")
              (pass "x")
              (uid "3")
              (gid "3")
              (nameORcomment "sys")
              (homedir (path (pathELEMENT "dev")))
              (shell (path (pathELEMENT "bin") (pathELEMENT "sh"))))
       (entry (login "nobody")
              (pass "x")
              (uid "65534")
              (gid "65534")
              (nameORcomment "nobody")
              (homedir (path (pathELEMENT "nonexistent")))
              (shell (path (pathELEMENT "bin") (pathELEMENT "sh"))))
       (entry (login "messagebus")
              (pass "x")
              (uid "103")
              (gid "107")
              nameORcomment
              (homedir
                (path (pathELEMENT "var")
                      (pathELEMENT "run")
                      (pathELEMENT "dbus")))
              (shell (path (pathELEMENT "bin") (pathELEMENT "false")))))

   Notice that when there’s no entry in a field (e.g.  ‘nameORcomment’
for messagebus) the symbol is inserted.  This is the “don’t throw away
any information” rule—we succesfully matched a ‘nameORcomment’ of 0
characters (since we used ‘*’ when defining it).  This is usually what
you want, because it allows you to e.g.  use ‘list-ref’ to pull out
elements (since they all have known offsets).

   If you’d prefer not to have symbols for empty matches, you can
replace the ‘*’ with a ‘+’ and add a ‘?’ after the ‘nameORcomment’ in
‘entry’.  Then it will try to parse 1 or more characters, fail
(inserting nothing into the parse tree), but continue because it didn’t
have to match the nameORcomment to continue.

Embedding Arithmetic Expressions
................................

We can parse simple mathematical expressions with the following PEG:

     (define-peg-string-patterns
       "expr <- sum
     sum <-- (product ('+' / '-') sum) / product
     product <-- (value ('*' / '/') product) / value
     value <-- number / '(' expr ')'
     number <-- [0-9]+")

   Then:
     (peg:tree (match-pattern expr "1+1/2*3+(1+1)/2")) ⇒
     (sum (product (value (number "1")))
          "+"
          (sum (product
                 (value (number "1"))
                 "/"
                 (product
                   (value (number "2"))
                   "*"
                   (product (value (number "3")))))
               "+"
               (sum (product
                      (value "("
                             (sum (product (value (number "1")))
                                  "+"
                                  (sum (product (value (number "1")))))
                             ")")
                      "/"
                      (product (value (number "2")))))))

   There is very little wasted effort in this PEG. The ‘number’
nonterminal has to be tagged because otherwise the numbers might run
together with the arithmetic expressions during the string concatenation
stage of parse-tree compression (the parser will see “1” followed by “/”
and decide to call it “1/”).  When in doubt, tag.

   It is very easy to turn these parse trees into lisp expressions:

     (define (parse-sum sum left . rest)
       (if (null? rest)
           (apply parse-product left)
           (list (string->symbol (car rest))
     	    (apply parse-product left)
     	    (apply parse-sum (cadr rest)))))

     (define (parse-product product left . rest)
       (if (null? rest)
           (apply parse-value left)
           (list (string->symbol (car rest))
     	    (apply parse-value left)
     	    (apply parse-product (cadr rest)))))

     (define (parse-value value first . rest)
       (if (null? rest)
           (string->number (cadr first))
           (apply parse-sum (car rest))))

     (define parse-expr parse-sum)

   (Notice all these functions look very similar; for a more complicated
PEG, it would be worth abstracting.)

   Then:
     (apply parse-expr (peg:tree (match-pattern expr "1+1/2*3+(1+1)/2"))) ⇒
     (+ 1 (+ (/ 1 (* 2 3)) (/ (+ 1 1) 2)))

   But wait!  The associativity is wrong!  Where it says ‘(/ 1 (* 2
3))’, it should say ‘(* (/ 1 2) 3)’.

   It’s tempting to try replacing e.g.  ‘"sum <-- (product ('+' / '-')
sum) / product"’ with ‘"sum <-- (sum ('+' / '-') product) / product"’,
but this is a Bad Idea.  PEGs don’t support left recursion.  To see why,
imagine what the parser will do here.  When it tries to parse ‘sum’, it
first has to try and parse ‘sum’.  But to do that, it first has to try
and parse ‘sum’.  This will continue until the stack gets blown off.

   So how does one parse left-associative binary operators with PEGs?
Honestly, this is one of their major shortcomings.  There’s no
general-purpose way of doing this, but here the repetition operators are
a good choice:

     (use-modules (srfi srfi-1))

     (define-peg-string-patterns
       "expr <- sum
     sum <-- (product ('+' / '-'))* product
     product <-- (value ('*' / '/'))* value
     value <-- number / '(' expr ')'
     number <-- [0-9]+")

     ;; take a deep breath...
     (define (make-left-parser next-func)
       (lambda (sum first . rest) ;; general form, comments below assume
         ;; that we're dealing with a sum expression
         (if (null? rest) ;; form (sum (product ...))
           (apply next-func first)
           (if (string? (cadr first));; form (sum ((product ...) "+") (product ...))
     	  (list (string->symbol (cadr first))
     		(apply next-func (car first))
     		(apply next-func (car rest)))
               ;; form (sum (((product ...) "+") ((product ...) "+")) (product ...))
     	  (car
     	   (reduce ;; walk through the list and build a left-associative tree
     	    (lambda (l r)
     	      (list (list (cadr r) (car r) (apply next-func (car l)))
     		    (string->symbol (cadr l))))
     	    'ignore
     	    (append ;; make a list of all the products
                  ;; the first one should be pre-parsed
     	     (list (list (apply next-func (caar first))
     			 (string->symbol (cadar first))))
     	     (cdr first)
                  ;; the last one has to be added in
     	     (list (append rest '("done"))))))))))

     (define (parse-value value first . rest)
       (if (null? rest)
           (string->number (cadr first))
           (apply parse-sum (car rest))))
     (define parse-product (make-left-parser parse-value))
     (define parse-sum (make-left-parser parse-product))
     (define parse-expr parse-sum)

   Then:
     (apply parse-expr (peg:tree (match-pattern expr "1+1/2*3+(1+1)/2"))) ⇒
     (+ (+ 1 (* (/ 1 2) 3)) (/ (+ 1 1) 2))

   As you can see, this is much uglier (it could be made prettier by
using ‘context-flatten’, but the way it’s written above makes it clear
how we deal with the three ways the zero-or-more ‘*’ expression can
parse).  Fortunately, most of the time we can get away with only using
right-associativity.

Simplified Functions
....................

For a more tantalizing example, consider the following grammar that
parses (highly) simplified C functions:

     (define-peg-string-patterns
       "cfunc <-- cSP ctype cSP cname cSP cargs cLB cSP cbody cRB
     ctype <-- cidentifier
     cname <-- cidentifier
     cargs <-- cLP (! (cSP cRP) carg cSP (cCOMMA / cRP) cSP)* cSP
     carg <-- cSP ctype cSP cname
     cbody <-- cstatement *
     cidentifier <- [a-zA-z][a-zA-Z0-9_]*
     cstatement <-- (!';'.)*cSC cSP
     cSC < ';'
     cCOMMA < ','
     cLP < '('
     cRP < ')'
     cLB < '{'
     cRB < '}'
     cSP < [ \t\n]*")

   Then:
     (match-pattern cfunc "int square(int a) { return a*a;}") ⇒
     (32
      (cfunc (ctype "int")
             (cname "square")
             (cargs (carg (ctype "int") (cname "a")))
             (cbody (cstatement "return a*a"))))

   And:
     (match-pattern cfunc "int mod(int a, int b) { int c = a/b;return a-b*c; }") ⇒
     (52
      (cfunc (ctype "int")
             (cname "mod")
             (cargs (carg (ctype "int") (cname "a"))
                    (carg (ctype "int") (cname "b")))
             (cbody (cstatement "int c = a/b")
                    (cstatement "return a- b*c"))))

   By wrapping all the ‘carg’ nonterminals in a ‘cargs’ nonterminal, we
were able to remove any ambiguity in the parsing structure and avoid
having to call ‘context-flatten’ on the output of ‘match-pattern’.  We
used the same trick with the ‘cstatement’ nonterminals, wrapping them in
a ‘cbody’ nonterminal.

   The whitespace nonterminal ‘cSP’ used here is a (very) useful
instantiation of a common pattern for matching syntactically irrelevant
information.  Since it’s tagged with ‘<’ and ends with ‘*’ it won’t
clutter up the parse trees (all the empty lists will be discarded during
the compression step) and it will never cause parsing to fail.


File: guile.info,  Node: PEG Internals,  Prev: PEG Tutorial,  Up: PEG Parsing

6.15.4 PEG Internals
--------------------

A PEG parser takes a string as input and attempts to parse it as a given
nonterminal.  The key idea of the PEG implementation is that every
nonterminal is just a function that takes a string as an argument and
attempts to parse that string as its nonterminal.  The functions always
start from the beginning, but a parse is considered successful if there
is material left over at the end.

   This makes it easy to model different PEG parsing operations.  For
instance, consider the PEG grammar ‘"ab"’, which could also be written
‘(and "a" "b")’.  It matches the string “ab”.  Here’s how that might be
implemented in the PEG style:

     (define (match-and-a-b str)
       (match-a str)
       (match-b str))

   As you can see, the use of functions provides an easy way to model
sequencing.  In a similar way, one could model ‘(or a b)’ with something
like the following:

     (define (match-or-a-b str)
       (or (match-a str) (match-b str)))

   Here the semantics of a PEG ‘or’ expression map naturally onto
Scheme’s ‘or’ operator.  This function will attempt to run ‘(match-a
str)’, and return its result if it succeeds.  Otherwise it will run
‘(match-b str)’.

   Of course, the code above wouldn’t quite work.  We need some way for
the parsing functions to communicate.  The actual interface used is
below.

Parsing Function Interface
..........................

A parsing function takes three arguments - a string, the length of that
string, and the position in that string it should start parsing at.  In
effect, the parsing functions pass around substrings in pieces - the
first argument is a buffer of characters, and the second two give a
range within that buffer that the parsing function should look at.

   Parsing functions return either #f, if they failed to match their
nonterminal, or a list whose first element must be an integer
representing the final position in the string they matched and whose cdr
can be any other data the function wishes to return, or ’() if it
doesn’t have any more data.

   The one caveat is that if the extra data it returns is a list, any
adjacent strings in that list will be appended by ‘match-pattern’.  For
instance, if a parsing function returns ‘(13 ("a" "b" "c"))’,
‘match-pattern’ will take ‘(13 ("abc"))’ as its value.

   For example, here is a function to match “ab” using the actual
interface.

     (define (match-a-b str len pos)
        (and (<= (+ pos 2) len)
             (string= str "ab" pos (+ pos 2))
             (list (+ pos 2) '()))) ; we return no extra information

   The above function can be used to match a string by running
‘(match-pattern match-a-b "ab")’.

Code Generators and Extensible Syntax
.....................................

PEG expressions, such as those in a ‘define-peg-pattern’ form, are
interpreted internally in two steps.

   First, any string PEG is expanded into an s-expression PEG by the
code in the ‘(ice-9 peg string-peg)’ module.

   Then, the s-expression PEG that results is compiled into a parsing
function by the ‘(ice-9 peg codegen)’ module.  In particular, the
function ‘compile-peg-pattern’ is called on the s-expression.  It then
decides what to do based on the form it is passed.

   The PEG syntax can be expanded by providing ‘compile-peg-pattern’
more options for what to do with its forms.  The extended syntax will be
associated with a symbol, for instance ‘my-parsing-form’, and will be
called on all PEG expressions of the form
     (my-parsing-form ...)

   The parsing function should take two arguments.  The first will be a
syntax object containing a list with all of the arguments to the form
(but not the form’s name), and the second will be the ‘capture-type’
argument that is passed to ‘define-peg-pattern’.

   New functions can be registered by calling ‘(add-peg-compiler! symbol
function)’, where ‘symbol’ is the symbol that will indicate a form of
this type and ‘function’ is the code generating function described
above.  The function ‘add-peg-compiler!’ is exported from the ‘(ice-9
peg codegen)’ module.


File: guile.info,  Node: Read/Load/Eval/Compile,  Next: Memory Management,  Prev: PEG Parsing,  Up: API Reference

6.16 Reading and Evaluating Scheme Code
=======================================

This chapter describes Guile functions that are concerned with reading,
loading, evaluating, and compiling Scheme code at run time.

* Menu:

* Scheme Syntax::               Standard and extended Scheme syntax.
* Scheme Read::                 Reading Scheme code.
* Annotated Scheme Read::       Reading Scheme code, for the compiler.
* Scheme Write::                Writing Scheme values to a port.
* Fly Evaluation::              Procedures for on the fly evaluation.
* Compilation::                 How to compile Scheme files and procedures.
* Loading::                     Loading Scheme code from file.
* Load Paths::                  Where Guile looks for code.
* Character Encoding of Source Files:: Loading non-ASCII Scheme code from file.
* Delayed Evaluation::          Postponing evaluation until it is needed.
* Local Evaluation::            Evaluation in a local lexical environment.
* Local Inclusion::             Compile-time inclusion of one file in another.
* Sandboxed Evaluation::        Evaluation with limited capabilities.
* REPL Servers::                Serving a REPL over a socket.
* Cooperative REPL Servers::    REPL server for single-threaded applications.


File: guile.info,  Node: Scheme Syntax,  Next: Scheme Read,  Up: Read/Load/Eval/Compile

6.16.1 Scheme Syntax: Standard and Guile Extensions
---------------------------------------------------

* Menu:

* Expression Syntax::
* Comments::
* Block Comments::
* Case Sensitivity::
* Keyword Syntax::
* Reader Extensions::


File: guile.info,  Node: Expression Syntax,  Next: Comments,  Up: Scheme Syntax

6.16.1.1 Expression Syntax
..........................

An expression to be evaluated takes one of the following forms.

SYMBOL
     A symbol is evaluated by dereferencing.  A binding of that symbol
     is sought and the value there used.  For example,

          (define x 123)
          x ⇒ 123

(PROC ARGS...)
     A parenthesised expression is a function call.  PROC and each
     argument are evaluated, then the function (which PROC evaluated to)
     is called with those arguments.

     The order in which PROC and the arguments are evaluated is
     unspecified, so be careful when using expressions with side
     effects.

          (max 1 2 3) ⇒ 3

          (define (get-some-proc)  min)
          ((get-some-proc) 1 2 3) ⇒ 1

     The same sort of parenthesised form is used for a macro invocation,
     but in that case the arguments are not evaluated.  See the
     descriptions of macros for more on this (*note Macros::, and *note
     Syntax Rules::).

CONSTANT
     Number, string, character and boolean constants evaluate “to
     themselves”, so can appear as literals.

          123     ⇒ 123
          99.9    ⇒ 99.9
          "hello" ⇒ "hello"
          #\z     ⇒ #\z
          #t      ⇒ #t

     Note that an application must not attempt to modify literal
     strings, since they may be in read-only memory.

(quote DATA)
’DATA
     Quoting is used to obtain a literal symbol (instead of a variable
     reference), a literal list (instead of a function call), or a
     literal vector.  ’ is simply a shorthand for a ‘quote’ form.  For
     example,

          'x                   ⇒ x
          '(1 2 3)             ⇒ (1 2 3)
          '#(1 (2 3) 4)        ⇒ #(1 (2 3) 4)
          (quote x)            ⇒ x
          (quote (1 2 3))      ⇒ (1 2 3)
          (quote #(1 (2 3) 4)) ⇒ #(1 (2 3) 4)

     Note that an application must not attempt to modify literal lists
     or vectors obtained from a ‘quote’ form, since they may be in
     read-only memory.

(quasiquote DATA)
‘DATA
     Backquote quasi-quotation is like ‘quote’, but selected
     sub-expressions are evaluated.  This is a convenient way to
     construct a list or vector structure most of which is constant, but
     at certain points should have expressions substituted.

     The same effect can always be had with suitable ‘list’, ‘cons’ or
     ‘vector’ calls, but quasi-quoting is often easier.

     (unquote EXPR)
     ,EXPR
          Within the quasiquote DATA, ‘unquote’ or ‘,’ indicates an
          expression to be evaluated and inserted.  The comma syntax ‘,’
          is simply a shorthand for an ‘unquote’ form.  For example,

               `(1 2 (* 9 9) 3 4)       ⇒ (1 2 (* 9 9) 3 4)
               `(1 2 ,(* 9 9) 3 4)      ⇒ (1 2 81 3 4)
               `(1 (unquote (+ 1 1)) 3) ⇒ (1 2 3)
               `#(1 ,(/ 12 2))          ⇒ #(1 6)

     (unquote-splicing EXPR)
     ,@EXPR
          Within the quasiquote DATA, ‘unquote-splicing’ or ‘,@’
          indicates an expression to be evaluated and the elements of
          the returned list inserted.  EXPR must evaluate to a list.
          The “comma-at” syntax ‘,@’ is simply a shorthand for an
          ‘unquote-splicing’ form.

               (define x '(2 3))
               `(1 ,x 4)                           ⇒ (1 (2 3) 4)
               `(1 ,@x 4)                         ⇒ (1 2 3 4)
               `(1 (unquote-splicing (map 1+ x)))  ⇒ (1 3 4)
               `#(9 ,@x 9)                        ⇒ #(9 2 3 9)

          Notice ‘,@’ differs from plain ‘,’ in the way one level of
          nesting is stripped.  For ‘,@’ the elements of a returned list
          are inserted, whereas with ‘,’ it would be the list itself
          inserted.


File: guile.info,  Node: Comments,  Next: Block Comments,  Prev: Expression Syntax,  Up: Scheme Syntax

6.16.1.2 Comments
.................

Comments in Scheme source files are written by starting them with a
semicolon character (‘;’).  The comment then reaches up to the end of
the line.  Comments can begin at any column, and the may be inserted on
the same line as Scheme code.

     ; Comment
     ;; Comment too
     (define x 1)        ; Comment after expression
     (let ((y 1))
       ;; Display something.
       (display y)
     ;;; Comment at left margin.
       (display (+ y 1)))

   It is common to use a single semicolon for comments following
expressions on a line, to use two semicolons for comments which are
indented like code, and three semicolons for comments which start at
column 0, even if they are inside an indented code block.  This
convention is used when indenting code in Emacs’ Scheme mode.


File: guile.info,  Node: Block Comments,  Next: Case Sensitivity,  Prev: Comments,  Up: Scheme Syntax

6.16.1.3 Block Comments
.......................

In addition to the standard line comments defined by R5RS, Guile has
another comment type for multiline comments, called “block comments”.
This type of comment begins with the character sequence ‘#!’ and ends
with the characters ‘!#’.

   These comments are compatible with the block comments in the Scheme
Shell ‘scsh’ (*note The Scheme shell (scsh)::).  The characters ‘#!’
were chosen because they are the magic characters used in shell scripts
for indicating that the name of the program for executing the script
follows on the same line.

   Thus a Guile script often starts like this.

     #! /usr/local/bin/guile -s
     !#

   More details on Guile scripting can be found in the scripting section
(*note Guile Scripting::).

   Similarly, Guile (starting from version 2.0) supports nested block
comments as specified by R6RS and SRFI-30
(http://srfi.schemers.org/srfi-30/srfi-30.html):

     (+ 1 #| this is a #| nested |# block comment |# 2)
     ⇒ 3

   For backward compatibility, this syntax can be overridden with
‘read-hash-extend’ (*note ‘read-hash-extend’: Reader Extensions.).

   There is one special case where the contents of a comment can
actually affect the interpretation of code.  When a character encoding
declaration, such as ‘coding: utf-8’ appears in one of the first few
lines of a source file, it indicates to Guile’s default reader that this
source code file is not ASCII. For details see *note Character Encoding
of Source Files::.


File: guile.info,  Node: Case Sensitivity,  Next: Keyword Syntax,  Prev: Block Comments,  Up: Scheme Syntax

6.16.1.4 Case Sensitivity
.........................

Scheme as defined in R5RS is not case sensitive when reading symbols.
Guile, on the contrary is case sensitive by default, so the identifiers

     guile-whuzzy
     Guile-Whuzzy

   are the same in R5RS Scheme, but are different in Guile.

   It is possible to turn off case sensitivity in Guile by setting the
reader option ‘case-insensitive’.  For more information on reader
options, *Note Scheme Read::.

     (read-enable 'case-insensitive)

   It is also possible to disable (or enable) case sensitivity within a
single file by placing the reader directives ‘#!fold-case’ (or
‘#!no-fold-case’) within the file itself.


File: guile.info,  Node: Keyword Syntax,  Next: Reader Extensions,  Prev: Case Sensitivity,  Up: Scheme Syntax

6.16.1.5 Keyword Syntax
.......................


File: guile.info,  Node: Reader Extensions,  Prev: Keyword Syntax,  Up: Scheme Syntax

6.16.1.6 Reader Extensions
..........................

 -- Scheme Procedure: read-hash-extend chr proc
 -- C Function: scm_read_hash_extend (chr, proc)
     Install the procedure PROC for reading expressions starting with
     the character sequence ‘#’ and CHR.  PROC will be called with two
     arguments: the character CHR and the port to read further data
     from.  The object returned will be the return value of ‘read’.
     Passing ‘#f’ for PROC will remove a previous setting.


File: guile.info,  Node: Scheme Read,  Next: Annotated Scheme Read,  Prev: Scheme Syntax,  Up: Read/Load/Eval/Compile

6.16.2 Reading Scheme Code
--------------------------

 -- Scheme Procedure: read [port]
 -- C Function: scm_read (port)
     Read an s-expression from the input port PORT, or from the current
     input port if PORT is not specified.  Any whitespace before the
     next token is discarded.

   The behaviour of Guile’s Scheme reader can be modified by
manipulating its read options.

 -- Scheme Procedure: read-options [setting]
     Display the current settings of the global read options.  If
     SETTING is omitted, only a short form of the current read options
     is printed.  Otherwise if SETTING is the symbol ‘help’, a complete
     options description is displayed.

   The set of available options, and their default values, may be had by
invoking ‘read-options’ at the prompt.

     scheme@(guile-user)> (read-options)
     (square-brackets keywords #f positions)
     scheme@(guile-user)> (read-options 'help)
     positions         yes   Record positions of source code expressions.
     case-insensitive  no    Convert symbols to lower case.
     keywords          #f    Style of keyword recognition: #f, 'prefix or 'postfix.
     r6rs-hex-escapes  no    Use R6RS variable-length character and string hex escapes.
     square-brackets   yes   Treat `[' and `]' as parentheses, for R6RS compatibility.
     hungry-eol-escapes no   In strings, consume leading whitespace after an
                             escaped end-of-line.
     curly-infix       no    Support SRFI-105 curly infix expressions.
     r7rs-symbols      no    Support R7RS |...| symbol notation.

   Note that Guile also includes a preliminary mechanism for setting
read options on a per-port basis.  For instance, the ‘case-insensitive’
read option is set (or unset) on the port when the reader encounters the
‘#!fold-case’ or ‘#!no-fold-case’ reader directives.  Similarly, the
‘#!curly-infix’ reader directive sets the ‘curly-infix’ read option on
the port, and ‘#!curly-infix-and-bracket-lists’ sets ‘curly-infix’ and
unsets ‘square-brackets’ on the port (*note SRFI-105::).  There is
currently no other way to access or set the per-port read options.

   The boolean options may be toggled with ‘read-enable’ and
‘read-disable’.  The non-boolean ‘keywords’ option must be set using
‘read-set!’.

 -- Scheme Procedure: read-enable option-name
 -- Scheme Procedure: read-disable option-name
 -- Scheme Syntax: read-set! option-name value
     Modify the read options.  ‘read-enable’ should be used with boolean
     options and switches them on, ‘read-disable’ switches them off.

     ‘read-set!’ can be used to set an option to a specific value.  Due
     to historical oddities, it is a macro that expects an unquoted
     option name.

   For example, to make ‘read’ fold all symbols to their lower case
(perhaps for compatibility with older Scheme code), you can enter:

     (read-enable 'case-insensitive)

   For more information on the effect of the ‘r6rs-hex-escapes’ and
‘hungry-eol-escapes’ options, see (*note String Syntax::).

   For more information on the ‘r7rs-symbols’ option, see (*note Symbol
Read Syntax::).


File: guile.info,  Node: Annotated Scheme Read,  Next: Scheme Write,  Prev: Scheme Read,  Up: Read/Load/Eval/Compile

6.16.3 Reading Scheme Code, For the Compiler
--------------------------------------------

When something goes wrong with a Scheme program, the user will want to
know how to fix it.  This starts with identifying where the error
occured: we want to associate a source location with each component part
of source code, and propagate that source location information through
to the compiler or interpreter.

   For that, Guile provides ‘read-syntax’.

 -- Scheme Procedure: read-syntax [port]
     Read an s-expression from the input port PORT, or from the current
     input port if PORT is not specified.

     If, after skipping white space and comments, no more bytes are
     available from PORT, return the end-of-file object.  *Note Binary
     I/O::.  Otherwise, return an annotated datum.  An annotated datum
     is a syntax object which associates a source location with a datum.
     For example:

          (call-with-input-string "  foo" read-syntax)
          ; ⇒ #<syntax:unknown file:1:2 foo>
          (call-with-input-string "(foo)" read-syntax)
          ; ⇒
          ; #<syntax:unknown file:1:0
          ;   (#<syntax unknown file:1:1 foo>)>

     As the second example shows, all fields of pairs and vectors are
     also annotated, recursively.

   Most users are familiar with syntax objects in the context of macros,
which use syntax objects to associate scope information with
identifiers.  *Note Macros::.  Here we use syntax objects to associate
source location information with any datum, but without attaching scope
information.  The Scheme compiler (‘compile’) and the interpreter
(‘eval’) can accept syntax objects directly as input, allowing them to
associate source information with resulting code.  *Note Compilation::,
and *Note Fly Evaluation::.

   Note that there is a legacy interface for getting source locations
into the Scheme compiler or interpreter, which is to use a side table
that associates “source properties” with each subdatum returned by
‘read’, instead of wrapping the datums directly as in ‘read-syntax’.
This has the disadvantage of not being able to annotate all kinds of
datums.  *Note Source Properties::, for more information.


File: guile.info,  Node: Scheme Write,  Next: Fly Evaluation,  Prev: Annotated Scheme Read,  Up: Read/Load/Eval/Compile

6.16.4 Writing Scheme Values
----------------------------

Any scheme value may be written to a port.  Not all values may be read
back in (*note Scheme Read::), however.

 -- Scheme Procedure: write obj [port]
     Send a representation of OBJ to PORT or to the current output port
     if not given.

     The output is designed to be machine readable, and can be read back
     with ‘read’ (*note Scheme Read::).  Strings are printed in double
     quotes, with escapes if necessary, and characters are printed in
     ‘#\’ notation.

 -- Scheme Procedure: display obj [port]
     Send a representation of OBJ to PORT or to the current output port
     if not given.

     The output is designed for human readability, it differs from
     ‘write’ in that strings are printed without double quotes and
     escapes, and characters are printed as per ‘write-char’, not in
     ‘#\’ form.

   As was the case with the Scheme reader, there are a few options that
affect the behavior of the Scheme printer.

 -- Scheme Procedure: print-options [setting]
     Display the current settings of the read options.  If SETTING is
     omitted, only a short form of the current read options is printed.
     Otherwise if SETTING is the symbol ‘help’, a complete options
     description is displayed.

   The set of available options, and their default values, may be had by
invoking ‘print-options’ at the prompt.

     scheme@(guile-user)> (print-options)
     (quote-keywordish-symbols reader highlight-suffix "}" highlight-prefix "{")
     scheme@(guile-user)> (print-options 'help)
     highlight-prefix          {       The string to print before highlighted values.
     highlight-suffix          }       The string to print after highlighted values.
     quote-keywordish-symbols  reader  How to print symbols that have a colon
                                       as their first or last character. The
                                       value '#f' does not quote the colons;
                                       '#t' quotes them; 'reader' quotes them
                                       when the reader option 'keywords' is
                                       not '#f'.
     escape-newlines           yes     Render newlines as \n when printing
                                       using `write'.
     r7rs-symbols              no      Escape symbols using R7RS |...| symbol
                                       notation.

   These options may be modified with the print-set!  syntax.

 -- Scheme Syntax: print-set! option-name value
     Modify the print options.  Due to historical oddities, ‘print-set!’
     is a macro that expects an unquoted option name.


File: guile.info,  Node: Fly Evaluation,  Next: Compilation,  Prev: Scheme Write,  Up: Read/Load/Eval/Compile

6.16.5 Procedures for On the Fly Evaluation
-------------------------------------------

Scheme has the lovely property that its expressions may be represented
as data.  The ‘eval’ procedure takes a Scheme datum and evaluates it as
code.

 -- Scheme Procedure: eval exp module_or_state
 -- C Function: scm_eval (exp, module_or_state)
     Evaluate EXP, a list representing a Scheme expression, in the
     top-level environment specified by MODULE_OR_STATE.  While EXP is
     evaluated (using ‘primitive-eval’), MODULE_OR_STATE is made the
     current module.  The current module is reset to its previous value
     when ‘eval’ returns.  XXX - dynamic states.  Example: (eval ’(+ 1
     2) (interaction-environment))

 -- Scheme Procedure: interaction-environment
 -- C Function: scm_interaction_environment ()
     Return a specifier for the environment that contains
     implementation–defined bindings, typically a superset of those
     listed in the report.  The intent is that this procedure will
     return the environment in which the implementation would evaluate
     expressions dynamically typed by the user.

   *Note Environments::, for other environments.

   One does not always receive code as Scheme data, of course, and this
is especially the case for Guile’s other language implementations (*note
Other Languages::).  For the case in which all you have is a string, we
have ‘eval-string’.  There is a legacy version of this procedure in the
default environment, but you really want the one from ‘(ice-9
eval-string)’, so load it up:

     (use-modules (ice-9 eval-string))

 -- Scheme Procedure: eval-string string [#:module=#f] [#:file=#f]
          [#:line=#f] [#:column=#f] [#:lang=(current-language)]
          [#:compile?=#f]
     Parse STRING according to the current language, normally Scheme.
     Evaluate or compile the expressions it contains, in order,
     returning the last expression.

     If the MODULE keyword argument is set, save a module excursion
     (*note Module System Reflection::) and set the current module to
     MODULE before evaluation.

     The FILE, LINE, and COLUMN keyword arguments can be used to
     indicate that the source string begins at a particular source
     location.

     Finally, LANG is a language, defaulting to the current language,
     and the expression is compiled if COMPILE? is true or there is no
     evaluator for the given language.

 -- C Function: scm_eval_string (string)
 -- C Function: scm_eval_string_in_module (string, module)
     These C bindings call ‘eval-string’ from ‘(ice-9 eval-string)’,
     evaluating within MODULE or the current module.

 -- C Function: SCM scm_c_eval_string (const char *string)
     ‘scm_eval_string’, but taking a C string in locale encoding instead
     of an ‘SCM’.

 -- Scheme Procedure: apply proc arg ... arglst
 -- C Function: scm_apply_0 (proc, arglst)
 -- C Function: scm_apply_1 (proc, arg1, arglst)
 -- C Function: scm_apply_2 (proc, arg1, arg2, arglst)
 -- C Function: scm_apply_3 (proc, arg1, arg2, arg3, arglst)
 -- C Function: scm_apply (proc, arg, rest)
     Call PROC with arguments ARG ... and the elements of the ARGLST
     list.

     ‘scm_apply’ takes parameters corresponding to a Scheme level
     ‘(lambda (proc arg1 . rest) ...)’.  So ARG1 and all but the last
     element of the REST list make up ARG ..., and the last element of
     REST is the ARGLST list.  Or if REST is the empty list ‘SCM_EOL’
     then there’s no ARG ..., and (ARG1) is the ARGLST.

     ARGLST is not modified, but the REST list passed to ‘scm_apply’ is
     modified.

 -- C Function: scm_call_0 (proc)
 -- C Function: scm_call_1 (proc, arg1)
 -- C Function: scm_call_2 (proc, arg1, arg2)
 -- C Function: scm_call_3 (proc, arg1, arg2, arg3)
 -- C Function: scm_call_4 (proc, arg1, arg2, arg3, arg4)
 -- C Function: scm_call_5 (proc, arg1, arg2, arg3, arg4, arg5)
 -- C Function: scm_call_6 (proc, arg1, arg2, arg3, arg4, arg5, arg6)
 -- C Function: scm_call_7 (proc, arg1, arg2, arg3, arg4, arg5, arg6,
          arg7)
 -- C Function: scm_call_8 (proc, arg1, arg2, arg3, arg4, arg5, arg6,
          arg7, arg8)
 -- C Function: scm_call_9 (proc, arg1, arg2, arg3, arg4, arg5, arg6,
          arg7, arg8, arg9)
     Call PROC with the given arguments.

 -- C Function: scm_call (proc, ...)
     Call PROC with any number of arguments.  The argument list must be
     terminated by ‘SCM_UNDEFINED’.  For example:

          scm_call (scm_c_public_ref ("guile", "+"),
                    scm_from_int (1),
                    scm_from_int (2),
                    SCM_UNDEFINED);

 -- C Function: scm_call_n (proc, argv, nargs)
     Call PROC with the array of arguments ARGV, as a ‘SCM*’.  The
     length of the arguments should be passed in NARGS, as a ‘size_t’.

 -- Scheme Procedure: primitive-eval exp
 -- C Function: scm_primitive_eval (exp)
     Evaluate EXP in the top-level environment specified by the current
     module.


File: guile.info,  Node: Compilation,  Next: Loading,  Prev: Fly Evaluation,  Up: Read/Load/Eval/Compile

6.16.6 Compiling Scheme Code
----------------------------

The ‘eval’ procedure directly interprets the S-expression representation
of Scheme.  An alternate strategy for evaluation is to determine ahead
of time what computations will be necessary to evaluate the expression,
and then use that recipe to produce the desired results.  This is known
as “compilation”.

   While it is possible to compile simple Scheme expressions such as ‘(+
2 2)’ or even ‘"Hello world!"’, compilation is most interesting in the
context of procedures.  Compiling a lambda expression produces a
compiled procedure, which is just like a normal procedure except
typically much faster, because it can bypass the generic interpreter.

   Functions from system modules in a Guile installation are normally
compiled already, so they load and run quickly.

   Note that well-written Scheme programs will not typically call the
procedures in this section, for the same reason that it is often bad
taste to use ‘eval’.  By default, Guile automatically compiles any files
it encounters that have not been compiled yet (*note ‘--auto-compile’:
Invoking Guile.).  The compiler can also be invoked explicitly from the
shell as ‘guild compile foo.scm’.

   (Why are calls to ‘eval’ and ‘compile’ usually in bad taste?  Because
they are limited, in that they can only really make sense for top-level
expressions.  Also, most needs for “compile-time” computation are
fulfilled by macros and closures.  Of course one good counterexample is
the REPL itself, or any code that reads expressions from a port.)

   Automatic compilation generally works transparently, without any need
for user intervention.  However Guile does not yet do proper dependency
tracking, so that if file ‘A.scm’ uses macros from ‘B.scm’, and B.SCM
changes, ‘A.scm’ would not be automatically recompiled.  To forcibly
invalidate the auto-compilation cache, pass the ‘--fresh-auto-compile’
option to Guile, or set the ‘GUILE_AUTO_COMPILE’ environment variable to
‘fresh’ (instead of to ‘0’ or ‘1’).

   For more information on the compiler itself, see *note Compiling to
the Virtual Machine::.  For information on the virtual machine, see
*note A Virtual Machine for Guile::.

   The command-line interface to Guile’s compiler is the ‘guild compile’
command:

 -- Command: guild compile [‘option’...] FILE...
     Compile FILE, a source file, and store bytecode in the compilation
     cache or in the file specified by the ‘-o’ option.  The following
     options are available:

     ‘-L DIR’
     ‘--load-path=DIR’
          Add DIR to the front of the module load path.

     ‘-o OFILE’
     ‘--output=OFILE’
          Write output bytecode to OFILE.  By convention, bytecode file
          names end in ‘.go’.  When ‘-o’ is omitted, the output file
          name is as for ‘compile-file’ (see below).

     ‘-x EXTENSION’
          Recognize EXTENSION as a valid source file name extension.

          For example, to compile R6RS code, you might want to pass ‘-x
          .sls’ so that files ending in ‘.sls’ can be found.

     ‘-W WARNING’
     ‘--warn=WARNING’
          Enable specific warning passes; use ‘-Whelp’ for a list of
          available options.  The default is ‘-W1’, which enables a
          number of common warnings.  Pass ‘-W0’ to disable all
          warnings.

     ‘-O OPT’
     ‘--optimize=OPT’
          Enable or disable specific compiler optimizations; use
          ‘-Ohelp’ for a list of available options.  The default is
          ‘-O2’, which enables most optimizations.  ‘-O0’ is recommended
          if compilation speed is more important than the speed of the
          compiled code.  Pass ‘-Ono-OPT’ to disable a specific compiler
          pass.  Any number of ‘-O’ options can be passed to the
          compiler, with later ones taking precedence.

     ‘--r6rs’
     ‘--r7rs’
          Compile in an environment whose default bindings, reader
          options, and load paths are adapted for specific Scheme
          standards.  *Note R6RS Support::, and *Note R7RS Support::.

     ‘-f LANG’
     ‘--from=LANG’
          Use LANG as the source language of FILE.  If this option is
          omitted, ‘scheme’ is assumed.

     ‘-t LANG’
     ‘--to=LANG’
          Use LANG as the target language of FILE.  If this option is
          omitted, ‘rtl’ is assumed.

     ‘-T TARGET’
     ‘--target=TARGET’
          Produce code for TARGET instead of %HOST-TYPE (*note
          %host-type: Build Config.).  Target must be a valid GNU
          triplet, such as ‘armv5tel-unknown-linux-gnueabi’ (*note
          (autoconf)Specifying Target Triplets::).

     Each FILE is assumed to be UTF-8-encoded, unless it contains a
     coding declaration as recognized by ‘file-encoding’ (*note
     Character Encoding of Source Files::).

   The compiler can also be invoked directly by Scheme code.  These
interfaces are in their own module:

     (use-modules (system base compile))

 -- Scheme Procedure: compile exp [#:env=#f] [#:from=(current-language)]
          [#:to=value] [#:opts='()]
          [#:optimization-level=(default-optimization-level)]
          [#:warning-level=(default-warning-level)]
     Compile the expression EXP in the environment ENV.  If EXP is a
     procedure, the result will be a compiled procedure; otherwise
     ‘compile’ is mostly equivalent to ‘eval’.

     For a discussion of languages and compiler options, *Note Compiling
     to the Virtual Machine::.

 -- Scheme Procedure: compile-file file [#:output-file=#f]
          [#:from=(current-language)] [#:to='rtl]
          [#:env=(default-environment from)] [#:opts='()]
          [#:optimization-level=(default-optimization-level)]
          [#:warning-level=(default-warning-level)]
          [#:canonicalization='relative]
     Compile the file named FILE.

     Output will be written to a OUTPUT-FILE.  If you do not supply an
     output file name, output is written to a file in the cache
     directory, as computed by ‘(compiled-file-name FILE)’.

     FROM and TO specify the source and target languages.  *Note
     Compiling to the Virtual Machine::, for more information on these
     options, and on ENV and OPTS.

     As with ‘guild compile’, FILE is assumed to be UTF-8-encoded unless
     it contains a coding declaration.

 -- Scheme Parameter: default-optimization-level
     The default optimization level, as an integer from 0 to 9.  The
     default is 2.
 -- Scheme Parameter: default-warning-level
     The default warning level, as an integer from 0 to 9.  The default
     is 1.

   *Note Parameters::, for more on how to set parameters.

 -- Scheme Procedure: compiled-file-name file
     Compute a cached location for a compiled version of a Scheme file
     named FILE.

     This file will usually be below the ‘$HOME/.cache/guile/ccache’
     directory, depending on the value of the ‘XDG_CACHE_HOME’
     environment variable.  The intention is that ‘compiled-file-name’
     provides a fallback location for caching auto-compiled files.  If
     you want to place a compile file in the ‘%load-compiled-path’, you
     should pass the OUTPUT-FILE option to ‘compile-file’, explicitly.

 -- Scheme Variable: %auto-compilation-options
     This variable contains the options passed to the ‘compile-file’
     procedure when auto-compiling source files.  By default, it enables
     useful compilation warnings.  It can be customized from ‘~/.guile’.


File: guile.info,  Node: Loading,  Next: Load Paths,  Prev: Compilation,  Up: Read/Load/Eval/Compile

6.16.7 Loading Scheme Code from File
------------------------------------

 -- Scheme Procedure: load filename [reader]
     Load FILENAME and evaluate its contents in the top-level
     environment.

     READER if provided should be either ‘#f’, or a procedure with the
     signature ‘(lambda (port) ...)’ which reads the next expression
     from PORT.  If READER is ‘#f’ or absent, Guile’s built-in ‘read’
     procedure is used (*note Scheme Read::).

     The READER argument takes effect by setting the value of the
     ‘current-reader’ fluid (see below) before loading the file, and
     restoring its previous value when loading is complete.  The Scheme
     code inside FILENAME can itself change the current reader procedure
     on the fly by setting ‘current-reader’ fluid.

     If the variable ‘%load-hook’ is defined, it should be bound to a
     procedure that will be called before any code is loaded.  See
     documentation for ‘%load-hook’ later in this section.

 -- Scheme Procedure: load-compiled filename
     Load the compiled file named FILENAME.

     Compiling a source file (*note Read/Load/Eval/Compile::) and then
     calling ‘load-compiled’ on the resulting file is equivalent to
     calling ‘load’ on the source file.

 -- Scheme Procedure: primitive-load filename
 -- C Function: scm_primitive_load (filename)
     Load the file named FILENAME and evaluate its contents in the
     top-level environment.  FILENAME must either be a full pathname or
     be a pathname relative to the current directory.  If the variable
     ‘%load-hook’ is defined, it should be bound to a procedure that
     will be called before any code is loaded.  See the documentation
     for ‘%load-hook’ later in this section.

 -- C Function: SCM scm_c_primitive_load (const char *filename)
     ‘scm_primitive_load’, but taking a C string instead of an ‘SCM’.

 -- Variable: current-reader
     ‘current-reader’ holds the read procedure that is currently being
     used by the above loading procedures to read expressions (from the
     file that they are loading).  ‘current-reader’ is a fluid, so it
     has an independent value in each dynamic root and should be read
     and set using ‘fluid-ref’ and ‘fluid-set!’ (*note Fluids and
     Dynamic States::).

     Changing ‘current-reader’ is typically useful to introduce local
     syntactic changes, such that code following the ‘fluid-set!’ call
     is read using the newly installed reader.  The ‘current-reader’
     change should take place at evaluation time when the code is
     evaluated, or at compilation time when the code is compiled:

          (eval-when (compile eval)
            (fluid-set! current-reader my-own-reader))

     The ‘eval-when’ form above ensures that the ‘current-reader’ change
     occurs at the right time.

 -- Variable: %load-hook
     A procedure to be called ‘(%load-hook FILENAME)’ whenever a file is
     loaded, or ‘#f’ for no such call.  ‘%load-hook’ is used by all of
     the loading functions (‘load’ and ‘primitive-load’, and
     ‘load-from-path’ and ‘primitive-load-path’ documented in the next
     section).

     For example an application can set this to show what’s loaded,

          (set! %load-hook (lambda (filename)
                             (format #t "Loading ~a ...\n" filename)))
          (load-from-path "foo.scm")
          ⊣ Loading /usr/local/share/guile/site/foo.scm ...

 -- Scheme Procedure: current-load-port
 -- C Function: scm_current_load_port ()
     Return the current-load-port.  The load port is used internally by
     ‘primitive-load’.


File: guile.info,  Node: Load Paths,  Next: Character Encoding of Source Files,  Prev: Loading,  Up: Read/Load/Eval/Compile

6.16.8 Load Paths
-----------------

The procedure in the previous section look for Scheme code in the file
system at specific location.  Guile also has some procedures to search
the load path for code.

 -- Variable: %load-path
     List of directories which should be searched for Scheme modules and
     libraries.  When Guile starts up, ‘%load-path’ is initialized to
     the default load path ‘(list (%library-dir) (%site-dir)
     (%global-site-dir) (%package-data-dir))’.  The ‘GUILE_LOAD_PATH’
     environment variable can be used to prepend or append additional
     directories (*note Environment Variables::).

     *Note Build Config::, for more on ‘%site-dir’ and related
     procedures.

 -- Scheme Procedure: load-from-path filename
     Similar to ‘load’, but searches for FILENAME in the load paths.
     Preferentially loads a compiled version of the file, if it is
     available and up-to-date.

   A user can extend the load path by calling ‘add-to-load-path’.

 -- Scheme Syntax: add-to-load-path dir
     Add DIR to the load path.

   For example, a script might include this form to add the directory
that it is in to the load path:

     (add-to-load-path (dirname (current-filename)))

   It’s better to use ‘add-to-load-path’ than to modify ‘%load-path’
directly, because ‘add-to-load-path’ takes care of modifying the path
both at compile-time and at run-time.

 -- Scheme Procedure: primitive-load-path filename
          [exception-on-not-found]
 -- C Function: scm_primitive_load_path (filename)
     Search ‘%load-path’ for the file named FILENAME and load it into
     the top-level environment.  If FILENAME is a relative pathname and
     is not found in the list of search paths, an error is signalled.
     Preferentially loads a compiled version of the file, if it is
     available and up-to-date.

     If FILENAME is a relative pathname and is not found in the list of
     search paths, one of three things may happen, depending on the
     optional second argument, EXCEPTION-ON-NOT-FOUND.  If it is ‘#f’,
     ‘#f’ will be returned.  If it is a procedure, it will be called
     with no arguments.  (This allows a distinction to be made between
     exceptions raised by loading a file, and exceptions related to the
     loader itself.)  Otherwise an error is signalled.

     For compatibility with Guile 1.8 and earlier, the C function takes
     only one argument, which can be either a string (the file name) or
     an argument list.

 -- Scheme Procedure: %search-load-path filename
 -- C Function: scm_sys_search_load_path (filename)
     Search ‘%load-path’ for the file named FILENAME, which must be
     readable by the current user.  If FILENAME is found in the list of
     paths to search or is an absolute pathname, return its full
     pathname.  Otherwise, return ‘#f’.  Filenames may have any of the
     optional extensions in the ‘%load-extensions’ list;
     ‘%search-load-path’ will try each extension automatically.

 -- Variable: %load-extensions
     A list of default file extensions for files containing Scheme code.
     ‘%search-load-path’ tries each of these extensions when looking for
     a file to load.  By default, ‘%load-extensions’ is bound to the
     list ‘("" ".scm")’.

   As mentioned above, when Guile searches the ‘%load-path’ for a source
file, it will also search the ‘%load-compiled-path’ for a corresponding
compiled file.  If the compiled file is as new or newer than the source
file, it will be loaded instead of the source file, using
‘load-compiled’.

 -- Variable: %load-compiled-path
     Like ‘%load-path’, but for compiled files.  By default, this path
     has two entries: one for compiled files from Guile itself, and one
     for site packages.  The ‘GUILE_LOAD_COMPILED_PATH’ environment
     variable can be used to prepend or append additional directories
     (*note Environment Variables::).

   When ‘primitive-load-path’ searches the ‘%load-compiled-path’ for a
corresponding compiled file for a relative path it does so by appending
‘.go’ to the relative path.  For example, searching for ‘ice-9/popen’
could find ‘/usr/lib/guile/3.0/ccache/ice-9/popen.go’, and use it
instead of ‘/usr/share/guile/3.0/ice-9/popen.scm’.

   If ‘primitive-load-path’ does not find a corresponding ‘.go’ file in
the ‘%load-compiled-path’, or the ‘.go’ file is out of date, it will
search for a corresponding auto-compiled file in the fallback path,
possibly creating one if one does not exist.

   *Note Installing Site Packages::, for more on how to correctly
install site packages.  *Note Modules and the File System::, for more on
the relationship between load paths and modules.  *Note Compilation::,
for more on the fallback path and auto-compilation.

   Finally, there are a couple of helper procedures for general path
manipulation.

 -- Scheme Procedure: parse-path path [tail]
 -- C Function: scm_parse_path (path, tail)
     Parse PATH, which is expected to be a colon-separated string, into
     a list and return the resulting list with TAIL appended.  If PATH
     is ‘#f’, TAIL is returned.

 -- Scheme Procedure: parse-path-with-ellipsis path base
 -- C Function: scm_parse_path_with_ellipsis (path, base)
     Parse PATH, which is expected to be a colon-separated string, into
     a list and return the resulting list with BASE (a list) spliced in
     place of the ‘...’ path component, if present, or else BASE is
     added to the end.  If PATH is ‘#f’, BASE is returned.

 -- Scheme Procedure: search-path path filename [extensions
          [require-exts?]]
 -- C Function: scm_search_path (path, filename, rest)
     Search PATH for a directory containing a file named FILENAME.  The
     file must be readable, and not a directory.  If we find one, return
     its full filename; otherwise, return ‘#f’.  If FILENAME is
     absolute, return it unchanged.  If given, EXTENSIONS is a list of
     strings; for each directory in PATH, we search for FILENAME
     concatenated with each EXTENSION.  If REQUIRE-EXTS? is true,
     require that the returned file name have one of the given
     extensions; if REQUIRE-EXTS? is not given, it defaults to ‘#f’.

     For compatibility with Guile 1.8 and earlier, the C function takes
     only three arguments.


File: guile.info,  Node: Character Encoding of Source Files,  Next: Delayed Evaluation,  Prev: Load Paths,  Up: Read/Load/Eval/Compile

6.16.9 Character Encoding of Source Files
-----------------------------------------

Scheme source code files are usually encoded in ASCII or UTF-8, but the
built-in reader can interpret other character encodings as well.  When
Guile loads Scheme source code, it uses the ‘file-encoding’ procedure
(described below) to try to guess the encoding of the file.  In the
absence of any hints, UTF-8 is assumed.  One way to provide a hint about
the encoding of a source file is to place a coding declaration in the
top 500 characters of the file.

   A coding declaration has the form ‘coding: XXXXXX’, where ‘XXXXXX’ is
the name of a character encoding in which the source code file has been
encoded.  The coding declaration must appear in a scheme comment.  It
can either be a semicolon-initiated comment, or the first block ‘#!’
comment in the file.

   The name of the character encoding in the coding declaration is
typically lower case and containing only letters, numbers, and hyphens,
as recognized by ‘set-port-encoding!’ (*note ‘set-port-encoding!’:
Ports.).  Common examples of character encoding names are ‘utf-8’ and
‘iso-8859-1’, as defined by IANA
(http://www.iana.org/assignments/character-sets).  Thus, the coding
declaration is mostly compatible with Emacs.

   However, there are some differences in encoding names recognized by
Emacs and encoding names defined by IANA, the latter being essentially a
subset of the former.  For instance, ‘latin-1’ is a valid encoding name
for Emacs, but it’s not according to the IANA standard, which Guile
follows; instead, you should use ‘iso-8859-1’, which is both understood
by Emacs and dubbed by IANA (IANA writes it uppercase but Emacs wants it
lowercase and Guile is case insensitive.)

   For source code, only a subset of all possible character encodings
can be interpreted by the built-in source code reader.  Only those
character encodings in which ASCII text appears unmodified can be used.
This includes ‘UTF-8’ and ‘ISO-8859-1’ through ‘ISO-8859-15’.  The
multi-byte character encodings ‘UTF-16’ and ‘UTF-32’ may not be used
because they are not compatible with ASCII.

   There might be a scenario in which one would want to read non-ASCII
code from a port, such as with the function ‘read’, instead of with
‘load’.  If the port’s character encoding is the same as the encoding of
the code to be read by the port, not other special handling is
necessary.  The port will automatically do the character encoding
conversion.  The functions ‘setlocale’ or by ‘set-port-encoding!’ are
used to set port encodings (*note Ports::).

   If a port is used to read code of unknown character encoding, it can
accomplish this in three steps.  First, the character encoding of the
port should be set to ISO-8859-1 using ‘set-port-encoding!’.  Then, the
procedure ‘file-encoding’, described below, is used to scan for a coding
declaration when reading from the port.  As a side effect, it rewinds
the port after its scan is complete.  After that, the port’s character
encoding should be set to the encoding returned by ‘file-encoding’, if
any, again by using ‘set-port-encoding!’.  Then the code can be read as
normal.

   Alternatively, one can use the ‘#:guess-encoding’ keyword argument of
‘open-file’ and related procedures.  *Note File Ports::.

 -- Scheme Procedure: file-encoding port
 -- C Function: scm_file_encoding (port)
     Attempt to scan the first few hundred bytes from the PORT for hints
     about its character encoding.  Return a string containing the
     encoding name or ‘#f’ if the encoding cannot be determined.  The
     port is rewound.

     Currently, the only supported method is to look for an Emacs-like
     character coding declaration (*note how Emacs recognizes file
     encoding: (emacs)Recognize Coding.).  The coding declaration is of
     the form ‘coding: XXXXX’ and must appear in a Scheme comment.
     Additional heuristics may be added in the future.


File: guile.info,  Node: Delayed Evaluation,  Next: Local Evaluation,  Prev: Character Encoding of Source Files,  Up: Read/Load/Eval/Compile

6.16.10 Delayed Evaluation
--------------------------

Promises are a convenient way to defer a calculation until its result is
actually needed, and to run such a calculation only once.  Also *note
SRFI-45::.

 -- syntax: delay expr
     Return a promise object which holds the given EXPR expression,
     ready to be evaluated by a later ‘force’.

 -- Scheme Procedure: promise? obj
 -- C Function: scm_promise_p (obj)
     Return true if OBJ is a promise.

 -- Scheme Procedure: force p
 -- C Function: scm_force (p)
     Return the value obtained from evaluating the EXPR in the given
     promise P.  If P has previously been forced then its EXPR is not
     evaluated again, instead the value obtained at that time is simply
     returned.

     During a ‘force’, an EXPR can call ‘force’ again on its own
     promise, resulting in a recursive evaluation of that EXPR.  The
     first evaluation to return gives the value for the promise.  Higher
     evaluations run to completion in the normal way, but their results
     are ignored, ‘force’ always returns the first value.


File: guile.info,  Node: Local Evaluation,  Next: Local Inclusion,  Prev: Delayed Evaluation,  Up: Read/Load/Eval/Compile

6.16.11 Local Evaluation
------------------------

Guile includes a facility to capture a lexical environment, and later
evaluate a new expression within that environment.  This code is
implemented in a module.

     (use-modules (ice-9 local-eval))

 -- syntax: the-environment
     Captures and returns a lexical environment for use with
     ‘local-eval’ or ‘local-compile’.

 -- Scheme Procedure: local-eval exp env
 -- C Function: scm_local_eval (exp, env)
 -- Scheme Procedure: local-compile exp env [opts=()]
     Evaluate or compile the expression EXP in the lexical environment
     ENV.

   Here is a simple example, illustrating that it is the variable that
gets captured, not just its value at one point in time.

     (define e (let ((x 100)) (the-environment)))
     (define fetch-x (local-eval '(lambda () x) e))
     (fetch-x)
     ⇒ 100
     (local-eval '(set! x 42) e)
     (fetch-x)
     ⇒ 42

   While EXP is evaluated within the lexical environment of
‘(the-environment)’, it has the dynamic environment of the call to
‘local-eval’.

   ‘local-eval’ and ‘local-compile’ can only evaluate expressions, not
definitions.

     (local-eval '(define foo 42)
                 (let ((x 100)) (the-environment)))
     ⇒ syntax error: definition in expression context

   Note that the current implementation of ‘(the-environment)’ only
captures “normal” lexical bindings, and pattern variables bound by
‘syntax-case’.  It does not currently capture local syntax transformers
bound by ‘let-syntax’, ‘letrec-syntax’ or non-top-level ‘define-syntax’
forms.  Any attempt to reference such captured syntactic keywords via
‘local-eval’ or ‘local-compile’ produces an error.


File: guile.info,  Node: Local Inclusion,  Next: Sandboxed Evaluation,  Prev: Local Evaluation,  Up: Read/Load/Eval/Compile

6.16.12 Local Inclusion
-----------------------

This section has discussed various means of linking Scheme code
together: fundamentally, loading up files at run-time using ‘load’ and
‘load-compiled’.  Guile provides another option to compose parts of
programs together at expansion-time instead of at run-time.

 -- Scheme Syntax: include file-name
     Open FILE-NAME, at expansion-time, and read the Scheme forms that
     it contains, splicing them into the location of the ‘include’,
     within a ‘begin’.

     If FILE-NAME is a relative path, it is searched for relative to the
     path that contains the file that the ‘include’ form appears in.

   If you are a C programmer, if ‘load’ in Scheme is like ‘dlopen’ in C,
consider ‘include’ to be like the C preprocessor’s ‘#include’.  When you
use ‘include’, it is as if the contents of the included file were typed
in instead of the ‘include’ form.

   Because the code is included at compile-time, it is available to the
macroexpander.  Syntax definitions in the included file are available to
later code in the form in which the ‘include’ appears, without the need
for ‘eval-when’.  (*Note Eval When::.)

   For the same reason, compiling a form that uses ‘include’ results in
one compilation unit, composed of multiple files.  Loading the compiled
file is one ‘stat’ operation for the compilation unit, instead of ‘2*N’
in the case of ‘load’ (once for each loaded source file, and once each
corresponding compiled file, in the best case).

   Unlike ‘load’, ‘include’ also works within nested lexical contexts.
It so happens that the optimizer works best within a lexical context,
because all of the uses of bindings in a lexical context are visible, so
composing files by including them within a ‘(let () ...)’ can sometimes
lead to important speed improvements.

   On the other hand, ‘include’ does have all the disadvantages of early
binding: once the code with the ‘include’ is compiled, no change to the
included file is reflected in the future behavior of the including form.

   Also, the particular form of ‘include’, which requires an absolute
path, or a path relative to the current directory at compile-time, is
not very amenable to compiling the source in one place, but then
installing the source to another place.  For this reason, Guile provides
another form, ‘include-from-path’, which looks for the source file to
include within a load path.

 -- Scheme Syntax: include-from-path file-name
     Like ‘include’, but instead of expecting ‘file-name’ to be an
     absolute file name, it is expected to be a relative path to search
     in the ‘%load-path’.

   ‘include-from-path’ is more useful when you want to install all of
the source files for a package (as you should!).  It makes it possible
to evaluate an installed file from source, instead of relying on the
‘.go’ file being up to date.


File: guile.info,  Node: Sandboxed Evaluation,  Next: REPL Servers,  Prev: Local Inclusion,  Up: Read/Load/Eval/Compile

6.16.13 Sandboxed Evaluation
----------------------------

Sometimes you would like to evaluate code that comes from an untrusted
party.  The safest way to do this is to buy a new computer, evaluate the
code on that computer, then throw the machine away.  However if you are
unwilling to take this simple approach, Guile does include a limited
“sandbox” facility that can allow untrusted code to be evaluated with
some confidence.

   To use the sandboxed evaluator, load its module:

     (use-modules (ice-9 sandbox))

   Guile’s sandboxing facility starts with the ability to restrict the
time and space used by a piece of code.

 -- Scheme Procedure: call-with-time-limit limit thunk limit-reached
     Call THUNK, but cancel it if LIMIT seconds of wall-clock time have
     elapsed.  If the computation is cancelled, call LIMIT-REACHED in
     tail position.  THUNK must not disable interrupts or prevent an
     abort via a ‘dynamic-wind’ unwind handler.

 -- Scheme Procedure: call-with-allocation-limit limit thunk
          limit-reached
     Call THUNK, but cancel it if LIMIT bytes have been allocated.  If
     the computation is cancelled, call LIMIT-REACHED in tail position.
     THUNK must not disable interrupts or prevent an abort via a
     ‘dynamic-wind’ unwind handler.

     This limit applies to both stack and heap allocation.  The
     computation will not be aborted before LIMIT bytes have been
     allocated, but for the heap allocation limit, the check may be
     postponed until the next garbage collection.

     Note that as a current shortcoming, the heap size limit applies to
     all threads; concurrent allocation by other unrelated threads
     counts towards the allocation limit.

 -- Scheme Procedure: call-with-time-and-allocation-limits time-limit
          allocation-limit thunk
     Invoke THUNK in a dynamic extent in which its execution is limited
     to TIME-LIMIT seconds of wall-clock time, and its allocation to
     ALLOCATION-LIMIT bytes.  THUNK must not disable interrupts or
     prevent an abort via a ‘dynamic-wind’ unwind handler.

     If successful, return all values produced by invoking THUNK.  Any
     uncaught exception thrown by the thunk will propagate out.  If the
     time or allocation limit is exceeded, an exception will be thrown
     to the ‘limit-exceeded’ key.

   The time limit and stack limit are both very precise, but the heap
limit only gets checked asynchronously, after a garbage collection.  In
particular, if the heap is already very large, the number of allocated
bytes between garbage collections will be large, and therefore the
precision of the check is reduced.

   Additionally, due to the mechanism used by the allocation limit (the
‘after-gc-hook’), large single allocations like ‘(make-vector #e1e7)’
are only detected after the allocation completes, even if the allocation
itself causes garbage collection.  It’s possible therefore for user code
to not only exceed the allocation limit set, but also to exhaust all
available memory, causing out-of-memory conditions at any allocation
site.  Failure to allocate memory in Guile itself should be safe and
cause an exception to be thrown, but most systems are not designed to
handle ‘malloc’ failures.  An allocation failure may therefore exercise
unexpected code paths in your system, so it is a weakness of the sandbox
(and therefore an interesting point of attack).

   The main sandbox interface is ‘eval-in-sandbox’.

 -- Scheme Procedure: eval-in-sandbox exp [#:time-limit 0.1]
          [#:allocation-limit #e10e6] [#:bindings all-pure-bindings]
          [#:module (make-sandbox-module bindings)] [#:sever-module? #t]
     Evaluate the Scheme expression EXP within an isolated "sandbox".
     Limit its execution to TIME-LIMIT seconds of wall-clock time, and
     limit its allocation to ALLOCATION-LIMIT bytes.

     The evaluation will occur in MODULE, which defaults to the result
     of calling ‘make-sandbox-module’ on BINDINGS, which itself defaults
     to ‘all-pure-bindings’.  This is the core of the sandbox: creating
     a scope for the expression that is “safe”.

     A safe sandbox module has two characteristics.  Firstly, it will
     not allow the expression being evaluated to avoid being cancelled
     due to time or allocation limits.  This ensures that the expression
     terminates in a timely fashion.

     Secondly, a safe sandbox module will prevent the evaluation from
     receiving information from previous evaluations, or from affecting
     future evaluations.  All combinations of binding sets exported by
     ‘(ice-9 sandbox)’ form safe sandbox modules.

     The BINDINGS should be given as a list of import sets.  One import
     set is a list whose car names an interface, like ‘(ice-9 q)’, and
     whose cdr is a list of imports.  An import is either a bare symbol
     or a pair of ‘(OUT . IN)’, where OUT and IN are both symbols and
     denote the name under which a binding is exported from the module,
     and the name under which to make the binding available,
     respectively.  Note that BINDINGS is only used as an input to the
     default initializer for the MODULE argument; if you pass
     ‘#:module’, BINDINGS is unused.  If SEVER-MODULE? is true (the
     default), the module will be unlinked from the global module tree
     after the evaluation returns, to allow MOD to be garbage-collected.

     If successful, return all values produced by EXP.  Any uncaught
     exception thrown by the expression will propagate out.  If the time
     or allocation limit is exceeded, an exception will be thrown to the
     ‘limit-exceeded’ key.

   Constructing a safe sandbox module is tricky in general.  Guile
defines an easy way to construct safe modules from predefined sets of
bindings.  Before getting to that interface, here are some general notes
on safety.

  1. The time and allocation limits rely on the ability to interrupt and
     cancel a computation.  For this reason, no binding included in a
     sandbox module should be able to indefinitely postpone interrupt
     handling, nor should a binding be able to prevent an abort.  In
     practice this second consideration means that ‘dynamic-wind’ should
     not be included in any binding set.
  2. The time and allocation limits apply only to the ‘eval-in-sandbox’
     call.  If the call returns a procedure which is later called, no
     limit is “automatically” in place.  Users of ‘eval-in-sandbox’ have
     to be very careful to reimpose limits when calling procedures that
     escape from sandboxes.
  3. Similarly, the dynamic environment of the ‘eval-in-sandbox’ call is
     not necessarily in place when any procedure that escapes from the
     sandbox is later called.

     This detail prevents us from exposing ‘primitive-eval’ to the
     sandbox, for two reasons.  The first is that it’s possible for
     legacy code to forge references to any binding, if the
     ‘allow-legacy-syntax-objects?’ parameter is true.  The default for
     this parameter is true; *note Syntax Transformer Helpers:: for the
     details.  The parameter is bound to ‘#f’ for the duration of the
     ‘eval-in-sandbox’ call itself, but that will not be in place during
     calls to escaped procedures.

     The second reason we don’t expose ‘primitive-eval’ is that
     ‘primitive-eval’ implicitly works in the current module, which for
     an escaped procedure will probably be different than the module
     that is current for the ‘eval-in-sandbox’ call itself.

     The common denominator here is that if an interface exposed to the
     sandbox relies on dynamic environments, it is easy to mistakenly
     grant the sandboxed procedure additional capabilities in the form
     of bindings that it should not have access to.  For this reason,
     the default sets of predefined bindings do not depend on any
     dynamically scoped value.
  4. Mutation may allow a sandboxed evaluation to break some invariant
     in users of data supplied to it.  A lot of code culturally doesn’t
     expect mutation, but if you hand mutable data to a sandboxed
     evaluation and you also grant mutating capabilities to that
     evaluation, then the sandboxed code may indeed mutate that data.
     The default set of bindings to the sandbox do not include any
     mutating primitives.

     Relatedly, ‘set!’ may allow a sandbox to mutate a primitive,
     invalidating many system-wide invariants.  Guile is currently quite
     permissive when it comes to imported bindings and mutability.
     Although ‘set!’ to a module-local or lexically bound variable would
     be fine, we don’t currently have an easy way to disallow ‘set!’ to
     an imported binding, so currently no binding set includes ‘set!’.
  5. Mutation may allow a sandboxed evaluation to keep state, or make a
     communication mechanism with other code.  On the one hand this
     sounds cool, but on the other hand maybe this is part of your
     threat model.  Again, the default set of bindings doesn’t include
     mutating primitives, preventing sandboxed evaluations from keeping
     state.
  6. The sandbox should probably not be able to open a network
     connection, or write to a file, or open a file from disk.  The
     default binding set includes no interaction with the operating
     system.

   If you, dear reader, find the above discussion interesting, you will
enjoy Jonathan Rees’ dissertation, “A Security Kernel Based on the
Lambda Calculus”.

 -- Scheme Variable: all-pure-bindings
     All “pure” bindings that together form a safe subset of those
     bindings available by default to Guile user code.

 -- Scheme Variable: all-pure-and-impure-bindings
     Like ‘all-pure-bindings’, but additionally including mutating
     primitives like ‘vector-set!’.  This set is still safe in the sense
     mentioned above, with the caveats about mutation.

   The components of these composite sets are as follows:
 -- Scheme Variable: alist-bindings
 -- Scheme Variable: array-bindings
 -- Scheme Variable: bit-bindings
 -- Scheme Variable: bitvector-bindings
 -- Scheme Variable: char-bindings
 -- Scheme Variable: char-set-bindings
 -- Scheme Variable: clock-bindings
 -- Scheme Variable: core-bindings
 -- Scheme Variable: error-bindings
 -- Scheme Variable: fluid-bindings
 -- Scheme Variable: hash-bindings
 -- Scheme Variable: iteration-bindings
 -- Scheme Variable: keyword-bindings
 -- Scheme Variable: list-bindings
 -- Scheme Variable: macro-bindings
 -- Scheme Variable: nil-bindings
 -- Scheme Variable: number-bindings
 -- Scheme Variable: pair-bindings
 -- Scheme Variable: predicate-bindings
 -- Scheme Variable: procedure-bindings
 -- Scheme Variable: promise-bindings
 -- Scheme Variable: prompt-bindings
 -- Scheme Variable: regexp-bindings
 -- Scheme Variable: sort-bindings
 -- Scheme Variable: srfi-4-bindings
 -- Scheme Variable: string-bindings
 -- Scheme Variable: symbol-bindings
 -- Scheme Variable: unspecified-bindings
 -- Scheme Variable: variable-bindings
 -- Scheme Variable: vector-bindings
 -- Scheme Variable: version-bindings
     The components of ‘all-pure-bindings’.

 -- Scheme Variable: mutating-alist-bindings
 -- Scheme Variable: mutating-array-bindings
 -- Scheme Variable: mutating-bitvector-bindings
 -- Scheme Variable: mutating-fluid-bindings
 -- Scheme Variable: mutating-hash-bindings
 -- Scheme Variable: mutating-list-bindings
 -- Scheme Variable: mutating-pair-bindings
 -- Scheme Variable: mutating-sort-bindings
 -- Scheme Variable: mutating-srfi-4-bindings
 -- Scheme Variable: mutating-string-bindings
 -- Scheme Variable: mutating-variable-bindings
 -- Scheme Variable: mutating-vector-bindings
     The additional components of ‘all-pure-and-impure-bindings’.

   Finally, what do you do with a binding set?  What is a binding set
anyway?  ‘make-sandbox-module’ is here for you.

 -- Scheme Procedure: make-sandbox-module bindings
     Return a fresh module that only contains BINDINGS.

     The BINDINGS should be given as a list of import sets.  One import
     set is a list whose car names an interface, like ‘(ice-9 q)’, and
     whose cdr is a list of imports.  An import is either a bare symbol
     or a pair of ‘(OUT . IN)’, where OUT and IN are both symbols and
     denote the name under which a binding is exported from the module,
     and the name under which to make the binding available,
     respectively.

   So you see that binding sets are just lists, and
‘all-pure-and-impure-bindings’ is really just the result of appending
all of the component binding sets.


File: guile.info,  Node: REPL Servers,  Next: Cooperative REPL Servers,  Prev: Sandboxed Evaluation,  Up: Read/Load/Eval/Compile

6.16.14 REPL Servers
--------------------

The procedures in this section are provided by
     (use-modules (system repl server))

   When an application is written in Guile, it is often convenient to
allow the user to be able to interact with it by evaluating Scheme
expressions in a REPL.

   The procedures of this module allow you to spawn a “REPL server”,
which permits interaction over a local or TCP connection.  Guile itself
uses them internally to implement the ‘--listen’ switch, *note
Command-line Options::.

 -- Scheme Procedure: make-tcp-server-socket [#:host=#f] [#:addr]
          [#:port=37146]
     Return a stream socket bound to a given address ADDR and port
     number PORT.  If the HOST is given, and ADDR is not, then the HOST
     string is converted to an address.  If neither is given, we use the
     loopback address.

 -- Scheme Procedure: make-unix-domain-server-socket
          [#:path="/tmp/guile-socket"]
     Return a UNIX domain socket, bound to a given PATH.

 -- Scheme Procedure: run-server [server-socket]
 -- Scheme Procedure: spawn-server [server-socket]
     Create and run a REPL, making it available over the given
     SERVER-SOCKET.  If SERVER-SOCKET is not provided, it defaults to
     the socket created by calling ‘make-tcp-server-socket’ with no
     arguments.

     ‘run-server’ runs the server in the current thread, whereas
     ‘spawn-server’ runs the server in a new thread.

 -- Scheme Procedure: stop-server-and-clients!
     Closes the connection on all running server sockets.

     Please note that in the current implementation, the REPL threads
     are cancelled without unwinding their stacks.  If any of them are
     holding mutexes or are within a critical section, the results are
     unspecified.


File: guile.info,  Node: Cooperative REPL Servers,  Prev: REPL Servers,  Up: Read/Load/Eval/Compile

6.16.15 Cooperative REPL Servers
--------------------------------

The procedures in this section are provided by
     (use-modules (system repl coop-server))

   Whereas ordinary REPL servers run in their own threads (*note REPL
Servers::), sometimes it is more convenient to provide REPLs that run at
specified times within an existing thread, for example in programs
utilizing an event loop or in single-threaded programs.  This allows for
safe access and mutation of a program’s data structures from the REPL,
without concern for thread synchronization.

   Although the REPLs are run in the thread that calls
‘spawn-coop-repl-server’ and ‘poll-coop-repl-server’, dedicated threads
are spawned so that the calling thread is not blocked.  The spawned
threads read input for the REPLs and to listen for new connections.

   Cooperative REPL servers must be polled periodically to evaluate any
pending expressions by calling ‘poll-coop-repl-server’ with the object
returned from ‘spawn-coop-repl-server’.  The thread that calls
‘poll-coop-repl-server’ will be blocked for as long as the expression
takes to be evaluated or if the debugger is entered.

 -- Scheme Procedure: spawn-coop-repl-server [server-socket]
     Create and return a new cooperative REPL server object, and spawn a
     new thread to listen for connections on SERVER-SOCKET.  Proper
     functioning of the REPL server requires that
     ‘poll-coop-repl-server’ be called periodically on the returned
     server object.

 -- Scheme Procedure: poll-coop-repl-server coop-server
     Poll the cooperative REPL server COOP-SERVER and apply a pending
     operation if there is one, such as evaluating an expression typed
     at the REPL prompt.  This procedure must be called from the same
     thread that called ‘spawn-coop-repl-server’.


File: guile.info,  Node: Memory Management,  Next: Modules,  Prev: Read/Load/Eval/Compile,  Up: API Reference

6.17 Memory Management and Garbage Collection
=============================================

Guile uses a _garbage collector_ to manage most of its objects.  While
the garbage collector is designed to be mostly invisible, you sometimes
need to interact with it explicitly.

   See *note Garbage Collection:: for a general discussion of how
garbage collection relates to using Guile from C.

* Menu:

* Garbage Collection Functions::
* Memory Blocks::
* Weak References::
* Guardians::


File: guile.info,  Node: Garbage Collection Functions,  Next: Memory Blocks,  Up: Memory Management

6.17.1 Function related to Garbage Collection
---------------------------------------------

 -- Scheme Procedure: gc
 -- C Function: scm_gc ()
     Finds all of the “live” ‘SCM’ objects and reclaims for further use
     those that are no longer accessible.  You normally don’t need to
     call this function explicitly.  Its functionality is invoked
     automatically as needed.

 -- C Function: SCM scm_gc_protect_object (SCM OBJ)
     Protects OBJ from being freed by the garbage collector, when it
     otherwise might be.  When you are done with the object, call
     ‘scm_gc_unprotect_object’ on the object.  Calls to
     ‘scm_gc_protect_object’/‘scm_gc_unprotect_object’ can be nested,
     and the object remains protected until it has been unprotected as
     many times as it was protected.  It is an error to unprotect an
     object more times than it has been protected.  Returns the SCM
     object it was passed.

     Note that storing OBJ in a C global variable has the same
     effect(1).

 -- C Function: SCM scm_gc_unprotect_object (SCM OBJ)

     Unprotects an object from the garbage collector which was protected
     by ‘scm_gc_unprotect_object’.  Returns the SCM object it was
     passed.

 -- C Function: SCM scm_permanent_object (SCM OBJ)

     Similar to ‘scm_gc_protect_object’ in that it causes the collector
     to always mark the object, except that it should not be nested
     (only call ‘scm_permanent_object’ on an object once), and it has no
     corresponding unpermanent function.  Once an object is declared
     permanent, it will never be freed.  Returns the SCM object it was
     passed.

 -- C Macro: void scm_remember_upto_here_1 (SCM obj)
 -- C Macro: void scm_remember_upto_here_2 (SCM obj1, SCM obj2)
     Create a reference to the given object or objects, so they’re
     certain to be present on the stack or in a register and hence will
     not be freed by the garbage collector before this point.

     Note that these functions can only be applied to ordinary C local
     variables (ie. “automatics”).  Objects held in global or static
     variables or some malloced block or the like cannot be protected
     with this mechanism.

 -- Scheme Procedure: gc-stats
 -- C Function: scm_gc_stats ()
     Return an association list of statistics about Guile’s current use
     of storage.

 -- Scheme Procedure: gc-live-object-stats
 -- C Function: scm_gc_live_object_stats ()
     Return an alist of statistics of the current live objects.

 -- Function: void scm_gc_mark (SCM X)
     Mark the object X, and recurse on any objects X refers to.  If X’s
     mark bit is already set, return immediately.  This function must
     only be called during the mark-phase of garbage collection,
     typically from a smob _mark_ function.

   ---------- Footnotes ----------

   (1) In Guile up to version 1.8, C global variables were not visited
by the garbage collector in the mark phase; hence,
‘scm_gc_protect_object’ was the only way in C to prevent a Scheme object
from being freed.


File: guile.info,  Node: Memory Blocks,  Next: Weak References,  Prev: Garbage Collection Functions,  Up: Memory Management

6.17.2 Memory Blocks
--------------------

In C programs, dynamic management of memory blocks is normally done with
the functions malloc, realloc, and free.  Guile has additional functions
for dynamic memory allocation that are integrated into the garbage
collector and the error reporting system.

   Memory blocks that are associated with Scheme objects (for example a
foreign object) should be allocated with ‘scm_gc_malloc’ or
‘scm_gc_malloc_pointerless’.  These two functions will either return a
valid pointer or signal an error.  Memory blocks allocated this way may
be released explicitly; however, this is not strictly needed, and we
recommend _not_ calling ‘scm_gc_free’.  All memory allocated with
‘scm_gc_malloc’ or ‘scm_gc_malloc_pointerless’ is automatically
reclaimed when the garbage collector no longer sees any live reference
to it(1).

   When garbage collection occurs, Guile will visit the words in memory
allocated with ‘scm_gc_malloc’, looking for live pointers.  This means
that if ‘scm_gc_malloc’-allocated memory contains a pointer to some
other part of the memory, the garbage collector notices it and prevents
it from being reclaimed(2).  Conversely, memory allocated with
‘scm_gc_malloc_pointerless’ is assumed to be “pointer-less” and is not
scanned for pointers.

   For memory that is not associated with a Scheme object, you can use
‘scm_malloc’ instead of ‘malloc’.  Like ‘scm_gc_malloc’, it will either
return a valid pointer or signal an error.  However, it will not assume
that the new memory block can be freed by a garbage collection.  The
memory must be explicitly freed with ‘free’.

   There is also ‘scm_gc_realloc’ and ‘scm_realloc’, to be used in place
of ‘realloc’ when appropriate, and ‘scm_gc_calloc’ and ‘scm_calloc’, to
be used in place of ‘calloc’ when appropriate.

   The function ‘scm_dynwind_free’ can be useful when memory should be
freed with libc’s ‘free’ when leaving a dynwind context, *Note Dynamic
Wind::.

 -- C Function: void * scm_malloc (size_t SIZE)
 -- C Function: void * scm_calloc (size_t SIZE)
     Allocate SIZE bytes of memory and return a pointer to it.  When
     SIZE is 0, return ‘NULL’.  When not enough memory is available,
     signal an error.  This function runs the GC to free up some memory
     when it deems it appropriate.

     The memory is allocated by the libc ‘malloc’ function and can be
     freed with ‘free’.  There is no ‘scm_free’ function to go with
     ‘scm_malloc’ to make it easier to pass memory back and forth
     between different modules.

     The function ‘scm_calloc’ is similar to ‘scm_malloc’, but
     initializes the block of memory to zero as well.

     These functions will (indirectly) call
     ‘scm_gc_register_allocation’.

 -- C Function: void * scm_realloc (void *MEM, size_t NEW_SIZE)
     Change the size of the memory block at MEM to NEW_SIZE and return
     its new location.  When NEW_SIZE is 0, this is the same as calling
     ‘free’ on MEM and ‘NULL’ is returned.  When MEM is ‘NULL’, this
     function behaves like ‘scm_malloc’ and allocates a new block of
     size NEW_SIZE.

     When not enough memory is available, signal an error.  This
     function runs the GC to free up some memory when it deems it
     appropriate.

     This function will call ‘scm_gc_register_allocation’.

 -- C Function: void * scm_gc_malloc (size_t SIZE, const char *WHAT)
 -- C Function: void * scm_gc_malloc_pointerless (size_t SIZE, const
          char *WHAT)
 -- C Function: void * scm_gc_realloc (void *MEM, size_t OLD_SIZE,
          size_t NEW_SIZE, const char *WHAT);
 -- C Function: void * scm_gc_calloc (size_t SIZE, const char *WHAT)
     Allocate SIZE bytes of automatically-managed memory.  The memory is
     automatically freed when no longer referenced from any live memory
     block.

     When garbage collection occurs, Guile will visit the words in
     memory allocated with ‘scm_gc_malloc’ or ‘scm_gc_calloc’, looking
     for pointers to other memory allocations that are managed by the
     GC. In contrast, memory allocated by ‘scm_gc_malloc_pointerless’ is
     not scanned for pointers.

     The ‘scm_gc_realloc’ call preserves the “pointerlessness” of the
     memory area pointed to by MEM.  Note that you need to pass the old
     size of a reallocated memory block as well.  See below for a
     motivation.

 -- C Function: void scm_gc_free (void *MEM, size_t SIZE, const char
          *WHAT)
     Explicitly free the memory block pointed to by MEM, which was
     previously allocated by one of the above ‘scm_gc’ functions.  This
     function is almost always unnecessary, except for codebases that
     still need to compile on Guile 1.8.

     Note that you need to explicitly pass the SIZE parameter.  This is
     done since it should normally be easy to provide this parameter
     (for memory that is associated with GC controlled objects) and help
     keep the memory management overhead very low.  However, in Guile
     2.x, SIZE is always ignored.

 -- C Function: void scm_gc_register_allocation (size_t SIZE)
     Informs the garbage collector that SIZE bytes have been allocated,
     which the collector would otherwise not have known about.

     In general, Scheme will decide to collect garbage only after some
     amount of memory has been allocated.  Calling this function will
     make the Scheme garbage collector know about more allocation, and
     thus run more often (as appropriate).

     It is especially important to call this function when large
     unmanaged allocations, like images, may be freed by small Scheme
     allocations, like foreign objects.

 -- C Function: void scm_dynwind_free (void *mem)
     Equivalent to ‘scm_dynwind_unwind_handler (free, MEM,
     SCM_F_WIND_EXPLICITLY)’.  That is, the memory block at MEM will be
     freed (using ‘free’ from the C library) when the current dynwind is
     left.

 -- Scheme Procedure: malloc-stats
     Return an alist ((WHAT .  N) ...)  describing number of malloced
     objects.  WHAT is the second argument to ‘scm_gc_malloc’, N is the
     number of objects of that type currently allocated.

     This function is only available if the ‘GUILE_DEBUG_MALLOC’
     preprocessor macro was defined when Guile was compiled.

   ---------- Footnotes ----------

   (1) In Guile up to version 1.8, memory allocated with ‘scm_gc_malloc’
_had_ to be freed with ‘scm_gc_free’.

   (2) In Guile up to 1.8, memory allocated with ‘scm_gc_malloc’ was
_not_ visited by the collector in the mark phase.  Consequently, the GC
had to be told explicitly about pointers to live objects contained in
the memory block, e.g., via SMOB mark functions (*note
‘scm_set_smob_mark’: Smobs.)


File: guile.info,  Node: Weak References,  Next: Guardians,  Prev: Memory Blocks,  Up: Memory Management

6.17.3 Weak References
----------------------

[FIXME: This chapter is based on Mikael Djurfeldt’s answer to a question
by Michael Livshin.  Any mistakes are not theirs, of course.  ]

   Weak references let you attach bookkeeping information to data so
that the additional information automatically disappears when the
original data is no longer in use and gets garbage collected.  In a weak
key hash, the hash entry for that key disappears as soon as the key is
no longer referenced from anywhere else.  For weak value hashes, the
same happens as soon as the value is no longer in use.  Entries in a
doubly weak hash disappear when either the key or the value are not used
anywhere else anymore.

   Object properties offer the same kind of functionality as weak key
hashes in many situations.  (*note Object Properties::)

   Here’s an example (a little bit strained perhaps, but one of the
examples is actually used in Guile):

   Assume that you’re implementing a debugging system where you want to
associate information about filename and position of source code
expressions with the expressions themselves.

   Hashtables can be used for that, but if you use ordinary hash tables
it will be impossible for the scheme interpreter to "forget" old source
when, for example, a file is reloaded.

   To implement the mapping from source code expressions to positional
information it is necessary to use weak-key tables since we don’t want
the expressions to be remembered just because they are in our table.

   To implement a mapping from source file line numbers to source code
expressions you would use a weak-value table.

   To implement a mapping from source code expressions to the procedures
they constitute a doubly-weak table has to be used.

* Menu:

* Weak hash tables::
* Weak vectors::


File: guile.info,  Node: Weak hash tables,  Next: Weak vectors,  Up: Weak References

6.17.3.1 Weak hash tables
.........................

 -- Scheme Procedure: make-weak-key-hash-table [size]
 -- Scheme Procedure: make-weak-value-hash-table [size]
 -- Scheme Procedure: make-doubly-weak-hash-table [size]
 -- C Function: scm_make_weak_key_hash_table (size)
 -- C Function: scm_make_weak_value_hash_table (size)
 -- C Function: scm_make_doubly_weak_hash_table (size)
     Return a weak hash table with SIZE buckets.  As with any hash
     table, choosing a good size for the table requires some caution.

     You can modify weak hash tables in exactly the same way you would
     modify regular hash tables, with the exception of the routines that
     act on handles.  Weak tables have a different implementation behind
     the scenes that doesn’t have handles.  *note Hash Tables::, for
     more on ‘hashq-ref’ et al.

   Note that in a weak-key hash table, the reference to the value is
strong.  This means that if the value references the key, even
indirectly, the key will never be collected, which can lead to a memory
leak.  The reverse is true for weak value tables.

 -- Scheme Procedure: weak-key-hash-table? obj
 -- Scheme Procedure: weak-value-hash-table? obj
 -- Scheme Procedure: doubly-weak-hash-table? obj
 -- C Function: scm_weak_key_hash_table_p (obj)
 -- C Function: scm_weak_value_hash_table_p (obj)
 -- C Function: scm_doubly_weak_hash_table_p (obj)
     Return ‘#t’ if OBJ is the specified weak hash table.  Note that a
     doubly weak hash table is neither a weak key nor a weak value hash
     table.


File: guile.info,  Node: Weak vectors,  Prev: Weak hash tables,  Up: Weak References

6.17.3.2 Weak vectors
.....................

 -- Scheme Procedure: make-weak-vector size [fill]
 -- C Function: scm_make_weak_vector (size, fill)
     Return a weak vector with SIZE elements.  If the optional argument
     FILL is given, all entries in the vector will be set to FILL.  The
     default value for FILL is the empty list.

 -- Scheme Procedure: weak-vector elem ...
 -- Scheme Procedure: list->weak-vector l
 -- C Function: scm_weak_vector (l)
     Construct a weak vector from a list: ‘weak-vector’ uses the list of
     its arguments while ‘list->weak-vector’ uses its only argument L (a
     list) to construct a weak vector the same way ‘list->vector’ would.

 -- Scheme Procedure: weak-vector? obj
 -- C Function: scm_weak_vector_p (obj)
     Return ‘#t’ if OBJ is a weak vector.

 -- Scheme Procedure: weak-vector-ref wvect k
 -- C Function: scm_weak_vector_ref (wvect, k)
     Return the Kth element of the weak vector WVECT, or ‘#f’ if that
     element has been collected.

 -- Scheme Procedure: weak-vector-set! wvect k elt
 -- C Function: scm_weak_vector_set_x (wvect, k, elt)
     Set the Kth element of the weak vector WVECT to ELT.


File: guile.info,  Node: Guardians,  Prev: Weak References,  Up: Memory Management

6.17.4 Guardians
----------------

Guardians provide a way to be notified about objects that would
otherwise be collected as garbage.  Guarding them prevents the objects
from being collected and cleanup actions can be performed on them, for
example.

   See R. Kent Dybvig, Carl Bruggeman, and David Eby (1993) "Guardians
in a Generation-Based Garbage Collector".  ACM SIGPLAN Conference on
Programming Language Design and Implementation, June 1993.

 -- Scheme Procedure: make-guardian
 -- C Function: scm_make_guardian ()
     Create a new guardian.  A guardian protects a set of objects from
     garbage collection, allowing a program to apply cleanup or other
     actions.

     ‘make-guardian’ returns a procedure representing the guardian.
     Calling the guardian procedure with an argument adds the argument
     to the guardian’s set of protected objects.  Calling the guardian
     procedure without an argument returns one of the protected objects
     which are ready for garbage collection, or ‘#f’ if no such object
     is available.  Objects which are returned in this way are removed
     from the guardian.

     You can put a single object into a guardian more than once and you
     can put a single object into more than one guardian.  The object
     will then be returned multiple times by the guardian procedures.

     An object is eligible to be returned from a guardian when it is no
     longer referenced from outside any guardian.

     There is no guarantee about the order in which objects are returned
     from a guardian.  If you want to impose an order on finalization
     actions, for example, you can do that by keeping objects alive in
     some global data structure until they are no longer needed for
     finalizing other objects.

     Being an element in a weak vector, a key in a hash table with weak
     keys, or a value in a hash table with weak values does not prevent
     an object from being returned by a guardian.  But as long as an
     object can be returned from a guardian it will not be removed from
     such a weak vector or hash table.  In other words, a weak link does
     not prevent an object from being considered collectable, but being
     inside a guardian prevents a weak link from being broken.

     A key in a weak key hash table can be thought of as having a strong
     reference to its associated value as long as the key is accessible.
     Consequently, when the key is only accessible from within a
     guardian, the reference from the key to the value is also
     considered to be coming from within a guardian.  Thus, if there is
     no other reference to the value, it is eligible to be returned from
     a guardian.


File: guile.info,  Node: Modules,  Next: Foreign Function Interface,  Prev: Memory Management,  Up: API Reference

6.18 Modules
============

When programs become large, naming conflicts can occur when a function
or global variable defined in one file has the same name as a function
or global variable in another file.  Even just a _similarity_ between
function names can cause hard-to-find bugs, since a programmer might
type the wrong function name.

   The approach used to tackle this problem is called _information
encapsulation_, which consists of packaging functional units into a
given name space that is clearly separated from other name spaces.

   The language features that allow this are usually called _the module
system_ because programs are broken up into modules that are compiled
separately (or loaded separately in an interpreter).

   Older languages, like C, have limited support for name space
manipulation and protection.  In C a variable or function is public by
default, and can be made local to a module with the ‘static’ keyword.
But you cannot reference public variables and functions from another
module with different names.

   More advanced module systems have become a common feature in recently
designed languages: ML, Python, Perl, and Modula 3 all allow the
_renaming_ of objects from a foreign module, so they will not clutter
the global name space.

   In addition, Guile offers variables as first-class objects.  They can
be used for interacting with the module system.

* Menu:

* General Information about Modules::  Guile module basics.
* Using Guile Modules::         How to use existing modules.
* Creating Guile Modules::      How to package your code into modules.
* Modules and the File System:: Installing modules in the file system.
* R6RS Version References::     Using version numbers with modules.
* R6RS Libraries::              The library and import forms.
* Variables::                   First-class variables.
* Module System Reflection::    First-class modules.
* Declarative Modules::         Allowing Guile to reason about modules.
* Accessing Modules from C::    How to work with modules with C code.
* provide and require::         The SLIB feature mechanism.
* Environments::                R5RS top-level environments.


File: guile.info,  Node: General Information about Modules,  Next: Using Guile Modules,  Up: Modules

6.18.1 General Information about Modules
----------------------------------------

A Guile module can be thought of as a collection of named procedures,
variables and macros.  More precisely, it is a set of “bindings” of
symbols (names) to Scheme objects.

   Within a module, all bindings are visible.  Certain bindings can be
declared “public”, in which case they are added to the module’s
so-called “export list”; this set of public bindings is called the
module’s “public interface” (*note Creating Guile Modules::).

   A client module “uses” a providing module’s bindings by either
accessing the providing module’s public interface, or by building a
custom interface (and then accessing that).  In a custom interface, the
client module can “select” which bindings to access and can also
algorithmically “rename” bindings.  In contrast, when using the
providing module’s public interface, the entire export list is available
without renaming (*note Using Guile Modules::).

   All Guile modules have a unique “module name”, for example ‘(ice-9
popen)’ or ‘(srfi srfi-11)’.  Module names are lists of one or more
symbols.

   When Guile goes to use an interface from a module, for example
‘(ice-9 popen)’, Guile first looks to see if it has loaded ‘(ice-9
popen)’ for any reason.  If the module has not been loaded yet, Guile
searches a “load path” for a file that might define it, and loads that
file.

   The following subsections go into more detail on using, creating,
installing, and otherwise manipulating modules and the module system.


File: guile.info,  Node: Using Guile Modules,  Next: Creating Guile Modules,  Prev: General Information about Modules,  Up: Modules

6.18.2 Using Guile Modules
--------------------------

To use a Guile module is to access either its public interface or a
custom interface (*note General Information about Modules::).  Both
types of access are handled by the syntactic form ‘use-modules’, which
accepts one or more interface specifications and, upon evaluation,
arranges for those interfaces to be available to the current module.
This process may include locating and loading code for a given module if
that code has not yet been loaded, following ‘%load-path’ (*note Modules
and the File System::).

   An “interface specification” has one of two forms.  The first
variation is simply to name the module, in which case its public
interface is the one accessed.  For example:

     (use-modules (ice-9 popen))

   Here, the interface specification is ‘(ice-9 popen)’, and the result
is that the current module now has access to ‘open-pipe’, ‘close-pipe’,
‘open-input-pipe’, and so on (*note Pipes::).

   Note in the previous example that if the current module had already
defined ‘open-pipe’, that definition would be overwritten by the
definition in ‘(ice-9 popen)’.  For this reason (and others), there is a
second variation of interface specification that not only names a module
to be accessed, but also selects bindings from it and renames them to
suit the current module’s needs.  For example:

     (use-modules ((ice-9 popen)
                   #:select ((open-pipe . pipe-open) close-pipe)
                   #:renamer (symbol-prefix-proc 'unixy:)))

or more simply:

     (use-modules ((ice-9 popen)
                   #:select ((open-pipe . pipe-open) close-pipe)
                   #:prefix unixy:))

   Here, the interface specification is more complex than before, and
the result is that a custom interface with only two bindings is created
and subsequently accessed by the current module.  The mapping of old to
new names is as follows:

     (ice-9 popen) sees:             current module sees:
     open-pipe                       unixy:pipe-open
     close-pipe                      unixy:close-pipe

   This example also shows how to use the convenience procedure
‘symbol-prefix-proc’.

   You can also directly refer to bindings in a module by using the ‘@’
syntax.  For example, instead of using the ‘use-modules’ statement from
above and writing ‘unixy:pipe-open’ to refer to the ‘pipe-open’ from the
‘(ice-9 popen)’, you could also write ‘(@ (ice-9 popen) open-pipe)’.
Thus an alternative to the complete ‘use-modules’ statement would be

     (define unixy:pipe-open (@ (ice-9 popen) open-pipe))
     (define unixy:close-pipe (@ (ice-9 popen) close-pipe))

   There is also ‘@@’, which can be used like ‘@’, but does not check
whether the variable that is being accessed is actually exported.  Thus,
‘@@’ can be thought of as the impolite version of ‘@’ and should only be
used as a last resort or for debugging, for example.

   Note that just as with a ‘use-modules’ statement, any module that has
not yet been loaded will be loaded when referenced by a ‘@’ or ‘@@’
form.

   You can also use the ‘@’ and ‘@@’ syntaxes as the target of a ‘set!’
when the binding refers to a variable.

 -- Scheme Procedure: symbol-prefix-proc prefix-sym
     Return a procedure that prefixes its arg (a symbol) with
     PREFIX-SYM.

 -- syntax: use-modules spec ...
     Resolve each interface specification SPEC into an interface and
     arrange for these to be accessible by the current module.  The
     return value is unspecified.

     SPEC can be a list of symbols, in which case it names a module
     whose public interface is found and used.

     SPEC can also be of the form:

           (MODULE-NAME [#:select SELECTION]
                        [#:prefix PREFIX]
                        [#:renamer RENAMER])

     in which case a custom interface is newly created and used.
     MODULE-NAME is a list of symbols, as above; SELECTION is a list of
     selection-specs; PREFIX is a symbol that is prepended to imported
     names; and RENAMER is a procedure that takes a symbol and returns
     its new name.  A selection-spec is either a symbol or a pair of
     symbols ‘(ORIG . SEEN)’, where ORIG is the name in the used module
     and SEEN is the name in the using module.  Note that SEEN is also
     modified by PREFIX and RENAMER.

     The ‘#:select’, ‘#:prefix’, and ‘#:renamer’ clauses are optional.
     If all are omitted, the returned interface has no bindings.  If the
     ‘#:select’ clause is omitted, PREFIX and RENAMER operate on the
     used module’s public interface.

     In addition to the above, SPEC can also include a ‘#:version’
     clause, of the form:

           #:version VERSION-SPEC

     where VERSION-SPEC is an R6RS-compatible version reference.  An
     error will be signaled in the case in which a module with the same
     name has already been loaded, if that module specifies a version
     and that version is not compatible with VERSION-SPEC.  *Note R6RS
     Version References::, for more on version references.

     If the module name is not resolvable, ‘use-modules’ will signal an
     error.

 -- syntax: @ module-name binding-name
     Refer to the binding named BINDING-NAME in module MODULE-NAME.  The
     binding must have been exported by the module.

 -- syntax: @@ module-name binding-name
     Refer to the binding named BINDING-NAME in module MODULE-NAME.  The
     binding must not have been exported by the module.  This syntax is
     only intended for debugging purposes or as a last resort.  *Note
     Declarative Modules::, for some limitations on the use of ‘@@’.


File: guile.info,  Node: Creating Guile Modules,  Next: Modules and the File System,  Prev: Using Guile Modules,  Up: Modules

6.18.3 Creating Guile Modules
-----------------------------

When you want to create your own modules, you have to take the following
steps:

   • Create a Scheme source file and add all variables and procedures
     you wish to export, or which are required by the exported
     procedures.

   • Add a ‘define-module’ form at the beginning.

   • Export all bindings which should be in the public interface, either
     by using ‘define-public’ or ‘export’ (both documented below).

 -- syntax: define-module module-name option ...
     MODULE-NAME is a list of one or more symbols.

          (define-module (ice-9 popen))

     ‘define-module’ makes this module available to Guile programs under
     the given MODULE-NAME.

     OPTION ... are keyword/value pairs which specify more about the
     defined module.  The recognized options and their meaning are shown
     in the following table.

     ‘#:use-module INTERFACE-SPECIFICATION’
          Equivalent to a ‘(use-modules INTERFACE-SPECIFICATION)’ (*note
          Using Guile Modules::).

     ‘#:autoload MODULE SYMBOL-LIST’
          Load MODULE when any of SYMBOL-LIST are accessed.  For
          example,

               (define-module (my mod)
                 #:autoload (srfi srfi-1) (partition delete-duplicates))
               ...
               (when something
                 (set! foo (delete-duplicates ...)))

          When a module is autoloaded, only the bindings in SYMBOL-LIST
          become available(1).

          An autoload is a good way to put off loading a big module
          until it’s really needed, for instance for faster startup or
          if it will only be needed in certain circumstances.

     ‘#:export LIST’
          Export all identifiers in LIST which must be a list of symbols
          or pairs of symbols.  This is equivalent to ‘(export LIST)’ in
          the module body.

     ‘#:re-export LIST’
          Re-export all identifiers in LIST which must be a list of
          symbols or pairs of symbols.  The symbols in LIST must be
          imported by the current module from other modules.  This is
          equivalent to ‘re-export’ below.

     ‘#:replace LIST’
          Export all identifiers in LIST (a list of symbols or pairs of
          symbols) and mark them as “replacing bindings”.  In the module
          user’s name space, this will have the effect of replacing any
          binding with the same name that is not also “replacing”.
          Normally a replacement results in an “override” warning
          message, ‘#:replace’ avoids that.

          In general, a module that exports a binding for which the
          ‘(guile)’ module already has a definition should use
          ‘#:replace’ instead of ‘#:export’.  ‘#:replace’, in a sense,
          lets Guile know that the module _purposefully_ replaces a core
          binding.  It is important to note, however, that this binding
          replacement is confined to the name space of the module user.
          In other words, the value of the core binding in question
          remains unchanged for other modules.

          Note that although it is often a good idea for the replaced
          binding to remain compatible with a binding in ‘(guile)’, to
          avoid surprising the user, sometimes the bindings will be
          incompatible.  For example, SRFI-19 exports its own version of
          ‘current-time’ (*note SRFI-19 Time::) which is not compatible
          with the core ‘current-time’ function (*note Time::).  Guile
          assumes that a user importing a module knows what she is
          doing, and uses ‘#:replace’ for this binding rather than
          ‘#:export’.

          A ‘#:replace’ clause is equivalent to ‘(export! LIST)’ in the
          module body.

          The ‘#:duplicates’ (see below) provides fine-grain control
          about duplicate binding handling on the module-user side.

     ‘#:re-export-and-replace LIST’
          Like ‘#:re-export’, but also marking the bindings as
          replacements in the sense of ‘#:replace’.

     ‘#:version LIST’
          Specify a version for the module in the form of LIST, a list
          of zero or more exact, nonnegative integers.  The
          corresponding ‘#:version’ option in the ‘use-modules’ form
          allows callers to restrict the value of this option in various
          ways.

     ‘#:duplicates LIST’
          Tell Guile to handle duplicate bindings for the bindings
          imported by the current module according to the policy defined
          by LIST, a list of symbols.  LIST must contain symbols
          representing a duplicate binding handling policy chosen among
          the following:

          ‘check’
               Raises an error when a binding is imported from more than
               one place.
          ‘warn’
               Issue a warning when a binding is imported from more than
               one place and leave the responsibility of actually
               handling the duplication to the next duplicate binding
               handler.
          ‘replace’
               When a new binding is imported that has the same name as
               a previously imported binding, then do the following:

                 1. If the old binding was said to be “replacing” (via
                    the ‘#:replace’ option above) and the new binding is
                    not replacing, the keep the old binding.
                 2. If the old binding was not said to be replacing and
                    the new binding is replacing, then replace the old
                    binding with the new one.
                 3. If neither the old nor the new binding is replacing,
                    then keep the old one.

          ‘warn-override-core’
               Issue a warning when a core binding is being overwritten
               and actually override the core binding with the new one.
          ‘first’
               In case of duplicate bindings, the firstly imported
               binding is always the one which is kept.
          ‘last’
               In case of duplicate bindings, the lastly imported
               binding is always the one which is kept.
          ‘noop’
               In case of duplicate bindings, leave the responsibility
               to the next duplicate handler.

          If LIST contains more than one symbol, then the duplicate
          binding handlers which appear first will be used first when
          resolving a duplicate binding situation.  As mentioned above,
          some resolution policies may explicitly leave the
          responsibility of handling the duplication to the next handler
          in LIST.

          If GOOPS has been loaded before the ‘#:duplicates’ clause is
          processed, there are additional strategies available for
          dealing with generic functions.  *Note Merging Generics::, for
          more information.

          The default duplicate binding resolution policy is given by
          the ‘default-duplicate-binding-handler’ procedure, and is

               (replace warn-override-core warn last)

     ‘#:pure’
          Create a “pure” module, that is a module which does not
          contain any of the standard procedure bindings except for the
          syntax forms.  This is useful if you want to create “safe”
          modules, that is modules which do not know anything about
          dangerous procedures.

 -- syntax: export variable ...
     Add all VARIABLEs (which must be symbols or pairs of symbols) to
     the list of exported bindings of the current module.  If VARIABLE
     is a pair, its ‘car’ gives the name of the variable as seen by the
     current module and its ‘cdr’ specifies a name for the binding in
     the current module’s public interface.

 -- syntax: define-public ...
     Equivalent to ‘(begin (define foo ...) (export foo))’.

 -- syntax: re-export variable ...
     Add all VARIABLEs (which must be symbols or pairs of symbols) to
     the list of re-exported bindings of the current module.  Pairs of
     symbols are handled as in ‘export’.  Re-exported bindings must be
     imported by the current module from some other module.

 -- syntax: export! variable ...
     Like ‘export’, but marking the exported variables as replacing.
     Using a module with replacing bindings will cause any existing
     bindings to be replaced without issuing any warnings.  See the
     discussion of ‘#:replace’ above.

   ---------- Footnotes ----------

   (1) In Guile 2.2 and earlier, _all_ the module bindings would become
available; SYMBOL-LIST was just the list of bindings that will first
trigger the load.


File: guile.info,  Node: Modules and the File System,  Next: R6RS Version References,  Prev: Creating Guile Modules,  Up: Modules

6.18.4 Modules and the File System
----------------------------------

Typical programs only use a small subset of modules installed on a Guile
system.  In order to keep startup time down, Guile only loads modules
when a program uses them, on demand.

   When a program evaluates ‘(use-modules (ice-9 popen))’, and the
module is not loaded, Guile searches for a conventionally-named file
from in the “load path”.

   In this case, loading ‘(ice-9 popen)’ will eventually cause Guile to
run ‘(primitive-load-path "ice-9/popen")’.  ‘primitive-load-path’ will
search for a file ‘ice-9/popen’ in the ‘%load-path’ (*note Load
Paths::).  For each directory in ‘%load-path’, Guile will try to find
the file name, concatenated with the extensions from ‘%load-extensions’.
By default, this will cause Guile to ‘stat’ ‘ice-9/popen.scm’, and then
‘ice-9/popen’.  *Note Load Paths::, for more on ‘primitive-load-path’.

   If a corresponding compiled ‘.go’ file is found in the
‘%load-compiled-path’ or in the fallback path, and is as fresh as the
source file, it will be loaded instead of the source file.  If no
compiled file is found, Guile may try to compile the source file and
cache away the resulting ‘.go’ file.  *Note Compilation::, for more on
compilation.

   Once Guile finds a suitable source or compiled file is found, the
file will be loaded.  If, after loading the file, the module under
consideration is still not defined, Guile will signal an error.

   For more information on where and how to install Scheme modules,
*Note Installing Site Packages::.


File: guile.info,  Node: R6RS Version References,  Next: R6RS Libraries,  Prev: Modules and the File System,  Up: Modules

6.18.5 R6RS Version References
------------------------------

Guile’s module system includes support for locating modules based on a
declared version specifier of the same form as the one described in R6RS
(*note R6RS Library Form: (r6rs)Library form.).  By using the
‘#:version’ keyword in a ‘define-module’ form, a module may specify a
version as a list of zero or more exact, nonnegative integers.

   This version can then be used to locate the module during the module
search process.  Client modules and callers of the ‘use-modules’
function may specify constraints on the versions of target modules by
providing a “version reference”, which has one of the following forms:

      (SUB-VERSION-REFERENCE ...)
      (and VERSION-REFERENCE ...)
      (or VERSION-REFERENCE ...)
      (not VERSION-REFERENCE)

   in which SUB-VERSION-REFERENCE is in turn one of:

      (SUB-VERSION)
      (>= SUB-VERSION)
      (<= SUB-VERSION)
      (and SUB-VERSION-REFERENCE ...)
      (or SUB-VERSION-REFERENCE ...)
      (not SUB-VERSION-REFERENCE)

   in which SUB-VERSION is an exact, nonnegative integer as above.  A
version reference matches a declared module version if each element of
the version reference matches a corresponding element of the module
version, according to the following rules:

   • The ‘and’ sub-form matches a version or version element if every
     element in the tail of the sub-form matches the specified version
     or version element.

   • The ‘or’ sub-form matches a version or version element if any
     element in the tail of the sub-form matches the specified version
     or version element.

   • The ‘not’ sub-form matches a version or version element if the tail
     of the sub-form does not match the version or version element.

   • The ‘>=’ sub-form matches a version element if the element is
     greater than or equal to the SUB-VERSION in the tail of the
     sub-form.

   • The ‘<=’ sub-form matches a version element if the version is less
     than or equal to the SUB-VERSION in the tail of the sub-form.

   • A SUB-VERSION matches a version element if one is EQV? to the
     other.

   For example, a module declared as:

      (define-module (mylib mymodule) #:version (1 2 0))

   would be successfully loaded by any of the following ‘use-modules’
expressions:

      (use-modules ((mylib mymodule) #:version (1 2 (>= 0))))
      (use-modules ((mylib mymodule) #:version (or (1 2 0) (1 2 1))))
      (use-modules ((mylib mymodule) #:version ((and (>= 1) (not 2)) 2 0)))


File: guile.info,  Node: R6RS Libraries,  Next: Variables,  Prev: R6RS Version References,  Up: Modules

6.18.6 R6RS Libraries
---------------------

In addition to the API described in the previous sections, you also have
the option to create modules using the portable ‘library’ form described
in R6RS (*note R6RS Library Form: (r6rs)Library form.), and to import
libraries created in this format by other programmers.  Guile’s R6RS
library implementation takes advantage of the flexibility built into the
module system by expanding the R6RS library form into a corresponding
Guile ‘define-module’ form that specifies equivalent import and export
requirements and includes the same body expressions.  The library
expression:

       (library (mylib (1 2))
         (export mybinding)
         (import (otherlib (3))))

   is equivalent to the module definition:

       (define-module (mylib)
         #:version (1 2)
         #:use-module ((otherlib) #:version (3))
         #:export (mybinding))

   Central to the mechanics of R6RS libraries is the concept of import
and export “levels”, which control the visibility of bindings at various
phases of a library’s lifecycle — macros necessary to expand forms in
the library’s body need to be available at expand time; variables used
in the body of a procedure exported by the library must be available at
runtime.  R6RS specifies the optional ‘for’ sub-form of an _import set_
specification (see below) as a mechanism by which a library author can
indicate that a particular library import should take place at a
particular phase with respect to the lifecycle of the importing library.

   Guile’s library implementation uses a technique called “implicit
phasing” (first described by Abdulaziz Ghuloum and R. Kent Dybvig),
which allows the expander and compiler to automatically determine the
necessary visibility of a binding imported from another library.  As
such, the ‘for’ sub-form described below is ignored by Guile (but may be
required by Schemes in which phasing is explicit).

 -- Scheme Syntax: library name (export export-spec ...) (import
          import-spec ...) body ...
     Defines a new library with the specified name, exports, and
     imports, and evaluates the specified body expressions in this
     library’s environment.

     The library NAME is a non-empty list of identifiers, optionally
     ending with a version specification of the form described above
     (*note Creating Guile Modules::).

     Each EXPORT-SPEC is the name of a variable defined or imported by
     the library, or must take the form ‘(rename (internal-name
     external-name) ...)’, where the identifier INTERNAL-NAME names a
     variable defined or imported by the library and EXTERNAL-NAME is
     the name by which the variable is seen by importing libraries.

     Each IMPORT-SPEC must be either an “import set” (see below) or must
     be of the form ‘(for import-set import-level ...)’, where each
     IMPORT-LEVEL is one of:

            run
            expand
            (meta LEVEL)

     where LEVEL is an integer.  Note that since Guile does not require
     explicit phase specification, any IMPORT-SETs found inside of ‘for’
     sub-forms will be “unwrapped” during expansion and processed as if
     they had been specified directly.

     Import sets in turn take one of the following forms:

            LIBRARY-REFERENCE
            (library LIBRARY-REFERENCE)
            (only IMPORT-SET IDENTIFIER ...)
            (except IMPORT-SET IDENTIFIER ...)
            (prefix IMPORT-SET IDENTIFIER)
            (rename IMPORT-SET (INTERNAL-IDENTIFIER EXTERNAL-IDENTIFIER) ...)

     where LIBRARY-REFERENCE is a non-empty list of identifiers ending
     with an optional version reference (*note R6RS Version
     References::), and the other sub-forms have the following
     semantics, defined recursively on nested IMPORT-SETs:

        • The ‘library’ sub-form is used to specify libraries for import
          whose names begin with the identifier “library.”

        • The ‘only’ sub-form imports only the specified IDENTIFIERs
          from the given IMPORT-SET.

        • The ‘except’ sub-form imports all of the bindings exported by
          IMPORT-SET except for those that appear in the specified list
          of IDENTIFIERs.

        • The ‘prefix’ sub-form imports all of the bindings exported by
          IMPORT-SET, first prefixing them with the specified
          IDENTIFIER.

        • The ‘rename’ sub-form imports all of the identifiers exported
          by IMPORT-SET.  The binding for each INTERNAL-IDENTIFIER among
          these identifiers is made visible to the importing library as
          the corresponding EXTERNAL-IDENTIFIER; all other bindings are
          imported using the names provided by IMPORT-SET.

     Note that because Guile translates R6RS libraries into module
     definitions, an import specification may be used to declare a
     dependency on a native Guile module — although doing so may make
     your libraries less portable to other Schemes.

 -- Scheme Syntax: import import-spec ...
     Import into the current environment the libraries specified by the
     given import specifications, where each IMPORT-SPEC takes the same
     form as in the ‘library’ form described above.


File: guile.info,  Node: Variables,  Next: Module System Reflection,  Prev: R6RS Libraries,  Up: Modules

6.18.7 Variables
----------------

Each module has its own hash table, sometimes known as an “obarray”,
that maps the names defined in that module to their corresponding
variable objects.

   A variable is a box-like object that can hold any Scheme value.  It
is said to be “undefined” if its box holds a special Scheme value that
denotes undefined-ness (which is different from all other Scheme values,
including for example ‘#f’); otherwise the variable is “defined”.

   On its own, a variable object is anonymous.  A variable is said to be
“bound” when it is associated with a name in some way, usually a symbol
in a module obarray.  When this happens, the name is said to be bound to
the variable, in that module.

   (That’s the theory, anyway.  In practice, defined-ness and bound-ness
sometimes get confused, because Lisp and Scheme implementations have
often conflated — or deliberately drawn no distinction between — a name
that is unbound and a name that is bound to a variable whose value is
undefined.  We will try to be clear about the difference and explain any
confusion where it is unavoidable.)

   Variables do not have a read syntax.  Most commonly they are created
and bound implicitly by ‘define’ expressions: a top-level ‘define’
expression of the form

     (define NAME VALUE)

creates a variable with initial value VALUE and binds it to the name
NAME in the current module.  But they can also be created dynamically by
calling one of the constructor procedures ‘make-variable’ and
‘make-undefined-variable’.

 -- Scheme Procedure: make-undefined-variable
 -- C Function: scm_make_undefined_variable ()
     Return a variable that is initially unbound.

 -- Scheme Procedure: make-variable init
 -- C Function: scm_make_variable (init)
     Return a variable initialized to value INIT.

 -- Scheme Procedure: variable-bound? var
 -- C Function: scm_variable_bound_p (var)
     Return ‘#t’ if VAR is bound to a value, or ‘#f’ otherwise.  Throws
     an error if VAR is not a variable object.

 -- Scheme Procedure: variable-ref var
 -- C Function: scm_variable_ref (var)
     Dereference VAR and return its value.  VAR must be a variable
     object; see ‘make-variable’ and ‘make-undefined-variable’.

 -- Scheme Procedure: variable-set! var val
 -- C Function: scm_variable_set_x (var, val)
     Set the value of the variable VAR to VAL.  VAR must be a variable
     object, VAL can be any value.  Return an unspecified value.

 -- Scheme Procedure: variable-unset! var
 -- C Function: scm_variable_unset_x (var)
     Unset the value of the variable VAR, leaving VAR unbound.

 -- Scheme Procedure: variable? obj
 -- C Function: scm_variable_p (obj)
     Return ‘#t’ if OBJ is a variable object, else return ‘#f’.


File: guile.info,  Node: Module System Reflection,  Next: Declarative Modules,  Prev: Variables,  Up: Modules

6.18.8 Module System Reflection
-------------------------------

The previous sections have described a declarative view of the module
system.  You can also work with it programmatically by accessing and
modifying various parts of the Scheme objects that Guile uses to
implement the module system.

   At any time, there is a “current module”.  This module is the one
where a top-level ‘define’ and similar syntax will add new bindings.
You can find other module objects with ‘resolve-module’, for example.

   These module objects can be used as the second argument to ‘eval’.

 -- Scheme Procedure: current-module
 -- C Function: scm_current_module ()
     Return the current module object.

 -- Scheme Procedure: set-current-module module
 -- C Function: scm_set_current_module (module)
     Set the current module to MODULE and return the previous current
     module.

 -- Scheme Procedure: save-module-excursion thunk
     Call THUNK within a ‘dynamic-wind’ such that the module that is
     current at invocation time is restored when THUNK’s dynamic extent
     is left (*note Dynamic Wind::).

     More precisely, if THUNK escapes non-locally, the current module
     (at the time of escape) is saved, and the original current module
     (at the time THUNK’s dynamic extent was last entered) is restored.
     If THUNK’s dynamic extent is re-entered, then the current module is
     saved, and the previously saved inner module is set current again.

 -- Scheme Procedure: resolve-module name [autoload=#t] [version=#f]
          [#:ensure=#t]
 -- C Function: scm_resolve_module (name)
     Find the module named NAME and return it.  When it has not already
     been defined and AUTOLOAD is true, try to auto-load it.  When it
     can’t be found that way either, create an empty module if ENSURE is
     true, otherwise return ‘#f’.  If VERSION is true, ensure that the
     resulting module is compatible with the given version reference
     (*note R6RS Version References::).  The name is a list of symbols.

 -- Scheme Procedure: resolve-interface name [#:select=#f] [#:hide='()]
          [#:prefix=#f] [#:renamer=#f] [#:version=#f]
     Find the module named NAME as with ‘resolve-module’ and return its
     interface.  The interface of a module is also a module object, but
     it contains only the exported bindings.

 -- Scheme Procedure: module-uses module
     Return a list of the interfaces used by MODULE.

 -- Scheme Procedure: module-use! module interface
     Add INTERFACE to the front of the use-list of MODULE.  Both
     arguments should be module objects, and INTERFACE should very
     likely be a module returned by ‘resolve-interface’.

 -- Scheme Procedure: reload-module module
     Revisit the source file that corresponds to MODULE.  Raises an
     error if no source file is associated with the given module.

   As mentioned in the previous section, modules contain a mapping
between identifiers (as symbols) and storage locations (as variables).
Guile defines a number of procedures to allow access to this mapping.
If you are programming in C, *note Accessing Modules from C::.

 -- Scheme Procedure: module-variable module name
     Return the variable bound to NAME (a symbol) in MODULE, or ‘#f’ if
     NAME is unbound.

 -- Scheme Procedure: module-add! module name var
     Define a new binding between NAME (a symbol) and VAR (a variable)
     in MODULE.

 -- Scheme Procedure: module-ref module name
     Look up the value bound to NAME in MODULE.  Like ‘module-variable’,
     but also does a ‘variable-ref’ on the resulting variable, raising
     an error if NAME is unbound.

 -- Scheme Procedure: module-define! module name value
     Locally bind NAME to VALUE in MODULE.  If NAME was already locally
     bound in MODULE, i.e., defined locally and not by an imported
     module, the value stored in the existing variable will be updated.
     Otherwise, a new variable will be added to the module, via
     ‘module-add!’.

 -- Scheme Procedure: module-set! module name value
     Update the binding of NAME in MODULE to VALUE, raising an error if
     NAME is not already bound in MODULE.

   There are many other reflective procedures available in the default
environment.  If you find yourself using one of them, please contact the
Guile developers so that we can commit to stability for that interface.


File: guile.info,  Node: Declarative Modules,  Next: Accessing Modules from C,  Prev: Module System Reflection,  Up: Modules

6.18.9 Declarative Modules
--------------------------

The first-class access to modules and module variables described in the
previous subsection is very powerful and allows Guile users to build
many tools to dynamically learn things about their Guile systems.
However, as Scheme godparent Mathias Felleisen wrote in “On the
Expressive Power of Programming Languages”, a more expressive language
is necessarily harder to reason about.  There are transformations that
Guile’s compiler would like to make which can’t be done if every
top-level definition is subject to mutation at any time.

   Consider this module:

     (define-module (boxes)
       #:export (make-box box-ref box-set! box-swap!))

     (define (make-box x) (list x))
     (define (box-ref box) (car box))
     (define (box-set! box x) (set-car! box x))
     (define (box-swap! box x)
       (let ((y (box-ref box)))
         (box-set! box x)
         y))

   Ideally you’d like for the ‘box-ref’ in ‘box-swap!’ to be inlined to
‘car’.  Guile’s compiler can do this, but only if it knows that
‘box-ref’’s definition is what it appears to be in the text.  However,
in the general case it could be that a programmer could reach into the
‘(boxes)’ module at any time and change the value of ‘box-ref’.

   To allow Guile to reason about the values of top-levels from a
module, a module can be marked as “declarative”.  This flag applies only
to the subset of top-level definitions that are themselves declarative:
those that are defined within the compilation unit, and not assigned
(‘set!’) or redefined within the compilation unit.

   To explicitly mark a module as being declarative, pass the
‘#:declarative?’ keyword argument when declaring a module:

     (define-module (boxes)
       #:export (make-box box-ref box-set! box-swap!)
       #:declarative? #t)

   By default, modules are compiled declaratively if the
‘user-modules-declarative?’ parameter is true when the module is
compiled.

 -- Scheme Parameter: user-modules-declarative?
     A boolean indicating whether definitions in modules created by
     ‘define-module’ or implicitly as part of a compilation unit without
     an explicit module can be treated as declarative.

   Because it’s usually what you want, the default value of
‘user-modules-declarative?’ is ‘#t’.

Should I Mark My Module As Declarative?
.......................................

In the vast majority of use cases, declarative modules are what you
want.  However, there are exceptions.

   Consider the ‘(boxes)’ module above.  Let’s say you want to be able
to go in and change the definition of ‘box-set!’ at run-time:

     scheme@(guile-user)> (use-modules (boxes))
     scheme@(guile-user)> ,module boxes
     scheme@(boxes)> (define (box-set! x y) (set-car! x (pk y)))

   However, considering that ‘(boxes)’ is a declarative module, it could
be that ‘box-swap!’ inlined the call to ‘box-set!’ – so it may be that
you are surprised if you call ‘(box-swap! x y)’ and you don’t see the
new definition being used.  (Note, however, that Guile has no guarantees
about what definitions its compiler will or will not inline.)

   If you want to allow the definition of ‘box-set!’ to be changed and
to have all of its uses updated, then probably the best option is to
edit the module and reload the whole thing:

     scheme@(guile-user)> ,reload (boxes)

   The advantage of the reloading approach is that you maintain the
optimizations that declarative modules enable, while also being able to
live-update the code.  If the module keeps precious program state, those
definitions can be marked as ‘define-once’ to prevent reloads from
overwriting them.  *Note Top Level::, for more on ‘define-once’.
Incidentally, ‘define-once’ also prevents declarative-definition
optimizations, so if there’s a limited subset of redefinable bindings,
‘define-once’ could be an interesting tool to mark those definitions as
works-in-progress for interactive program development.

   To users, whether a module is declarative or not is mostly
immaterial: besides normal use via ‘use-modules’, users can reference
and redefine public or private bindings programmatically or
interactively.  The only difference is that changing a declarative
definition may not change all of its uses.  If this use-case is
important to you, and if reloading whole modules is insufficient, then
you can mark all definitions in a module as non-declarative by adding
‘#:declarative? #f’ to the module definition.

   The default of whether modules are declarative or not can be
controlled via the ‘(user-modules-declarative?)’ parameter mentioned
above, but care should be taken to set this parameter when the modules
are compiled, e.g.  via ‘(eval-when (expand) (user-modules-declarative?
#f))’.  *Note Eval When::.

   Alternately you can prevent declarative-definition optimizations by
compiling at the ‘-O1’ optimization level instead of the default ‘-O2’,
or via explicitly passing ‘-Ono-letrectify’ to the ‘guild compile’
invocation.  *Note Compilation::, for more on compiler options.

   One final note.  Currently, definitions from declarative modules can
only be inlined within the module they are defined in, and within a
compilation unit.  This may change in the future to allow Guile to
inline imported declarative definitions as well (cross-module inlining).
To Guile, whether a definition is inlinable or not is a property of the
definition, not its use.  We hope to improve compiler tooling in the
future to allow the user to identify definitions that are out of date
when a declarative binding is redefined.


File: guile.info,  Node: Accessing Modules from C,  Next: provide and require,  Prev: Declarative Modules,  Up: Modules

6.18.10 Accessing Modules from C
--------------------------------

The last sections have described how modules are used in Scheme code,
which is the recommended way of creating and accessing modules.  You can
also work with modules from C, but it is more cumbersome.

   The following procedures are available.

 -- C Function: SCM scm_c_call_with_current_module (SCM MODULE, SCM
          (*FUNC)(void *), void *DATA)
     Call FUNC and make MODULE the current module during the call.  The
     argument DATA is passed to FUNC.  The return value of
     ‘scm_c_call_with_current_module’ is the return value of FUNC.

 -- C Function: SCM scm_public_variable (SCM MODULE_NAME, SCM NAME)
 -- C Function: SCM scm_c_public_variable (const char *MODULE_NAME,
          const char *NAME)
     Find a the variable bound to the symbol NAME in the public
     interface of the module named MODULE_NAME.

     MODULE_NAME should be a list of symbols, when represented as a
     Scheme object, or a space-separated string, in the ‘const char *’
     case.  See ‘scm_c_define_module’ below, for more examples.

     Signals an error if no module was found with the given name.  If
     NAME is not bound in the module, just returns ‘#f’.

 -- C Function: SCM scm_private_variable (SCM MODULE_NAME, SCM NAME)
 -- C Function: SCM scm_c_private_variable (const char *MODULE_NAME,
          const char *NAME)
     Like ‘scm_public_variable’, but looks in the internals of the
     module named MODULE_NAME instead of the public interface.
     Logically, these procedures should only be called on modules you
     write.

 -- C Function: SCM scm_public_lookup (SCM MODULE_NAME, SCM NAME)
 -- C Function: SCM scm_c_public_lookup (const char *MODULE_NAME, const
          char *NAME)
 -- C Function: SCM scm_private_lookup (SCM MODULE_NAME, SCM NAME)
 -- C Function: SCM scm_c_private_lookup (const char *MODULE_NAME, const
          char *NAME)
     Like ‘scm_public_variable’ or ‘scm_private_variable’, but if the
     NAME is not bound in the module, signals an error.  Returns a
     variable, always.

          static SCM eval_string_var;

          /* NOTE: It is important that the call to 'my_init'
             happens-before all calls to 'my_eval_string'. */
          void my_init (void)
          {
            eval_string_var = scm_c_public_lookup ("ice-9 eval-string",
                                                   "eval-string");
          }

          SCM my_eval_string (SCM str)
          {
            return scm_call_1 (scm_variable_ref (eval_string_var), str);
          }

 -- C Function: SCM scm_public_ref (SCM MODULE_NAME, SCM NAME)
 -- C Function: SCM scm_c_public_ref (const char *MODULE_NAME, const
          char *NAME)
 -- C Function: SCM scm_private_ref (SCM MODULE_NAME, SCM NAME)
 -- C Function: SCM scm_c_private_ref (const char *MODULE_NAME, const
          char *NAME)
     Like ‘scm_public_lookup’ or ‘scm_private_lookup’, but additionally
     dereferences the variable.  If the variable object is unbound,
     signals an error.  Returns the value bound to NAME in MODULE_NAME.

   In addition, there are a number of other lookup-related procedures.
We suggest that you use the ‘scm_public_’ and ‘scm_private_’ family of
procedures instead, if possible.

 -- C Function: SCM scm_c_lookup (const char *NAME)
     Return the variable bound to the symbol indicated by NAME in the
     current module.  If there is no such binding or the symbol is not
     bound to a variable, signal an error.

 -- C Function: SCM scm_lookup (SCM NAME)
     Like ‘scm_c_lookup’, but the symbol is specified directly.

 -- C Function: SCM scm_c_module_lookup (SCM MODULE, const char *NAME)
 -- C Function: SCM scm_module_lookup (SCM MODULE, SCM NAME)
     Like ‘scm_c_lookup’ and ‘scm_lookup’, but the specified module is
     used instead of the current one.

 -- C Function: SCM scm_module_variable (SCM MODULE, SCM NAME)
     Like ‘scm_module_lookup’, but if the binding does not exist, just
     returns ‘#f’ instead of raising an error.

   To define a value, use ‘scm_define’:

 -- C Function: SCM scm_c_define (const char *NAME, SCM VAL)
     Bind the symbol indicated by NAME to a variable in the current
     module and set that variable to VAL.  When NAME is already bound to
     a variable, use that.  Else create a new variable.

 -- C Function: SCM scm_define (SCM NAME, SCM VAL)
     Like ‘scm_c_define’, but the symbol is specified directly.

 -- C Function: SCM scm_c_module_define (SCM MODULE, const char *NAME,
          SCM VAL)
 -- C Function: SCM scm_module_define (SCM MODULE, SCM NAME, SCM VAL)
     Like ‘scm_c_define’ and ‘scm_define’, but the specified module is
     used instead of the current one.

   In some rare cases, you may need to access the variable that
‘scm_module_define’ would have accessed, without changing the binding of
the existing variable, if one is present.  In that case, use
‘scm_module_ensure_local_variable’:

 -- C Function: SCM scm_module_ensure_local_variable (SCM MODULE, SCM
          SYM)
     Like ‘scm_module_define’, but if the SYM is already locally bound
     in that module, the variable’s existing binding is not reset.
     Returns a variable.

 -- C Function: SCM scm_module_reverse_lookup (SCM MODULE, SCM VARIABLE)
     Find the symbol that is bound to VARIABLE in MODULE.  When no such
     binding is found, return ‘#f’.

 -- C Function: SCM scm_c_define_module (const char *NAME, void
          (*INIT)(void *), void *DATA)
     Define a new module named NAME and make it current while INIT is
     called, passing it DATA.  Return the module.

     The parameter NAME is a string with the symbols that make up the
     module name, separated by spaces.  For example, ‘"foo bar"’ names
     the module ‘(foo bar)’.

     When there already exists a module named NAME, it is used
     unchanged, otherwise, an empty module is created.

 -- C Function: SCM scm_c_resolve_module (const char *NAME)
     Find the module name NAME and return it.  When it has not already
     been defined, try to auto-load it.  When it can’t be found that way
     either, create an empty module.  The name is interpreted as for
     ‘scm_c_define_module’.

 -- C Function: SCM scm_c_use_module (const char *NAME)
     Add the module named NAME to the uses list of the current module,
     as with ‘(use-modules NAME)’.  The name is interpreted as for
     ‘scm_c_define_module’.

 -- C Function: void scm_c_export (const char *NAME, ...)
     Add the bindings designated by NAME, ...  to the public interface
     of the current module.  The list of names is terminated by ‘NULL’.


File: guile.info,  Node: provide and require,  Next: Environments,  Prev: Accessing Modules from C,  Up: Modules

6.18.11 provide and require
---------------------------

Aubrey Jaffer, mostly to support his portable Scheme library SLIB,
implemented a provide/require mechanism for many Scheme implementations.
Library files in SLIB _provide_ a feature, and when user programs
_require_ that feature, the library file is loaded in.

   For example, the file ‘random.scm’ in the SLIB package contains the
line

     (provide 'random)

   so to use its procedures, a user would type

     (require 'random)

   and they would magically become available, _but still have the same
names!_  So this method is nice, but not as good as a full-featured
module system.

   When SLIB is used with Guile, provide and require can be used to
access its facilities.


File: guile.info,  Node: Environments,  Prev: provide and require,  Up: Modules

6.18.12 Environments
--------------------

Scheme, as defined in R5RS, does _not_ have a full module system.
However it does define the concept of a top-level “environment”.  Such
an environment maps identifiers (symbols) to Scheme objects such as
procedures and lists: *note About Closure::.  In other words, it
implements a set of “bindings”.

   Environments in R5RS can be passed as the second argument to ‘eval’
(*note Fly Evaluation::).  Three procedures are defined to return
environments: ‘scheme-report-environment’, ‘null-environment’ and
‘interaction-environment’ (*note Fly Evaluation::).

   In addition, in Guile any module can be used as an R5RS environment,
i.e., passed as the second argument to ‘eval’.

   Note: the following two procedures are available only when the
‘(ice-9 r5rs)’ module is loaded:

     (use-modules (ice-9 r5rs))

 -- Scheme Procedure: scheme-report-environment version
 -- Scheme Procedure: null-environment version
     VERSION must be the exact integer ‘5’, corresponding to revision 5
     of the Scheme report (the Revised^5 Report on Scheme).
     ‘scheme-report-environment’ returns a specifier for an environment
     that is empty except for all bindings defined in the report that
     are either required or both optional and supported by the
     implementation.  ‘null-environment’ returns a specifier for an
     environment that is empty except for the (syntactic) bindings for
     all syntactic keywords defined in the report that are either
     required or both optional and supported by the implementation.

     Currently Guile does not support values of VERSION for other
     revisions of the report.

     The effect of assigning (through the use of ‘eval’) a variable
     bound in a ‘scheme-report-environment’ (for example ‘car’) is
     unspecified.  Currently the environments specified by
     ‘scheme-report-environment’ are not immutable in Guile.


File: guile.info,  Node: Foreign Function Interface,  Next: Foreign Objects,  Prev: Modules,  Up: API Reference

6.19 Foreign Function Interface
===============================

Sometimes you need to use libraries written in C or Rust or some other
non-Scheme language.  More rarely, you might need to write some C to
extend Guile.  This section describes how to load these “foreign
libraries”, look up data and functions inside them, and so on.

* Menu:

* Foreign Libraries::              Dynamically linking to libraries.
* Foreign Extensions::             Extending Guile in C with loadable modules.
* Foreign Pointers::               Pointers to C data or functions.
* Foreign Types::                  Expressing C types in Scheme.
* Foreign Functions::              Simple calls to C procedures.
* Void Pointers and Byte Access::  Pointers into the ether.
* Foreign Structs::                Packing and unpacking structs.
* More Foreign Functions::         Advanced examples.


File: guile.info,  Node: Foreign Libraries,  Next: Foreign Extensions,  Up: Foreign Function Interface

6.19.1 Foreign Libraries
------------------------

Just as Guile can load up Scheme libraries at run-time, Guile can also
load some system libraries written in C or other low-level languages.
We refer to these as dynamically-loadable modules as “foreign
libraries”, to distinguish them from native libraries written in Scheme
or other languages implemented by Guile.

   Foreign libraries usually come in two forms.  Some foreign libraries
are part of the operating system, such as the compression library
‘libz’.  These shared libraries are built in such a way that many
programs can use their functionality without duplicating their code.
When a program written in C is built, it can declare that it uses a
specific set of shared libraries.  When the program is run, the
operating system takes care of locating and loading the shared
libraries.

   The operating system components that can dynamically load and link
shared libraries when a program is run are also available
programmatically during a program’s execution.  This is the interface
that’s most useful for Guile, and this is what we mean in Guile when we
refer to “dynamic linking”.  Dynamic linking at run-time is sometimes
called “dlopening”, to distinguish it from the dynamic linking that
happens at program start-up.

   The other kind of foreign library is sometimes known as a module,
plug-in, bundle, or an extension.  These foreign libraries aren’t meant
to be linked to by C programs, but rather only to be dynamically loaded
at run-time – they extend some main program with functionality, but
don’t stand on their own.  Sometimes a Guile library will implement some
of its functionality in a loadable module.

   In either case, the interface on the Guile side is the same.  You
load the interface using ‘load-foreign-library’.  The resulting foreign
library object implements a simple lookup interface whereby the user can
get addresses of data or code exported by the library.  There is no
facility to inspect foreign libraries; you have to know what’s in there
already before you look.

   Routines for loading foreign libraries and accessing their contents
are implemented in the ‘(system foreign-library)’ module.

     (use-modules (system foreign-library))

 -- Scheme Procedure: load-foreign-library [library]
          [#:extensions=system-library-extensions]
          [#:search-ltdl-library-path?=#t] [#:search-path=search-path]
          [#:search-system-paths?=#t] [#:lazy?=#t] [#:global=#f]
     [#:rename-on-cygwin?=#t] Find the shared library denoted by LIBRARY
     (a string or ‘#f’) and link it into the running Guile application.
     When everything works out, return a Scheme object suitable for
     representing the linked object file.  Otherwise an error is thrown.

     If LIBRARY argument is omitted, it defaults to ‘#f’.  If ‘library’
     is false, the resulting foreign library gives access to all symbols
     available for dynamic linking in the main binary.

     It is not necessary to include any extension such as ‘.so’ in
     LIBRARY.  For each system, Guile has a default set of extensions
     that it will try.  On GNU systems, the default extension set is
     just ‘.so’; on Windows, just ‘.dll’; and on Darwin (Mac OS), it is
     ‘.bundle’, ‘.so’, and ‘.dylib’.  Pass ‘#:extensions EXTENSIONS’ to
     override the default extensions list.  If LIBRARY contains one of
     the extensions, no extensions are tried, so it is possible to
     specify the extension if you know exactly what file to load.

     Unless LIBRARY denotes an absolute file name or otherwise contains
     a directory separator (‘/’, and also ‘\’ on Windows), Guile will
     search for the library in the directories listed in SEARCH-PATHS.
     The default search path has three components, which can all be
     overriden by colon-delimited (semicolon on Windows) environment
     variables:

     ‘GUILE_EXTENSIONS_PATH’
          This is the main environment variable for users to add
          directories containing Guile extensions.  The default value
          has no entries.  This environment variable was added in Guile
          3.0.6.
     ‘LTDL_LIBRARY_PATH’
          Before Guile 3.0.6, Guile loaded foreign libraries using
          ‘libltdl’, the dynamic library loader provided by libtool.
          This loader used ‘LTDL_LIBRARY_PATH’, and for backwards
          compatibility we still support that path.

          However, ‘libltdl’ would not only open ‘.so’ (or ‘.dll’ and so
          on) files, but also the ‘.la’ files created by libtool.  In
          installed libraries – libraries that are in the target
          directories of ‘make install’ – ‘.la’ files are never needed,
          to the extent that most GNU/Linux distributions remove them
          entirely.  It is sufficient to just load the ‘.so’ (or ‘.dll’
          and so on) files, which are always located in the same
          directory as the ‘.la’ files.

          But for uninstalled dynamic libraries, like those in a build
          tree, the situation is a bit of a mess.  If you have a project
          that uses libtool to build libraries – which is the case for
          Guile, and for most projects using autotools – and you build
          ‘foo.so’ in directory ‘D’, libtool will put ‘foo.la’ in ‘D’,
          but ‘foo.so’ gets put into ‘D/.libs’.

          Users were mostly oblivious to this situation, as ‘libltdl’
          had special logic to be able to read the ‘.la’ file to know
          where to find the ‘.so’, even from an uninstalled build tree,
          preventing the existence of ‘.libs’ from leaking out to the
          user.

          We don’t use libltdl now, essentially for flexibility and
          error-reporting reasons.  But, to keep this old use-case
          working, if SEARCH-LTDL-LIBRARY-PATH? is true, we add each
          entry of ‘LTDL_LIBRARY_PATH’ to the default extensions load
          path, additionally adding the ‘.libs’ subdirextories for each
          entry, in case there are ‘.so’ files there instead of
          alongside the ‘.la’ files.
     ‘GUILE_SYSTEM_EXTENSIONS_PATH’
          The last path in Guile’s search path belongs to Guile itself,
          and defaults to the libdir and the extensiondir, in that
          order.  For example, if you install to ‘/opt/guile’, these
          would probably be ‘/opt/guile/lib’ and
          ‘/opt/guile/lib/guile/3.0/extensions’, respectively.  *Note
          Parallel Installations::, for more details on ‘extensionsdir’.

     Finally, if no library is found in the search path, and if LIBRARY
     is not absolute and does not include directory separators, and if
     SEARCH-SYSTEM-PATHS? is true, the operating system may have its own
     logic for where to locate LIBRARY.  For example, on GNU, there will
     be a default set of paths (often ‘/usr/lib’ and ‘/lib’, though it
     depends on the system), and the ‘LD_LIBRARY_PATH’ environment
     variable can add additional paths.  Other operating systems have
     other conventions.

     Falling back to the operating system for search is usually not a
     great thing; it is a recipe for making programs that work on one
     machine but not on others.  Still, when wrapping system libraries,
     it can be the only way to get things working at all.

     If LAZY? is true (the default), Guile will request the operating
     system to resolve symbols used by the loaded library as they are
     first used.  If GLOBAL? is true, symbols defined by the loaded
     library will be available when other modules need to resolve
     symbols; the default is ‘#f’, which keeps symbols local.

     If RENAME-ON-CYGWIN? is true (the default) – on Cygwin hosts only –
     the search behavior is modified such that a filename that starts
     with “lib” will be searched for under the name “cyg”, as is
     customary for Cygwin.

   The environment variables mentioned above are parsed when the
foreign-library module is first loaded and bound to parameters.  Null
path components, for example the three components of
‘GUILE_SYSTEM_EXTENSIONS_PATH="::"’, are ignored.

 -- Scheme Parameter: guile-extensions-path
 -- Scheme Parameter: ltdl-library-path
 -- Scheme Parameter: guile-system-extensions-path
     Parameters whose initial values are taken from
     ‘GUILE_EXTENSIONS_PATH’, ‘LTDL_LIBRARY_PATH’, and
     ‘GUILE_SYSTEM_EXTENSIONS_PATH’, respectively.  *Note Parameters::.
     The current values of these parameters are used when building the
     search path when ‘load-foreign-library’ is called, unless the
     caller explicitly passes a ‘#:search-path’ argument.

 -- Scheme Procedure: foreign-library? obj
     Return ‘#t’ if OBJ is a foreign library, or ‘#f’ otherwise.


File: guile.info,  Node: Foreign Extensions,  Next: Foreign Pointers,  Prev: Foreign Libraries,  Up: Foreign Function Interface

6.19.2 Foreign Extensions
-------------------------

One way to use shared libraries is to extend Guile.  Such loadable
modules generally define one distinguished initialization function that,
when called, will use the ‘libguile’ API to define procedures in the
current module.

   Concretely, you might extend Guile with an implementation of the
Bessel function, ‘j0’:

     #include <math.h>
     #include <libguile.h>

     SCM
     j0_wrapper (SCM x)
     {
       return scm_from_double (j0 (scm_to_double (x, "j0")));
     }

     void
     init_math_bessel (void)
     {
       scm_c_define_gsubr ("j0", 1, 0, 0, j0_wrapper);
     }

   The C source file would then need to be compiled into a shared
library.  On GNU/Linux, the compiler invocation might look like this:

     gcc -shared -o bessel.so -fPIC bessel.c

   A good default place to put shared libraries that extend Guile is
into the extensions dir.  From the command line or a build script,
invoke ‘pkg-config --variable=extensionsdir guile-3.0’ to print the
extensions dir.  *Note Parallel Installations::, for more details.

   Guile can load up ‘bessel.so’ via ‘load-extension’.

 -- Scheme Procedure: load-extension lib init
 -- C Function: scm_load_extension (lib, init)
     Load and initialize the extension designated by LIB and INIT.

   The normal way for a extension to be used is to write a small Scheme
file that defines a module, and to load the extension into this module.
When the module is auto-loaded, the extension is loaded as well.  For
example:

     (define-module (math bessel)
       #:export (j0))

     (load-extension "bessel" "init_math_bessel")

   This ‘load-extension’ invocation loads the ‘bessel’ library via
‘(load-foreign-library "bessel")’, then looks up the ‘init_math_bessel’
symbol in the library, treating it as a function of no arguments, and
calls that function.

   If you decide to put your extension outside the default search path
for ‘load-foreign-library’, probably you should adapt the Scheme module
to specify its absolute path.  For example, if you use ‘automake’ to
build your extension and place it in ‘$(pkglibdir)’, you might define a
build-parameters module that gets created by the build system:

     (define-module (math config)
       #:export (extensiondir))
     (define extensiondir "PKGLIBDIR")

   This file would be ‘config.scm.in’.  You would define a ‘make’ rule
to substitute in the absolute installed file name:

     config.scm: config.scm.in
             sed 's|PKGLIBDIR|$(pkglibdir)|' <$< >$

   Then your ‘(math bessel)’ would import ‘(math config)’, then
‘(load-extension (in-vicinity extensiondir "bessel")
"init_math_bessel")’.

   An alternate approach would be to rebind the ‘guile-extensions-path’
parameter, or its corresponding environment variable, but note that
changing those parameters applies to other users of
‘load-foreign-library’ as well.

   Note that the new primitives that the extension adds to Guile with
‘scm_c_define_gsubr’ (*note Primitive Procedures::) or with any of the
other mechanisms are placed into the module that is current when the
‘scm_c_define_gsubr’ is executed, so to be clear about what goes vwhere
it’s best to include the ‘load-extension’ in a module, as above.
Alternately, the C code can use ‘scm_c_define_module’ to specify which
module is being created:

     static void
     do_init (void *unused)
     {
       scm_c_define_gsubr ("j0", 1, 0, 0, j0_wrapper);
       scm_c_export ("j0", NULL);
     }

     void
     init_math_bessel ()
     {
       scm_c_define_module ("math bessel", do_init, NULL);
     }

   And yet...  if what we want is just the ‘j0’ function, it seems like
a lot of ceremony to have to compile a Guile-specific wrapper library
complete with an initialization function and wraper module to allow
Guile users to call it.  There is another way, but to get there, we have
to talk about function pointers and function types first.  *Note Foreign
Functions::, to skip to the good parts.


File: guile.info,  Node: Foreign Pointers,  Next: Foreign Types,  Prev: Foreign Extensions,  Up: Foreign Function Interface

6.19.3 Foreign Pointers
-----------------------

Foreign libraries are essentially key-value mappings, where the keys are
names of definitions and the values are the addresses of those
definitions.  To look up the address of a definition, use
‘foreign-library-pointer’ from the ‘(system foreign-library)’ module.

 -- Scheme Procedure: foreign-library-pointer lib name
     Return a “wrapped pointer” for the symbol NAME in the shared object
     referred to by LIB.  The returned pointer points to a C object.

     As a convenience, if LIB is not a foreign library, it will be
     passed to ‘load-foreign-library’.

   If we continue with the ‘bessel.so’ example from before, we can get
the address of the ‘init_math_bessel’ function via:

     (use-modules (system foreign-library))
     (define init (foreign-library-pointer "bessel" "init_math_bessel"))
     init
     ⇒ #<pointer 0x7fb35b1b4688>

   A value returned by ‘foreign-library-pointer’ is a Scheme wrapper for
a C pointer.  Pointers are a data type in Guile that is disjoint from
all other types.  The next section discusses ways to dereference
pointers, but before then we describe the usual type predicates and so
on.

   Note that the rest of the interfaces in this section are part of the
‘(system foreign)’ library:

     (use-modules (system foreign))

 -- Scheme Procedure: pointer-address pointer
 -- C Function: scm_pointer_address (pointer)
     Return the numerical value of POINTER.

          (pointer-address init)
          ⇒ 139984413364296 ; YMMV

 -- Scheme Procedure: make-pointer address [finalizer]
     Return a foreign pointer object pointing to ADDRESS.  If FINALIZER
     is passed, it should be a pointer to a one-argument C function that
     will be called when the pointer object becomes unreachable.

 -- Scheme Procedure: pointer? obj
     Return ‘#t’ if OBJ is a pointer object, or ‘#f’ otherwise.

 -- Scheme Variable: %null-pointer
     A foreign pointer whose value is 0.

 -- Scheme Procedure: null-pointer? pointer
     Return ‘#t’ if POINTER is the null pointer, ‘#f’ otherwise.

   For the purpose of passing SCM values directly to foreign functions,
and allowing them to return SCM values, Guile also supports some unsafe
casting operators.

 -- Scheme Procedure: scm->pointer scm
     Return a foreign pointer object with the ‘object-address’ of SCM.

 -- Scheme Procedure: pointer->scm pointer
     Unsafely cast POINTER to a Scheme object.  Cross your fingers!

   Sometimes you want to give C extensions access to the dynamic FFI. At
that point, the names get confusing, because “pointer” can refer to a
‘SCM’ object that wraps a pointer, or to a ‘void*’ value.  We will try
to use “pointer object” to refer to Scheme objects, and “pointer value”
to refer to ‘void *’ values.

 -- C Function: SCM scm_from_pointer (void *ptr, void (*finalizer)
          (void*))
     Create a pointer object from a pointer value.

     If FINALIZER is non-null, Guile arranges to call it on the pointer
     value at some point after the pointer object becomes collectable.

 -- C Function: void* scm_to_pointer (SCM obj)
     Unpack the pointer value from a pointer object.


File: guile.info,  Node: Foreign Types,  Next: Foreign Functions,  Prev: Foreign Pointers,  Up: Foreign Function Interface

6.19.4 Foreign Types
--------------------

From Scheme’s perspective, foreign pointers are shards of chaos.  The
user can create a foreign pointer for any address, and do with it what
they will.  The only thing that lends a sense of order to the whole is a
shared hallucination that certain storage locations have certain types.
When making Scheme wrappers for foreign interfaces, we hide the madness
by explicitly representing the the data types of parameters and fields.

   These “foreign type values” may be constructed using the constants
and procedures from the ‘(system foreign)’ module, which may be loaded
like this:

     (use-modules (system foreign))

   ‘(system foreign)’ exports a number of values expressing the basic C
types.

 -- Scheme Variable: int8
 -- Scheme Variable: uint8
 -- Scheme Variable: uint16
 -- Scheme Variable: int16
 -- Scheme Variable: uint32
 -- Scheme Variable: int32
 -- Scheme Variable: uint64
 -- Scheme Variable: int64
 -- Scheme Variable: float
 -- Scheme Variable: double
     These values represent the C numeric types of the specified sizes
     and signednesses.

   In addition there are some convenience bindings for indicating types
of platform-dependent size.

 -- Scheme Variable: int
 -- Scheme Variable: unsigned-int
 -- Scheme Variable: long
 -- Scheme Variable: unsigned-long
 -- Scheme Variable: short
 -- Scheme Variable: unsigned-short
 -- Scheme Variable: size_t
 -- Scheme Variable: ssize_t
 -- Scheme Variable: ptrdiff_t
 -- Scheme Variable: intptr_t
 -- Scheme Variable: uintptr_t
     Values exported by the ‘(system foreign)’ module, representing C
     numeric types.  For example, ‘long’ may be ‘equal?’ to ‘int64’ on a
     64-bit platform.

 -- Scheme Variable: void
     The ‘void’ type.  It can be used as the first argument to
     ‘pointer->procedure’ to wrap a C function that returns nothing.

   In addition, the symbol ‘*’ is used by convention to denote pointer
types.  Procedures detailed in the following sections, such as
‘pointer->procedure’, accept it as a type descriptor.


File: guile.info,  Node: Foreign Functions,  Next: Void Pointers and Byte Access,  Prev: Foreign Types,  Up: Foreign Function Interface

6.19.5 Foreign Functions
------------------------

The most natural thing to do with a dynamic library is to grovel around
in it for a function pointer: a “foreign function”.  Load the ‘(system
foreign)’ module to use these Scheme interfaces.

     (use-modules (system foreign))

 -- Scheme Procedure: pointer->procedure return_type func_ptr arg_types
          [#:return-errno?=#f]
 -- C Function: scm_pointer_to_procedure (return_type, func_ptr,
          arg_types)
 -- C Function: scm_pointer_to_procedure_with_errno (return_type,
          func_ptr, arg_types)

     Make a foreign function.

     Given the foreign void pointer FUNC_PTR, its argument and return
     types ARG_TYPES and RETURN_TYPE, return a procedure that will pass
     arguments to the foreign function and return appropriate values.

     ARG_TYPES should be a list of foreign types.  ‘return_type’ should
     be a foreign type.  *Note Foreign Types::, for more information on
     foreign types.

     If RETURN-ERRNO? is true, or when calling
     ‘scm_pointer_to_procedure_with_errno’, the returned procedure will
     return two values, with ‘errno’ as the second value.

   Finally, in ‘(system foreign-library)’ there is a convenient wrapper
function, joining together ‘foreign-libary-pointer’ and
‘procedure->pointer’:

 -- Scheme Procedure: foreign-library-function lib name
          [#:return-type=void] [#:arg-types='()] [#:return-errno?=#f]
     Load the address of NAME from LIB, and treat it as a function
     taking arguments ARG-TYPES and returning RETURN-TYPE, optionally
     also with errno.

     An invocation of ‘foreign-library-function’ is entirely equivalent
     to:
          (pointer->procedure RETURN-TYPE
                              (foreign-library-pointer LIB NAME)
                              ARG-TYPES
                              #:return-errno? RETURN-ERRNO?).

   Pulling all this together, here is a better definition of ‘(math
bessel)’:

     (define-module (math bessel)
       #:use-module (system foreign)
       #:use-module (system foreign-library)
       #:export (j0))

     (define j0
       (foreign-library-function "libm" "j0"
                                 #:return-type double
                                 #:arg-types (list double)))

   That’s it!  No C at all.

   Before going on to more detailed examples, the next two sections
discuss how to deal with data that is more complex than, say, ‘int8’.
*Note More Foreign Functions::, to continue with foreign function
examples.


File: guile.info,  Node: Void Pointers and Byte Access,  Next: Foreign Structs,  Prev: Foreign Functions,  Up: Foreign Function Interface

6.19.6 Void Pointers and Byte Access
------------------------------------

Wrapped pointers are untyped, so they are essentially equivalent to C
‘void’ pointers.  As in C, the memory region pointed to by a pointer can
be accessed at the byte level.  This is achieved using _bytevectors_
(*note Bytevectors::).  The ‘(rnrs bytevectors)’ module contains
procedures that can be used to convert byte sequences to Scheme objects
such as strings, floating point numbers, or integers.

   Load the ‘(system foreign)’ module to use these Scheme interfaces.

     (use-modules (system foreign))

 -- Scheme Procedure: pointer->bytevector pointer len [offset
          [uvec_type]]
 -- C Function: scm_pointer_to_bytevector (pointer, len, offset,
          uvec_type)
     Return a bytevector aliasing the LEN bytes pointed to by POINTER.

     The user may specify an alternate default interpretation for the
     memory by passing the UVEC_TYPE argument, to indicate that the
     memory is an array of elements of that type.  UVEC_TYPE should be
     something that ‘array-type’ would return, like ‘f32’ or ‘s16’.

     When OFFSET is passed, it specifies the offset in bytes relative to
     POINTER of the memory region aliased by the returned bytevector.

     Mutating the returned bytevector mutates the memory pointed to by
     POINTER, so buckle your seatbelts.

 -- Scheme Procedure: bytevector->pointer bv [offset]
 -- C Function: scm_bytevector_to_pointer (bv, offset)
     Return a pointer pointer aliasing the memory pointed to by BV or
     OFFSET bytes after BV when OFFSET is passed.

   In addition to these primitives, convenience procedures are
available:

 -- Scheme Procedure: dereference-pointer pointer
     Assuming POINTER points to a memory region that holds a pointer,
     return this pointer.

 -- Scheme Procedure: string->pointer string [encoding]
     Return a foreign pointer to a nul-terminated copy of STRING in the
     given ENCODING, defaulting to the current locale encoding.  The C
     string is freed when the returned foreign pointer becomes
     unreachable.

     This is the Scheme equivalent of ‘scm_to_stringn’.

 -- Scheme Procedure: pointer->string pointer [length] [encoding]
     Return the string representing the C string pointed to by POINTER.
     If LENGTH is omitted or ‘-1’, the string is assumed to be
     nul-terminated.  Otherwise LENGTH is the number of bytes in memory
     pointed to by POINTER.  The C string is assumed to be in the given
     ENCODING, defaulting to the current locale encoding.

     This is the Scheme equivalent of ‘scm_from_stringn’.

   Most object-oriented C libraries use pointers to specific data
structures to identify objects.  It is useful in such cases to reify the
different pointer types as disjoint Scheme types.  The
‘define-wrapped-pointer-type’ macro simplifies this.

 -- Scheme Syntax: define-wrapped-pointer-type type-name pred wrap
          unwrap print
     Define helper procedures to wrap pointer objects into Scheme
     objects with a disjoint type.  Specifically, this macro defines:

        • PRED, a predicate for the new Scheme type;
        • WRAP, a procedure that takes a pointer object and returns an
          object that satisfies PRED;
        • UNWRAP, which does the reverse.

     WRAP preserves pointer identity, for two pointer objects P1 and P2
     that are ‘equal?’, ‘(eq? (WRAP P1) (WRAP P2)) ⇒ #t’.

     Finally, PRINT should name a user-defined procedure to print such
     objects.  The procedure is passed the wrapped object and a port to
     write to.

     For example, assume we are wrapping a C library that defines a
     type, ‘bottle_t’, and functions that can be passed ‘bottle_t *’
     pointers to manipulate them.  We could write:

          (define-wrapped-pointer-type bottle
            bottle?
            wrap-bottle unwrap-bottle
            (lambda (b p)
              (format p "#<bottle of ~a ~x>"
                      (bottle-contents b)
                      (pointer-address (unwrap-bottle b)))))

          (define grab-bottle
            ;; Wrapper for `bottle_t *grab (void)'.
            (let ((grab (foreign-library-function libbottle "grab_bottle"
                                                  #:return-type '*)))
              (lambda ()
                "Return a new bottle."
                (wrap-bottle (grab)))))

          (define bottle-contents
            ;; Wrapper for `const char *bottle_contents (bottle_t *)'.
            (let ((contents (foreign-library-function libbottle "bottle_contents"
                                                      #:return-type '*
                                                      #:arg-types  '(*))))
              (lambda (b)
                "Return the contents of B."
                (pointer->string (contents (unwrap-bottle b))))))

          (write (grab-bottle))
          ⇒ #<bottle of Château Haut-Brion 803d36>

     In this example, ‘grab-bottle’ is guaranteed to return a genuine
     ‘bottle’ object satisfying ‘bottle?’.  Likewise, ‘bottle-contents’
     errors out when its argument is not a genuine ‘bottle’ object.

   As another example, currently Guile has a variable, ‘scm_numptob’, as
part of its API. It is declared as a C ‘long’.  So, to read its value,
we can do:

     (use-modules (system foreign))
     (use-modules (rnrs bytevectors))
     (define numptob
       (foreign-library-pointer #f "scm_numptob"))
     numptob
     (bytevector-uint-ref (pointer->bytevector numptob (sizeof long))
                          0 (native-endianness)
                          (sizeof long))
     ⇒ 8

   If we wanted to corrupt Guile’s internal state, we could set
‘scm_numptob’ to another value; but we shouldn’t, because that variable
is not meant to be set.  Indeed this point applies more widely: the C
API is a dangerous place to be.  Not only might setting a value crash
your program, simply accessing the data pointed to by a dangling pointer
or similar can prove equally disastrous.


File: guile.info,  Node: Foreign Structs,  Next: More Foreign Functions,  Prev: Void Pointers and Byte Access,  Up: Foreign Function Interface

6.19.7 Foreign Structs
----------------------

Finally, one last note on foreign values before moving on to actually
calling foreign functions.  Sometimes you need to deal with C structs,
which requires interpreting each element of the struct according to the
its type, offset, and alignment.  The ‘(system foreign)’ module has some
primitives to support this.

     (use-modules (system foreign))

 -- Scheme Procedure: sizeof type
 -- C Function: scm_sizeof (type)
     Return the size of TYPE, in bytes.

     TYPE should be a valid C type, like ‘int’.  Alternately TYPE may be
     the symbol ‘*’, in which case the size of a pointer is returned.
     TYPE may also be a list of types, in which case the size of a
     ‘struct’ with ABI-conventional packing is returned.

 -- Scheme Procedure: alignof type
 -- C Function: scm_alignof (type)
     Return the alignment of TYPE, in bytes.

     TYPE should be a valid C type, like ‘int’.  Alternately TYPE may be
     the symbol ‘*’, in which case the alignment of a pointer is
     returned.  TYPE may also be a list of types, in which case the
     alignment of a ‘struct’ with ABI-conventional packing is returned.

   Guile also provides some convenience methods to pack and unpack
foreign pointers wrapping C structs.

 -- Scheme Procedure: make-c-struct types vals
     Create a foreign pointer to a C struct containing VALS with types
     ‘types’.

     VALS and ‘types’ should be lists of the same length.

 -- Scheme Procedure: parse-c-struct foreign types
     Parse a foreign pointer to a C struct, returning a list of values.

     ‘types’ should be a list of C types.

   For example, to create and parse the equivalent of a ‘struct {
int64_t a; uint8_t b; }’:

     (parse-c-struct (make-c-struct (list int64 uint8)
                                    (list 300 43))
                     (list int64 uint8))
     ⇒ (300 43)

   As yet, Guile only has convenience routines to support
conventionally-packed structs.  But given the ‘bytevector->pointer’ and
‘pointer->bytevector’ routines, one can create and parse tightly packed
structs and unions by hand.  See the code for ‘(system foreign)’ for
details.


File: guile.info,  Node: More Foreign Functions,  Prev: Foreign Structs,  Up: Foreign Function Interface

6.19.8 More Foreign Functions
-----------------------------

It is possible to pass pointers to foreign functions, and to return them
as well.  In that case the type of the argument or return value should
be the symbol ‘*’, indicating a pointer.  For example, the following
code makes ‘memcpy’ available to Scheme:

     (use-modules (system foreign))
     (define memcpy
       (foreign-library-function #f "memcpy"
                                 #:return-type '*
                                 #:arg-types (list '* '* size_t)))

   To invoke ‘memcpy’, one must pass it foreign pointers:

     (use-modules (rnrs bytevectors))

     (define src-bits
       (u8-list->bytevector '(0 1 2 3 4 5 6 7)))
     (define src
       (bytevector->pointer src-bits))
     (define dest
       (bytevector->pointer (make-bytevector 16 0)))

     (memcpy dest src (bytevector-length src-bits))

     (bytevector->u8-list (pointer->bytevector dest 16))
     ⇒ (0 1 2 3 4 5 6 7 0 0 0 0 0 0 0 0)

   One may also pass structs as values, passing structs as foreign
pointers.  *Note Foreign Structs::, for more information on how to
express struct types and struct values.

   “Out” arguments are passed as foreign pointers.  The memory pointed
to by the foreign pointer is mutated in place.

     ;; struct timeval {
     ;;      time_t      tv_sec;     /* seconds */
     ;;      suseconds_t tv_usec;    /* microseconds */
     ;; };
     ;; assuming fields are of type "long"

     (define gettimeofday
       (let ((f (foreign-library-function #f "gettimeofday"
                                          #:return-type int
                                          #:arg-types (list '* '*)))
             (tv-type (list long long)))
         (lambda ()
           (let* ((timeval (make-c-struct tv-type (list 0 0)))
                  (ret (f timeval %null-pointer)))
             (if (zero? ret)
                 (apply values (parse-c-struct timeval tv-type))
                 (error "gettimeofday returned an error" ret))))))

     (gettimeofday)
     ⇒ 1270587589
     ⇒ 499553

   As you can see, this interface to foreign functions is at a very low,
somewhat dangerous level(1).

   The FFI can also work in the opposite direction: making Scheme
procedures callable from C. This makes it possible to use Scheme
procedures as “callbacks” expected by C function.

 -- Scheme Procedure: procedure->pointer return-type proc arg-types
 -- C Function: scm_procedure_to_pointer (return_type, proc, arg_types)
     Return a pointer to a C function of type RETURN-TYPE taking
     arguments of types ARG-TYPES (a list) and behaving as a proxy to
     procedure PROC.  Thus PROC’s arity, supported argument types, and
     return type should match RETURN-TYPE and ARG-TYPES.

   As an example, here’s how the C library’s ‘qsort’ array sorting
function can be made accessible to Scheme (*note ‘qsort’: (libc)Array
Sort Function.):

     (define qsort!
       (let ((qsort (foreign-library-function
                     #f "qsort" #:arg-types (list '* size_t size_t '*))))
         (lambda (bv compare)
           ;; Sort bytevector BV in-place according to comparison
           ;; procedure COMPARE.
           (let ((ptr (procedure->pointer int
                                          (lambda (x y)
                                            ;; X and Y are pointers so,
                                            ;; for convenience, dereference
                                            ;; them before calling COMPARE.
                                            (compare (dereference-uint8* x)
                                                     (dereference-uint8* y)))
                                          (list '* '*))))
             (qsort (bytevector->pointer bv)
                    (bytevector-length bv) 1 ;; we're sorting bytes
                    ptr)))))

     (define (dereference-uint8* ptr)
       ;; Helper function: dereference the byte pointed to by PTR.
       (let ((b (pointer->bytevector ptr 1)))
         (bytevector-u8-ref b 0)))

     (define bv
       ;; An unsorted array of bytes.
       (u8-list->bytevector '(7 1 127 3 5 4 77 2 9 0)))

     ;; Sort BV.
     (qsort! bv (lambda (x y) (- x y)))

     ;; Let's see what the sorted array looks like:
     (bytevector->u8-list bv)
     ⇒ (0 1 2 3 4 5 7 9 77 127)

   And voilà!

   Note that ‘procedure->pointer’ is not supported (and not defined) on
a few exotic architectures.  Thus, user code may need to check
‘(defined? 'procedure->pointer)’.  Nevertheless, it is available on many
architectures, including (as of libffi 3.0.9) x86, ia64, SPARC, PowerPC,
ARM, and MIPS, to name a few.

   ---------- Footnotes ----------

   (1) A contribution to Guile in the form of a high-level FFI would be
most welcome.


File: guile.info,  Node: Foreign Objects,  Next: Smobs,  Prev: Foreign Function Interface,  Up: API Reference

6.20 Foreign Objects
====================

This chapter contains reference information related to defining and
working with foreign objects.  *Note Defining New Foreign Object
Types::, for a tutorial-like introduction to foreign objects.

 -- C Type: scm_t_struct_finalize
     This function type returns ‘void’ and takes one ‘SCM’ argument.

 -- C Function: SCM scm_make_foreign_object_type (SCM name, SCM slots,
          scm_t_struct_finalize finalizer)
     Create a fresh foreign object type.  NAME is a symbol naming the
     type.  SLOTS is a list of symbols, each one naming a field in the
     foreign object type.  FINALIZER indicates the finalizer, and may be
     ‘NULL’.

   We recommend that finalizers be avoided if possible.  *Note Foreign
Object Memory Management::.  Finalizers must be async-safe and
thread-safe.  Again, *note Foreign Object Memory Management::.  If you
are embedding Guile in an application that is not thread-safe, and you
define foreign object types that need finalization, you might want to
disable automatic finalization, and arrange to call
‘scm_manually_run_finalizers ()’ yourself.

 -- C Function: int scm_set_automatic_finalization_enabled (int
          enabled_p)
     Enable or disable automatic finalization.  By default, Guile
     arranges to invoke object finalizers automatically, in a separate
     thread if possible.  Passing a zero value for ENABLED_P will
     disable automatic finalization for Guile as a whole.  If you
     disable automatic finalization, you will have to call
     ‘scm_run_finalizers ()’ periodically.

     Unlike most other Guile functions, you can call
     ‘scm_set_automatic_finalization_enabled’ before Guile has been
     initialized.

     Return the previous status of automatic finalization.

 -- C Function: int scm_run_finalizers (void)
     Invoke any pending finalizers.  Returns the number of finalizers
     that were invoked.  This function should be called when automatic
     finalization is disabled, though it may be called if it is enabled
     as well.

 -- C Function: void scm_assert_foreign_object_type (SCM type, SCM val)
     When VAL is a foreign object of the given TYPE, do nothing.
     Otherwise, signal an error.

 -- C Function: SCM scm_make_foreign_object_0 (SCM type)
 -- C Function: SCM scm_make_foreign_object_1 (SCM type, void *val0)
 -- C Function: SCM scm_make_foreign_object_2 (SCM type, void *val0,
          void *val1)
 -- C Function: SCM scm_make_foreign_object_3 (SCM type, void *val0,
          void *val1, void *val2)
 -- C Function: SCM scm_make_foreign_object_n (SCM type, size_t n, void
          *vals[])
     Make a new foreign object of the type with type TYPE and initialize
     the first N fields to the given values, as appropriate.

     The number of fields for objects of a given type is fixed when the
     type is created.  It is an error to give more initializers than
     there are fields in the value.  It is perfectly fine to give fewer
     initializers than needed; this is convenient when some fields are
     of non-pointer types, and would be easier to initialize with the
     setters described below.

 -- C Function: void* scm_foreign_object_ref (SCM obj, size_t n);
 -- C Function: scm_t_bits scm_foreign_object_unsigned_ref (SCM obj,
          size_t n);
 -- C Function: scm_t_signed_bits scm_foreign_object_signed_ref (SCM
          obj, size_t n);
     Return the value of the Nth field of the foreign object OBJ.  The
     backing store for the fields is as wide as a ‘scm_t_bits’ value,
     which is at least as wide as a pointer.  The different variants
     handle casting in a portable way.

 -- C Function: void scm_foreign_object_set_x (SCM obj, size_t n, void
          *val);
 -- C Function: void scm_foreign_object_unsigned_set_x (SCM obj, size_t
          n, scm_t_bits val);
 -- C Function: void scm_foreign_object_signed_set_x (SCM obj, size_t n,
          scm_t_signed_bits val);
     Set the value of the Nth field of the foreign object OBJ to VAL,
     after portably converting to a ‘scm_t_bits’ value, if needed.

   One can also access foreign objects from Scheme.  *Note Foreign
Objects and Scheme::, for some examples.

     (use-modules (system foreign-object))

 -- Scheme Procedure: make-foreign-object-type name slots
          [#:finalizer=#f]
     Make a new foreign object type.  See the above documentation for
     ‘scm_make_foreign_object_type’; these functions are exactly
     equivalent, except for the way in which the finalizer gets attached
     to instances (an internal detail).

     The resulting value is a GOOPS class.  *Note GOOPS::, for more on
     classes in Guile.

 -- Scheme Syntax: define-foreign-object-type name constructor (slot
          ...) [#:finalizer=#f]
     A convenience macro to define a type, using
     ‘make-foreign-object-type’, and bind it to NAME.  A constructor
     will be bound to CONSTRUCTOR, and getters will be bound to each of
     SLOT....


File: guile.info,  Node: Smobs,  Next: Scheduling,  Prev: Foreign Objects,  Up: API Reference

6.21 Smobs
==========

A “smob” is a “small object”.  Before foreign objects were introduced in
Guile 2.0.12 (*note Foreign Objects::), smobs were the preferred way to
for C code to define new kinds of Scheme objects.  With the exception of
the so-called “applicable SMOBs” discussed below, smobs are now a legacy
interface and are headed for eventual deprecation.  *Note Deprecation::.
New code should use the foreign object interface.

   This section contains reference information related to defining and
working with smobs.  For a tutorial-like introduction to smobs, see
“Defining New Types (Smobs)” in previous versions of this manual.

 -- Function: scm_t_bits scm_make_smob_type (const char *name, size_t
          size)
     This function adds a new smob type, named NAME, with instance size
     SIZE, to the system.  The return value is a tag that is used in
     creating instances of the type.

     If SIZE is 0, the default _free_ function will do nothing.

     If SIZE is not 0, the default _free_ function will deallocate the
     memory block pointed to by ‘SCM_SMOB_DATA’ with ‘scm_gc_free’.  The
     WHAT parameter in the call to ‘scm_gc_free’ will be NAME.

     Default values are provided for the _mark_, _free_, _print_, and
     _equalp_ functions.  If you want to customize any of these
     functions, the call to ‘scm_make_smob_type’ should be immediately
     followed by calls to one or several of ‘scm_set_smob_mark’,
     ‘scm_set_smob_free’, ‘scm_set_smob_print’, and/or
     ‘scm_set_smob_equalp’.

 -- C Function: void scm_set_smob_free (scm_t_bits tc, size_t (*free)
          (SCM obj))
     This function sets the smob freeing procedure (sometimes referred
     to as a “finalizer”) for the smob type specified by the tag TC.  TC
     is the tag returned by ‘scm_make_smob_type’.

     The FREE procedure must deallocate all resources that are directly
     associated with the smob instance OBJ.  It must assume that all
     ‘SCM’ values that it references have already been freed and are
     thus invalid.

     It must also not call any libguile function or macro except
     ‘scm_gc_free’, ‘SCM_SMOB_FLAGS’, ‘SCM_SMOB_DATA’,
     ‘SCM_SMOB_DATA_2’, and ‘SCM_SMOB_DATA_3’.

     The FREE procedure must return 0.

     Note that defining a freeing procedure is not necessary if the
     resources associated with OBJ consists only of memory allocated
     with ‘scm_gc_malloc’ or ‘scm_gc_malloc_pointerless’ because this
     memory is automatically reclaimed by the garbage collector when it
     is no longer needed (*note ‘scm_gc_malloc’: Memory Blocks.).

   Smob free functions must be thread-safe.  *Note Foreign Object Memory
Management::, for a discussion on finalizers and concurrency.  If you
are embedding Guile in an application that is not thread-safe, and you
define smob types that need finalization, you might want to disable
automatic finalization, and arrange to call ‘scm_manually_run_finalizers
()’ yourself.  *Note Foreign Objects::.

 -- C Function: void scm_set_smob_mark (scm_t_bits tc, SCM (*mark) (SCM
          obj))
     This function sets the smob marking procedure for the smob type
     specified by the tag TC.  TC is the tag returned by
     ‘scm_make_smob_type’.

     Defining a marking procedure is almost always the wrong thing to
     do.  It is much, much preferable to allocate smob data with the
     ‘scm_gc_malloc’ and ‘scm_gc_malloc_pointerless’ functions, and
     allow the GC to trace pointers automatically.

     Any mark procedures you see currently almost surely date from the
     time of Guile 1.8, before the switch to the Boehm-Demers-Weiser
     collector.  Such smob implementations should be changed to just use
     ‘scm_gc_malloc’ and friends, and to lose their mark function.

     If you decide to keep the mark function, note that it may be called
     on objects that are on the free list.  Please read and digest the
     comments from the BDW GC’s ‘gc/gc_mark.h’ header.

     The MARK procedure must cause ‘scm_gc_mark’ to be called for every
     ‘SCM’ value that is directly referenced by the smob instance OBJ.
     One of these ‘SCM’ values can be returned from the procedure and
     Guile will call ‘scm_gc_mark’ for it.  This can be used to avoid
     deep recursions for smob instances that form a list.

     It must not call any libguile function or macro except
     ‘scm_gc_mark’, ‘SCM_SMOB_FLAGS’, ‘SCM_SMOB_DATA’,
     ‘SCM_SMOB_DATA_2’, and ‘SCM_SMOB_DATA_3’.

 -- C Function: void scm_set_smob_print (scm_t_bits tc, int (*print)
          (SCM obj, SCM port, scm_print_state* pstate))
     This function sets the smob printing procedure for the smob type
     specified by the tag TC.  TC is the tag returned by
     ‘scm_make_smob_type’.

     The PRINT procedure should output a textual representation of the
     smob instance OBJ to PORT, using information in PSTATE.

     The textual representation should be of the form ‘#<name ...>’.
     This ensures that ‘read’ will not interpret it as some other Scheme
     value.

     It is often best to ignore PSTATE and just print to PORT with
     ‘scm_display’, ‘scm_write’, ‘scm_simple_format’, and ‘scm_puts’.

 -- C Function: void scm_set_smob_equalp (scm_t_bits tc, SCM (*equalp)
          (SCM obj1, SCM obj2))
     This function sets the smob equality-testing predicate for the smob
     type specified by the tag TC.  TC is the tag returned by
     ‘scm_make_smob_type’.

     The EQUALP procedure should return ‘SCM_BOOL_T’ when OBJ1 is
     ‘equal?’ to OBJ2.  Else it should return ‘SCM_BOOL_F’.  Both OBJ1
     and OBJ2 are instances of the smob type TC.

 -- C Function: void scm_assert_smob_type (scm_t_bits tag, SCM val)
     When VAL is a smob of the type indicated by TAG, do nothing.  Else,
     signal an error.

 -- C Macro: int SCM_SMOB_PREDICATE (scm_t_bits tag, SCM exp)
     Return true if EXP is a smob instance of the type indicated by TAG,
     or false otherwise.  The expression EXP can be evaluated more than
     once, so it shouldn’t contain any side effects.

 -- C Function: SCM scm_new_smob (scm_t_bits tag, void *data)
 -- C Function: SCM scm_new_double_smob (scm_t_bits tag, void *data,
          void *data2, void *data3)
     Make a new smob of the type with tag TAG and smob data DATA, DATA2,
     and DATA3, as appropriate.

     The TAG is what has been returned by ‘scm_make_smob_type’.  The
     initial values DATA, DATA2, and DATA3 are of type ‘scm_t_bits’;
     when you want to use them for ‘SCM’ values, these values need to be
     converted to a ‘scm_t_bits’ first by using ‘SCM_UNPACK’.

     The flags of the smob instance start out as zero.

 -- C Macro: scm_t_bits SCM_SMOB_FLAGS (SCM obj)
     Return the 16 extra bits of the smob OBJ.  No meaning is predefined
     for these bits, you can use them freely.

 -- C Macro: scm_t_bits SCM_SET_SMOB_FLAGS (SCM obj, scm_t_bits flags)
     Set the 16 extra bits of the smob OBJ to FLAGS.  No meaning is
     predefined for these bits, you can use them freely.

 -- C Macro: scm_t_bits SCM_SMOB_DATA (SCM obj)
 -- C Macro: scm_t_bits SCM_SMOB_DATA_2 (SCM obj)
 -- C Macro: scm_t_bits SCM_SMOB_DATA_3 (SCM obj)
     Return the first (second, third) immediate word of the smob OBJ as
     a ‘scm_t_bits’ value.  When the word contains a ‘SCM’ value, use
     ‘SCM_SMOB_OBJECT’ (etc.)  instead.

 -- C Macro: void SCM_SET_SMOB_DATA (SCM obj, scm_t_bits val)
 -- C Macro: void SCM_SET_SMOB_DATA_2 (SCM obj, scm_t_bits val)
 -- C Macro: void SCM_SET_SMOB_DATA_3 (SCM obj, scm_t_bits val)
     Set the first (second, third) immediate word of the smob OBJ to
     VAL.  When the word should be set to a ‘SCM’ value, use
     ‘SCM_SMOB_SET_OBJECT’ (etc.)  instead.

 -- C Macro: SCM SCM_SMOB_OBJECT (SCM obj)
 -- C Macro: SCM SCM_SMOB_OBJECT_2 (SCM obj)
 -- C Macro: SCM SCM_SMOB_OBJECT_3 (SCM obj)
     Return the first (second, third) immediate word of the smob OBJ as
     a ‘SCM’ value.  When the word contains a ‘scm_t_bits’ value, use
     ‘SCM_SMOB_DATA’ (etc.)  instead.

 -- C Macro: void SCM_SET_SMOB_OBJECT (SCM obj, SCM val)
 -- C Macro: void SCM_SET_SMOB_OBJECT_2 (SCM obj, SCM val)
 -- C Macro: void SCM_SET_SMOB_OBJECT_3 (SCM obj, SCM val)
     Set the first (second, third) immediate word of the smob OBJ to
     VAL.  When the word should be set to a ‘scm_t_bits’ value, use
     ‘SCM_SMOB_SET_DATA’ (etc.)  instead.

 -- C Macro: SCM * SCM_SMOB_OBJECT_LOC (SCM obj)
 -- C Macro: SCM * SCM_SMOB_OBJECT_2_LOC (SCM obj)
 -- C Macro: SCM * SCM_SMOB_OBJECT_3_LOC (SCM obj)
     Return a pointer to the first (second, third) immediate word of the
     smob OBJ.  Note that this is a pointer to ‘SCM’.  If you need to
     work with ‘scm_t_bits’ values, use ‘SCM_PACK’ and ‘SCM_UNPACK’, as
     appropriate.

 -- Function: SCM scm_markcdr (SCM X)
     Mark the references in the smob X, assuming that X’s first data
     word contains an ordinary Scheme object, and X refers to no other
     objects.  This function simply returns X’s first data word.