netrik hacker's manual
>========================<

[This file contains a description of the file loading module. See hacking.txt or hacking.html for an overview of the manual.]

load.c

Every open input resource has a descriptor associated with it. The descriptor is created when opening the with init_load().

The struct for this descriptor is declared in load.h, and contains:

• The URL of the resource is stored by "url", which points to a "struct Url" containing the (split up) effective URL
• "type" is an enum of type "Res_type", describing what kind of resource it is. (RES_STDIN for stdin, RES_FILE for local files, RES_HTTP for HTTP connections handled by builtin HTTP code, or RES_PIPE for piping data from wget; it may also be set to RES_FAIL during the loading process, indicating that the loading failed)
• The union "handle" contains data necessary to handle the resource itself, e.g. a file handle.
• "buf" is a pointer to a string, where every loaded data block is stored for processing.
• "buf_end" contains the current end of the data in the buffer. (Not always the whole buffer is filled, escpecially at EOF.)
• "buf_ptr" points to the current read position inside the buffer.
• The "user_break" flag indicates when loading was interrupted by a user break and the document wasn't loaded completely.

An additional "form" argument can be given to submit the data of an HTML form along with the HTTP request. See Submitting in hacking-links.* for details on this.

Calling this function with only a base URL and no target means that a page is to be reloaded from history.

For all other target URLs, first the effective URL is determined. This is done by merge_urls(), which first splits the URL up into its components using split_url(), and then merges it with the base URL components. A pointer to the resulting merged URL ist stored in "res->url".

If merging the URLs failed for some reason, the resource type is set to "RES_FAIL", and init_load() returns normally. Thus, there is no special error handling necessary in the calling function; all necessary handling is done in load().

If the URL is "-", meaning load from stdin, "res->url->proto.type" is set to PT_INTERNAL (meaning no relative links are allowed, and the page is not to be kept in history).

No URL merging is done if a page from history is reloaded. The split URL passed as the base is already the destination URL.

Now having the destination URL, the resource is opened appropriately, depending on what resource type is set in the URL.

If it is a HTTP URL, by default http_init_load() is used to open a connection to the server and read the HTTP headers. "res->type" is set to HTTP.

Alternatively, "cfg.wget" may be set (using --no-builtin-http), in which case "wget" is used instead of the builtin HTTP code. This is done by init_wget(), which starts wget with the correct filename, and initiates reading its standard output through a pipe. The popen() function returns our pipe end as a stream; we use this as the input. "res->type" is set to "RES_PIPE".

FTP URLs are always fetched using wget.

Local files are simply opened as the input stream, and "res->type" is set to "RES_FILE".

Internal URLs presently always mean loading from stdin. (Error pages also use "PT_INTERNAL", but init_load() returns immediataly after setting it, so this needn't be handled in the switch.) The standard input is reopened as a stream, and this one is used as the input. As we need some way to read user commands from the terminal, "stderr" is reopend in place of "stdin". (Some programs reopen "/dev/tty" instead -- no idea which one is better...) This way the normal "stdin" descriptor points to a terminal again, while the input file is read from the pipe that is (hopefully) connected to the original standard input. "res->type" is set to "RES_STDIN".

Things get a bit more complicated if there was no protocol specification in the URL, and no base was supplied (meaning the URL has to be treated as an absolute URL in any case). In this case we have to guess whether it is a local file or an HTTP URL.

First we try to open a local file with the path returned by merge_urls(). (Should be identical to the given URL.) If this succeeds, we set the protocol to "PT_FILE" and the resource type to "RES_FILE", and that's it.

If opening the local file fails, and the URL string doesn't start with '/', the URL is assumed to be an HTTP URL. As these URLs are split in another way, we have to prepend the "http://" to the URL string and call merge_urls() again. Afterwards, we proceed just as with any other HTTP URL.

After opening an HTTP connection (regardless whether it was PT_HTTP or PT_UNKNOWN recognized as HTTP), we need to check for HTTP redirects. If a redirect is set (a "Location:" header exists), the loading has to be repeated with the redirect URL. The new "main_url" is taken from the "Location:" value; the "base_url" is set to "res->url", i.e. the original absolute target URL, so the redirect will be relative to that; the "form" is also cleared, as redirects always use a simple GET. With this new URL(s), the whole process is repeated. (Except for allocation of the resource descriptor, which is reused.) In this reiteration the protocol type always has to be PT_HTTP; redirections to other resource types are not allowed. All this can happen up to five times; when more, it's probably a redirection loop, and we abort.

If opening a file fails, init_load() returns immediately, only setting "res->type" to "RES_FAIL" and "res->url->proto" to "PT_INTERNAL". "RES_FAIL" is then handled appropriately in load(); this way, the caller needn't bother about it. (Except an additional error message and setting an error code in parse_syntax(); see Warning Mesages in hacking-layout.* for that.) "PT_INTERNAL" means that the page isn't to be kept in history, that following relative URLs isn't allowed etc. Thus, the link/page history handling functions also do not need special handling for the error pages.

HTTP loading errors inside http_init_load() are handled the same way.

Every call to load() reads one data block (of size BUF_SIZE) into a buffer. The reading function (parse_syntax() or parse_header()) then processes the data, keeping track of the current read position inside the buffer by "res->buf_ptr"; when it reaches the end of the data block, load() is called again to read the next block.

If "res->type" is "RES_HTTP", read() is used to read a data block from the socket; otherwise, fread() is used to read data from the input stream. (It doesn't matter if this stream is a normal file (RES_FILE), stdin (RES_STDIN), or a pipe (RES_PIPE).)

"RES_FAIL" means that opening the input resource failed for some reason, or an error emerged in a previous load() call. In this case, no data is read; an empty buffer is returned, which normally would mean EOF. This causes parse_syntax() to generate an empty page, or stops parsing at this point if some data already has been read before the error occured. For the latter case, some (little) additional handling is necessary in parse_syntax(), to ensure that an appropriate error message is printed and an error code returned to main(). (Causing a keywait before starting the pager.)

The data block is stored in the "buf[]" referenced by the descriptor, and "buf_end" is set to the end of the data inside the buffer; "buf_ptr" is set to the beginning of the buffer.

url.c

This file contains a couple of functions for handling of URLs, which are used chiefly by the file loading module.

The parser is very similar to the HTML parser in parse_syntax.c (see Parsing in hacking-layout.*): The URL string is processed char by char in a loop. In every iteration, one char is examined, and action is taken (in a switch statement) depending on what character it is, and in what mode the parser currently is.

There is a parsing mode for every URL component. (There are also some additional ones for constructs like the "://" after the protocol specification.) Every time we encounter some special character seperating different components, the mode is switched to the one fitting the next component, and its beginning position (normaly the char after the one causing the mode change) is stored in "word_start". Everything between the previous "word_start" and the current char is stored to the respective split URL field of the component parsed up to now, using store_component().

url: "http://domain:80/dir1/dir2/name.ext?params#fragment"
             ^     ^url_char
             word_start
parse_mode: PM_HOST

url: "http://domain:80/dir1/dir2/name.ext?params#fragment"
           url_char^^
                    word_start
parse_mode: PM_PORT
components->host: "domain"

If the separating char is not the one that would introduce the component normally following now, that means that component is missing, and we immediately have to proceed with the next one.

url: "http://domain/dir1/dir2/name.ext#fragment"
                              url_char^^
                                       word_start
parse_mode: PM_PARAMS

This is done by setting "recycle", meaning that the current character is to be parsed again. In this new iteration the parser will see the separating char again, thus introducing a second mode change. store_component() will store a NULL string for that component, and the parser will go on with parsing the next one.

url: "http://domain/dir1/dir2/name.ext#fragment"
                              url_char^^
                                       word_start
parse_mode: PM_FRAG
components->params: NULL

The first mode change is a bit more tricky: At the beginning of the URL, we do not know if it is a full qualified one (starting with a protocol specification), or a relative URL without a protocol. At first we assume that it starts with the protocol. If a ":" follows the first word, our guess was right, and we proceed normally with the host.

url: "http://domain:80/dir1/dir2/name.ext?params#fragment"
  url_char^^
           word_start
parse_mode: PM_PROTO_END1
components->proto.str: "http"

If any other separating char occurs instead, we have to skip protocol, host, and port, and switch immediately to path parsing. We also do a "recycle" then, as the current char needs to be parsed in "PM_PATH" mode.

url: "dir1/dir2/name.ext?params#fragment"
          ^url_char
      ^word_start
parse_mode: PM_PATH

Path parsing is also a bit more complicated, as the directory name and the file name are stored separately. For that purpose, while parsing the path we keep track where the last '/' was (in "name_start"); everything before it (inclusive) belongs to the directory, and what follows it is the file name.

url: "http://domain:80/dir1/dir2/name.ext?params#fragment"
                      ^          ^       ^url_char
             word_start          name_start
                      |<- dir ->|
                                 |<name>|
parse_mode: PM_PARA
components->dir: "/dir1/dir2/"
components->name: "name.ext"

"full_url" and "path" aren't filled in split_url(), as this URL won't be used directly; they are only necessary for the final URL created in merge_urls(). The "absolute" and "local" flags are also set only in merge_urls().

If an error occurs during URL parsing (either the protocol specification contains an unknown protocol type, or an unexpected character is encountered), split_url() sets "proto.type" to "PT_INTERNAL" and immediately returns. As "PT_INTERNAL" normally can't ever be generated in split_url(), the caller knows it was an error by this. Misusing "PT_INTERNAL" for this is surely a bit confusing; it has a big advantage, though: The created (empty) pages are correctly handled as temporary by all the page history functions without needing any exception handling.

http.c

The functions for handling HTTP resources are a bit more complicated, and reside in a source file on their own.

Things get slightly more interesting when some form data is to be POSTed. (GET needn't be handled here, as the form data is already encoded into the URL before calling http_init_load() in this case.) Besides of adding the "Content-Type:" and "Content-Encoding:" header fields, the form data has to be submitted inside the body of the request. For that purpose, the form data is encoded using mime_encode() or url_encode() (see hacking-links.*) -- dependending on the desired encoding stored in the "method" field of the form item -- and stored in "form_data", which is a string submitted at the end of the request.

During the whole exection time of get_http_socket(), user breaks are enabled, so that all the slow functions (DNS lookup, connecting to server, submitting request), which are often even waiting for a timeout on failure, can be interrupted.

After the connection has been established with get_http_socket(), parse_header() is used to read and parse the HTTP headers.

Errors in HTTP loading are handled by setting "RES_FAIL" and "PT_INTERNAL", just like file loading errors are handled in init_load().

The parser is very similar to the one in parse_syntax() (see Parsing in hacking-layout.*): The input is processed character-wise by an FSM parser, where each char is read from a buffer which is refilled (more or less) transparently each time all chars have been processed.

The parser first skips the status line (we do not care about the return code, for now...), and afterwards parses all header lines, extracting the name and value of each, and storing them inside the "headers" structure of the HTTP handle so they can be used later. (Presently, the only one used is the "Location:" header, which is cheked in init_load() to handle redirections.)

The parser tries to be as tolerant as possible about broken (or unknown...) syntax. For that matter, '\r' characters are completely ignored, so both the DOS-like '\r\n' linefeeds and unix-like '\n' do work. Illegal characters in the header name are just ignored, as well as a missing space after the ':' separating header name and value, and spaces at the beginning of a line which can't be a folding.

There are a couple of both correct and broken example files in the test/ directory. Use them by piping the contents to tcplisten and pointing netrik to the listening port. (You can also concatenate some HTML file to avoid "No data" errors.)

The data (from the TCP socket) ending before the whole header was parsed is treated as an HTTP loading error, except when a user break was performed, in which case parse_header() simply returns without taking any more action.

User Breaks

The loading of a file or HTTP page can be interrupted by sending SIGINT. interrupt.c contains a couple of functions to faciliate the SIGINT handling. Those are called from various places in load.c and http.c; main() calls init_int() at startup, which sets some constants used in the other functions.

At the beginning of init_load(), hold_int() is called. This function uses sigprocmask() to block SIGINT. The signal is put on hold, i.e. it doesn't have it's usual effect of aborting the program anymore, but it's also not discarded; instead, it's stored, and awaits its release.

The signal is released using enable_load(), which is called in two places: during read() or fread() in load(), and during get_http_socket() in http_init_load(). (These are the functions which may take fairly long, and need a way to be interrupted directly.)

enable_load() sets int_handler() as the handler for SIGINT (using sigaction()), and then unblocks the signal with procsigmask(). However, not only arriving new signals will invoke int_handler() now; if some signal was sent between the hold_int() and enable_int(), it will be deliverd after enable_int(), too. This way, every SIGINT during the whole loading process will cause a break, not only if it was sent during the periods where it can be handled.

Before calling enable_load(), a return point is set using setjmp(), and stored in "label_int". int_handler() does nothing else but immediately jumping to that return point with longjmp(). That means, when an interrupt occurs before or during the read()/fread() in load(), the signal handler jumps to another position in load(). Here, "res->user_break" is set to indicate the interrupt, and load() returns, instead of continuing the read(). Likewise in http_init_load().

At the end of load() and http_init_load(), hold_int() is called, so SIGINT will be put on hold again till the next call of load().

uninit_load() calls disable_int(). This function used sigaction() to set the SIGINT handler to SIG_IGN, and then unblocks the signal. Thus, any SIGINTs arriving after file loading has finished are discarded. They are only enabled again in init_load(), when the next file load beginns.