User Manual for Wsmake 0.6.x www.wsmake.org www.wsmake.org 0.5.5 September 3 2001 mlb Updated for Wsmake 0.6.3 0.5.4 June 30 2001 mlb Updated for Wsmake 0.6.1 0.5.3 May 05 2001 mlb Updated for Wsmake 0.5.13 0.5.2 Mar 18 2001 mlb 0.5.1 Jan 14 2001 mlb Edition fix 0.5 Aug 14 2000 mlb Updated for Wsmake 0.5.10. 0.4 Dec 27 1999 mlb rkb Updated for Wsmake 0.5.9. 0.3 Nov 16 1999 mlb rkb Updated for Wsmake 0.5.8. New layout for introduction. 2001 $Date: 2001/09/04 01:52:38 $ 2001Michael Brownlow Permission to use, copy, modify, and distribute The Wsmake User Manual for any purpose and without fee is hereby granted in perpetuity, provided that the above copyright notice and this paragraph appear in all copies. The introduction first discusses the need for a tool like Wsmake, then explains how Wsmake is designed to facilitate that need. Areas of need discussed are website implementation advantages and potential resource efficiencies that may be gained. The design discussion familiarizes the user/developer with the terminology and organization of Wsmake. This chapter describes how to use Wsmake. It covers the command line options and several ways Wsmake can be used. Wsmake uses a make file named wsmakefile to understand what it is supposed to do. This chapter describes how a wsmakefile file is built and what options are allowed for each section. Overviews of the examples provided with the Wsmake package are included here. This section describes the licenses covering the use of Wsmake. Welcome. This text is the user manual for the program Wsmake, a website pre-processor. The basic function of this tool is to process a set of files (e.g. webpages), from a "source" location into an "output" location. Along the way, various types of modifications are done to achieve a desired result.

The structure of a webpage has both common and unique content. Common content is information that is the same from page to page. Unique content is information that is associated with a single page. A benefit of separating common from unique content is the ability to change the common content of the structure without interfering with the unique content. The ideal way to manage the common content is to have a single source for each common content part and, by some mechanism other than manual effort, replicate the common content throughout the site. The ability to re-use the common structure has potential to lower maintenance time and reduce errors. An example of common content is the header and footer information for a web page. The header and footer could contain navigation control, ads, or contact information. An example of unique content could be news items, a company memo, or a price list. Wsmake is a tool which provides the framework for you to define common, re-usable structures, which are then used, along with the unique content, to generate the pages for your website.

There are several ways for webpage structure to be created dynamically when a client makes a request. Webserver directives, modules, CGI scripts, and embedded languages are example implementation possibilities. However, the common attribute for these is that the page being sent to the user is often created dynamically at the time of the request. Wsmake is different because it works externaly to the webserver's response to requests. It creates the webpage only when changes are made to it, not when the client makes the request. The created webpage is a static entity which is served directly, with no need to create the webpage structure on the fly. This can potentially save computing resources.

For the same reasons that Wsmake can yield lower system resource consumption, there can be a time savings associated with manual labor by: Eliminating the need for common content generation code in CGI and embedded scripts. Making a change in only one place in order to update an item of common content. When dynamic page creation on the server is not available, it is possible to save hours of tedious work with Wsmake. For instance, consider a situation where there is a company name change which affects every webpage on a website. If the site has many pages, the most efficient way to make the change is to change one occurrence of the name and let Wsmake do the rest.

Wsmake was made to simplify website maintenance. It does this using a parser, a configuration file, and configuration file syntax which describes the website to be generated. The default configuration filename is wsmakefile and files that end with .ws, but you can use a specific name.

Inputs to Wsmake are: These inputs alter the behavior of the make process and amount of feedback. This file defines the website structure and make process. Any file on the filesystem in any format. Certain conditions may need to be set for binary files.

Wsmake was made to reduce the effort required to maintain a website. However, it is not limited to a website filesystem. It's programming is abstract enough to allow a developer to order any group of files to be compiled into a final version. A website filesystem happens to be a good example. As such, it is the primary focus for this discussion. Processing first handles the command line options and arguments. It then loads the configuration file(s). The configuration file is needed for Wsmake to know what to do. Much the same way that GNU Make uses a Makefile. Each configuration file can define one website. In the file there are page groups. In each group, the parser is setup and one or more pages are declared. Wsmake loads the entire configuration file and then processes each group in the order they occur. Each page represents a source file which will be compiled into one or more output files.

Each PageGroup section is made up of attributes and declarations that identify website components and structure. The section also defines how the output should be generated. These attributes and declarations will be described in order of terminology dependency. Detailed definitions are described in Chapter 3.

SubTag means Substitution Tag. The SubTag has two parts, a tag name, and a value. When the parser in Wsmake parses the source text, it looks for a tag name. When one is found, the value for the tag is put in place of the tag before the text output is written. You can also create a SubTag whose value is the output of an external command.

A SubTagGroup is a group of SubTags. Defining a SubTagGroup allows a set of SubTags to be re-used. Each SubTagGroup has a name attribute which can be referenced by some of the other parts of a processing section.

A PageOrder is used to define the order of a set of files that will be compiled into an output page. PageOrders use PagePart declarations to identify those files. The focal point of a PageOrder is the unique content. This data is not known to the PageOrder. However, a placeholder for the unique content is placed in the order to identify its occurrence. The label "data" is used for the placeholder. This allows a PageOrder to be re-used to create the common content for many unique content (data) pages. PageParts define a source file and whether or not it should be parsed in preparation for a Page output. They can define SubTags and reference SubTagGroups. Each page part also has a name attribute. The PagePart name is what is referenced by PageOrders. For any given PageOrder, PageParts can be referenced either before or after the unique content. For instance: header unique content footer In this case "header" represents a PagePart that will be pre-pended to the unique content. Similarly, "footer" represents a PagePart that will be appended to the unique content.

A Theme is a way to group SubTags, SubTagGroups, and PageOrders into one place so that the group as a whole can be referenced by Pages.

A Page represents a page on the website. It defines which PageOrder(s) and which unique content source file will be composed into a website page. It also defines whether or not the unique content file will be parsed, and the output path for the new composite file. A Page may also include SubTag definitions and references to SubTagGroups. The following discussions describe how each Page is handled by Wsmake.

The parser works on a line by line basis. It reads in a line from the source page, manipulates it, and writes it out to the output page. The parser will create the file if it does not exist. It also creates directories when necessary.

There are three references that a Page can use: subtaggroup, pageorder, and theme. When references are used, the output page takes on the characteristics of the referenced information. In the case of a referenced SubTagGroup, the SubTags in that SubTagGroup are added to the list of SubTags already defined for the Page. For referenced PageOrders, the order they are defined is the order in which they are applied, so that the first pageorder referenced will be used for header and footer additions to the output. Then the next pageorder adds to the previous definitions, and so on. When a Theme is referenced, the page acquires all of the properties defined in the theme.

Each Page has the capability of being cloned. This means that Wsmake will process the source of the page not only for the original definition, but for each clone as well. Clones are treated just like real Pages, and can have all of the properties that the Page can have. Depending on where the Clone is defined inside a Page, certain properties of the Page are inherited by the clone.

There are three types of output: Normal Feedback, Error Feedback, and File Output. See Chapter 3 for a detailed description of these types.

Several "debug" levels are provided which can alter the verbosity. The default feedback consists of showing the user the current state of updated Pages, and the process of making (parsing) the particular Pages that are in need of an update. With debugging set to higher levels, more information is reported about what Wsmake read from the configuration file.

When Wsmake fails in some part of its run-time, errors are reported to the standard error output for your operating environment. Types of errors include: Configuration file load errors, Database errors, Synchronization errors, and Make errors.

The files created by Wsmake will take on the same permissions as the source version it was created from. The timestamp is not replicated by default.

The basic usage of Wsmake is: shell$ wsmake -f configfilename This runs Wsmake using the configuration file named configfilename. The defaults for the output are to report the status of pages which have been updated. Typical output might look like: shell$ wsmake -f configfilename Upd Part : templates/header.html PoU Webpage: html/index.html [...] shell$ The output shows all Pages that are in need of updating. The 3 character state on the left of these lines can be one of the following: --- This means there was no change since the last run Err This means that there was an error in determining the state of the Page. This could be because of: Incorrect Permissions Non-existent directories Database problems New This means that the Page was not in the database on the last run, but now is. And it will be updated. Frc This means that the Page will be updated, regardless of it's state, by a request from the user, or because a clone's original was updated. Upd This means that the source for this Page was modified, so the output will be updated. NoO This means that the Page has no output file and will be updated. NoS This means that the Page has no source file Old This means that the output file is old and will be updated. PoU This means that one of the PageOrders that the Page uses has a PagePart whose source has been modified. So the output page will be updated. ThU This means that one of the Themes that the Page uses has been been updated. So the output page will be updated. Dep This means that one of the files that the Page has been made to depend on has been been updated. So the output page will be updated. StU This means that one of the SubTags that the Page uses has been updated. So the output page will be updated. SGU This means that one of the SubTagGroups that the Page uses has been updated. So the output page will be updated. For all cases the page will be updated unless the state is ---, Err, NoS, or Unk. The next word indicates the type of page: Part, Webpage, or Clone. When square brackets appear after a target, it represents the start and stop of the make process for that page. The characters in between can be: . A piece of the page. i A file was included by one of the pieces. X A piece of the page source was missing.

Wsmake has the following command line options: wsmake [-Fshvct] [-m|-e] [-d #] [-f configfile] [target] ... -F,--force Force files to be updated regardless of their state. -s,--silent,--quiet Silent. No output. -h,--help This help. -v,--version Show version and copyright information. -c,--clean Clean the website. -t,--times Sync timestamps between source and output files. -m,--match Targets match anywhere in web paths. -e,--end-match Targets match at end of web paths. -d #, --debug=# Debug output level. (0-3) 0 - No output except for errors 1 - Level 0 and minimal output (default) 2 - Level 1 and verbose output 3 - Level 2 and config file loading info -f file, --file=file, --wsmakefile=file Configuration filename (looks for wsmakefile and files ending with .ws if option not given targets Specific web paths, matching from front, to make. If not specified, all pages will be made. "clean" is a special target name. When used, all other targets are ignored and the website is cleaned. To clean only certain groups of files, use the -c or --clean option with the targets but without the clean target. Note that updates to the configuration file will not update the website on the next run. Using the -F or --force option is one solution to this problem. Another is to use a depend attribute (discussed in more detail in chapter 3).

Output all details: shell$ wsmake -d 3 -f website.conf Force build of entire website: shell$ wsmake -Ff website.conf Don't show any output except for errors: shell$ wsmake -d 0 -f website.conf Don't show any output at all: shell$ wsmake -s -f website.conf Make just index.html: shell$ wsmake -f website.conf index.html For config files named wsmakefile or ending with .ws, make all pages that end with ".html": shell$ wsmake -e .html Make all files in the help directory: shell$ wsmake -f website.conf help Clean the website: shell$ wsmake -f website.conf clean or shell$ wsmake -cf website.conf

The Wsmake configuration file is made up of sections. The first section defined in a configuration file is called Website. Currently there can only be one Website section for each configuration file. All of the other sections can have more than one instance, however, there are restrictions to their location. In any given config file, the general layout is: Where "..." indicates an item which can occur more than once. As an example, SubTagGroups can have multiple instantiations, but must be in the PageGroup section, which must be in the Website section. The order that the items should be declared is the same order of their dependencies, e.g., if you define a PageOrder that uses (references) a certain PagePart, that PagePart must be defined before the PageOrder. Details on what a section can reference are described later in this chapter.

Each section has the following syntax: SectionName { } The section name goes where "SectionName" is defined. The beginning and ending curly brace indicate that the contents within apply to that Section. Note that the first curly brace must be on the same line separated by whitespace after the section name. Also note that the ending curly brace must be on a line of its own. (this behavior is tentative, and will change in the future)

Attributes in each section are limited to one "line." The line can span several physical lines by using the escape charcter ("\") followed by a carriage return. The maximum line length is 2048 characters. Attributes are defined within the following syntax: AttributeName value The attribute's name goes where "AttributeName" is. The "value" is all characters after the attribute name and space. This means you do not need quotations to include whitespace. However, leading and trailing whitespace are removed before processing, so quotations are necessary for leading and trailing whitespace. The valid quotation recognized is double-quotes. Back-quotes are recognized for SubTags and have special meaning (described later). (note again that the behavior described is tentative) AttributeName " The Name value " In order to use a double quote inside the value, escape it with a '\': AttributeName "The \"Name\" value" The only escape sequences processed are the double quotes and carriage returns for multi-line attributes. Any other '\' usage is literal. To have double quotes in the value, always escape them and enclose the entire value in double quotes.

Comments can be defined by using the '#' character. Any data following a '#' is ignored by Wsmake, unless it is within quotations of a value: url http://mysite.com/index.html#about comment url "http://mysite.com/index.html#about"

SubTag attributes are a special type of attribute. They are defined in the same fashion as an AttributeName, but the value is made up of two parts: SubTag "SubTag Name" "SubTag Value" The value is composed of both the SubTag Name and SubTag Value together. The entire attribute must be all on one line. Note that quotes are not mandatory for SubTags. So it is possible to do something like this: SubTag SubTag Name SubTag Value However, Wsmake will split the string on the first whitespace character. So, in the case above, the subtag name will be "SubTag", and the value will be "Name SubTag Value". A command-based SubTag can be created by using back quotes around the value part: SubTag "DATE" `date` In this case, whenever the string DATE is found in the source, it will be replaced by the output of the command "date". Note that the date command will be run only once before any pages are parsed, and the output from the command is the same across all pages. (note again that this is tentative)

The Website section holds all information about a Wsmake project. There can only be one Website section for each configuration file. The PageGroup subsection is the only sub-section that can be defined within a Website section. There can be multiple PageGroups. Name Required / Optional Description name optional The name of the website maintainer optional The maintainer of the website url optional The url of the website include optional Include a file as part of the configuration The name, maintainer, and url settings are identifiers only and do not play a role in the parse process. When the value of an include attribute does not start with a '/', the file is looked for relative to the configuration file. Website { name "Mike's Website" maintainer "Mike" url "http://wsmake.org/~mike/" include "pagegroups.ws" }

The PageGroup section holds common information for a group of Pages. There can be multiple PageGroup sections in the Website section. The common content items fall into two categories: attributes and subsections. The following subsections can be defined within a PageGroup section: SubTagGroup PagePart PageOrder Theme Attributes will be discussed first. Table 3-2 lists the PageGroup attributes. Name Required / Optional Description lastmod_tagname optional Internal subtag, replaced with last modification date of source file wsmake_version_tagname optional Internal subtag, replaced with version of Wsmake output_page_tagname optional Internal subtag, replaced with output_page of current page source_tagname optional Internal subtag, replaced with the full path of the source page output_tagname optional Internal subtag, replaced with the full path of the output page header_tagname optional Internal subtag, replaced with the full path of the header output for use with CGI scripts footer_tagname optional Internal subtag, replaced with the full path of the footer output for use with CGI scripts include_tagname optional Internal subtag, replaced with contents of file referenced (see details below) time_format optional Format of time/date printouts. This is the same format as defined by the strftime function in C (see the manpage for strftime for details) subtag_format optional Specifies a generality about all subtags in a pagegroup (see details below) database_format required Format of database. One of DB, CSV, or Timestamp. Defaults to CSV. database_filename required *** Filename of database for the PageGroup source_dir required * The default source directory output_dir required * The default output directory part_dir optional * The default PagePart directory include_dir optional The backup include directory command optional The default external command to use on the data after the wsmake parser. Default is none. options optional The default options for the default command depend optional A file that this PageGroup depends on include optional Include a file as part of the configuration * Can be overridden in PageParts and Pages. If part_dir is not specified, source_dir will be used for PageParts. ** Can be overridden in Pages. *** The filename is optional when the Timestamp format is requested. But even if one is given, no data is stored in it. When the Timestamp format is requested, only the source pages have bearing on the update process. This means changing SubTags, Depends, or other entities that would normally be tracked, will not have there state cached. The include_tagname subtag is used to specify a tag name for including a file during the make process. For instance: include_tagname "include(%s)" The specified tag name requires the characters "%s" somewhere in the string to represent the name of the file to be included. This means that for every occurrence in the source where the string "include(" is found, with some filename text and a ")" after it, Wsmake will replace it with the contents of the filename given: This is HTML. Now a file will be included: include(/web/include.txt). This will take the entire string "include(/web/include.txt)" and replace it with the contents of the file /web/include.txt. The start and stop string before and after the filename is required. If the filename doesn't begin with a '/' character, the file is first looked for relative to the source file that is being parsed, and the in the directory defined by include_dir, if specified. Globs, however, are only looked for relative to the source file. The subtag_format definition can be used to abstract a common part of all subtags so that you can define them without the common part. For example, if every SubTag you use on a website is of the form: `subtag` Then you can specify a ssubtag_format of: subtag_format "`%s`" Then when you specify SubTags, it is not necessary to include the extra information in the subtag_format: SubTag "test" "here is the value" With the subtag_format definition above, Wsmake will look for `test` when it parses the source data. As in the Website section, the include files will be looked for relative to the configuration file if not prefixed with a '/'. The same is true for the depend attribute. PageGroup { database_filename "website.db" lastmod_tagname "`LASTMOD`" wsmake_version_tagname "`WSMAKEVERSION`" web_path_tagname "`FILEPATH`" source_tagname "`FILESOURCE`" output_tagname "`FILEOUTPUT`" include_tagname "include(%s)" header_tagname "`CGI_HEADER`" footer_tagname "`CGI_FOOTER`" source_dir "/web/source" output_dir "/web/docs" part_dir "/web/templates" include_dir "/web/includes" command "myparser.pl" options "<%i >%o" depend "pagegroups.ws" include "pages.ws" }

The SubTagGroup section defines a group of SubTags which can be re-used by PageParts and Pages. The following attributes can be defined in a SubTagGroup section: Name Required / Optional Description name required The name of the SubTagGroup SubTag optional A SubTag attribute There can only be one name for a SubTagGroup. SubTags may be defined 0, 1, or many times. SubTagGroup { name "default" SubTag "`EMAILLNK`" "<a href=\"mailto:`EMAIL`\">`EMAIL`</a>" SubTag "`EMAIL`" "mike@wsmake.org" SubTag "`FONTFACE`" "Arial, Helvetica" SubTag "`HOST`" "http://wsmake.org/~mike/" } Note in this example how a SubTag was used inside of another SubTag. This is valid as long as the SubTag being used inside is defined after its use. Remember that SubTags are parsed in the order that they are declared. You cannot nest a SubTag within its own value.

The PagePart section defines a file to be used as a piece of a PageOrder. It also references any PageOrders it needs to complete itself. The following attributes can be defined in a PagePart section: Name Required / Optional Description name required The name of the PagePart filename required The filename of the pagepart (no directory!) directory optional * The directory that the file is in SubTag optional A SubTag to use for this PagePart subtaggroup optional A SubTagGroup to use for this PagePart pageorder optional A PageOrder to use for this PagePart theme optional A Theme to use for this PagePart parse optional A value of NO, YES, or INCLUDES_ONLY which tells how Wsmake should parse this PagePart. The default is YES depend optional A file that this PagePart depends on * If the directory is not defined here, it will use the part_dir attribute defined in PageGroup. There can only be one name, filename, directory, and parse attribute for each PagePart. The rest can have multiples definitions, and they are applied to the PagePart in the order that they are defined. PagePart { name "header" filename "header.html" directory "templates" SubTag "`INFO`" "Header piece" subtaggroup "default" parse YES }

The PageOrder section defines a group of files to be used by PageParts and Pages for ordering content. The following attributes can be defined in a PageOrder section: Name Required / Optional Description name required The name of the PageOrder part optional A PagePart name to use data required A placeholder for the unique content of the page. It has no value. PageOrder { name "default" part "header" data part "footer" } Any page that uses this pageorder will have the PagePart whose name is "header" prepended to it and will have the PagePart whose name is "footer" appended to it in the output file. The "part" definitions are applied in the order that they are defined. There can only be one name and data definition for each PageOrder.

The Theme section defines a set of SubTags, SubTagGroups, and PageOrders to be reused by Pages. The following attributes can be defined in a Theme section: Name Required / Optional Description name required The name of the Theme SubTag optional A SubTag to use for this Theme subtaggroup optional A SubTagGroup to use for this Theme pageorder optional A PageOrder to use for this Theme command optional The external command to use on the data after the wsmake parser. options optional The command line options to use for the parser specified by command depend optional A file that this Theme depends on Theme { name "home" SubTag "`SECTION`" "Home" subtaggroup "home" pageorder "home" }

The Page section defines a Page on the website. Each Page represents an item to be made by the make process. The following attributes can be defined in a Page section: Name Required / Optional Description web_page required ** The web page of the Page, i.e., the directory after the http://servername/ including the page filename. source_page required ** A string that, when appended to the source_dir attribute, represents the filename of the source for this Page. output_page optional A string that, when appended to the output_dir attribute, represents the filename of the output for this Page. If this is not set, the corresponding web_page or source_page value will be used. source_dir optional The directory prefix to web_page and/or source_page that represents the source location output_dir optional The directory prefix to output_page that represents the output location time_format optional Format of time/date printouts. This is the same format as defined by the strftime function in C (see the manpage for strftime for details) subtag_format optional Specifies a generality about all subtags in a pagegroup (see details in PageGroup section above) SubTag optional A SubTag to use for this Page subtaggroup optional A SubTagGroup to use for this Page pageorder optional A PageOrder to use for this Page theme optional A Theme to use for this Page page_type optional A value of NORMAL, or CGI which tells Wsmake the way it should output the header/footer information. The default is NORMAL parse optional A value of NO, YES, or INCLUDES_ONLY which tells how Wsmake should parse this PagePart. The default is YES cgi_header optional This defines the name of the header file to output to (relative to the location of the output) for CGI page_types cgi_footer optional This defines the name of the footer file to output to (relative to the location of the output) for CGI page_types command optional The external command to use on the data after the wsmake parser. options optional The command line options to use for the parser specified by command depend optional A file that this Page depends on ** Either web_page or source_page may be defined. They are both considered the same. It may be more readable to use web_page if both the source_page and output_page are the same value. There can only be one web_page/source_page, source_dir, output_dir, page_type, parse, cgi_header, and cgi_footer definition for each Page. When the page_type is CGI, the output of the header and footer information (if any) goes to the files defined by cgi_header and cgi_footer. If cgi_header or cgi_footer is not defined, "header.html" or "footer.html" is used as a default, respectively. For options, '%i' is replaced by the full path to the wsmake parsed source file and '%o' is replaced by the full path to the output file at make time. Other options are '%s', the full path to the source file, '%w' the web path of the page, '%ds', the full path to the directory of the source file, '%do', the full path to the directory of the output file. If options is not specified, but the command is, the default options from the PageGroup section are not used. Page { web_page "index.html" source_dir "html" output_dir "docs" SubTag "`FONTFACE`" "Courier" subtaggroup "default" pageorder "default" page_type NORMAL parse YES }

In this chapter we overview the set of examples that come with the Wsmake package. The locations given are relative to the doc directory in the source package.

Location: examples/ex1/README This example covers the minimal use of Wsmake. It describes what to expect when running Wsmake as well as the use of relative directories and the use of comments.

Location: examples/ex2/README This example covers the use of SubTags for tag substitution in a webpage.

Location: examples/ex3/README This example demonstrates how to use SubTagGroups with webpages.

Location: examples/ex4/README This example covers page ordering and describes how the dependencies work out when updates are made.

Location: examples/ex5/README This example shows how to use the include_tagname attribute to have files included during the make process.

Location: examples/ex6/README This example shows how PageParts can have PageOrders. It also describes the order in which Wsmake creates the pages that utilize this.

Location: examples/ex7/README This example shows one way of making a menu for a website using a combination of PageOrders, SubTagGroups, and SubTags.

Location: examples/ex8/README This example shows how you can use CGI scripts with Wsmake. It shows how header_tagname and footer_tagname can be used by the script to generate the final page.

Location: examples/ex9/README This example shows how to use other pre-processors with Wsmake.

Location: examples/ex10/README This example shows how to use clones to make multiple output pages from a single source page. It then shows how to use the output from one page made by wsmake as input to another page.

Location: examples/ex11/README This example explains what themes are and how they are used.

Location: examples/ex12/README This example explains how to use the depend attribute for pages.

Location: examples/ex13/README This example explains how to use the include attribute.

Wsmake may be copied only under the terms of either the Artistic License or the GNU General Public License, which may be found in the Wsmake source distribution. If you have any doubt about using Wsmake in a way that may violate the license, please contact mike@wsmake.org and we'll discuss it.