=begin rakudoc :type<fundamental>

Overview

The module contains the RakuDoc::Processor class.

The class method 'render' is the principal one for rendering RakuDoc v2. The Raku compiler expects to call render with an AST and to return a Str. It can be called with the option :pre-finalised to get the processed data, see the example in Debugging.

The aim is for a RakuDoc::Processor object (an RPO) to be as generic as possible, allowing for other classes to instantiate it, and to attach templates that will generate more specific formats, such as HTML, MarkDown, or EPub pages.

An RPO relies on Templates, PromiseStrings, and ScopedData.

It is also necessary to have a good understanding of RakuDoc v2.

Some environment variables provide for some output control.

The following describes choices that are left by RakuDoc v2 to the renderer.

Templates

An RPO contains a default set of Text templates.

The aim of the text templates is that they are the lowest level templates needed to render any RakuDoc source.

When a renderer is needed to output a new format, eg., HTML, the renderer instantiates an RPO and attaches new templates to it using the add-template( Pair $p, :source ) method or the add-templates( Hash $h, :source ) method. Setting the source option to some value will help determine which module introduced the templates.

The design of the Templates object means that new additions to the object push the previous definition onto a linked list, and so they will always be available by default.

The set of template keys needed to create a renderer is tabulated here.

All blocks make be prefixed by num. The renderer handles the enumeration before handling the block. The num handling sets the config variable numeration, which is the enumeration string as defined by the specification.

Consequently, all block templates need to handle the numeration value, which is '' for non-num blocks.

All aspects of the output can be defined using the templates.

Process

The generic render method transforms the AST of the input file using a handle method for each type of Block and Markup code. A template exists for each block and markup code.

After all the blocks have been processed, the Table of Contents, Index, and Footnotes are rendered, with templates for each item and for the list of items.

Any forward references and numerations are then resolved.

The intermediate structure of the processed file can then be accessed before it is finally rendered into a string.

The final rendering is done by the 'final' template.

Once a string has been obtained, it can be post-processed using the post-process method. For example, text output can be transformed by wrapping lines.

Choices left by RakuDoc v2 to renderer

There are a few choices left by the specification to the renderer. This section contains a description of those currently known (please raise issues if others are discovered), except for rendering options regarding customisability that are covered separately below.

• not all schemas listed for L<> and P<> are implemented, namely man:, rakudoc:, isbn:

• the toc: schema is implemented for P<> only in relation to level. Limiting the ToC to tables or other named blocks is not yet implemented.

• The index: schema is implemented in relation to level. The index structure is compiled as follows: each item of an index contains a label (the key of a hash structure), an array of references, and a sub-index of children with the same structure.

  The reference array may be empty if the sub-index has elements, and vice versa.

  The level of an index is the depth of the label. But if a label only has elements in a sub-index and no references of its own, AND the index is constrained to some level, then the label may appear without a link to the body of the text. The renderer provides an ordered list of items. It is for the template to determine whether to filter 'empty' labels.

• numitem and numdefn counters are reset if the next =block is not the same type. This behaviour is extended to para and nested.

Warnings

In order to offer the maximum flexibility to a RakuDoc author, if there is an error condition after the RakuDoc source has been compiled, a set of warnings are generated.

By default these are rendered using the 'warnings' template, and the rendered version is appended to the output file.

Customisability

RakuDoc allows for customisability. This renderer handles customisability via the template mechanism and Plugins.

Custom blocks

When custom block is detected according to the RakuDoc v2 spelling rules, an RPO will check whether a template exists with that name.

If a template has been attached to an RPO, then the rendered content of the block (including rendered embedded RakuDoc instructions) will be provided to the template as contents and the verbatim contents with no renderering will be provided as raw.

So a new block can be created by attaching a template with the blockname to the RPO.

Custom markup codes

RakuDoc v2 allows for custom markup codes in two ways:

• the < M< DISPLAY | FUNCTION ; list of comma delimited strings > >

• a single Unicode (not ASCII ) character with the upper property.

Markup M

When a < M< DISPLAY | FUNCTION ; LIST > > is encountered, the renderer will look for a template named in the FUNCTION position. If no such template exists, the text will be rendered verbatim and a warning logged.

Markup UNICODE[upper]

When some custom markup is encountered, the renderer will

• check that the character, eg Ɵ (Greek Capital Theta), has the unicode property Upper;

• look for a template called (eg) markup-Ɵ. For clarity, the Unicode character chosen for the markup code is prefixed by markup- in order to name the template.

Thus in order to create a new markup code, create an appropriately named template and add it to the RPO.

Plugins

Typically an output, such as HTML, will require the content of the text to be organised in some way, but it will also need additional files such CSS or JS content. The templates can handle the content organisation, but obtaining JS and CSS libraries, or converting SASS to CSS involves other code to be run.

These can be organised using plugins.

A plugin can attach assets (such as Javascript content) to a hash in its data workspace. Other plugins can operate on the data in these workspaces. An example is the RakuDoc::Plugin::HTML::SCSS plugin.

To enable plugins to work, the add-plugins( @plugin-list ) method of the RakuDoc-Processor (rdp) object is called by the renderer. The @plugin-list must contain the full name of each plugin.

Each plugin must have a enable method that takes an rdp. Typically, the enable method will attach its config attribute to the workspace, and attach its templates.

gather-flatten( $key ) / gather-flatten( @keys )

Typically many of the plugins will define SCSS, Javascript, or links. Since the relative order in which (eg) css style sheets are attached matter, each attribute needs to be defined as a sequence of 'Str',<order> tuples.

This method takes all the $key attributes, sorts them by order, then forms a single sequence that is then used in the templates, eg. to form stylesheets in the head part of an HTML file.

Global functions

Each output format requires a different escaping strategy both for content and file names.

This is handled by subclassing Processor object and over-riding escape and mangle methods for each output. See for example how RakuDoc::To::HTML manages this.

Since these functions are also needed inside templates, they can be called as follows:

• $tmpl.globals.escape.($s), where $s is a string. Note the mandatory . before the ( )

• $tmpl.globals.mangle.($s)

Global data accessible in templates

More information is available in Templates.

All templates can attach items to the Table of Contents, Index, Footnotes, and Warnings structures of the rendered source using the helper methods.

Currently, templates can add but not substitute or remove items from structures. For example, within a template with a custom block, the ToC structure has been changed when the block is specified, so adding to the ToC within the custom block's templates will add information to the ToC structure immediately under the heading for the block. It is necessary, therefore to add a :caption to the custom block in order to over-ride the default ToC behaviour.

The following helper methods are available:

• add-to-toc( %h ), where %h is a hash with the following keys:

• :caption(Str)

• :target( Str )

• :level( Int )

• add-to-index( %h ), where %h is a hash with the following keys:

• :contents(Str)

• :target(Str)

• :place( Str }

• add-to-footnotes( %h ), where %h is a hash with the following keys:

• :retTarget(Str)

• :fnTarget(Str)

• :fnNumber(Str}

• add-to-warnings( $warning )

In addition, within a template, it is possible to attach data to globals, and to retrieve the data in another template.

For an example of this, see test xt/030-customisation-data.rakutest. (test number may be changed, but otherwise the filename will be the same).

Debugging

When developing new templates or renderers based on an RPO, several debug options can be attached to the RPO, eg.

    my RakuDoc::Processor $rdp .=new( :test );
    $rdp.debug( AstBlock, Templates);
    $ast = Q:to/QAST/.AST;
    =begin rakudoc
    =head This is a header

    Some text

    =end rakudoc
    QAST

    $rv = $rdp.render( $ast, :pre-finalised );
    'myOutput'.IO.spurt: $rdp.finalise

Commentary on the code

• new( :test ) this attaches the test templates to the RPO

• the list inside the debug method may contain several entities as described below

• creation of an AST from a string, or it can be slurped in from a file.

• :pre-finalised returns the RakuDoc::Processed object

• if called with :pre-finalised, calling finalise returns a Str that can then be stored as a file.

Debug options

The following options are available:

• None removes all previous debugging

• All adds all debugging options

• AstBlock diagnostic whenever a new Block is called

• BlockType information about which RakuAST::Doc::Block is being processed

• Scoping produces scope level diagnostics when a new level is started/ended

• Templates indicates which templates are called and the params they are called with

• MarkUp like BlockType but gives the MarkUp letter

It is also possible to get the result of one template (so as to reduce the amount of output information). This is done eg for the 'table' template: $rdp.verbose( 'table' ); $rv = $rdp.render( $ast, :pre-finalised );

The Test and Pretty options described in the Templates documentation can be set on an RPO, eg. $rdp.test( True ); $rdp.pretty( True );

Bear in mind that the pretty flag overrides the test flag, and both override the debug and verbose flags.

Environment variables

By setting the environment variable POSTPROCESSING=1 the text output will be naively wrapped.

For example,

POSTPROCESSING=1 bin/RenderTextify --pretty rakudociem-ipsum

If the environment variable WIDTH is also set, the text output will be wrapped to the value.

WIDTH by default is set at 80 chars. To set at 70, use:

POSTPROCESSING=1 WIDTH=70 bin/RenderTextify rakudociem-ipsum

Paragraph id

Since sometimes it is necessary to target a specific paragraph, some paragraphs get an automated id based on the last Seven hex chars of the SHA1 encoding of its contents.

Seven chars should be adequate so long as the number of paragraphs in a single document is below 16,000.

Based on the expectation that we would see collision in a repository with 2^(2N) objects when using object names shortened to first N bits Stackoverflow question

Long texts or books will probably need more to avoid a conflict. This can be done by setting paragraph-id-length in the %structure-data to the required number of hex digits.

Writing a new Render

Typically, it should only be necessary to change the templates of an existing renderer to alter the way output is formated. But for a new type of format, a new renderer will be needed.

It is best to look at the code for the HTML renderer to see how to write a renderer. The following are only very brief notes.

A new renderer for a new output format will need: =numitem A processor class, which is sub-classed from RakuDoc::Processor. The escape method will need to be over-ridden.

=begin numitem A Renderer class which must be called RakuDoc::To::NEWFORMAT. This is required so that the code

raku --rakudoc=NEWFORMAT input-file.rakudoc > output-filename.newformat

will work. =end numitem =numitem The Renderer class must contain the method render( $ast ), which is needed by the raku compiler, as noted above.

=numitem The class should also define and attach templates. Note that if any needed templates are not included, then the default 'text' templates will be used.

=place semantic:AUTHOR :caption<Credits>

=place semantic:VERSION :!toc

=place semantic:LICENSE =end rakudoc