Grant Proposal - Make documentation installable and accessible

Raku has code for handling the installation of distributions through CompUnit::Repository::Installation. It handles the part of the installation process that involves copying the files into their final location, path normalization, precompilation, as well as providing an interface for querying what is installed. Unfortunately it does not yet handle anything related to documentation.

Benefits to the Raku community

Developers can query for documentation

Currently there is no useful way of finding documentation for Raku distributions. There is a program called rakudoc, but it is generally only useful for core documentation. This would allow developers to query for documentation similar to how they can query for module names.

Developers more likely to write documentation

Since there is not a general way of accessing documentation, documentation in Raku distributions has been nearly non-existent compared to those in the Perl community.

Solves at least half of Raku/problem-solving/issues/393

The issue seen at Raku/problem-solving#393 is about how installing Raku distributions involves installing files that have not been declared in the META6.json file. While that issue affects more than documentation files, the solution implemented by this grant would act as prior art for what needs to be done to e.g. declaratively install bin/ files from a distribution.

Unblocks other RakuDoc related work

One example I can think of is related to the work being done on rakudoc renderers, particularly in regards to generating online html documenation a site like raku.land might use. These tools will be able to leverage the aforementioned META6.json updates.

Deliverables

A META6.json spec for defining documentation related data is created.
Updates to core Raku code that allow it to install, uninstall, and query-for documentation files as defined by the updated META6.json spec.
Docs for CompUnit::Repository::Installation and modules are updated to mention the above changes.
Examples are added (via the roast as spec tests) that show how to access documentation using core Raku functionality.

Note: Integration into rakudoc is not intendended to be part of this. Such an integration will be little more than copying and pasting the aforementioned examples, but there is some other unrelated work that would need to be done as well (making it installable via zef and how the user interacts with the CLI in regards to e.g. api/version/auth and multiple matching namespaces).

Project details

We need to update the META6.json spec to allow declaring our documentation files simlar to how we declare our source files with provides. Something like:

"provides-docs" : {
    "Text::Foo" : "docs/Text/Foo.rakudoc",
    "Text::Foo::DocInModule": "lib/Text/Foo/DocInModule.rakumod",
}

This will allow us to know where to look up the documentation for a given namespace, including know if the source is in a module or a stand alone documention file.

When installing a distribution Raku could then iterate through provides-docs to install (and precompile) the given files. There is a bit of nuance involved in this part due to the way rakudo hashes files to be used in namespace look ups. Rakudo does something like $filename = sha1("Namespace::Being::Installed") when installing modules which is also used when looking up namespaces, but that would be ambigious when we have something like:

"provides": {
    "Namespace::Being::Installed": "lib/Namespace/Being/Installed.rakumod"
},
"provides-docs": {
    "Namespace::Being::Installed": "docs/Namespace/Being/Installed.rakudoc"
}

The Raku logic for querying installed files could then be extended to also search documentation namespaces. This will involve adding a new method similar to the existing candidates method, or alternatively a new argument on the existing method. Again, this will be some nuance here involving the namespace normalization and file system location.

At last we can write the spec tests to show how these pieces are used. We'll want to ensure cases such as the ambigious namespace example above are covered.

Biography

I am Nick Logan, ugexe on github. I have been a co-author of the Raku module manager Zef, and a core developer of the Raku programming language for over 10 years. I am also a member of the Raku Steering Council.

Requested amount

$2500 USD

There has been a recent effort to draw a clear distinction between RakuDoc (.rakudoc) and Raku (.rakumod) files. The RakuDoc V2 spec now draws a clearer border and there are ideas for a tool to convert .rakumod files to .rakudoc. The idea being to separate the safe and unsafe bits of processing documentation. An unsafe tool can do the conversion (unsafe because it needs to potentially run BEGIN code during compilation) and a safe tool can then process the .rakudoc files. (That tool can be safe if it by design only parses RakuDoc but not Raku code).

Leaning on this distinction, it might make sense that the provides and provides-docs sets are exclusive, so that provides exclusively contains files that need a Raku parser while provides-docs exclusively contains files that can be parsed with a RakuDoc parser. That way a tool processing the documentation could then do the following:

Retrieve all provides files. (All of them being Raku files.)
Convert all those files to RakuDoc files. (Creating a bunch of RakuDoc files.)
Retrieve all provides-docs files. (All of them being RakuDoc files.)
Process all the files created in step 2 and retrieved in step 3.

ugexe/rakudoc-grant.md