=pod =encoding utf8 =head1 NAME Muldis::D::Dialect::PTMD_STD - How to format Plain Text Muldis D =head1 VERSION This document is Muldis::D::Dialect::PTMD_STD version 0.79.1. =head1 PREFACE This document is part of the Muldis D language specification, whose root document is L; you should read that root document before you read this one, which provides subservient details. =head1 DESCRIPTION This document outlines the grammar of the I dialect named C. The fully-qualified name of this Muldis D dialect, in combination with the base language spec it is bundled with, is C (when the bundled base language version is substituted for the C). This dialect is designed to exactly match the Muldis D system catalog (the possible representation of Muldis D code that is visible to or updateable by Muldis D programs at runtime) as to what non-critical meta-data it explicitly stores; so code in the C dialect should be round-trippable with the system catalog with the result maintaining all the details that were started with. Since it matches the system catalog, this dialect should be able to exactly represent all possible Muldis D base language code (and probably all extensions too), rather than a subset of it. That said, the C dialect does provide a choice of multiple syntax options for writing Muldis D value literals and DBMS entity (eg type and routine) declarations, so several very distinct C code artifacts may parse into the same system catalog entries. There is even a considerable level of abstraction in some cases, so that it is easier for programmers to write and understand typical C code, and so that this code isn't absurdly verbose. This dialect is designed to be as small as possible while meeting the above criteria, and is designed such that a parser that handles all of this dialect can be fairly small and simple. Likewise, a code generator for this dialect from the system catalog can be fairly small and simple. A significant quality of the C dialect is that it is designed to work easily for a single-pass parser, or at least a single-pass lexer; all the context that one needs to know for how to parse or lex any arbitrary substring of code is provided by prior code, or any required lookahead is just by a few characters in general. Therefore, a C parser can easily work on a streaming input like a file-handle where you can't go back earlier in the stream. Often this means a parser can work with little RAM. Also the dialect is designed that any amount of whitespace can be added or omitted next to most non-alphanumeric characters (which happen to be next to alphanumeric tokens) without that affecting the meaning of the code at all, except obviously for within character string literals. And long binary or character or numeric or identifier strings can be split into arbitrary-size substrings, without affecting the meaning. And many elements are identified by name rather than ordinal position, so to some degree the order they appear has no effect on the meaning. So programmers can easily format (separate, indent, linewrap, order) code how they like, and making an automated code reformatter shouldn't be difficult. Often, named elements can also be omitted entirely for brevity, in which case the parser would use context to supply default values for those elements. Given that plain text is (more or less) universally unambiguously portable between all general purpose languages that could be used to implement a DBMS, it is expected that every single Muldis D implementation will natively accept input in the C dialect, which isn't dependent on any specific host language and should be easy enough to process, so it should be considered the safest official Muldis D dialect to write in by default, when you don't have a specific reason to use some other dialect. See also the dialects L and L, which are derived directly from C, and represent possible Perl 6 and 5 concrete syntax trees for it; in fact, most of the details in common with those other dialects are described just in the current file, for all 3 dialects. =head1 GENERAL STRUCTURE A C Muldis D code file consists just of a full or partial Muldis D C routine definition, which begins with a language name declaration, and otherwise is simply an ordered sequence of imperative routine calls, where earlier routine calls are to system-defined data-definition routines (their arguments are values to put in the system catalog), and later ones are then to user-defined routines that the earlier statements either loaded or defined. This is conceptually what a C file is, and it can even be that literally, but C provides a canonical further abstraction which should be used when doing data-definition. And so you typically use syntax resembling routine and type declarations in a general purpose programming language, where simply declaring such an entity will cause it to be written into the system catalog for subsequent use. The grammar in this file is formatted as a hybrid between various EBNF flavors and Perl 6 rules (see L for details on the latter) with further changes. It is only meant to be illustrative and human readable, and would need significant changes to actually be a functional parser, which are different for each parser toolkit. The grammar consists mainly of named I which define matching rules. Loosely speaking, each parser match of a token corresponds to a capture I or node element in the concrete syntax tree resulting from the parse; in practice, the parser may make various alterations to the match when generating a node, such as adding guide keywords corresponding to the token name, or by merging series of trivial tokens or doing escaped character substitutions. No explicit capture syntax such as parenthesis is used in the grammar. To help understand the grammar in this file, here are a few guidelines: 1. The grammar is exactly the same as that of a Perl 6 rule except where these guidelines state otherwise; this includes that square brackets mean grouping not optionality, and that when multiple sub-pattern alternatives match, the one that is the longest wins. 2. The grammar portion that actually declares a token, that is what associates a token name with its definition body, is formatted like EBNF, as C<< ::= ... >> rather than the Perl 6 way like C or C. 3. All non-quoted whitespace is not significant and just is formatting the grammar itself; rather, whitespace rules in the grammar are spelled out explicitly such as with C<\s*> (optional whitespace) and C<\s+> (mandatory whitespace). The root grammar token for the entire dialect is C. =head1 BOOTLOADER Grammar: ::= [\s+ [ | | ** \s+ ]]? A C node has 1..N ordered elements where the first element is a C node and then either: 1. there is exactly one (second) element that is a C node or a C node; 2. there are 1..N ordered elements where each is a C node; 3. there are no other elements, making the bootloader a no-op. See the pod sections in this file named L, L, L, and L for more details about the aforementioned tokens/nodes. When Muldis D is being compiled and invoked piecemeal, such as because the Muldis D implementing virtual machine (VM) is attached to an interactive user terminal, or the VM is embedded in a host language where code in the host language invokes Muldis D code at various times, the conceptual C is usually split up, and so not every Muldis D code fragment would then have its own C. Usually a C would be supplied to the Muldis D VM just once as a VM configuration step, which provides a context for further interaction with the VM that just involves Muldis D code that isn't itself qualified with a C. =head1 LANGUAGE NAME Grammar: ::= \s* ':' \s* \s* ':' \s* \s* ':' \s* [\s* ':' \s* ]? ::= Muldis_D ::= ::= ::= PTMD_STD ::= As per the VERSIONING pod section of L, code written in Muldis D must start by declaring the fully-qualified Muldis D language name it is written in. The C dialect formats this name as a C node having 4-5 ordered elements: =over =item C This is the Muldis D language base name; it is simply the bareword character string C. =item C This is the base authority; it is a character string formatted as per a specific-context C value literal; it is typically the delimited character string C. =item C This is the base version number; it is a character string formatted as per C; it is typically a character string like C<1.2.3>. =item C This is the dialect name; it is simply the bareword character string C. =item C Optional; this is a set of chosen pragma/parser-config options as per a C SCVL; see the L pod section for more details. =back Examples: Muldis_D:"http://muldis.com":"1.2.3":PTMD_STD Muldis_D:"http://muldis.com":"1.2.3":PTMD_STD:{ ... } =head1 VALUE LITERALS AND SELECTORS Grammar: ::= | ::= | | | | | | | | | | | | | | | | | ::= | | | | | | | A C node is a Muldis D value literal, which is a common special case of a Muldis D value selector. Unlike value selectors in general, which must be composed beneath a C or C because they actually represent a Muldis D value expression tree of a function or updater or type definition, a C node does I represent an expression tree, but rather a value constant; by definition, a C can be completely evaluated at compile time. A C consisting directly of a C is hence just a serialized Muldis D value. The PTMD_STD grammar subsection for value literals (having the root grammar token C) is completely self-defined and can be used in isolation from the wider grammar as a Muldis D sub-language; for example, a hosted-data Muldis D implementation may have an object representing a Muldis D value, which is initialized using code written in that sub-language. Every grammar token, and corresponding capture node, representing a Muldis D value literal is similarly formatted and has 1-3 elements; the following pod section L describes the similarities once for all of them, in terms of an alternate C token definition which is called C. And then the other pod sections specific to each kind of value literal then just focus on describing their unique aspects, namely their I. An C node represents a conceptually opaque Muldis D value, such that every one of these values is defined with its own literal syntax that is compact and doesn't look like a collection of other nodes; this includes the basic numeric and string literals. A C node represents a conceptually transparent Muldis D value, such that every one of these values is defined visibly in terms of a collection of other nodes; this includes the basic tuple and relation selectors. =head2 Value Literal Common Elements A I (or I) is a value literal that can be properly interpreted in a context that is expecting I value but has no expectation that said value belongs to a specific data type; in the general case, a GCVL includes explicit I meta-data (such as, "this is an C" or "this is a C"); but with a few specific data types (see the C node description for details) that meta-data may be omitted for brevity because the main literal has mutually uniquely identifying characteristics. For example, each element of a generic Muldis D collection value, such as a member of an array or tuple, could potentially have any type at all. In contrast, a I (or I) is a value literal that does not include explicit value kind meta-data, even when the main literal doesn't have uniquely identifying characteristics, because the context of its use supplies said meta-data. For example, in a tuple value literal it is assumed that a value literal in an attribute name position must denote a C. The grammar token C|C denotes a GCVL, as do most short-named grammar tokens, like C or C; in contrast, a grammar token containing C denotes a SCVL, like C or C. Every GCVL has 1-3 elements, illustrated by this grammar: ::= [ \s* ':' \s* [ \s* ':' \s*]? ]? ::= Bool | Order | RatRoundMeth | Int | NNInt | PInt | Rat | NNRat | PRat | Blob | OctetBlob | Text | Name | NameChain | DeclNameChain | Comment | Instant | Duration | UTC [Instant | DateTime | Date | Time] | Float [Instant | DateTime | Date | Time] | UTCDuration | RatRoundRule | String | BString | OString | UCPString | DH? Scalar | DH? Tuple | Database | DH? Relation | DH? Set | DH? [Maybe | Single] | DH? Array | DH? Bag ::= ::= | | | | | | | | | | | | | | | | | | | | | | | | | So a C|C node has 1-3 elements in general: =over =item C This is a character string of the format C<< <[A..Z]> <[ a..z A..Z ]>+ >>; it identifies the data type of the value literal in broad terms and is the only external meta-data of C generally necessary to interpret the latter; what grammars are valid for C depend just on C. For all values of just the 7 data types [C, C, C, C, C, C, C], the C portion of a GCVL may be omitted for brevity, but the code parser should still be able to infer it easily by examining the first few characters of the C, which for each of said 7 data types has a mutually uniquely identifying format, which is also distinct from all possible C. Note that omission of C is only allowed when the GCVL doesn't include a C element. Note that, by the above criteria, the C type could easily have been an 8th member of the list; however, it is excluded on purpose because in practice Muldis D code can contain SCVL C literals almost anywhere as meta-data for that code, and so having an explicit C of C means that that particular C literal is to be treated as ordinary data like with any GCVL. For just these certain special values of other data types, the same option of omitting the C (and C) applies: C, C, C, C. =item C This is a Muldis D data type name, for example C; it identifies a specific subtype of the generic type denoted by C, and serves as an assertion that the Muldis D value denoted by C is a member of the named subtype. Iff C is C<[|DH]Scalar> then C is mandatory; otherwise, C is optional for all C, except that C must be omitted when C is one of the 2 [C, C]; this isn't because those 2 types can't be subtyped, but because in practice doing so isn't useful. How a Muldis D parser treats a C node with a C element depends on the wider context. In the general case where the C is an C beneath the context of a C or C node, the C is treated as if it had an extra parent C node that invokes the C function and whose 2 argument nodes are as follows: C gets the C without the C element, and C gets the C element. This means that in general the C assertion is done at runtime. In the common special case where both C is an C and C refers to a system-defined type, then the C assertion is done at compile time, and then the C element is simply eliminated, so the C ends up simply as itself with no new C parent. =item C This is mandatory for all C. =back For GCVL and SCVL examples, see the subsequent documentation sections. =head1 OPAQUE VALUE LITERALS =head2 Boolean Literals Grammar: ::= [Bool \s* ':' \s*]? ::= false | ⊥ | true | ⊤ A C node represents a logical boolean value. It is interpreted as a Muldis D C value as follows: The C is a bareword character string formatted as per a C SCVL, and it maps directly to the C possrep of the C type. Examples: Bool:true false ⊤ ⊥ =head2 Order-Determination Literals Grammar: ::= [Order \s* ':' \s*]? ::= increase | same | decrease An C node represents an order-determination. It is interpreted as a Muldis D C value as follows: The C is a bareword character string formatted as per a C SCVL, and it maps directly to the C possrep of the C type. Examples: Order:same decrease =head2 Rational Rounding Method Literals Grammar: ::= [ RatRoundMeth \s* ':' \s* [ \s* ':' \s*]? ]? ::= half_down | half_up | half_even | to_floor | to_ceiling | to_zero | to_inf A C node represents a rational rounding method. It is interpreted as a Muldis D C value as follows: The C is a bareword character string formatted as per a C SCVL, and it maps directly to the only possrep of the C type. Examples: RatRoundMeth:half_up to_zero =head2 General Purpose Integer Numeric Literals Grammar: ::= [ [Int | NNInt | PInt] \s* ':' \s* [ \s* ':' \s*]? ]? ::= \s* ';' \s* | ::= ::= [0 | \-?] ::= ? ::= <[ 1..9 A..Z ]> ::= [[_?<[ 0..9 A..Z ]>+]+] ** [\s* '~' \s*] ::= [0 | \-?] ::= ? ::= <[ 1..9 ]> ::= [[_?<[ 0..9 ]>+]+] ** [\s* '~' \s*] An C node represents an integer numeric value. It is interpreted as a Muldis D C value as follows: If the C is composed of a C plus C, then the C is interpreted as a base-I integer where I might be between 2 and 36, and the C says which possible value of I to use. Assuming all C column values are between zero and I-minus-one, the C contains that I-minus-one. So to specify, eg, bases [2,8,10,16], use C of [1,7,9,F]. If the C is a C, then it is interpreted as a base 10 integer. Fundamentally the I part of an C node consists of a string of digits and uppercased (but not lowercased) letters, where each digit (C<0..9>) represents its own number and each letter (C) represents a number in [10..35]. A I may optionally contain underscore characters (C<_>), which exist just to help with visual formatting, such as for C<10_000_000>, and these are ignored/stripped by the parser. A I may optionally be split into 1..N segments where each segment is separated by a tilde token (C<~>); this segmenting ability is provided to support code that contains very long numeric literals while still being well formatted (no extra long lines); the tilde tokens are also ignored/stripped by the parser, and the I is interpreted as if all its alphanumeric characters were contiguous. If the C of a C node is C or C rather than C, then the C node is interpreted simply as an C node whose C is C or C, and the allowed I is appropriately further restricted. Examples: Int:1;11001001 # binary # 7;0 # octal # 7;644 # octal # -34 # decimal # 42 # decimal # F;DEADBEEF # hexadecimal # Z;-HELLOWORLD # base-36 # 3;301 # base-4 # B;A09B # base-12 # =head2 General Purpose Rational Numeric Literals Grammar: ::= [ [Rat | NNRat | PRat] \s* ':' \s* [ \s* ':' \s*]? ]? ::= \s* ';' \s* | ::= \. | \s* \/ \s* | \s* \* \s* \s* \^ \s* ::= \. | \s* \/ \s* | \s* \* \s* \s* \^ \s* A C node represents a rational numeric value. It is interpreted as a Muldis D C value as follows: Fundamentally a C node is formatted and interpreted like an C node, and any similarities won't be repeated here. The differences of interpreting a C being composed of a C plus C versus the C being a C are as per the corresponding differences of interpreting an C. Also interpreting a C or C is as per a C or C. If the I part of a C node contains a radix point (C<.>), then it is interpreted as is usual for a programming language with such a literal. If the I part of a C node contains a solidus (C), then the rational's value is interpreted as the leading integer (a numerator) divided by the trailing positive integer (a denominator); that is, the two integers collectively map to the C possrep of the C type. If the I part of a C node contains a asterisk (C<*>) plus a circumflex accent (C<^>), then the rational's value is interpreted as the leading integer (a mantissa) multiplied by the result of the middle positive integer (a radix) taken to the power of the trailing integer (an exponent); that is, the three integers collectively map to the C possrep of the C type. Examples: Rat:1;-1.1 -1.5 # same val as prev # 3.14159 A;0.0 F;DEADBEEF.FACE Z;0.000AZE Rat:6;500001/1000 B;A09B/A Rat:1;1011101101*10^-11011 45207196*10^37 1/43 314159*10^-5 =head2 General Purpose Binary String Literals Grammar: ::= [ [Blob | OctetBlob] \s* ':' \s* [ \s* ':' \s*]? ]? ::= \s* ';' \s* ::= <[137F]> ::= [ <[']> <[ 0..9 A..F ]>* <[']> ] ** [\s* '~' \s*] A C node represents a general purpose bit string. It is interpreted as a Muldis D C value as follows: Fundamentally the I part of a C node consists of a delimited string of digits and uppercased (but not lowercased) letters, where each digit (C<0..9>) represents its own number and each letter (C) represents a number in [10..15]; this string is qualified with a C character (C<[137F]>), similarly to how an C is qualified by a C. Each character of the delimited string specifies a sequence of one of [1,2,3,4] bits, depending on whether C is [1,3,7,F]. If the C of a C node is C rather than C, then the C node is interpreted simply as an C node whose C is C, and the delimited string is appropriately further restricted. Examples: Blob:1;'00101110100010' # binary # 3;'' F;'A705E' # hexadecimal # 7;'523504376' =head2 General Purpose Character String Literals Grammar: ::= [ Text \s* ':' \s* [ \s* ':' \s*]? ]? ::= [ <[']> [<-[\\\'\t\n\f\r]> | ]* <[']> ] ** [\s* '~' \s*] ::= '\b' | '\a' | '\q' | '\h' | '\s' | '\t' | '\n' | '\f' | '\r' | '\c<' [ [<[ A..Z ]>+] ** ' ' | [0 | <[ 1..9 ]> <[ 0..9 ]>*] | <[ 1..9 A..Z ]> ';' [0 | <[ 1..9 A..Z ]> <[ 0..9 A..Z ]>*] ] '>' A C node represents a general purpose character string. It is interpreted as a Muldis D C value as follows: The C is interpreted generally as is usual for a programming language with such a delimited character string literal. A C may optionally be split into 1..N segments where each segment is delimited by apostrophes/single-quotes (C<'>) and separated by a tilde token (C<~>); this segmenting ability is provided to support code that contains long string literals while still being well formatted (no extra long lines); the tilde tokens and adjoining string delimiters are ignored/stripped by the parser, and the C is interpreted as if it just consisted of a single delimited string. All Muldis D delimited character string literals (generally the 3 C, C, C) may contain some characters denoted with escape sequences rather than literally. The Muldis D parser would substitute the escape sequences with the characters they represent, so the resulting character string values don't contain those escape sequences. Currently there are 2 classes of escape sequences, called I and I. The meanings of the simple escape sequences are: Esc | Unicode | Unicode | Chr | Literal character used Seq | Codepoint | Character Name | Lit | for when not escaped ----+-----------+-----------------+-----+------------------------------ \b | F;5C | REVERSE SOLIDUS | \ | esc seq lead (aka backslash) \a | F;27 | APOSTROPHE | ' | delim Text literals \q | F;22 | QUOTATION MARK | " | delim quoted Name literals \h | F;23 | NUMBER SIGN | # | delim Comment lit (aka hash) \s | F;20 | SPACE | | space char \t | F;9 | CHAR... TAB... | | control char horizontal tab \n | F;A | LINE FEED (LF) | | ctrl char line feed / newline \f | F;C | FORM FEED (FF) | | control char form feed \r | F;D | CARR. RET. (CR) | | control char carriage return One design decision of PTMD_STD that is distinct from typical other languages is that an escape sequence for any character used as a delimiter I contains that literal character. For example, while in SQL or Perl character strings delimited by C<'>, they typically escape literal apostrophes/single-quotes as C<''> or C<\'>; while this is unambiguous, the task of parsing such code is considerably more difficult than it could be. In contrast, while in PTMD_STD character strings delimited by C<'>, a literal of the same is escaped with C<\a>; so parsing such code is an order of magnitude easier because the parser doesn't have to understand the internals of any character string literal in order to separate out the character string from its surrounding code. Another design decision of PTMD_STD that is distinct at least from Perl is that non-"space" whitespace characters in character string literals must never appear literally, but must instead be denoted with escape sequences. The main reason for this is to ensure that the actual values being selected by the string literals were not variable per the kind of linebreaks used to format the Muldis D source code itself. There is currently just one complex escape sequence, of the format C<< \c<...> >>, that supports specifying characters in terms of their Unicode abstract codepoint name or number. If the C<...> consists of just uppercased (not lowercased) letters and the space character, then the C<...> is interpreted as a Unicode character name. If the C<...> looks like an C, sans that underscores and tilde segmentation aren't allowed here, then the C<...> is interpreted as a Unicode abstract codepoint number. One reason for this feature is to empower more elegant passing of Unicode-savvy PTMD_STD source code through a communications channel that is more limited, such as to 7-bit ASCII. Examples: Text:'Ceres' 'サンプル' '' 'Perl' '\c\c\c<65>' =head2 DBMS Entity Name Literals Grammar: ::= Name \s* ':' \s* [ \s* ':' \s*]? ::= | ::= <[ a..z A..Z _ ]><[ a..z A..Z 0..9 _ - ]>* ::= [ <["]> [<-[\\\"\t\n\f\r]> | ]* <["]> ] ** [\s* '~' \s*] ::= NameChain \s* ':' \s* [ \s* ':' \s*]? ::= ['.' \s*]? ** [\s* '.' \s*] ::= DeclNameChain \s* ':' \s* [