=pod

=begin text

+-----------------------------------+
| Lingua::Translit Developer Manual |
+-----------------------------------+

=end text

=head1 Conventions used in this manual

=over 2

=item *

Every non absolute path is relative to the source code's directory.

=back

=head1 Adding a transliteration

If you want to add a new transliteration to
L<Lingua::Translit> just...

=over 2

=item *

write an XML file (the "transliteration table")

=item *

build a development version containing your table

=item *

write and run some tests to check if your transliteration is working as
expected.

=item *

integrate your table into the set of upstream tables and consider
contributing it

=back

=head2 Writing a transliteration table

Each XML-file consists of meta data and a set of transliteration rules.

The B<meta data> tags cover the name of the transliteration, a short
description and the information whether the transliteration can be used
in both directions. For example:

  <name>DIN 1460</name>
  <desc>DIN 1460: Cyrillic to Latin</desc>
  <reverse>true</reverse>

The B<rules> can be simple one to one mappings
 
 <rule>
    <from>X</from>
    <to>Y</to>
  </rule>

...but you can also specify a context in which the rule should be evaluated
only:
 
  <rule>
    <from>A</from>
    <to>B</to>
    <context>
      <after>x</after>
      <before>y</before>
    </context>
  </rule>

To get an easy start, you can copy the file F<xml/template.xml>, rename it
to your needs and edit it right away.

=begin html

Template:

  <a href="template.xml.html">template.xml</a>
  <br/>

Complete example:

  <a href="Common_DEU.xml.html">Common_DEU.xml</a>

=end html

=begin text

  Template:         xml/template.xml
  Complete example: xml/common_deu.xml

=end text

Although editing an XML-file is technically quite easy, some things
have to be respected.
The most important thing to keep in mind is that the rules are applied
I<in sequence> - one after another.
Therefore the order of rules is important if you specify a context 
or transliterate multiple characters.

=head3 Unicode notation

If you are determining characters that are non-ASCII characters, use an
entity that represents the Unicode code point in hex-notation to specify
them and leave a comment on the character. 

  <rule>
    <from>&#x0410;</from>	  <!-- CYRILLIC CAPITAL LETTER A --> 
    <to>A</to>
  </rule>

This assures that the correct character is transformed and it can be
exactly determined, if it is not represented correctly.

=head3 Specifying a context

The context is evaluated as a Perl regular expression. So for
specifying the context I<literal ASCII characters>, I<entities>
or I<meta characters> can be used.

If a character has two mappings depending on the context,
the context-sensitive rule must be applied first and then the 
context-free rule. Otherwise every character is replaced at once through
the context-free rule and the context-sensitive rule will never match.

1. rule
 
  <rule>
    <from>&#x0393;&#x03BA;</from> <!-- GREEK CAPITAL LETTER GAMMA &
                                       GREEK SMALL LETTER KAPPA -->
    <to>Gk</to>
    <context>
      <after>\b</after>		  <!-- word initial -->
    </context>
  </rule>

2. rule

  <rule>
    <from>&#x0393;&#x03BA;</from> <!-- GREEK CAPITAL LETTER GAMMA &
                                       GREEK SMALL LETTER KAPPA -->
    <to>Nk</to>
  </rule>

The following pattern matching contexts are available:

=over 4

=item C<E<lt>afterE<gt>> 

if the transliteration rule should only be applied after a certain 
character (corresponds to Perl's I<lookbehind>)

=item C<E<lt>beforeE<gt>>

if the rule should only be applied before a certain character 
(corresponds to Perl's I<lookahead>)

=item C<E<lt>afterE<gt> E<amp> E<lt>beforeE<gt>>

if the rule should only be applied if the character is in between 
two characters

=back

=head3 Multiple characters

As all rules are applied in sequence, and hence the order of rules is
important, all rules concerning multiple characters must precede
all single character rules.

1. rule

  <rule>
    <from>&#x03B1;&#x03C5;</from> <!-- GREEK SMALL LETTER ALPHA &
				       GREEK SMALL LETTER UPSILON -->
    <to>au</to>
  </rule>

2. rule

  <rule>
    <from>&#x03B1;</from>	  <!-- GREEK SMALL LETTER ALPHA -->
    <to>a</to>
  </rule>

If you switch the order of the rules in the above example, every single
alpha would be transliterated first and the digraph pattern will never
match.

=head2 Building a development version

Your new transliteration table has to be converted to a Perl data
structure and stored in F<xml/tables.dump> in order to be put to use and
tested as a development version of L<Lingua::Translit>.

I<xml2dump.pl> is a tool that processes XML transliteration table
definitions and converts them to Perl data structures. Normally, all
stable transliteration tables are processed once and stored in
F<xml/tables.dump> and included in the L<Lingua::Translit::Tables>
module once at build time.

=head3 Using xml2dump.pl

To accomplish this task the I<xml2dump.pl> tool comes in handy:

  alinke$ ./xml2dump.pl -v -o tables.dump mytable.xml 
  Parsing mytable.xml... (MyTable: rules=2, contexts=1)
  1 transliteration table(s) dumped to tables.dump.

It reads an XML definition, processes it and dumps the resulting data
structure to a given file S<(C<-o> switch)>.

Your transliteration table is now ready to be included by
L<Lingua::Translit::Tables> so it can be tested and evaluated.

=head3 Building a temporary Lingua::Translit

Use the standard toolchain to build a temporary development version of
L<Lingua::Translit> which contains nothing but your new
transliteration table.

  alinke$ perl Makefile.PL && make

Given the resulting development version, it's time to test the
transliteration table for completeness and correct functionality.

=head2 Testing the transliteration table

To verify that your set of transliteration rules works correctly,
you need to make some tests using your favorite Perl test framework.
For an easy and complete example that utilizes the
L<Test::More|Test::More>
framework, have a look at the following example:

=begin html

  <a href="11_tr_Common_DEU.t.html">t/11_tr_Common_DEU.t</a>

=end html

L<Lingua::Translit> comes with a ready to use test template that you
could use as a starting point and suite it to your transliterations
specific needs. It is located at F<t/xx_tr_template.t.pl> - to follow
L<Lingua::Translit>'s naming convention, rename it to F<NN_tr_NAME.t>.

=begin html

Online version of the template:

  <a href="xx_tr_template.t.html">t/xx_tr_template.t.pl</a>

=end html

=begin text

  t/11_tr_Common_DEU.t

=end text

=head3 Hints on what to test

=over 2

=item *

If your transliteration is straight forward (only "1:1" mappings), just
test a small text and have a look at the result. At best, everything is
correct an you are ready.

=item *

If the transliteration is reversible, you should check, if both directions
are transliterating correctly.

=item *

All the context-sensitive and multi-character transliterations should
be tested explicitly, to assure, that the error-prone mappings also
work as expected.

=back

=head3 Running the Tests

While testing it is convenient to define the environment variable
I<PERL5LIB> (have a look at I<perlrun>(1)) so that the Perl interpreter
C<knows> where your development version of
L<Lingua::Translit> is located. The following example
session assumes that you are using I<bash>(1) or a similar shell:

  alinke$ export PERL5LIB="blib/lib"
  alinke$ perl t/66_tr_mytest.t
  1..2
  ok 1 - MyTable: not reversible
  ok 2 - MyTable: transliteration

If all tests work as expected and hence your transliteration table is
ready for usage, clean up your shell's environment and prepare to
integrate your table into the existing set of transliteration tables:

  alinke$ unset PERL5LIB

=head2 Integrating the new table

Change to the F<xml/> directory and let I<make>(1) call I<xml2dump.pl>
in order to build a data structure ("F<tables.dump>") from all
available XML transliteration tables, including yours:

  alinke$ make all-tables

Now, clean up the old files from the development version you used to
write your tests. Change into the source directory's root and run

  alinke$ make distclean && perl Makefile.PL && make

The result is a complete version of L<Lingua::Translit> that contains
all upstream tables, as well as your own addition.

  alinke$ make test

...assures everything is alright and ready for installation or
packaging. Congratulations!

=head2 Contributing your table

If you like to contribute your transliteration table under the terms of
the GPL/Artistic License, it can be included in the official upstream
version. To accomplish this, create a patch of your changes and send it
along with a description and comments to C<perl@lingua-systems.com> so
it can be part of the next release.

=begin html

  <div id="footer">
    <a href="http://validator.w3.org/check?uri=referer"><img
      src="http://www.w3.org/Icons/valid-xhtml10-blue"
      alt="Valid XHTML 1.0 Strict" height="31" width="88" /></a>
    <br/>
    &copy; 2008 Alex Linke &amp; Rona Linke
    <br/>
    &copy; 2009 Lingua-Systems Software GmbH
  </div>

=end html

=cut

# vim: sts=2 enc=utf-8 textwidth=72 wrap