locale gen tools: Set all UTF-8 to same rollup CTYPE

The CLDR CTYPE definitions are still used for non-UTF8 encodings,
but all UTF-8 locales now share a single "master" CTYPE that knows
all reasonab

locale gen tools: Set all UTF-8 to same rollup CTYPE

The CLDR CTYPE definitions are still used for non-UTF8 encodings,
but all UTF-8 locales now share a single "master" CTYPE that knows
all reasonable character sets.

show more ...

cldr2def: Add 6 Arabic locales: AE EG JO MA QA SA

The lack of Arabic support on BSD seems to be a glaring omission, so
lets help rectify that by adding since 6 locales to the cldr2def tool:
ar_AE:

cldr2def: Add 6 Arabic locales: AE EG JO MA QA SA

The lack of Arabic support on BSD seems to be a glaring omission, so
lets help rectify that by adding since 6 locales to the cldr2def tool:
ar_AE: United Arab Emirates
ar_EG: Egypt
ar_JO: Jordan
ar_MA: Morocco
ar_QA: Qatar
ar_SA: Saudi Arabia

There are obviously more (e.g. Iraq, Kuwait, Algeria, Libya, Tunisia,
Syria, etc.) but I selected these as being the most likely to be
used in my limited opinion.

They are UTF-8 locales only.

show more ...

cldr2def: Change locale from common.UTF-8 to xx_Comm_US.UTF-8

Some applications check the format of the locale so we need to stick
to the xx_Yyyy_ZZ format.

cldr2def: Modify tool to create a "common" UTF-8 locale

With the switch to CLDR definitions, each locale has specific encoding.
E.g. en_US, fr_FR has latin characters, el_GR has Greek characters,
et

cldr2def: Modify tool to create a "common" UTF-8 locale

With the switch to CLDR definitions, each locale has specific encoding.
E.g. en_US, fr_FR has latin characters, el_GR has Greek characters,
etc. Seeing a filename written in Chinese characters while in en_US.UTF-8
will result in a bunch of "???" because the characters are illegal for
the locale's CTYPE definition.

Previously there was one shared CTYPE for all locales. This was handy
but incorrect. It was also not updated and became slightly bitrotted.

I am going to create a "fake" locale called "common.UTF-8". It is the
same as en_US.UTF-8, but with an LC_CTYPE that is practically universal
in that all common character sets (and symbols) are defined. To use it,
one can do either:
1) Set LANG env. variable to "common.UTF-8"
2) Set LC_CTYPE env. variable to "common.UTF-8"

e.g. "env LANG=fr_FR.UTF-8 LC_CTYPE=common.UTF-8 ls"

This commit only updates the tool. The locale comes later.
In addition to making changes that support the "common" base, the tool
was also tweaked to favor en_US.UTF-8 as the "LOCALES+=" value. This
should reduce a lot of the renaming churn we've been seeing on new
generations.

show more ...

locales polish: Remove ISO-8859-1 encoding from 27 locales

The ISO-8859-1 encoding is considered obsolete compared to ISO-8859-19
(which in itself is obsolete compared to UTF-8). To help avoid some

locales polish: Remove ISO-8859-1 encoding from 27 locales

The ISO-8859-1 encoding is considered obsolete compared to ISO-8859-19
(which in itself is obsolete compared to UTF-8). To help avoid some
potential mismatches, remove the ISO-8859-1 encoding in the following
cases:

1) European localities that already have ISO-8859-15 (25 of 27)
* ca_(AD|ES|FR|IT)
* da_dk
* de_(AT|CH|DE)
* en_GB
* (es|eu)_ES
* fi_FI
* fr_(BE|CH|FR)
* is_IS
* it_(CH|IT)
* (nb|nn)_NO
* nl_(BE|NL)
* pt_PT
* sv_(FI|SE)

2) Localities where ISO-8859-1 can't represent local currency symbol
* en_PH (PH Peso)
* es_CR (Colon)

show more ...

locales polish: Remove ISO8859-15 from 8 non-European places

Let's limit usage of the "obsolete" ISO-8859-15 to countries in
Europe. Users of the eight locales affected should use UTF-8 instead.

locale polishing: lt_LT, et_EE, and en_IE changes

lt_LT: Remove ISO8859-4. If an ISO encoding is needed, -13 would be
used instead.
et_EE: Estonia could use ISO8859-4 (not present), the newe

locale polishing: lt_LT, et_EE, and en_IE changes

lt_LT: Remove ISO8859-4. If an ISO encoding is needed, -13 would be
used instead.
et_EE: Estonia could use ISO8859-4 (not present), the newer ISO8859-13
(not present) or ISO8859-15 (present). To be uniform with other
European countries that use -15, change the default from UTF-8
to ISO8859-15. Do not bring in -4 or -13.
en_IE: Ireland only had UTF-8. To be uniform with the UK and Western
Europe, bring in ISO8859-15 and set that as the alias for the
shortname. Do not bring in -1.

show more ...

cldr2def: Update Makefile to generate new POSIX source files

We've been using CLDR v2.0.1 because it was the last version that
provided generated POSIX source files. (The next release, v21, only
pr

cldr2def: Update Makefile to generate new POSIX source files

We've been using CLDR v2.0.1 because it was the last version that
provided generated POSIX source files. (The next release, v21, only
provided the java tool to generate them). The last release is version
27.0.1. This alternation to the makefile allows the generation of the
desired 72 base locales with a single command (after the tool is
downloaded and installed in the same CLDR release directory)

Note that kk_KZ locale is changing to kk_Cyrl_KZ.

While here, make the necessary updates to charmaps.xml to generate
to locale source files.

show more ...

Add 17 new locales and really remove Latin

Now that locale defintions are generates, it's easy to add more.
I've added the following new locale defintions:

* en_HK ISO-8859-1 (Hong Kong/English)

Add 17 new locales and really remove Latin

Now that locale defintions are generates, it's easy to add more.
I've added the following new locale defintions:

* en_HK ISO-8859-1 (Hong Kong/English)
* en_HK UTF-8
* en_PH ISO-8859-1 (Phillipines/English)
* en_PH UTF-8
* en_SG ISO-8859-1 (Singapore/English)
* en_SG UTF-8
* es_AR ISO-8859-1 (Argentina/Spanish)
* es_AR UTF-8
* es_CR ISO-8859-1 (Costa Rica/Spanish)
* es_CR UTF-8
* es_MX ISO-8859-1 (Mexico/Spanish)
* es_MX UTF-8
* se_FI UTF-8 (Finland/Northern Sami)
* se_NO UTF-8 (Norway/Northern Sami)
* sv_FI ISO-8859-1 (Finland/Swedish)
* sv_FI ISO-8859-15
* sv_FI UTF-8

There were a few places la_LN (Latin) was hidden so I've really
removed it now.

show more ...

Activate kk_KZ, lv_LV, and pt_BR locales

These three locales already existed, but they weren't being updated.
I've adjusted the xml configuration to get them building (minus the
kk_KZ.PT154 locale w

Activate kk_KZ, lv_LV, and pt_BR locales

These three locales already existed, but they weren't being updated.
I've adjusted the xml configuration to get them building (minus the
kk_KZ.PT154 locale which will be removed).

While here, update the tools/tools/locale Makefile to replace all
six LC categories in /usr/src/share with the "make install" target,
and then install them on a live system with "make post-install" target.

show more ...

locales: Import unused FreeBSD tool to generate locales

Five years ago, Edwin Groothuis put in a *LOT* of work to create a perl-
based tool that would use the latest Unicode release from The Unicode

locales: Import unused FreeBSD tool to generate locales

Five years ago, Edwin Groothuis put in a *LOT* of work to create a perl-
based tool that would use the latest Unicode release from The Unicode
Consortium <http://unicode.org> and the latest Unicode CLDR release from
the same organization, and generate all the locale defintions for numeric,
monetary, time, and messages, including generating a readable Makefile.

The tool has the ability to define the conversions from Unicode to lower
resolution formats. While there were some minor bugs in it, the tool now
works, but for reasons unknown to me right now, it was never imported into
FreeBSD, nor used to improve their localization 5 years ago.

I've generated the new locale definitions which will go in soon. Note
that the Unicode Consortium no longer provides the "posix" definitions
the tool reads, so these definitions are 3 years old. Supposedly they
provide tools to generate the POSIX definitions, but I haven't found them
yet. Once I do, I'll regenerate the locales with the latest CLDR.

Refer: https://wiki.freebsd.org/LocaleNewApproach

show more ...