token.hpp

Go to the documentation of this file.

//==================================================================================================
/// \file
/// This header-file is part of module \alib_strings of the \aliblong.
///
/// Copyright 2013-2026 A-Worx GmbH, Germany.
/// Published under #"mainpage_license".
//==================================================================================================
ALIB_EXPORT namespace alib {  namespace strings::util  {
 
 
//==================================================================================================
/// Tokens in the context of \alib_strings_nl, are human-readable "words" or "symbols" that
/// represent a certain value or entity of software. Tokens may be used with configuration files,
/// mathematical or general expressions, programming languages, communication protocols and so forth.
///
/// This struct contains attributes to describe a token, a method to parse the attributes from a
/// (resource) string and finally method #".Match" that matches a given string against the token
/// definition.
///
/// ## Token Format: ##
/// With the construction, respectively the #"Token::Define;definition" of a token, special formats
/// are detected. These formats are:
/// - <em>"snake_case"</em><br>
/// - <em>"kebab-case"</em><br>
/// - <em>"CamelCase"</em><br>
///
/// \note
///   Information about such case formats is given in this
///   \https{Wikipedia article,en.wikipedia.org/wiki/Letter_case#Special_case_styles}.
///
/// \note
///   If the name indicates a mix of \e snake_case, \e kebab-case or \e CamelCase formats
///   (e.g., \e "System_Propery-ValueTable"), then snake_case supersedes both others and kebab-case
///   supersedes CamelCase.
///
/// The format detection is only performed when more than one minimum length is given. In this case,
/// the number of "segments" (e.g., "camel humps") has to match the number of length values.
///
///
/// ## Character Case Sensitivity: ##
/// Independent of the token format (normal or snake_case, kebab-case, CamelCase), character case
/// sensitivity can be chosen. With \e CamelCase and case-sensitive parsing, the first character of
/// the first hump may be defined lower or upper case (called "lowerCamelCase" vs. "UpperCamelCase").
///
/// If none of the special formats is detected, the tokens can optionally be abbreviated by just
/// providing a minimum amount of starting characters as specified by the then single entry
/// in #"minLengths".
/// Otherwise, each segment of the token (e.g., "camel hump") can (again optionally) be shortened
/// on its own.
/// As an example, if for token <c>"SystemProperty"</c> the minimum lengths given are
/// \c 3 and \c 4, the minimum abbreviation is <c>"SysProp"</c>, while <c>"SystProper"</c> also
/// matches.<br>
///
///
/// ## Limitation To Seven Segments: ##
/// This class supports minimum length definitions for up to \c 7 "camel humps", respectively
/// segments. Should a name contain even more segments, those cannot be abbreviated.
/// Providing more than \c 7 values for minimum segment lengths with the definition string results
/// in a definition error (see below).
///
///
/// ## Special Treatment For CamelCase: ##
/// ### Omitable Last Camel Hump: ####
/// The minimum length values provided must be greater than \c 0, with one exception:
/// With \e CamelCase format and case-insensitive definition, the last "camel hump" may have a
/// minimum length of \c 0 and hence may be omitted when matched.
/// If so, the "normalized" version of the token, which can be received by
/// #"AppendableTraits;appending" an instance to an #"^AString", will have
/// the last letter of the defined name converted to lower case.<br>
/// The rationale for this specific approach is to support the English plural case. This can be best
/// explained in a sample. If a token was defined using definition string:
///
///      "MilliSecondS Ignore 1 1 0"
///
/// then all of the following words match:
///
///      milliseconds
///      MilliSecs
///      millis
///      MSec
///      MSecs
///      MSs
///      ms
///
/// In the case that the rightfully (normalized) spelled token name is to be written, then with
/// the last character converted to lower case, the token becomes
///
///      MilliSeconds
///
/// This is performed with methods #"GetExportName" (which is also used by the specialization of
/// functor #"AppendableTraits" for this type.
/// Hence, when appending a #"%Token" to an #"%AString", if omitable, the last character
/// of the token name is converted to lower case.
///
/// If the above is not suitable, or for any other reasons a different "normalized" name is wanted
/// when writing the token, then method #Define offers a next mechanism to explicitly define
/// any custom string to be written.
///
/// ### Rollback: ####
/// \e CamelCase supports a simple "rollback" mechanism, which is needed, for example, for token
///
///      "SystemTemperature Ignore 1 1 0"
///
/// and given match argument
///
///      system
///
/// All six characters are matching the first hump, but then there are not characters left to
/// match the start of the second hump \c "Temperature". In this case, a loop of retries is
/// performed by rolling back characters from the back of the hump (\c 'm') and ending with the
/// first optional character of that hump (\c 'y'). The loop will be broken when
/// character \c 't' is found.
///
/// However: This is not continued in the case that the term that was rolled back does not match,
/// yet. This means, that certain (very unlikely!) tokens, with nested repeating character sequences
/// in camel humps, cannot be abbreviated to certain (unlikely wanted) lengths.
///
/// ## Handling Definition Errors: ###
///
/// The definition strings passed to method #Define are considered static (resourced) data.
/// In other words, this definition data should be compile-time defined and not be customizable
/// by end-users, but only by experts.
/// Therefore, only in debug-compilations of the library, a due testing of correctness of the
/// definitions is available.
///
/// The source code of overloaded utility namespace function
/// #"LoadResourcedTokens(resources::ResourcePool&)"
/// demonstrates how error codes, defined with enumeration #".DbgDefinitionError", can be handled in
/// debug-compilations by raising debug-assertions.
//==================================================================================================
class Token {
  public:
    /// Format types detected with #".detectFormat".
    enum class Formats : int8_t {
        Normal        = 0,  ///< Normal, optionally abbreviated words.
        SnakeCase     = 2,  ///< snake_case using underscores.
        KebabCase     = 4,  ///< kebab-case using hyphens.
        CamelCase     = 8,  ///< UpperCamelCase or lowerCamelCase.
    };
 
    #if ALIB_DEBUG
        /// Error codes which are written in field #".format" in the case that method #Define
        /// suffers a parsing error.<br>
        /// This enum, as well as the error detection, is only available in debug-compilations
        /// of the library.
    enum class DbgDefinitionError : int8_t {
        OK                              =   0, ///< All is fine.
        EmptyName                       = - 1, ///< No token name found.
        ErrorReadingSensitivity         = - 2, ///< Sensitivity value not found.
        ErrorReadingMinLengths          = - 3, ///< Error parsing the list of minimum lengths.
        TooManyMinLengthsGiven          = - 4, ///< A maximum of \c 7 minimum length values was exceeded.
        InconsistentMinLengths          = - 5, ///< The number of given minimum length values is greater than \c 1
                                               ///< but does not match the number of segments in the identifier.
        NoCaseSchemeFound               = - 6, ///< More than one minimum length value was given but no
                                               ///< segmentation scheme could be detected.
        MinLenExceedsSegmentLength      = - 7, ///< A minimum length is specified to be higher than the token
                                               ///< name, respectively the according segment name.
        DefinitionStringNotConsumed     = - 8, ///< The definition string was not completely consumed.
        ZeroMinLengthAndNotLastCamelHump= - 9, ///< A minimum length of \c 0 was specified for a segment that is not
                                               ///< a last camel case hump.
    };
    #endif
  protected:
 
    /// The tokens' definition string part.
    String      definitionName;
 
    /// The tokens' optional explicit export name.
    String      exportName                                                             =NULL_STRING;
 
 
    /// Defines the "case type" as well as the letter case sensitivity of this token.
    Formats     format;
 
    /// The minimum abbreviation length per segment. If only one is given (second is \c -1), then
    /// the field #".format" indicates normal tokens.
    /// Otherwise, the token is either snake_case, kebab-case or CamelCase.
    int8_t      minLengths[7]                                                      ={0,0,0,0,0,0,0};
 
    /// Letter case sensitivity. This is combined with the format bits.
    static constexpr Formats   ignoreCase    = Formats(1);
 
  //################################################################################################
  //  Constructors
  //################################################################################################
  public:
    /// Parameterless constructor. Creates an "undefined" token.
    Token()
    : format(Formats( ALIB_REL_DBG( Formats::Normal, DbgDefinitionError::EmptyName ) ))           {}
 
    /// Constructor used with function names that do not contain snake_case, kebab-case or
    /// CamelCase name scheme.
    /// \note Of course, the name may follow such a scheme.
    ///       With this constructor, it just will not  be detected.
    /// @param name           The function name.
    /// @param sensitivity    The letter case sensitivity of reading the function name.
    /// @param minLength      The minimum starting portion of the function name to read..
    /// @param exportName     An optional export name. If \b not given, the \p{name} is
    ///                       used with method #".GetExportName".
    ALIB_DLL
    Token(const String& name, lang::Case sensitivity, int8_t minLength,
          const String& exportName= NULL_STRING );
 
 
    /// Constructor with at least two minimum length values, used to define tokens that follow
    /// snake_case, kebab-case or CamelCase naming schemes.
    ///
    /// @param name           The function name.
    /// @param sensitivity    The letter case sensitivity of reading the function name.
    /// @param minLength1     The minimum starting portion of the first segment to read.
    /// @param minLength2     The minimum starting portion of the second segment to read.
    /// @param minLength3     The minimum starting portion of the third segment to read.
    ///                       Defaults to \c 1.
    /// @param minLength4     The minimum starting portion of the fourth segment to read.
    ///                       Defaults to \c 1.
    /// @param minLength5     The minimum starting portion of the fifth segment to read.
    ///                       Defaults to \c 1.
    /// @param minLength6     The minimum starting portion of the sixth segment to read.
    ///                       Defaults to \c 1.
    /// @param minLength7     The minimum starting portion of the seventh segment to read.
    ///                       Defaults to \c 1.
    ALIB_DLL
    Token( const String& name, lang::Case sensitivity, int8_t minLength1, int8_t minLength2,
           int8_t minLength3= -1, int8_t minLength4= -1, int8_t minLength5= -1,
           int8_t minLength6= -1, int8_t minLength7= -1                                 );
 
    #if ALIB_ENUMRECORDS
    /// Constructor using a (usually resourced) string to read the definitions.
    /// Invokes #Define.
    ///
    /// \par Availability
    ///   This method is available only if the module \alib_enumrecords is included in
    ///   the \alibbuild.
    /// @param definitionSrc  The input string.
    /// @param separator      Separation character used to parse the input.
    ///                       Defaults to <c>';'</c>.
    Token( const String& definitionSrc, character separator = ';'  )
    { Define( definitionSrc, separator ); }
    #endif
 
  //################################################################################################
  //  Interface
  //################################################################################################
  public:
    #if ALIB_DEBUG
    /// Tests if this token was well defined.
    ///
    /// \note
    ///   This method is only available in debug-compilations.
    ///   Definition strings are considered static data (preferably resourced).
    ///   Therefore, in debug-compilations, this method should be invoked and with that,
    ///   the consistency of the resources be tested. In the case of failure, a debug
    ///   assertion should be raised.
    ///
    /// @return #"DbgDefinitionError::OK", if this token is well
    ///         defined, a different error code otherwise.
    DbgDefinitionError DbgGetError()
    {
    return  int(format) >= 0 ?  DbgDefinitionError::OK
                             :  DbgDefinitionError(format);
}
#endif
 
    /// Returns the definition name used for parsing the token.
    ///
    /// \note
    ///   To receive the "normalized" name of this token, method #".GetExportName" can be used, or
    ///   a token can simply be #"alib_strings_assembly_ttostring;appended" to an instance of type
    ///   #"^AString".
    ///
    /// @return This token's #".definitionName".
    const String&   GetDefinitionName()                                                      const {
        ALIB_ASSERT_ERROR( int8_t(format) >= 0, "STRINGS/TOK",
          "Error {} in definition of token \"{}\". Use DbgGetError() in debug-compilations!",
          int8_t(format), definitionName)
        return definitionName;
    }
 
    /// If field #".exportName" is not \e nulled (hence explicitly given with resourced definition
    /// string or with a constructor), this is appended.
    ///
    /// Otherwise appends the result of #"Token::GetDefinitionName" to
    /// the \p{target}. If the token is defined \e CamelCase and the minimum length of the last
    /// segment is defined \c 0, then the last character written is converted to lower case.
    ///
    /// As a result, in most cases it is \b not necessary to provide a specific #".exportName"
    /// with the definition. Instead, this method should provide a reasonable output.
    ///
    /// \see Documentation section <b>Omitable Last Camel Hump</b> of this classes'
    ///      #"Token;documentation", for more information about why the
    ///      character conversion to lower case might be performed.
    ///
    /// @param target The #"%AString" that method #"%Append(const TAppendable&)" was invoked on.
    ALIB_DLL
    void            GetExportName(AString& target)                                            const;
 
    /// Returns the format of this token.
    ///
    /// \note Same as methods #".Sensitivity" and #".GetMinLength", this method is usually not
    ///       of interest to standard API usage.
    ///       These three informational methods are rather provided to support the unit tests.
    /// @return This token's format, used with method #".Match".
    Formats         GetFormat()                                                              const {
        ALIB_ASSERT_ERROR( int8_t(format) >= 0, "STRINGS/TOK",
          "Error {} in definition of token \"{}\". Use DbgGetError() in debug-compilations!",
          int8_t(format), definitionName)
        return Formats( int8_t(format) & ~int8_t(ignoreCase) );
    }
 
    /// Returns the letter case sensitivity of this token.
    ///
    /// \note Same as methods #".GetFormat" and #".GetMinLength", this method is usually not
    ///       of interest to standard API usage.
    ///       These three informational methods are rather provided to support the unit tests.
    /// @return The letter case sensitivity used with method #".Match".
    lang::Case            Sensitivity()                                                        const
    {             return (int(format) & 1 ) == 1 ? lang::Case::Ignore : lang::Case::Sensitive; }
 
    /// Returns the minimum length to be read. In case that this token is not of
    /// snake_case, kebab-case or CamelCase naming scheme, only \c 0 is allowed for parameter
    /// \p{idx} and this defines the minimal abbreviation length. If one of the naming schemes
    /// applies, parameter \p{idx} may be as high as the number of segments found in the
    /// name (and a maximum of \c 6, as this class supports only up to seven segments).
    ///
    /// The first index that exceeds the number of segments, will return \c -1 for the length.
    /// If even higher index values are requested, then the returned value is undefined.
    ///
    /// @param idx The index of the minimum length to receive.
    ///
    /// \note Same as methods #".GetFormat" and #".Sensitivity", this method is usually not
    ///       of interest to standard API usage.
    ///       These three informational methods are rather provided to support the unit tests.
    ///
    /// @return The minimum length of segment number \p{idx}.
    int8_t          GetMinLength( int idx )                                                  const {
        ALIB_ASSERT_ERROR( idx >= 0 && idx <= 6 , "STRINGS/TOK", "Index {} out of range.", idx  )
 
        return (idx >= 0 && idx <= 6) ? minLengths[idx] : -1;
    }
 
    #if ALIB_ENUMRECORDS
    /// Defines or redefines this token by parsing the attributes from the given substring.
    /// This method is usually invoked by code that loads tokens and other data from
    /// #"ResourcePool;resources" of \alib {lang;Camp} objects.
    ///
    /// The expected format is defined as a list of the following values, separated by
    /// the character given with parameter \p{separator}:
    /// - The #".definitionName" of the token. Even if the letter case is ignored, this should
    ///   contain the name in "normalized" format, as it may be used with #".GetExportName",
    ///   if no specific name to export is given.
    /// - Letter case sensitivity. This can be "Sensitive" or "Ignore"
    ///   (respectively, what is defined with resourced
    ///   #"alib_enums_records;ALib Enum Records" of the type #"lang::Case;2"),
    ///   can be abbreviated to just one character (i.e., <c>'s'</c> and
    ///   <c>'i'</c>) and itself is not parsed taking the letter-case into account.
    /// - Optionally, the standard export string is used with the method #".GetExportName", and
    ///   when appended to an #"%AString". Output names defined with this function must not start
    ///   with a digit, because a digit in this position of \p{definition}, indicates that
    ///   no export name is given.
    /// - The list of minimum length for each segment of the name. The number of values have
    ///   to match the number of segments. A value of \c 0 specifies that no abbreviation
    ///   must be done and therefore is the same as specifying the exact length of the segment.
    ///
    /// \note The given \p{definition} string has to survive the use of the token, which
    ///       is naturally true if the string resides in resources.
    ///       (String contents are not copied. Instead, this class later refers to substrings
    ///       of the given \p{definition}.)
    ///
    /// \par Availability
    ///   This method is available only if the module \alib_enumrecords is included in
    ///   the \alibbuild.
    /// @param definition  The input string.
    /// @param separator   Separation character used to parse the input.
    ///                    Defaults to <c>';'</c>.
    ALIB_DLL
    void            Define( const String& definition, character separator = ';' );
    #endif
 
    /// Matches a given string with this token. See this class's description for details.
    ///
    /// @param needle      The potentially abbreviated input string to match.
    /// @return \c true if \p{needle} matches this token, \c false otherwise.
    ALIB_DLL
    bool Match( const String& needle );
 
  protected:
    /// Detects snake_case, kebab-case or CamelCase.
    ALIB_DLL
    void    detectFormat();
 
}; // struct Token
 
    } // namespace alib[::strings::util]
 
    /// Type alias in namespace #"%alib".
using     Token=     strings::util::Token;
 
} // namespace [alib]
 
 
namespace alib {  namespace strings {
#if DOXYGEN
namespace APPENDABLES {
#endif
/// Specialization of functor #"AppendableTraits" for type #"util::Token".
template<typename TAllocator>
struct AppendableTraits<util::Token, character,TAllocator> {
    /// Appends the result of #"Token::GetExportName" to the \p{target}.
    /// @param target The #"%AString" that method #"%Append(const TAppendable&)" was invoked on.
    /// @param src    The #"%Token" to append.
    inline void operator()( TAString<character,TAllocator>& target, const util::Token& src )
    { src.GetExportName(target); }
};
#if DOXYGEN
}   // namespace alib::strings[::APPENDABLES]
#endif
}} // namespace [alib::strings]
 
 
ALIB_ENUMS_MAKE_BITWISE( alib::strings::util::Token::Formats )