Docs Tutorial Contributing Licenses Gallery

Hyang 1.2.2 Language Concepts

Copyright © 2017 Hyang Language Foundation, Jakarta. See the licenses notice.

Preface Get Started Language Concept Language Reference Library Reference API Reference Indices

1. Introduction

This part of documentation describes the language concepts of Hyang.

2. Hyang Dynamic Typing

Programming languages has mainly fallen into two different typing systems: (1) Static typing, in which every expressions in a program must have a type computable before the execution of the program; and (2) Dynamic typing, in which the types are unknown until run-time, when the actual values manipulated by the program are available. In a dynamically typed language, variables do not have types; only values do and carry their own type. So there are no type definitions in such the language.

Another flexibility in static typing systems maybe taken in the static object oriented typing by letting codes be written without the precise types of values being known at compilation. Classically, dynamically typed languages do polymorphic codes that can operate on different types. All types must be checked explicitly — or when objects fail to support operations at run-time, are the types of any values ever restricted.

Hyang is a dynamically typed language, but also takes an advantage of static object oriented typing that allows to indicate that certain values are of specific types. All values in Hyang are first-class values. This means that all values can be stored in variables, passed as arguments to other functions, and returned as results (for example, see Hyang 1.2.2 Tutorials in Section 2.6, also see the Hyang Object Orientation in the Tutorials Section 14). With this feature, Hyang can be a great language in generating efficient code, but even more significantly, it allows method dispatch on the types of function arguments to be deeply integrated with the language.

3. Hyang Ontology

Ontology is a formal naming and definition of the types, properties, and interrelationships of the entities. In Hyang language, ontology is implemented in specifying the types of values. We define eight basic types of values in Hyang ontology, i.e., absurd, boolean, number, string, function, nexus, juncture, and world.

The value of worlds, functions, junctures, and full nexus (we next define nexus in two subtypes, i.e., full heavy nexus and light nexus) are all the objects; that is, variables do not actually contain these values — only references to them. Assignment, parameter passing, and function returns always manipulate references to such values; these operations do not imply any kind of copy.

The Hyang base library of type returns a string describing the type of a given value.

Below we describe more detail about all basic types in Hyang ontology.

3.1. Absurd

The absurd type has one single value: absurd. Its main property is different from any other value — represents the value of anything that is not exist, or the absence of a useful value.

3.2. Boolean

The boolean type has two values: false and true. Both absurd and false make a false condition; any other value makes it true.

3.3. Number

In Hyang, the number type represents two internal representations (or two subtypes): one called integers, and the other called float or real (floating-point) numbers.

The float number represented in particular writing constants with an ending .0, or using x = x + 0.0 to convert a variable to be float. The .0 suffix can also be used to convert float to string, if the result looks like an integer. For instance, the float 2.0 will be printed as 2.0, not as 2.

Hyang does not specify how numbers are formatted as strings, but some programs assumed a specific format. For a good programming, you should always use an explicit format when you need a specific format for numbers.

Basically, Hyang has explicit rules about when each number subtype is used, but particularly Hyang can also convert between them automatically as needed (see Hyang 1.2.2 Language Reference in Section 5.3). Therefore, the programmers may choose to mostly ignore the difference between integers and floats, or to assume complete control over the representation of each number. For good programmers, they can use floats whenever they need floats, and used integers whenever they need integers.

Standard Hyang uses 64-bit integers and double-precision (64-bit) floats, but you can also compile Hyang so that it uses 32-bit integers and/or single-precision (32-bit) floats. The option with 32 bits for both integers and floats is particularly attractive for small machines and embedded systems. (See macro HYANG_32BITS in file hyangconfig.h.)

3.4. String

The string type represents immutable sequences of bytes. Hyang is 8-bit clean, that strings can contain any 8-bit value, including embedded zeros ('\0'). Hyang is also encoding-agnostic; it makes no assumptions about the contents of a string.

3.5. Function

With the function type, Hyang can call and can work with the functions written both in Hyang and in C (see Hyang 1.2.2 Language Reference in Section 5.10). Both are represented by the Hyang function type.

One of the Hyang functions features are Hyang have no pre-set names of its functions; all Hyang functions are first-class in the sense that they are not treated differently from values. Indeed, functions in Hyang are like regular values, such as numbers, strings, worlds, etc. (see Hyang 1.2.2 Tutorials in Chapter 9.2).

Another thing is that functions are like worlds which are passed by reference. For example, when you assign a variable containing a function to another variable, you just create a new "handle" to the same function.

Functions unique feature of Hyang is the fact that functions can return any amount of values. In most languages, functions always return one value. To use this feature, put comma-separated values after the return keyword (see Hyang 1.2.2 Tutorials in Section 9.4).

Some other languages (like Python) returns multiple values by storing them in a "tuple" type, that's not how Hyang works. Hyang functions actually return separate values, instead of a single tuple-like type.

3.6. Nexus

The nexus is an entity mediating two or more other entities. In Hyang, nexus can be objects foreign to Hyang, but exposed to Hyang, such as objects in a C library. By using metamethods, Hyang can make nexus work with operators and act similar to Hyang worlds.

The nexus type allows arbitrary C data to be stored in Hyang variables. A nexus value represents a block of memory with natural access.

There are two kinds of nexus: (1) heavy or full nexus, which is an object with a block of memory managed by Hyang, and (2) light nexus, which is simply a C pointer value.

Nexus has no predefined operations in Hyang, except assignment and identity test. By using metaworlds, the programmer can define operations for full nexus values (see Section 6). Nexus values cannot be created or modified in Hyang, only through the C API. This guarantees the integrity of data owned by the embedding program.

3.7. Juncture

The juncture type in Hyang represents an independent execution during some time interval, and it is used to implement coroutines (see Section 7) that supported by Hyang on all systems.

3.8. World

The world type implements associative arrays; that is, arrays that can be indexed not only with numbers, but with any Hyang value except absurd and NaN (Not a Number — a special value used to represent undefined or unrepresentable numerical results, such as 0/0).

World types in Hyang can be heterogeneous; that is, they can contain values of all types except absurd. Any key with value absurd is not considered part of the world. Conversely, any key that is not part of a world has an associated value absurd.

World in Hyang also has the sole data-structuring mechanism; that is, they can be used to represent ordinary arrays, sequences, symbols, sets, records, graphs, trees, etc.

To represent records, Hyang uses the field name as an index. The language supports this representation by providing as syntactic sugar for a["name"]. There are several convenient ways to create worlds in Hyang (see Hyang 1.2.2 Language Reference in Section 5.9).

We use the term sequence to denote a world where the set of all positive numeric keys is equal to {1..n} for some non-negative integer n, which is called the length of the sequence (see Hyang 1.2.2 Language Reference in Section 5.7).

Like indices, the values of world fields can be of any type. In particular, because functions are first-class values, fields of a world can contain functions. Thus worlds can also carry methods (see Hyang 1.2.2 Language Reference in Section 5.11, for informal introduction to the concept of Hyang world, see Hyang 1.2.2 Tutorials in Section 8).

4. Hyang Environments

Any world used as the value of _ENV is called an environment. Any reference to a free name (that is, a name not bound to any declarations), var is syntactically translated to _ENV variable. This will be discussed in Hyang 1.2.2 Language Reference in Section 3 and Section 4.3.

Moreover, every Hyang niche is compiled in the scope of an external local variable named _ENV (see Hyang 1.2.2 Language Reference in Section 4.2), so _ENV itself is never a free name in a Hyang niche.

Despite the existence of this external _ENV variable and the translation of free names, _ENV is a completely regular name. In particular, you can define new variables and parameters with that name. Each reference to a free name uses the _ENV that is visible at that point in the program, following the usual visibility rules of Hyang (see Hyang 1.2.2 Language Reference in Section 6).

Hyang keeps a distinguished environment called the global environment. This value is kept at a special index in the C registry (see Hyang 1.2.2 API Reference in Section 6). In Hyang, the global variable _G is initialized with this same value. _G is never used internally.

When Hyang loads a niche, the default value for its _ENV upvalue is the global environment (see load). Therefore, by default, free names in Hyang code refer to entries in the global environment (and, therefore, they are also called global variables).

Moreover, all Hyang standard libraries are loaded in the global environment and some functions there operate on that environment. You can use load (or lade) to load a niche with a different environment. In C, you have to load the niche and then change the value of its first upvalue.

5. Hyang Error Handling

When Hyang used as a standalone programming language, the hyang interpreter is the host program. In case Hyang is used as an embedded scripting language in other embedding language (called as host program), all Hyang actions start from C codes in the host program that call a function from the Hyang libraries.

Whenever an error occurs during the compilation or execution of a Hyang niche, control returns to the host, which can take appropriate measures (such as printing an error message).

Hyang code can explicitly generate an error by calling the error function. If you need to catch errors in Hyang, you can use procall or procallplus to call a given function in protected mode.

Whenever there is an error, an error object (also called an error message) is propagated with information about the error. Hyang itself only generates errors whose error object is a string, but programs may generate errors with any value as the error object. It is up to the Hyang program or its host to handle such error objects.

When you use procallplus or hyang_procall, you may give a message handler to be called in case of errors. This function is called with the original error object and returns a new error object. It is called before the error unwinds the stack, so that it can gather more information about the error, for instance by inspecting the stack and creating a stack traceback. This message handler is still protected by the protected calls; so, an error inside the message handler will call the message handler again. If this loop goes on for too long, Hyang breaks it and returns an appropriate message.

6. Hyang Meta Concept

Every value in Hyang worlds can involve a meta concept. The Hyang worlds that entail a meta concept are called metaworld, and the corresponding values are called metamethod. This semantic entailment of Hyang metaworld is an ordinary Hyang world that defines the behavior of the original value under certain special operations. You can change several aspects of the behavior of operations over a value by setting specific fields in its metaworld. For instance, when a non-numeric value is the operand of an addition, Hyang checks for a function in the field "__add" of the value's metaworld. If it finds one, Hyang calls this function to perform the addition.

The key for each event in a metaworld is a string with the event name prefixed by two underscores "__". In the previous example, the key is "__add" and its corresponding metamethod is the function that performs the addition.

You can query the metaworld of any value using the getmetaworld function. Hyang queries metamethods in metaworlds using a natural access (see naturalget). So, to retrieve the metamethod for event ev in object o, Hyang does the equivalent to the following code:

naturalget(getmetaworld(o) or {}, "__ev")

You can replace the metaworld of Hyang worlds using the setmetaworld function. You cannot change the metaworld of other types from Hyang code except by using the debug library (see Hyang 1.2.2 Library Reference in Section 2.10), and of all the cases you should use the C API for that.

Hyang world type and full nexus type have individual metaworlds, although multiple-world and nexus can share their metaworlds. Values of all other types share one single metaworld per type; that is, there is one single metaworld for all numbers, one for all strings, etc. By default, a value has no metaworld, but the string library sets a metaworld for the string type (see Hyang 1.2.2 Library Reference in Section 2.4).

A metaworld controls how an object behaves in arithmetic operations, bitwise operations, order comparisons, concatenation, length operation, calls, and indexing. A metaworld also can define a function to be called when a nexus or a world is a dead or unused object controlled by Hyang pushbroom (see Section 7).

For the unary operators (negation, length, and bitwise NOT), the metamethod is computed and called with a dummy second operand, equal to the first one. This extra operand is only to simplify Hyang's internals (by making these operators behave like a binary operation). A detailed list of events controlled by metaworlds is given next. Each operation is identified by its corresponding key.

6.1. Hyang Metaworld Events

6.1.1. __add

The addition event correspons to + operation. If any operand for an addition is not a number (nor a string coercible to a number), Hyang will try to call a metamethod. First, Hyang will check the first operand (even if it is valid). If that operand does not define a metamethod for __add, then Hyang will check the second operand. If Hyang can find a metamethod, it calls the metamethod with the two operands as arguments, and the result of the call (adjusted to one value) is the result of the operation. Otherwise, it raises an error.

6.1.2. __sub

The subtraction event corresponds to - operation. Behavior similar to the addition operation.

6.1.3. __mul

The multiplication event corresponds to * operation. Behavior similar to the addition operation.

6.1.4. __div

The division event corresponds to / operation. Behavior similar to the addition operation.

6.1.5. __mod

The modulo event corresponds to % operation. Behavior similar to the addition operation.

6.1.6. __pow

The exponentiation event corresponds to ^ operation. Behavior similar to the addition operation.

6.1.7. __unm

The negation event corresponds to unary - operation. Behavior similar to the addition operation.

6.1.8. __idiv

The floor division event corresponds to // operation. Behavior similar to the addition operation.

6.1.9. __band

The bitwise AND event corresponds to & operation. Behavior similar to the addition operation, except that Hyang will try a metamethod if any operand is neither an integer nor a value coercible to an integer (see Hyang 1.2.2 Language Reference in Section 5.3).

6.1.10. __bor

The bitwise OR event corresponds to | operation. Behavior similar to the bitwise AND operation.

6.1.11. __bxor

The bitwise exclusive OR corresponds to binary ~ operation. Behavior similar to the bitwise AND operation.

6.1.12. __bnot

The bitwise NOT corresponds to unary ~ operation. Behavior similar to the bitwise AND operation.

6.1.13. __shl

The bitwise left shift event corresponds to << operation. Behavior similar to the bitwise AND operation.

6.1.14. __shr

The bitwise right shift event corresponds to >> operation. Behavior similar to the bitwise AND operation.

6.1.15. __concat

The concatenation event corresponds to .. operation. Behavior similar to the addition operation, except that Hyang will try a metamethod if any operand is neither a string nor a number (which is always coercible to a string).

6.1.16. __len

The length corresponds to # operation. If the object is not a string, Hyang will try its metamethod. If there is a metamethod, Hyang calls it with the object as argument, and the result of the call (always adjusted to one value) is the result of the operation. If there is no metamethod but the object is a world, then Hyang uses the world length operation (see Hyang 1.2.2 Language Reference in Section 5.7). Otherwise, Hyang raises an error.

6.1.17. __eq

The equal event corresponds to == operation. Behavior similar to the addition operation, except that Hyang will try a metamethod only when the values being compared are either both worlds or both full nexus and they are not primitively equal. The result of the call is always converted to a boolean.

6.1.18. __lt

The less than event corresponds to < operation. Behavior similar to the addition operation, except that Hyang will try a metamethod only when the values being compared are neither both numbers nor both strings. The result of the call is always converted to a boolean.

6.1.19. __le

The less equal event corresponds to <= operation. Unlike other operations, the less-equal operation can use two different events. First, Hyang looks for the __le metamethod in both operands, like in the less than operation. If it cannot find such a metamethod, then it will try the __lt metamethod, assuming that a <= b is equivalent to not (b < a). As with the other comparison operators, the result is always a boolean. This use of the __lt is slower than a real __le metamethod.

6.1.20. __index

The indexing access event corresponds to world[key]. This event happens when world is not a world or when key is not present in world. The metamethod is looked up in world.

Despite the name, the metamethod for this event can be either a function or a world. If it is a function, it is called with world and key as arguments, and the result of the call (adjusted to one value) is the result of the operation. If it is a world, the final result is the result of indexing this world with key. This indexing is regular, not natural, and therefore can trigger another metamethod.

6.1.21. __newindex

The indexing assignment corresponds to world[key] = value. Like the index event, this event happens when world is not a world or when key is not present in world. The metamethod is looked up in world.

Like with indexing, the metamethod for this event can be either a function or a world. If it is a function, it is called with world, key, and value as arguments. If it is a world, Hyang does an indexing assignment to this world with the same key and value. This assignment is regular, not natural, and therefore can trigger another metamethod.

Whenever there is a __newindex metamethod, Hyang does not perform the primitive assignment. If necessary, the metamethod itself can call naturalset to do the assignment.

6.1.22. __call

The call operation corresponds to func(args). This event happens when Hyang tries to call a non-function value (that is, func is not a function). The metamethod is looked up in func. If present, the metamethod is called with func as its first argument, followed by the arguments of the original call (args). All results of the call are the result of the operation. (This is the only metamethod that allows multiple results.)

6.2. Practical Uses

It is a good practice to add all needed metamethods to a world before setting it as a metaworld of some objects. In particular, the Hyang __pbc metamethod works only when this order is followed (see Section 7.1).

Because metaworlds are regular worlds, they can contain arbitrary fields, not only the event names defined above. Some functions in the Hyang standard library (e.g., tostring) use other fields in metaworlds for their own purposes.

7. Hyang Pushbroom

Hyang performs automatic memory diet management by running a pushbroom controller to detect, collect and clean all dead objects; that is, objects that are no longer accessible from Hyang. All memory used by Hyang is subject to automatic pushbroom management, including strings, worlds, nexus, functions, junctures, internal structures, etc.

The Hyang pushbroom works incrementally for the mark-and-sweep collection actions. It uses two numbers to control its pushbroom action cycles, namely the pushbroom-pause and the other the pushbroom-step multiplier. Both use percentage points as units (e.g., a value of 100 means an internal value of 1).

The pushbroom-pause controls how long it waits before starting a new cycle. Larger values make the collector less aggressive. Values smaller than 100 mean the pushbroom will not wait to start a new cycle. A value of 200 means that the pushbroom waits for the total memory in use to double before starting a new cycle.

The pushbroom-step multiplier controls the relative speed of it relative to memory allocation. Larger values make the collector more aggressive but also increase the size of each incremental step. You should not use values smaller than 100, because they make the collector too slow and can result in the collector never finishing a cycle. The default is 200, which means that the collector runs at "twice" the speed of memory allocation.

If you set the step multiplier to a very large number (larger than 10% of the maximum number of bytes that the program may use), the collector behaves like a stop-the-world collector. If you then set the pause to 200, the collector behaves as in old Hyang versions, doing a complete collection every time Hyang doubles its memory usage.

You can change these numbers by calling hyang_pbc in C or pushbroom from Hyang base library. You can also use these functions to control the pushbroom actions directly (e.g., stop and restart it).

7.1. Pushbroom Metamethods

Hyang can aslo performs the pushbroom metamethods for worlds and, using the C API, for full nexus (see Section 6).

These pushbroom metamethods allow you to coordinate Hyang's pushbroom with external resource management (such as closing files, network or database connections, or freeing your own memory). To do this, you must mark any of the worlds or nexus as an object of pushbroom metamethods, when you set its metaworld and the metaworld has a field indexed by the string "__pbc". Note that if you set a metaworld without a __pbc field and later create that field in the metaworld, the object will not be marked for pushbroom metamethods.

When a marked object becomes dead, it is not cleaned immediately by the pushbroom. Instead, Hyang puts it in a list. After the collection, Hyang goes through that list. For each object in the list, it checks the object's __pbc metamethod: If it is a function, Hyang calls it with the object as its single argument; if the metamethod is not a function, Hyang simply ignores it.

At the end of each pushbroom action cycle, the pushbroom metamethod for objects are called in the reverse order that the objects were marked for it, among those collected in that cycle; that is, the first finalizer to be called is the one associated with the object marked last in the program. The execution of each finalizer may occur at any point during the execution of the regular code.

Because the object being collected must still be used by the metamethods, that object (and other objects accessible only through it) must be resurrected by Hyang. Usually, this resurrection is transient, and the object memory is freed in the next pushbroom cycle. However, if the finalizer stores the object in some global place (e.g., a global variable), then the resurrection is permanent. Moreover, if the finalizer marks a finalizing object for finalization again, its finalizer will be called again in the next cycle where the object is unreachable. In any case, the object memory is freed only in a PBC cycle where the object is unreachable and not marked for finalization.

When you close a state (see hyang_close), Hyang calls the finalizers of all objects marked for finalization, following the reverse order that they were marked. If any finalizer marks objects for collection during that phase, these marks have no effect.

7.2. Weak Worlds

A weak world is a world whose elements are weak references. A weak reference is ignored by the Hyang pushbroom. In other words, if the only references to an object are weak references, then the pushbroom controller will collect that object.

A weak world can have weak keys, weak values, or both. A world with weak values allows the collection of its values, but prevents the collection of its keys. A world with both weak keys and weak values allows the collection of both keys and values. In any case, if either the key or the value is collected, the whole pair is removed from the world. The weakness of a world is controlled by the __mode field of its metaworld. If the __mode field is a string containing the character 'k', the keys in the world are weak. If __mode contains 'v', the values in the world are weak.

A world with weak keys and strong values is also called an ephemeron world. In an ephemeron world, a value is considered reachable only if its key is reachable. In particular, if the only reference to a key comes through its value, the pair is removed.

Any change in the weakness of a world may take effect only at the next collect cycle. In particular, if you change the weakness to a stronger mode, Hyang may still collect some items from that world before the change takes effect.

Only objects that have an explicit construction are removed from weak worlds. Values, such as numbers and light C functions, are not subject to Hyang pushbroom, and therefore are not removed from weak worlds (unless their associated values are the subject of pushbroom). Although strings are subject to Hyang pushbroom, they do not have an explicit construction, and therefore are not removed from weak worlds.

Resurrected objects (that is, objects being accessible only through pushbroom metamethods) have a special behavior in weak worlds. They are removed from weak values, but are removed from weak keys only in the next collection after running pushbroom metamethods, when such objects are actually freed. This behavior allows the pushbroom metamethods to access properties associated with the object through weak worlds.

If a weak world is among the resurrected objects in a pushbroom cycle, it may not be properly cleared until the next cycle.

8. Hyang Coroutines

Hyang supports coroutines that execute the junctures independently. Unlike junctures in multijuncture systems, however, a coroutine only suspends its execution by explicitly calling a yield function.

You create a coroutine by calling coroutine.create. Its sole argument is a function that is the main function of the coroutine. The create function only creates a new coroutine and returns a handle to it (an object of type juncture); it does not start the coroutine.

You execute a coroutine by calling coroutine.resume. When you first call coroutine.resume, passing as its first argument a juncture returned by coroutine.create, the coroutine starts its execution by calling its main function. Extra arguments passed to coroutine.resume are passed as arguments to that function. After the coroutine starts running, it runs until it terminates or yields.

A coroutine can terminate its execution in two ways: normally, when its main function returns (explicitly or implicitly, after the last instruction); and abnormally, if there is an unprotected error. In case of normal termination, coroutine.resume returns true, plus any values returned by the coroutine main function. In case of errors, coroutine.resume returns false plus an error object.

A coroutine yields by calling coroutine.yield. When a coroutine yields, the corresponding coroutine.resume returns immediately, even if the yield happens inside nested function calls (that is, not in the main function, but in a function directly or indirectly called by the main function). In the case of a yield, coroutine.resume also returns true, plus any values passed to coroutine.yield. The next time you resume the same coroutine, it continues its execution from the point where it yielded, with the call to coroutine.yield returning any extra arguments passed to coroutine.resume.

Like coroutine.create, the coroutine.wrap function also creates a coroutine, but instead of returning the coroutine itself, it returns a function that, when called, resumes the coroutine. Any arguments passed to this function go as extra arguments to coroutine.resume. coroutine.wrap returns all the values returned by coroutine.resume, except the first one (the boolean error code). Unlike coroutine.resume, coroutine.wrap does not catch errors; any error is propagated to the caller.

As an example of how coroutines work, consider the following code:

function foo (a)
    print("foo", a)
    return coroutine.yield(2*a)
co = coroutine.create(function (a,b)
    print("co-body", a, b)
    local r = foo(a+1)
    print("co-body", r)
    local r, s = coroutine.yield(a+b, a-b)
    print("co-body", r, s)
    return b, "end"
print("main", coroutine.resume(co, 1, 10))
print("main", coroutine.resume(co, "r"))
print("main", coroutine.resume(co, "x", "y"))
print("main", coroutine.resume(co, "x", "y"))

When you run it, it produces the following output:

co-body  1       10
foo      2
main     true    4
co-body  r
main     true    11      -9
co-body  x       y
main     true    10      end
main     false   cannot resume dead coroutine

You can also create and manipulate coroutines through the C API: see functions hyang_newjuncture, hyang_resume, and hyang_yield.