Giancarlo Niccolai

Giancarlo Niccolai http://www.niccolai.cc/ en-us Nucleus CMS v3.33 � Weblog http://backend.userland.com/rss http://www.niccolai.cc//nucleus/nucleus2.gif Giancarlo Niccolai http://www.niccolai.cc/ Concept-Relation algebra http://www.niccolai.cc/index.php?itemid=478
The idea is based on a first try where I modeled the analysis of complex systems through networks of "emergence", connected by possibly recursive causal relationships, as described by Edgard Morin in its masterwork "The Method".

Here I publish a first draft of the idea extended to generic networks of arbitrary concept bound by arbitrary relations.The Concept-Relation algebra tries to provide a method to analyze networks of arbitrary concepts and relations. Of course, its predictive and normative ability depend on the quality of the analysis in the specific fields where it is applied, but the new Falcon engine, with its ability to dynamically create symbolic code, and it's orientation towards evolutionary programming, might be easily turned in an engine actually finding unknown relations binding arbitrary concepts.

In this period, I have contacted the VFF Institute which sent the first draft paper around to its member; I have received positive feedback and help to fix some details, mainly from Enrico Faggioli.

CR Algebra -- introduction.

The document is an informal introduction to the topic, and is still incomplete/immature, but I decided to publish it on my blog for the benefit of the people following the development of Falcon, to start reasoning about it together. There will be time for a scientific publication later on. Also, the publishing of Falcon to the Debian distro in this days has already brought a bit of attention to our work, so I'd like the people coming in and having a look to see what's we're working at and is soon to be available to them.

Anyhow, the audience of my personal blog is pretty limited, so we all might consider this document (marked as draft-3) as a restricted circulation draft :-).

---

In this days, I've been working at the networks between relations, or at relation-transformation function in the CR-Algebra realm. For instance, the "implies" relation that should transform a relation between to concepts into another; as in the case of a "contain" relation (a simple flat relation), and the "imply" relation with a "not contain" relation, which of course reverse its meaning. I'll try to publish something about that asap.

Also, I have several idea on how to implement and expose to the language this CR algebra. For one thing, I want to make it an actual programming paradigm, in the sense that you'd write program based on relations of concepts to make things "happen". More about this in later posts.

]]> Falcon http://www.niccolai.cc/index.php?itemid=478 Tue, 1 May 2012 15:52:40 -0700 Reflexive serialization http://www.niccolai.cc/index.php?itemid=475
Why did it take more than one year to have a "simple" thing as a skeleton of module serialization?

The reason lies in the way the new engine is conceived and developed. Although it is universally known that one rule of good design is to start with simple things and add modular complexity as things get nastier, this approach was simply not viable in the case of the new Falcon engine.

Mainly, we have three logic entities that are intertwined into a single, irreducible unit:
- The organic virtual machine
- The class-delegated data model
- The syntactic tree code definition

The virtual machine runs the syntactic tree, which operates on the data, which is then stored in the virtual machine. The syntree entities are reflected as data via class-delegation, and then the VM can manipulate them as data or execute them as code.

Besides this nuclear intertwined entities, the garbage collector, the storage system, the compiler(s), the Virtual File System and other minor ancillary entities interact more or less deeply with the central core of the system.

To be sure that this reflexive architecture could be working, it was necessary to bring it to a quasi-complete development stage. Solving just one of the numerous problems and challenges that are involved into this massive central core wasn't enough, as it wouldn't have been possible to use the success as a step towards further achievements. Or, said otherwise, it was an all-or-nothing bet.

We got all.

For instance, the Module class, which is responsible to present a module to a script, (i.e. finding Mantras on dynamic requests, adding or removing them dynamically etc.), is also in charge to provide the serialization layer with all the code needed to store any entity in the module. The serialization system itself is exposed to the scripts, and can be used to send arbitrary data, or code, or data and code entities anywhere in the world, or store them on any stream. The pre-compiled module that is stored in a fam file is the same thing that can be created in a script directly with a couple of statements. The data structures in the module, as the classes, could be inspected and changed, and the code, the functions, might be dynamically created or altered.

The module is then just an organized container that interacts with the Falcon linking system, so that foreign code can be automatically invoked, or so that forward declarations are served to the scripts through the help of the engine. But otherwise, it's an unnecessary entity, as you can dynamically create, inspect, alter, run and destroy any code from anywhere.

This is reality now.

There are a few secondary issues I want to take care of, to complete the public API; should be less than one day work. Then we have just to complete the engine withe a few constructs (switch and class states still missing), and we can move to the alpha release.

]]> General http://www.niccolai.cc/index.php?itemid=475 Sun, 25 Mar 2012 13:49:02 -0700 Falcon has mantras http://www.niccolai.cc/index.php?itemid=468

However, I introduced a concept in Falcon called "Mantra"; it's not visible at language level, at least not for now; it's a concept used in the engine. A Mantra is a basic entity that can be invoked, or better summoned in the engine or in a module. Namely, now it's either a Function or a Class. After searching a bit in the literature for a concept capturing this two concepts in one definition, I found nothing precise, so I picked this ... rather spiritual ... but adequate name.

(Yes, there are function objects, functionoids, functors and function classes, but that isn't quite a synthesis of the two distant concept of function and class; rather, is a way to express and re-define one of the concepts through the other one).

The need arose in the serialization. Some of the serialized entities are meant to be fully serialized, so that they can be re-built out of simple elements (PSteps, expressions, statements, values etc.), while others are just references to native entities, as for instance, C functions, or classes provided as native code by third party modules, or even directly created in the engine by embedding applications. The engine needs to know if those entities CAN be described in the VM language, or must just be "summoned" in an environment; if they are to be summoned, it must know what elements to search and load to restore them in memory, in the state they were found when serialized. And if the element can be expressed as VM code, the engine (or the user) might still wish not to do that, and just reference that entity by its logical place in the engine target environment. To allow this flexibility, the entities that can be treated this way should have a common storage behavior that can be overridden, which is Class::store &c does; but having the Class and Function hierarcies completely separated required either to write some glue code knowing both of them or doing some other terrible hack.

The best solution, is that of putting in common the things we know about Function and Class, and give them a common name (some times ago I wrote an article as programming being giving names to things). For instance: we know that they may come from modules, may be either opaque or expressed in a form that the engine can understand, they can be searched by name either in the engine or in a module providing them, they can be associated to symbols and they can be exposed to the script via Class handlers (yes, both Class and Function have Class derived handlers).

I found no better name for them than Mantra.

Other than this serialization issue, I felt somewhat uneasy at the idea that I had get/add function/class all around. They are often duplicate methods, with duplicate maps, just being different for the pointer types. Yes, modules need to do special thing for classes, but some special behavior doesn't really justify a completely different set of methods taking care of them. And the same thing goes for the Engine registered classes and known functions. And, similarly, I have different dynamic load methods for classes and functions... This Mantra entity allows me to simplify all those aspects, and it's ready to accept other basic concepts we might want to introduce in future.

Now both Function and Class derive from Mantra. Each mantra must present a handler Class (to be handled by the engine and exposed to scripts) via a virtual handler() method. As its descendant, Mantra itself must expose a handler; in its case it's ClassMantra; of course, it's a Class.

So, ClassMantra <-- Class <-- Mantra, and ClassMantra --handles--> Mantra. Of course, Class needs a handler as well: MetaClass <-- ClassMantra <-- Class <-- Mantra, and MetaClass --handles-->Class. The engine doesn't expose MetaClass, but if it were to, then MetaClass would be able to be a handler for a MetaClass instance.

Something similar goes with functions, but the chain is less spectacular, so I won't bore you with that.

We have now reached a point where we can serialize or invoke everything in the engine. In the channel, we're trying to find an ergonomic way to have this ability exposed to the engine users, so that they can cherry pick what to serialize and what to ask the remote side for invocation, giving sensible defaults where the user doesn't express any preference, and necessary constraints where one way is precluded; but that's just a matter of how to show it, as the problem of to do it is now solved.

]]> Falcon http://www.niccolai.cc/index.php?itemid=468 Sat, 25 Feb 2012 17:29:10 -0800 Everyone has an interface in the heart http://www.niccolai.cc/index.php?itemid=466
First, let's deal with the details. The title is "每个人心里都有一个路卡", literally, "Every person has a (network) card in the heart". The term "interface" as in "connector" has another "preferred" word in Chinese (which is 接口 - jiēkǒu - connection port); however, 路卡 - lùkǎ - is still used by specialists to describe the computer cards, especially the network cards, and it's also used sometimes to refer to the user interface (GUI) of a program. It bears an idea of motion and roadways, ad 路 - lù - literally means "road". It is evident that Kelly is using the meaning of "networking, interface" which is associated with this word.

The music is a gentle, cherishing electronic trance tune, where Kelly's warm, embracing voice is just perfect. If you're curious, you can easily find it on Youtube or on the net.

This song tells a story of how's life in this electronic era, in the Age of Information, where humanity is, on one side, pushed aside and turned into bits (in IT terms), but on the other, exalted and important like never before. In this age where computers seems to rule the order of the universe, staying human has become even more important than ever, and while it might be more difficult to hold our own humanity, we became able to express it through new means, or we may say now, through new media.

In this sense, we might now recognize a true interface in our soul ("heart", "soul" and "core" are interchangeable concepts all grouped under the Chinese word 心 - xīn), which allows us to communicate with other human being as never before. While there is a risk to lose our own humanity in the process, there is also the opportunity to enrich it and bring it to a new level of understanding, that was never so near and achievable as in this information era.

And in fact, when listening this song, I feel a bit like being One with the singer, as she probably wanted.

每个人心里都有一个路卡
Měi-gè rén xīnlǐ dōu yǒu yī-gè lùkǎ
Ogni persona nel cuore ha un'interfaccia
Everyone has an interface in the heart

词/曲：查可欣
Parole/musica: Kelly Cha
Song/Music: Kelly Cha

每个人心里都有一个路卡
Měi-gè rén xīnlǐ dōu yǒu yī-gè lùkǎ
Ogni persona nel cuore ha un'interfaccia
Everyone has an interface in the heart

每个人心里都有说不出口的话
Měi-gè rén xīnlǐ dōu yǒu shuō-bu-chūkǒu de huà
Ogni persona nel cuore ha parole che non riesce a esprimere
Everyone has words she can't bring up to her mouth

听周围那么嘈杂
Tīng zhōuwéi nàme cáozá
Ascolta... quant'è rumoroso qua attorno
Listen... how is noisy around here

淹没我没遮拦的表达
Yānmò wǒ méi zhēlán de biǎodá
Mi sommerge, ma non può offuscare le mie opinioni
It submerges me, but it can't obfuscate my opinion

是我总忍不过黑夜
Shì wǒ zǒng rěn bùguò hēiyè
È che sopporto a malapena l'oscurità della notte
Just, I barely stand the darkness of the night

总有那么多害怕
Zǒng yǒu nàme duō hàipà
E il sopportarla, quant'è spaventoso
And standing it, how is it dreadful

总担心如果爸爸妈妈离开我
Zǒng dānxīn rúguǒ bàba māmā líkāi wǒ
Mi rende sempre ansiosa come se papà e mamma mi lasciassero
It always makes me anxious, as if dad and mom left me

谁会真的爱我呢
Shuí huì zhēn de ài wǒ ne
E chi può davvero amarmi?
And is there someone that can really love me?

就算竖起一个手指
Jiùsuàn shù qǐ yīgè shǒuzhǐ
Anche se c'è un dito alzato
Eveni if there is a raised finger

困难也不会消失
Kùnnán yě bù huì xiāoshī
non nascondo che sia difficile,
I won't hide it's difficult,

没什么是可靠的
Méishénme shì kěkào de
niente di cui fidarsi.
and that there's nothing I can assure.

-----

我就是你
Wǒ jiù shì nǐ
Ed ecco che io sono te
But then, I am finally you

你就是我自己
Nǐ jiù shì wǒ zìjǐ
E tu sei esattamente me stessa
And you are finally myself

保护你我才能安全
Bǎohù nǐ wǒ cáinéng ānquán
protetti, tu ed io possiamo essere sicuri
Protected, you and me can be safe

才能找到我的家
Cáinéng zhǎodào wǒ de jiā
possiamo trovare la strada di casa
we can find our way back home

发现我也是路卡
Fāxiàn wǒ yěshì lùkǎ
e scopro che anche io sono un'interfaccia.
and I find out that I am an interface.
-----

每个人心里都有一个路卡
Měi-gè rén xīnlǐ dōu yǒu yī-gè lùkǎ
Ogni persona nel cuore ha un'interfaccia
Everyone has an interface in the heart

每个人都怀疑是否被别人遗忘
Měi gèrén dōu huáiyí shì fǒubèi biérén yíwàng
Ogni persona teme di essere dimenticata dagli altri
Everyone is afraid of being forgotten by the others

眼泪打湿了笑容
Yǎnlèi dǎ shīle xiàoróng
Le lacrime bagnano il sorriso
Tears wet the smile

时间只有一个方向
Shíjiān zhǐ yǒu yī-gè fāngxiàng
Il tempo ha una sola direzione
Time has just one way

习惯动作是把自己关上
Xíguàn-dòngzuò shì bǎ zìjǐ guānshàng
I gesti abituali mi tengono intrappolata
Ordinary gestures keep me trapped

总没有太多的自信
Zǒng méiyǒu tài-duō de zìxìn
Non ho mai avuto abbastanza fiducia in me
I never had enough trust in myself

渴望交流但又抗拒陌生人在近距离的地方
Kěwàng jiāoliú dànyòu kàngjù mòshēng rén zài jìnjùlí de dìfāng
Bramo relazionarmi, eppure faccio resistenza quando gli estranei si avvicinano
I am thirsty of relations, but I resist when a stranger gets nearby

最想有真正的自由
Zuìxiǎng yǒu zhēnzhèng de zìyóu
Ciò che più desidero è l'assoluta libertà
What I desire the most is the absolute freedom

可怎么就那么贪心
Kě zěnme jiù nàme tānxīn
Ma, questo, quant'è egoista!
but how much is that egoistic!

还坚持保守每个秘密
Hái jiānchí bǎoshǒu měi gè mìmì
E tengo in piedi e proteggo ogni mio segreto
And I stand on and defend every secret of mine.

-----

我就是你
Wǒ jiùshì nǐ
Ed ecco che io sono te
And I am finally you

你就是我自己
Nǐ jiùshì wǒ zìjǐ
E tu sei esattamente me stessa
And you are finally myself

了解你我才能完整
Liǎojiě nǐ wǒ cáinéng wánzhěng
Comprendo che tu ed io possiamo completarci
I understand that you and me can complete each other

才能找到我的家
Cáinéng zhǎodào wǒ de jiā
Possiamo trovare la via di casa
We can find our way back home

其实我就是路卡
Qíshí wǒ jiù shì lùkǎ
E sono davvero un'interfaccia
And truly, I am an interface

其实你也是路卡
Qíshí nǐ yě shì lùkǎ
E anche tu sei davvero un'interfaccia.
And truly, you are an interface too.

]]> Chinese http://www.niccolai.cc/index.php?itemid=466 Sun, 5 Feb 2012 17:15:34 -0800 ... and the big cirlce is now closed! http://www.niccolai.cc/index.php?itemid=463
The effort to fire up my new business was quite distressing, and the complexity of the things that we're doing in Falcon required an effort that was beyond my expectation. In particular, I've been coding 12-16 hours straight a day since last Tuesday (it's Saturday) to complete the reflective syntax support; now the basics and the skeleton of the system is done, but I need some help to fill in the gaps, so I am describing the system here. This text will go also into the VM/Falcon specifications when we organize it.

One fast note; as a preliminary work for this step, I removed the PCode system. For those who didn't follow the development, PCode were a set of pre-ordered expression PSteps to be specially executed by the VM. In the beginning, that seemed an optimization, so that the PCode could run multiple Expression PSteps before returning the control to the VM for new code to run, but that required the expressions to behave differently from all the other psteps (by not removing themselves when their execution was complete, and by considering their CodeFrame host untouchable), and also would have required re-compilation of the Host pcode when the expressions were changed. In the meanwhile, I found a PStep execution pattern that doesn't require the explicit intervention of the VM, and that is even more efficient than the original PCode idea; so we removed the PCode, and now the expressions, the statements and in general all the PSteps share the same exact behavior and have very similar execution patterns.

Syntax Tree Reflection

Falcon VM executes directly a tree-represented source code through a set of minimal code units called "PStep" that are executed in turn. However, not all the PStep represent a concrete entity of a source program; some of them are just generic instructions to the VM to perform some task.

Generics

A source program written in Falcon is represented by three categories of specialized PStep:

Statement: A statement is roughly a line of code in a Falcon source, eventually comprising a list of statements to be conditionally executed. Branches, loops, or even single instructions are statements.

Expression: An expression is a tree of simpler expressions bind by operators, which evaluate to a single result. Many statements use an expression to obtain a value that is then used to configure their behavior; for instance, the while statement repeats all the sub-statements it holds while the expression it bears is evaluated as "true". Those expressions are called "selector". Expressions found alone on a line form a special statement called AutoExpression whose sole purpose is that to evaluate them and discard or use their evaluated value in some specific way.

SynTree: A SynTree is a collection of sequentially ordered Statements. It represents a block of code that must either be executed or skipped. Some of the statements in a SynTree may control its execution status. Also, syntrees are provided with a selector expression, which might be used for special purposes depending on the statement they are included in, and of an optional target symbol that is bound to receive the result of the selector. For instance, the if statement holds a series of syntrees, which are interpreted as alternative branches if they hold a selector. In a try/catch statement, the selector indicates the kind of data that is to be caught, and the target, if present, is used as a variable where to store the value of the caught entity.

This entities fully represent a Falcon program source file, and they are descendant of a common class called TreeStep. A TreeStep is namely an entity that can be represented as a source Falcon instruction, or set of instructions, or seen the other way, it's the representation of a Falcon source token as an executable VM PStep element. The VM is not bound to execute just TreeSteps, as there are PSteps that can be injected in the VM as complements of the direct code tree representation (for instance, logic expression shortcut gates are PSteps that are onwed by an Expression entity and injected in the VM on need). However, all the code that can be represented as a Falcon source program is derived from the TreeStep class.

This distinction is very important because the TreeStep class has a very important optional feature that is not provided to all the PSteps: reflection. Each TreeStep must expose a Class, an entity derived from the Falcon::Class base class, which represents a handler through which the Falcon VM and other PStep are empowered to manipulate unknown data. In other words, TreeSteps are data known by the Falcon VM. As it's known the Class handler allows the VM to expose methods and properties to the user, to create new instances of the entity, to manage it's serialization and deserialization processes and to handle it's lifetime through the GC marking system.

Internals

The reflection is controlled by a set of files named engine/synclasses*, and the TreeSteps are under the engine/psteps/. The reason to centralize the Class handlers for the PSteps instead of spreading them writing somewhat nearer to their pstep is twofold. First, the PStep should not care about the fact of being handled by a Class or not. Other than declaring what class they are supported by, a TreeStep should totally ignore its handler. Second, 90% of the handler class can be mainly written by deriving from a common base b>ClassTreeStep and just adding the "virtual constructor" semantic needed to create an entity of the correct type on VM request. Also, many of those Class handler that require a specific behavior (nearly all the statements and some expressions) can have 50% to 90% of their behavior inherited from the base ClassTreeStep handler.

As such, a dictionary of class handlers is provided in include/falcon/synclasses_list.h, and some preprocessor macros are used to expand it to generate the classes. A include/falcon/synclasses_id.h is provided to store some special class ID used by the lexer and the parser to determine the context as they compile the source, or in some cases, at runtime by the interactive compiler.

Script interface

The ClassTreeStep base class exposes some methods to the scripts that are available for all the TreeStep elements.

Note: The standard of Falcon Class protocol indicates that a class exposed as "Name" to the script is named like ClassName. So, ClassWhile is seen by the Falcon sources as class named "While".

arity: size of the elements that can be directly accessed.

Operator[]; the index operator can be accessed to set or get the nth element. Some statements providing fixed but optional blocks can allow some element to be nil. For instance, the for statements access the main block, the forfirst, formiddle and forlast block respectively at index 0,1,2 and 3. Setting one block to nil means removing it.

insert(pos, element): Inserts a new element. The element must be of the kind accepted by the parent element (SynTree accepts statements, Statement accept SynTrees, Expression accept other Expressions). If the element has a fixed arity, an exception will be raised. The position (pos) has the same semantic of the [] accessor (negative index start from bottom), and if it is out of range, a new entity will be inserted at end (added).
remove(pos): removes the nth element. Elements having a fixed arity will raise an exception.

selector: A property returning or accepting an Expression. Setting it to nil means to remove the selector. Some elements do not accept a selector; other require a selector and can't accept a nil.

parent: the parent TreeStep. It's a read-only property; can be nil if the TreeStep is currently unparented.

The ClassSynTree handler adds a target class that expses a Symbol (internally handled by ClassSymbol) and can accept a nil in case the symbol is absent.

Each TreeStep element has a parent which is either 0 or a valid TreeStep. A TreeStep can accept as sub-element another TreeStep only if it has not a parent. Reparenting is not allowed. However, it is possible to get an element that currently has a parent and clone() it. The cloning process creates an exactly equal subtree, but the cloned element is unparented and used as the root of the new tree, so it can be stored into any TreeStep accepting it.

Garbage collecting

Parenting is very important for GC marking. When an TreeStep entity is found in a variable, it is marked; but actually, the mark is not applied directly to it, but it escalates up to the topmost parent. Since the TreeStep cannot be unparented once they have a parent, marking the topmost node of a tree (the unparented one) has the same meaning as marking the whole tree. The check for livelyness is performed on the topmost node, that can keep alive its tree alone. Values stored in the tree (for instance, items and symbols) are separately GC-locked, and cannot be disposed until the tree they are hosted in is killed. This grants the topmost parent of a tree full ownership of its sub-elements and to every entity the sub-element may relate to, so that it's possible to plainly delete the children of a node when the node is destroyed or substituted with another unparented node. Functions expose a RootTreeStep which is never exposed to the source files (it has not any handler Class) which can parent the topmost TreeStep and will propagate it's marking to the host function, which might in turn propagate the marking to its host class, if it's a method, and/or module.

Serialization

Since full reflection is granted to each TreeStep element, the task of saving pre-compiled modules is fully delegated to the Store/Restore system. The TreeStep class flatten and unflatten methods perform automatic storage of all the elements that can be accessed through TreeStep::selector, TreeStep::arity and TreeStep::nth virtual methods.

The Store system automatically finds the generator class of an item; so the only specifics that a TreeStep sublcass must respect in order to be serialized is that to offer a public empty constructor.

Note: The Storer system performs serialization in two steps: first, entities are unflattened; they must declare if some part of them might be subject to separate serialization (i.e. if they have other "items" that could be serialized), then they are stored. In the Class::flatten method, the class handler stores sub-elements that have their own class handler taking care of their own serialization, while in the Class::store method, the handler saves the low level data specific for that entity, if there is any. The Class::restore method is called to allow the handler to create a blank entity and eventually fill it with the specific data saved on the stream, and finally the Class::unflatten method is called to allow the entity to hook all the items that were separately restored.

The vast majority of expressions and statements are fully described through their class, selector and elements, so it's unnecessary to provide them with specific flatten/store methods. For those entities having some special attributes (for instance, the return statement that can have a doubt clause) and/or multiple selectors, (for instance, the for/to statement that has two or possibly three expressions defining the loop ends and its step, and a target symbol) it is necessary to reimplement the store/restore, and eventually flatten/unflatten methods. Of course, it is possible to use the base TreeStep::flatten/unflatten methods by invoking it directly, but the code is pretty simple (it just iterates over TreeStep::nth() arity() times, using the child step TreeStep::cls() to compose the item in the flattened array).

Constructors and other details.

To expose the TreeSteps to source scripts so that it is possible to manipulate programs runtime, it is useful to provide some script-level constructor to the Class handling the TreeStep. This is done by implementing the op_create class, which must use the parameters to create and, if useful or necessary configure the TreeStep before returning it to the script via VMContext::stackResult. As the entity is created anew, it should be delegated to the GC via the common FALCON_GC_STORE, where the handler class is the same class creating the entity ("this" is ok), and the stored entity is the just created TreeStep.

As Falcon has the ability to accept more complex structures and variable parameters as function (and constructor) parameters, the constructors exposed to Falcon need not to mimic those available in C++. For instance, the GenArray constructor accepts all the parameters as expressions that, in C++, must be added later on.

Note: Array class is the handler for the array items; GenArray is the handler for the ExprArray TreeStep, which GENERATES an array item. Similarly, all the expressions meant to generate a language item are prefixed Gen*.

For some basic case (n-ary fixed expressions, including zero-ary parameterless ones, variable length of similar tokens as the GenArray constructor), the support is included in the SynClasses system, which can generate standardized Class::op_create code. Specific behavior requires custom implementations.

Specialized TreeStep that offer some elements which cannot be easily captured through the selector/arity abstractions that the basic ClassTreeStep offers. For instance, the forstatements have targets (the for/in may have multiple ones), and expressions that are not seen as selectors. This special behaviors are to be exposed by reimplementing Class::op_setProperty and op_getProperty (which often involve also hasProperty, enumerateProperties and enumeratePV).

Note: All the classes in the ClassTreeStep hierarcy are derived from DerivedFrom. This class abstracts the behavior of a single-parent Class exposed to scripts. To create a class that the scripts see as derived from TreeStep, you need not to derive from ClassTreeStep, but instead to derive from DerivedFrom, passing the concrete instance of the handler ClassTreeStep to it. As any crucial class, ClassTreeStep is published by the engine, and it can be accessed through Engine::instance()->treeStepClass(). All the other TreeStep handler classes (at least, for the elements that are part of the core Falcon language) are members of SynClasses and instantiated at the construction of SynClasses singleton.

]]> Falcon http://www.niccolai.cc/index.php?itemid=463 Sat, 31 Dec 2011 07:41:00 -0800 The big circle of serialization http://www.niccolai.cc/index.php?itemid=458

Storage of built-in type values on static streams.

Saving and restoring of pre-compiled source code modules.

Assistance in creation of stateful programs across multiple sessions (game save/load, session data on web base applications).

Cooperative programming and live data sharing across network nodes.

Most notably the serialization process must be able to restore the status of a program at a different time, at least for those items that have been serialized. For instance, if an object belonging to a certain foreign class (coded in C++) is serialized and it must be deserialized on another program, the system must be able to dynamically verify the presence of that foreign code on which the class is based, or, if not available, try and load it to make it locally visible to all the entities that must interact with that entity. The thing is even more critical if you think of the case of hyper classes, where Falcon classes and foreign classes are seamlessly merged, and they may come from different Falcon modules and native dynamic libraries.

Of course, once a so complex system is setup, it is worth to use it as much pervasively as possible. To make it really useful in serializing any generic object, it must also provide a transparent mechanism to flatten cycles and multiple references to object. Without this feature, the serialization mechanism would be unsafe or partial even on simple arrays and dictionaries.

And, since modules have items to serialize, and some of them may be as complex as object instances, it's worth to integrate this mechanism in the precompile module save/load process (fam). Contrarily with respect to what happens in other programmings languages, fam files are totally standalone modules (similarly to java .class files, but with an internal link-time resolution process that allow to compile them at a place where not al the libraries that they rely on need to be actually available).

In the old engine, module serialization was implemented by kinda removing the problem: a fam module didn't contain the items, but just a set of instructions on how to generate them at link time. As a result, only a limited set of standard items could have been stored in the module. For instance, even declaring a dictionary of fully known static data would have required the VM to construct the dictionary at runtime. This is how it's done in every scripting language I know of, but the fact that we did had a way to serialize dictionaries and that I could not apply it to module pre-compilation bugged me. This problem become more evident in case of the attributes (static data about symbols that could have been queried also form a generic code without the need to run the module through a VM in advance). The data you could store in an attribute was limited to the data that the fam module generator was able to understand. At times, having an array of strings in an attribute could have been useful, but that was not possible.

So, the ability to serialize any kind of value that the falcon parser is able to understand, or even, that the virtual machine is able to generate, was too important to be overlooked and relegated outside the fam module loading process. But, since we had items in module, and ways to restore their value associating their class to them, it was worth to see if the mechanism could have been extended to the module code.

In Falcon, data and code are different, even if they can be pretty seamlessly merged. However, the code is bound to be an acyclic directed graph, while data in items needs not. Also, types of code entities is enumerable, while items are not. This might have suggested a different approach in the serialization of code and data. But, there was a detail that prevented this naive approach: in a non far future, we want to introduce self-modifying code, or in other words, reflect statements into language items. This means that items might potentially contain code, and more specifically, they might directly point to a code entity they are a part of. While the code itself stays directed and acyclic, once the border of values in the code leaf has been crossed it is possible that the entity they point to is the same code that contains them.

With this in mind, extending the same serialization of the items to the fam processing ceased to be an option and became a requirement.

Serialization of items happens through the help of their class, as items are opaque to the VM and the Falcon Class entity represents the item handlers. This means that every single grammar entity must be represented by a Class that the Falcon VM can use. For instance, if we have a child of PStep representing the While construct, we must have a class derived from Class called ClassWhile that knows how to serialize a While instance. In a Falcon program a While instance would be represented by an item containing a While C++ instance and a ClassWhile instance providing the scripting interface to the while construct.

It is not necessary to fully expose the interface of all the grammar constructs to the scripts by now, but all the classes handling the serialization of all the construct need to be put in place. While this is surely a lot of work, it's not terribly longer than writing a singe class knowing how to deserialize any grammar item stored in a file. And it has an interesting advantage: the grammar structures need not to be a closed enumeration anymore, and it becomes pretty simple to create new grammar constructs dynamically from third party modules. As long as those modules are available on the target environment, it is not necessary anymore that all the code entities are known by and declared in the central engine. While the parser is still not extensible (but it is easy to make it so, or even derive a new parser and use that one instead), this means that new module could bring in in new processing modes, and even whole new programming paradigms, as the grammar elements(the Psteps), are exactly what the VM understands and processes.

The serializer processor is now already able to query the classes for their instance to have a say in the process. In other words, instances can provide serialization handlers that must be processed by the virtual machine. This means that, contrarily to the old engine, a VM in place and ready to run is now a necessary part of the fam generation and restore method. However, in the most common cases, the VM won't be excited, and the overhead with respect to a simple serialization is totally marginal, (even more if compared to disk write times). Of course, deserializing a tree of instructions, instead of a flat table of code, is way more complex and cpu/memory intensive. OTOH, the deserialization of the code table is not the only thing that a scripting language must do in order to restore a pre-compiled module (and then run it), so I am pretty sure that the net cost of this serialization method is not as high as it could seem.

]]> General http://www.niccolai.cc/index.php?itemid=458 Tue, 25 Oct 2011 05:21:05 -0700 Nesting Ajax http://www.niccolai.cc/index.php?itemid=456
Initially, the only modular element in the picture was a so called "Service", a type of standardized module that could have been interacting with other services in a page, and that would have had to be "configured" (actually, programmed) on-site. For instance, a "Form" could have been a service, as a whole "Wiki". The idea is quite interesting for self-contained entities as the "Login" service, where the rendering of the login form, the check of the authorization level and other related activities can be tracked to a single conceptual entity; or, in simple database-table editing, where the form must just read and then store data in the required record; you just need, and most of the time just want, a very simple way to perform some consistency checks before storing data in the database, and then you're done.

But modern web application design requires far more. Initially, I thought that it was simply possible to delegate everything client-side to third-party client libraries, as jquery or mootools; in fact, Nest is pretty forgiving on the overall site structure and can be pretty shy, to the point that you won't even notice it's there... and so, not even use it. Especially when it comes to AJAX, while those client-side libraries have interesting pure AJAX frameworks where you just have to provide your server-side programming, the concepts behind Nest were too heavy-weight to delegate everything to a completely separate and stand-alone server-side AJAX interface. For instance, when activating the persistent login and database session management facility, having to javascriptize all the data that the server-side function of an AJAX framework need might be hard and clumsy.

So I thought a way to circumvent that, without necessarily overseed or outrule client-side javascript libraries, and I came out with the idea of "widgets". They are visual elements of the final page that are rooted inside the Nest application, and thus, they can use all the facilities used by Nest, as, for instance, the current status of the variables are they are set by existing services, or session data as it is handled by the session manager.

Widgets expose an AJAX interface that is published by Nest, and some simple javascript calls to excite the exposed AJAX interface and enforce actions that the AJAX interface wants them to perform. A simple javascript glue code (approx 150 lines) is used to bridge the Nest server-side widget representation and the client-side rendering.

Widgets can be configured right on-page (where they are rendered before sending them to the browser) so to link with any foreign javascript code or library. Also, while they have a pretty useful AJAX interface, they can use third party AJAX support. They themselves or the rest of the web application can interact with AJAX functions exposed by Nest independently by their widget AJAX interface, or could attach to the Falcon WebAPI simple AJAX framework (which doesn't require Nest to provide web AJAX features).

ATM this new feature is not very well integrated with the rest of Nest; i.e. the database oriented Form service is not using them. It will take a few days figure out how to work this feature in the existing framework.
]]> Falcon http://www.niccolai.cc/index.php?itemid=456 Sun, 25 Sep 2011 15:38:36 -0700 First day back on Falcon http://www.niccolai.cc/index.php?itemid=455
Today I have fixed a couple of nifty details that were left open: the booleanization of items (a = "value"; if a ...), which is now overridable, and safe and consistent cleanup of the execution context in interactive compiler.

Also, I have added grammar for both range declaration ([x:y:z]) and range accessor (x[a:b:c]). I dragged those a bit because they were extremely delicate and complex to integrate in the new array declaration syntax (which actually opens a simple, but powerful sub-parser for improved parsing of functional expressions).

Now I am searching for some ppl at #falcon implementing the implementation of range access in strings and arrays (functions are already in, we just need to copy some glue code from the old engine).

Well, we're back in action.
]]> Falcon http://www.niccolai.cc/index.php?itemid=455 Thu, 22 Sep 2011 10:27:44 -0700 Target #3 Reached (and how) http://www.niccolai.cc/index.php?itemid=453 try/catch statement working in the new organic virtual machine. Funny enough, it just took 3 days (less, actually) instead of the year it took with the old machine, and I was even able to add a finally keyword that would have been nearly impossible to be added in the old engine, with just a few touches more. Another proof, if it was needed, of the potential of the new engine. Here follows some details about what it took and what this is determining.
Again, the twist was that if finding a name to solve the problem. The new name, and the new concept introduced in the virtual machine is the code barrier.

After fumbling around a bit adding a separate try-frame stack, I tried to stuff the "finally" construct in. Other languages making good usage of exception handling, as Java, delegate finally to the role of cleaning up code sections not just in case of error throwing, but even in case of block break-out. This seemed desirable in Falcon as well. We have (willfully) a limited scope protection which makes variables declared inside the try block available also outside it. So, in Falcon it is possible to provide cleanups strategies that can be implemented in other languages through the finally keyword only. For instance, in falcon it is possible to write something like that:



try

   file = InputStream( ... )

   ....

catch IoError in e

   ...

end



// if file is an open file, it will be true here, else it will still be nil, and so, false:

if file: file.close()

while this simple finalization rule require a finally block in other languages. So, finally wasn't terribly necessary to implement cleanup strategies per-se, but the fact of providing a cleanup strategy on unstructured instruction issuing was terribly fascinating. Unstructured instructions are those statements that break the flow of the code and interrupt the flow. C has break, continue, return, goto and switch (partially unstructured), C++ adds throw; in Falcon we have:

break

continue

return

raise

Moreover, (this is a thing that I wasn't able to write on the blog yet), I have decided to experiment "break values", that is, funcitonal commands or values that instruct foreign code to perform unstructured operations. For instance, it is now possible to write:



function breakMe()

   ...

   if : break

   ...

end



...

while cond

  ...

   breakMe()

  ...

end

Break and continue propagate through the call stack past the call frame, and reach the innermost loop instruction. Actually, this is not a necessity of the new engine, but somehow it seemed to me that it was an interesting ability; however, it's an experimental feature, we might turn it off if it hurts more than help. Anyhow, the interesting thing in that is that you can now see "break" and "continue" as values, so you can "return break", and thus communicate the will to break out from functional loops using a nicer way than the old return oob(0).

This feature already required the ability to unroll the code stack in search for some "landing point" where it was safe to resume the VM control. Of course, the unroll process should have take the call stack status into consideration; willing to enlarge the scope of break/continue statements (scoping them similarly to "raise", that is, globally), it was necessary to consider eventual call frames to be unrolled while progressing backward in the search for a landing code frame.

So, this mechanism was present, but I didn't formalize it (didn't give it a "name") until I saw that "raise" and error catching was more or less doing the same thing. I then formalized the idea of "code barriers" that is, code frames that required a temporary or definitive stop of the code stack unroll process fired by a non-structured instruction.

return: stops at code frames marked as function callers by a call frame (call barrier).

continue: stops at a "next loop" code barrier.

break: stops at a "loop cleanup" code barrier.

raise: stops at a "catch barrier" matching the raise item type.

Notice that the nature of this barriers are different. In case of the of the return barrier, the barrier status is determined by a value stored in the return frame; also, the nature of this barrier allows to unroll the call frames without performing a reverse traversal, if conditions are favorable (but now you can consider that a mere optimization). In the case of the catch barrier, the barrier may be active or not depending of the fact that a catch clause can block the raised value or not. In the cases of continue and break, their barriers are mandatory and active unconditionally, each time they are met.

Yet, the mere fact of considering all this phenomenons under a common name allowed me to simplify the code and use just one function to handle all of them. For optimization reasons, the function is actually an inline template, where the template parameter is a code that implements the different behaviors of each barrier.

But the fact of naming this operation as "code unroll" (on unstructured statement), allowed a niftier thing. The finally block could have been named a code barrier as well: an optional code barrier suspending the code unroll process when met.

This allowed to deal almost transparently with break, continue and raise crossing call frames, and the fact that return stopped at first call frame could have been considered an incidental, non-influent detail now.

But this even allowed to sneak in a feature that is almost dreamed of by other languages, even major ones as Java: the ability to deal cleanly double-raise from finally blocks.

In fact, the engine keeps track of the ongoing process during finally allows to deal correctly with errors happening from within it. Our engine provides "sub-errors" natively, so it came pretty natural to me to deal with error raising from within finally handlers (even nested ones) by adding the error they thrown to the ongoing, already-traveling error.

ATM, I think it's sensible to let finally to override the ongoing operation; for instance, if they were excited on a break or continue, they could neutralize it by issuing an explicit return, or conversely, if excited on a clean termination of a try block or on an exception raisal, they could nullify it by issuing a continue or break operation. Also, as raise could throw not just exceptions, but any kind of item, a plain item raised by something inside the try block could (and IMHO should) be overridden by an explicit item raise in the finally block. However, those override operations would all be explicit, and not a side-effect of the language structure as in the case of error reaping in throwing exceptions from a Java finally block.

Yet, as we have full knowledge of what's going on during the finally operation, we might decide to do something else, and issuing an error if finally tries to override the ongoing process, we we find this to be more sensible.

One final note: actually, the time needed to code all this things in was somewhere about 4 hours. The rest of the 3 days were needed, in part, to refine the concepts of unroll process and code barriers, and in part to complete some elements of the engine that will be useful elsewhere: for instance, the Requirement class, a generalization of the Inheritance class, that allows for link-time resolution of extern symbols on even deep structures, as, for instance, the catch blocks deep inside try statements.

(Actually, the catch blocks are implemented as an instance of the select statement, which switches over the type of a value -- so that statement internal structure is complete as well).

]]> Falcon http://www.niccolai.cc/index.php?itemid=453 Wed, 17 Aug 2011 16:46:25 -0700 Target #2 reached http://www.niccolai.cc/index.php?itemid=451
What's still missing in the module loading process:

Precompiled module serialization-deserialization.

Binary module load (but it's a cut-&-paste from old engine).

Namespace definition.

Macrocompilation.

But as usual, this element of the new engine is already more powerful that the corresponding one we previously had in the old engine, being able to finely address sources or other module generation devices, differentiating module URIs from module logical names since before the load starts and making the load/import relationship explicit at any step of the process. This makes possible some progress that were not feasible with the old engine, as, for instance, a finer control of plugin modules (and their dependencies) and the import/in self statement that shall allow to copy the global namespaces of other modules (creating module "collections").

Also, this second target makes the new engine structurally complete. This means that now the new engine, for how still experimental and rudimentary, is able to stand alone and test itself. Every part of the "main loop" is complete; everything else that must be done from now on is just "gap filling" work. The overall structure of the engine, the modules, the garbage collecting system, the symbol integration and sharing, the item model and anything that's vital to the Falcon Programming Language is either completed, advanced, or drawn, however is not anymore "on paper". It's here.

This also means that "outsiders" might now be able to work on the new engine to help complete it. So, from now on, we should be able to proceed faster.
]]> General http://www.niccolai.cc/index.php?itemid=451 Tue, 2 Aug 2011 14:54:09 -0700