Wow, looking at the dates I see that more than 2 months have passed since my last article.

The effort to fire up my new business was quite distressing, and the complexity of the things that we're doing in Falcon required an effort that was beyond my expectation. In particular, I've been coding 12-16 hours straight a day since last Tuesday (it's Saturday) to complete the reflective syntax support; now the basics and the skeleton of the system is done, but I need some help to fill in the gaps, so I am describing the system here. This text will go also into the VM/Falcon specifications when we organize it.

One fast note; as a preliminary work for this step, I removed the PCode system. For those who didn't follow the development, PCode were a set of pre-ordered expression PSteps to be specially executed by the VM. In the beginning, that seemed an optimization, so that the PCode could run multiple Expression PSteps before returning the control to the VM for new code to run, but that required the expressions to behave differently from all the other psteps (by not removing themselves when their execution was complete, and by considering their CodeFrame host untouchable), and also would have required re-compilation of the Host pcode when the expressions were changed. In the meanwhile, I found a PStep execution pattern that doesn't require the explicit intervention of the VM, and that is even more efficient than the original PCode idea; so we removed the PCode, and now the expressions, the statements and in general all the PSteps share the same exact behavior and have very similar execution patterns.

We're dealing with a pretty critical element of the Falcon new engine: the serialization process. Serialization is extremely important in the new engine because it actually serves many purposes.

• Storage of built-in type values on static streams.


• Saving and restoring of pre-compiled source code modules.


• Assistance in creation of stateful programs across multiple sessions (game save/load, session data on web base applications).


• Cooperative programming and live data sharing across network nodes.

Falcon has an elected web application platform called "Nest" (Falcon Nest...) which we're intensely developing and that's becoming ready for production services by the hour.

Initially, the only modular element in the picture was a so called "Service", a type of standardized module that could have been interacting with other services in a page, and that would have had to be "configured" (actually, programmed) on-site. For instance, a "Form" could have been a service, as a whole "Wiki". The idea is quite interesting for self-contained entities as the "Login" service, where the rendering of the login form, the check of the authorization level and other related activities can be tracked to a single conceptual entity; or, in simple database-table editing, where the form must just read and then store data in the required record; you just need, and most of the time just want, a very simple way to perform some consistency checks before storing data in the database, and then you're done.

I've been full force on some other projects for sometime now, but I urged to get back to the new engine -- also for business reasons. My intention is to dedicate a bit of care to it every day from now on, but today I worked all the day on it to get back in touch with the new API and the problems that were left open on the ground.

Today I have fixed a couple of nifty details that were left open: the booleanization of items (a = "value"; if a ...), which is now overridable, and safe and consistent cleanup of the execution context in interactive compiler.

Also, I have added grammar for both range declaration ([x:y:z]) and range accessor (x[a:b:c]). I dragged those a bit because they were extremely delicate and complex to integrate in the new array declaration syntax (which actually opens a simple, but powerful sub-parser for improved parsing of functional expressions).

Now I am searching for some ppl at #falcon implementing the implementation of range access in strings and arrays (functions are already in, we just need to copy some glue code from the old engine).

Well, we're back in action.

And so, we got try/catch statement working in the new organic virtual machine. Funny enough, it just took 3 days (less, actually) instead of the year it took with the old machine, and I was even able to add a finally keyword that would have been nearly impossible to be added in the old engine, with just a few touches more. Another proof, if it was needed, of the potential of the new engine. Here follows some details about what it took and what this is determining.

The new engine has now partial capability of load submodules and supports import/from statements in all flavors.

What's still missing in the module loading process:

• Precompiled module serialization-deserialization.


• Binary module load (but it's a cut-&-paste from old engine).


• Namespace definition.


• Macrocompilation.

But as usual, this element of the new engine is already more powerful that the corresponding one we previously had in the old engine, being able to finely address sources or other module generation devices, differentiating module URIs from module logical names since before the load starts and making the load/import relationship explicit at any step of the process. This makes possible some progress that were not feasible with the old engine, as, for instance, a finer control of plugin modules (and their dependencies) and the import/in self statement that shall allow to copy the global namespaces of other modules (creating module "collections").

Also, this second target makes the new engine structurally complete. This means that now the new engine, for how still experimental and rudimentary, is able to stand alone and test itself. Every part of the "main loop" is complete; everything else that must be done from now on is just "gap filling" work. The overall structure of the engine, the modules, the garbage collecting system, the symbol integration and sharing, the item model and anything that's vital to the Falcon Programming Language is either completed, advanced, or drawn, however is not anymore "on paper". It's here.

This also means that "outsiders" might now be able to work on the new engine to help complete it. So, from now on, we should be able to proceed faster.

I have been enlightened by a Great Truth(TM) today. Programming (without adjectives) may be something else, but Good Programming (TM(tm)) is all about giving names to things, or, I may say, "dubbing".

I've been struggling to reintroduce auto-operators (+=, ++, postfix ++ & family) in the Organic Virtual Machine for a couple of days; and a couple of "my" days, so, something 16 work hours per day.

After a couple of days of work, I have found that the only sensible way to allow dynamic loading of modules is that to create references for static data and let those to be separately managed by the garbage collector.

Consider strings, arrays of flat items or even user-defined classes that might go in the static data of a dynamically loaded module. If some item is take from by the user of the loaded module and sent around, it must stay alive as long as needed. If necessary, it must even keep alive the module that is hosting it, otherwise it should just stay around even after the module has died. I wouldn't love to keep a plugin around just because it has returned "OK" from inside a function it exposes, and that OK is kept somewhere for future records...

Adding the requirement that anything that goes or may ever go outside the dynamic module boundaries must reference the module back would be a nightmare. It may make sense for (some) classes and functions (and I think we can lower this requirement to the external/native code ones) but it's totally unreasonable for i.e. strings. Also the reverse requirement, that is, that a module must stays alive 'till all the data born from it are gone is pretty heavy.

So, the only sensible way to handle plugin-based dynamic (dischargeable) modules is that of making their static data references as soon as the module is linked dynamically in a virtual machine. When the module is gone, the reference items might still be around and keeping alive the related data.

For this reason, re-introducing the references was in order before completing the module loading system.

In the process, I also improved the usability of references that was somewhat limited in the old engine. Here follows a talk about it in our #falcon chan at irc.freenode.net:

As a part of the redesign of the new Falcon engine, I have cleaned the dependencies of the Falcon interactive mode to readline (GNU/GPL) and editline (BSD). For who doesn't know, these are two libraries that allow history navigation and a bunch of other features at a console prompt, which came pretty handy while trying code snippets and expressions in the interactive mode.

Readline is widely available, embedded in bash and a bunch of other important system tools all over the POSIX world. Editline is a son of a lesser God, but it still does the job fine, and has a more detailed, extensible API. Unluckily, readline has superior support for internationalization, while utf-8 and wchar_t structures have been introduced in editline only recently, and they are supported only through the system-wide setlocale() facility. Finally, editline is hardly portable outside Posix world (but that's a minor problem, as MS-Windows console has now a "reduced" but still useable console line editor burned in).

We'll have to improve the editline support. I think we can find volunteers helping out in this task, and then sending the patches upstream.

The problem with symbols in a programming language is in the fact that they represent a value, but are NOT a value. You can take them as flexible pointers to values, and this means that the real values must be stored somewhere else.

There is also another problem: symbols are themselves data, and so, they themselves must reside somewhere. Since values and symbols are never in a 1:1 relationship, you need a way to ensure the following conditions at the same time:

• The value of a variable (named by a symbol) must stay alive at least as long as the symbol is accessible.


• Symbols must stay alive as long as there is some grammar element or code referencing them.


• Values, symbols and their container must be collected as soon as possible when they are not used anymore. In short, they must not leak.

One of the main reasons to rewrite the Falcon engine was to fully support Functional programming at parser level.

There are some extensions, or we may call them "dialect", that greatly rely on functional programming, up to the level of writing entire modules of several hundred lines in functional sequences. Of course, it is necessary to have full compiler support into those long sequences, so that an error in the sequence body gets pinpointed and the compilation can proceed in search for further errors.

Well, the A register still exists; but now it's completely ruled out from the inner (new) function management, and it doesn't play any role in the call frame.

I took the occasion to get rid of other minor details in the generation of the call frame and in return routine.

As a result, I achieved another 5-10% performance increase in the non-optimized recursive fib test with respect to the old engine. Now it's consistently 20%-33% faster than ever before. And the VM model is now even simpler and more linear.

Today, I have completed the OOP model constructs in the Falcon new engine. Some details (for instance, Class states, or a direct way to generate a sub-classed prototype via a single call) remain yet to be dealt with, but they are largely things that I can complete later or that I can leave to be completed by the other developers in the project.

What I want to talk about in this entry is 1) the findings about the engine that I have discovered, and 2) the short term plan.

I think it's pretty natural to make some silly mistake when you dig into a very deep, complex and obscure programming structure. Especially if this structure is relatively new.

However, the last couple of errors I nailed (completing the HyperClass system in the new engine) are disturbing me a bit.

Here are the updated references.

Comments are welcome.

The Docbook source.
The PDF version.