Internal vs. External Events in Databases

A database generally comprises data that records the (instantaneous) state of entities in the real world. If the historical states of the data are of interest, then a history of values may be recorded for these entities.  This history of values will usually be timestamped, and thus this historical list of data values can be considered as recording the occurrence of an event in the real world.  In other words, the real-world event changed the state of a real-world entity, whose updated state must then be recorded in the database.  Let's call these real-world events external events.

But now we must also consider that the process of updating the database to record the new real-world value is an event unto itself. Let's call this an internal event, since it is an event that is intimately tied to the database itself.  This event may be a human operator who manually types in a new value or hardware/software that captures the new real-world value and updates the database.  Consider that the act of recording the new value may or may not take place at the same time as the real-world value itself changed.  In other words the external event is distinct from the internal event.

I believe it is important to differentiate between external and internal events when developing a data model that captures the history of data states.  For one, recording internal events can help to audit data entry errors.  For example, erroneous data can sometimes be detected and even corrected if it is examined in the context of other temporally-proximal data entry events (e.g., a data entry operator that repeated a value from a previous data entry record).

Recording internal events explicitly can also help to troubleshoot querying anomalies caused by failure to take into account the difference in time between the occurrence of an external and its corresponding internal event.  For example, why did a query that was run at time t not reflect the updated state of the real world event that occurred at time t-1? It will not if the data entry--the internal event--occurred at time t+1.


Recording internal events can aid in the determining the time periods during which a database is "out of sync" with real-world entity values, due to data input errors.  Ideally, a database will always correctly reflect the state of the real-world, but in practice this is rarely the case. Inevitably, bad data will enter a system, and, at best, is corrected at some point in the future.  When corrections are made and recorded as internal events, the original and the corrected internal event timestamps can be used to determine when and for how long the database maintained inaccurate state.  If internal events can be marked as having been invalid, it is then also possible to generate reports that either ignore or include erroneous data states.  The advantage is that the database is not attempting to forget or otherwise hide the fact that erroneous data existed.  Much as database designers contend that data should never be deleted, one can argue that erroneous data should also not be deleted, but simply flagged.  Knowing that a database was temporarily maintaining bad data can be just as important as storing the correct values and their history.

Note that both external and internal events can also be used to record persons and comments associated with the event, in addition to just timestamps.

Internal events are most commonly recorded in log files, rather than as data in the database itself.  It can be very useful though to record internal events directly in the database, as this avoids the need to join log file output (of internal events) with database records of external events when performing troubleshooting or auditing tasks.

OSGi to eliminate app server restarts?

As I've been pondering ways to improve the configurability of my work project across multiple deployments, OSGi has suggested itself as a potential solution. Primarily, I would like to allow other deployments to select the set of modules, and the implementations of these modules, that are placed into use in various production environments. Though I soon realized that our current Spring-based configuration mechanism is probably sufficient for managing this level of configurability; one merely needs to change the implementation classes that are defined in a XML file. And so OSGi might be too heavy weight of a solution. The main benefit of an OSGi solution, as I see it, is the ability to dynamically swap in and out these subsystems. But this dynamicism is not really necessary for my project, since deployments are fairly static once configured.

But then I thought to myself, how nice would it be to be able to replace modules during development without having to restart the application server? This is one of the larger time sinks developers face while developing web apps. In fact, it's enough of a problem that it spawned a commercial solution, JRebel (why I haven't tried this out already, is beyond me!). Shouldn't an OSGI app, if designed properly, be able to address this problem as well? If only a minimal skeleton of code is needed to bootstrap the app, and the majority of the code is comprised of OSGi bundles, then shouldn't we be able to dynamically update pieces of our running web app? I need to look into this further.

One other benefit of OSGi is the "runtime enforcement of module boundaries" [ref]. That's a very appealing feature, as well.

Since the Spring application framework now has OSGi support, it may not take too much to move to an OSGi-based design.

Just found a reason to use Spring Framework's "Lookup Method Injection" feature. Our system had a parser bean that recently began to maintain state, and so could not be reused without remembering to reset the state. It seemed more pure and safer to simply instantiate a new parser on each use. Even though the parser was configured as a prototype Spring bean, it would only be instantiated once (for a given web user session), since its parent "loader" bean was a singleton. Lookup method injection allowed Spring to provide the loader with a new instance of the parser for each usage, and without creating Spring library dependencies.

Domain Modeling with Scala

[started writing this in Sep 2008; will post now, though not really a complete thought.]

What I dislike most about implementing a domain model with Hibernate (in Java) is how relationships are not treated as first class elements of the model implementation. Consider how with ER diagrams, the concept of a relationship holds the same importance as the entities.

The problem is that the developer must write all of the relationship maintenance code, which is especially cumbersome and error-pone for bidirectional relationships (short circuiting infinite mutual recursive calls; add and remove methods; etc.).

I'm wondering if there might be some way of using Scala traits to mix-in a bidir relationship to the two connected entity classses...

Domain-Driven Design

Started reading Domain Driven Design. Will try to write about my thoughts on the book and the subject as I proceed.

The key point thus far is that the domain very much represents a common language that will be used to communicate ideas, requirements, and behaviors (both of the system and the business) between developers and domain experts. At the same time it must be comprise a functional piece of a software system. This dual role will help keep the requirements of the users and the design of the software in sync. The model should not in any way be concealed from the domain experts (and users). It should be used as the language of communication. When either side finds it difficult to communicate aspects of the software or of the business using only the lexicon of the model, it is quite likely that the model is not "right", in that it either does not properly "map" to reality, is overly complicated or focused on technical (software) details, or is incomplete.

Accept that that the model will never be perfect, but must evolve continually evolve as everyone (developers and domain experts) works towards the development of the software. It is important to remember that even the domain experts, while knowing the business side very well, may not know how to formulate their understanding into a formal (and executable) model.
The model does not need to model everything known about the business, only what is pertinent to the current goals of the software system. It is a model of some reality, after all, and not a duplication of reality.

Declarative Application Design

Some miscellaneous thoughts on designing applications to support declarative instantiations.

I find that as an application design matures, it evolves towards a declarative design, where the procedural/imperative code implements the mechanism/machinery of the application, and the declarative "code" implements the "policy", which is to say that it defines an instance of the application for a particular customer (business rules, UI look & fool, etc.). E.g., Data table columns are declared, rather than being specified in code; user interface is declared in *.xhtml files; etc.

It seems like all frameworks evolve from particular application instances, and that as developers seek to generalize their solutions, they create this separation between the mechanism and the policy implementations. At this point a framework emerges.

The need for a clear separation between the mechanism and the policy is certainly coming to the forefront as a concern on my current project, as the application is being adopted by other customers and will need to be configurable for each customers' particular needs.

A highly declarative application design would mostly get around the web application server restart problem, since the declarative portions are not Java, but text configuration files, which can be reloaded by the application. We already see this working with UI design, which is easily modified and redeployed with .xhtml files.

With a declarative design, I begin to think of unit tests as those which test the mechanism implementations, and the acceptance tests as those that test the policy.