Scala "for" iteration with indexes

In Scala, to iterate through a collection of items while keeping an index, Seq.zipWithIndex:


for (e <- items.zipWithIndex) {
  println(e._1 + " at index " + e._2)
}


(I find this especially useful when writing Scala code that calls into Java library setter methods that are index-based.)

Getting Started is the Hardest Part

Too often, when I'm trying to get started on a small, personal software project, I'm stymied by the time it takes to get the development environment and project infrastructure setup. With a full-time job as a developer, an addiction to cycling, and the responsibilities associated with being the parent of a two-year child, it's hard to find the mental energy and time to work on even a small software idea.  So when I do have an hour of mental energy available, the last thing I want to spend it on is project setup and configuration task. Maven archetypes to the rescue! Archetypes allow you to setup your project nearly instantly, and if you have appropriate Maven support in your IDE, you'll be ready to code within second (okay, minutes). If--and this is a big if--you can find an appropriately up-to-date archetype that provides the exact stack of technologies upon which your project will rely. So far, I don't seem to have such luck (can any one tell me where I can find well designed sampling of Scala-based Maven archetypes?) So instead of trying to start off with someone else's half-baked archetype each time I need to start a project, I've decided to take the time create my own archetype(s) that I can reuse and evolve for my own needs. The following Maven reference page was all I needed to figure out how to generated my own custom archetypes: http://maven.apache.org/archetype/maven-archetype-plugin/advanced-usage.html.

Find Most Recently Modified File

To find the most recently modified file in the current directory tree:

find . -type f -printf '%T@\t%t\t%p\n' | sort -nr | head -n 1 | cut -f 2,3

How to nest single quotes in BASH command

You can't nest single quotes in a bash command, since there's no way to escape it, but you can do something like this instead:

alias a='perl -e '\''print "a\n"'\'

UNIX command to analyze Amazon S3 logs

I wanted to monitor the download activity on a particular file that I made publicly available on my Amazon S3 account.  Here's how I do it with from a bash command-line:

~/dev/s3-curl/s3curl.pl --id= -- https://s3.amazonaws.com/ 2> /dev/null | xpath -e '//Contents/Key/text()' 2> /dev/null | grep '^logs/' | xargs -i ~/dev/s3-curl/s3curl.pl --id= -- https://s3.amazonaws.com//{} 2> /dev/null | grep 'GET\.OBJECT.*'

where and are variables representing the name of your S3 bucket and the filename you'd like to monitor and is the name of the profile you configured in your ~/.s3curl file.

The above command assumes these S3 Logging options for the bucket:

• Enabled is checked
• Target Bucket is the same as the bucket containing the file being monitored
• Target Prefix is "logs/"

In a nutshell, this works by first downloading the file listing of the bucket, then extracting the log file names, then downloading each log file in turn and finally grep'ing them for the download activity (GET.OBJECT).

You'll need to have these command-line tools available:

• s3curl
• xpath

Internal vs. External Events in Databases

A database generally comprises data that records the (instantaneous) state of entities in the real world. If the historical states of the data are of interest, then a history of values may be recorded for these entities.  This history of values will usually be timestamped, and thus this historical list of data values can be considered as recording the occurrence of an event in the real world.  In other words, the real-world event changed the state of a real-world entity, whose updated state must then be recorded in the database.  Let's call these real-world events external events.

But now we must also consider that the process of updating the database to record the new real-world value is an event unto itself. Let's call this an internal event, since it is an event that is intimately tied to the database itself.  This event may be a human operator who manually types in a new value or hardware/software that captures the new real-world value and updates the database.  Consider that the act of recording the new value may or may not take place at the same time as the real-world value itself changed.  In other words the external event is distinct from the internal event.

I believe it is important to differentiate between external and internal events when developing a data model that captures the history of data states.  For one, recording internal events can help to audit data entry errors.  For example, erroneous data can sometimes be detected and even corrected if it is examined in the context of other temporally-proximal data entry events (e.g., a data entry operator that repeated a value from a previous data entry record).

Recording internal events explicitly can also help to troubleshoot querying anomalies caused by failure to take into account the difference in time between the occurrence of an external and its corresponding internal event.  For example, why did a query that was run at time t not reflect the updated state of the real world event that occurred at time t-1? It will not if the data entry--the internal event--occurred at time t+1.


Recording internal events can aid in the determining the time periods during which a database is "out of sync" with real-world entity values, due to data input errors.  Ideally, a database will always correctly reflect the state of the real-world, but in practice this is rarely the case. Inevitably, bad data will enter a system, and, at best, is corrected at some point in the future.  When corrections are made and recorded as internal events, the original and the corrected internal event timestamps can be used to determine when and for how long the database maintained inaccurate state.  If internal events can be marked as having been invalid, it is then also possible to generate reports that either ignore or include erroneous data states.  The advantage is that the database is not attempting to forget or otherwise hide the fact that erroneous data existed.  Much as database designers contend that data should never be deleted, one can argue that erroneous data should also not be deleted, but simply flagged.  Knowing that a database was temporarily maintaining bad data can be just as important as storing the correct values and their history.

Note that both external and internal events can also be used to record persons and comments associated with the event, in addition to just timestamps.

Internal events are most commonly recorded in log files, rather than as data in the database itself.  It can be very useful though to record internal events directly in the database, as this avoids the need to join log file output (of internal events) with database records of external events when performing troubleshooting or auditing tasks.

Configuring Spring-managed Hibernate event listeners with EntityManagerFactory

I'm in the process of migrating my work's web application from a pure Hibernate persistence architecture over to a JPA-based architecture (with Hibernate as the JPA implementation provider). The application uses Spring XML context files for its configuration. Previously, the persistence-related configuration defined a couple of custom Hibernate event listeners, as Spring-defined beans, and these were passed along to Spring's LocalSessionFactoryBean via its eventListeners property in a straightforward manner:

  <bean id="hibernateSessionFactory" class="org.springframework.orm.hibernate3.annotation.LocalSessionFactoryBean">
    ...
    <property name="eventListeners">
    <map>
      <entry key="post-load">
        <list>
          <bean ref="myPostLoadEventListenerBean" />
        </list>
    </map>
    </property>
  </bean>

However, after moving to a JPA-based configuration, and replacing the Hibernate-specific LocalSessionFactoryBean with a JPA-specific LocalContainerEntityManagerFactoryBean, the only way to declaratively specify the Hibernate event listeners is via the jpaPropertyMap. However, with this method, the event listeners can no longer be Spring-instantiated objects, but rather can only be specified as class names, which Hibernate will "conveniently" use to instantiate the event listener objects on its own. This is hardly convenient, and in the case where one's custom event listener classes rely upon Spring's dependency injection for initialization, Hibernate's initialization mechanism is downright limiting. What we would like to do, but cannot, is specify our event listener beans within the JPA-compliant jpaPropertyMap, as follows:

  <bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
    ...
    <property name="persistenceProvider">
      <bean class="org.hibernate.ejb.ConfigurableListenerBeansHibernatePersistence">
        <property name="postLoadEventListeners">
          <list>
            <bean ref="myPostLoadEventListener" />
          </list>
        </property>
      </bean>
    </property>
    <property name="jpaPropertyMap">
      <map>
         <entry key="hibernate.ejb.event.post-load">
           <list>
             <bean class="org.hibernate.ejb.event.EJB3PostLoadEventListener" />
             <bean ref="myPostLoadEventListener" />
           </list>
         </entry>
      </map>
    </property>
  </bean>

The limitation is ultimately introduced by Hibernate's EventListenerConfigurator, which only knows how to handle event listener class names, rather than event listener objects. That's fine, I suppose, but Spring should provide some factory class of its own that allows the configuration of JPA provider-specific properties (such as Hibernate's event listeners), to get around this limitation by taking the desired beans and then programmatically updating the underlying PersistenceProvider. But Spring does not provide this, as the LocalContainerEntityManagerFactoryBean simply passes along the jpaProperties without further consideration. In the end, Spring and Hibernate classes collude in a such a way that there is no way to accomplish what was easily done with the Hibernate-specific Spring configuration.

The solution I've come up with involves replacing Hibernate's HibernatePersistence class with an extended version of the class, ConfigurableListenerBeansHibernatePersistence, that specifically allows event listeners to be specified as objects, thus allowing Spring-instantiated bean to be injected. We then tell Spring's LocalContainerEntityManagerFactoryBean to use our special HibernatePersistence implementation, instead of the default HibernatePersistence class. Yes, this sidesteps the JPA-compliant interfaces, but I find no other way to accomplish the required injection of event listeners. The new Spring XML configuration looks like this:

  <bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
    ...
    <property name="persistenceProvider">
      <bean class="org.hibernate.ejb.ConfigurableListenerBeansHibernatePersistence">
        <property name="postLoadEventListeners">
          <list>
            <bean ref="myPostLoadEventListener" />
          </list>
        </property>
      </bean>
    </property>
    <property name="jpaPropertyMap">
      <map>
      </map>
    </property>
  </bean>

Seems like Spring should provide a similar solution out-of-the-box.

Note that ConfigurableListenerBeansHibernatePersistence also takes it upon itself to maintain the default event listeners that Hibernate normally creates on its own (Ejb3*EventListeners and various Default*EventListeners), and appends the application-provided listeners to these. Contrast this with the default configuration behavior, which requires a developer to specify the Hibernate-provided event listeners in addition to the listeners being added by the application. (Hibernate's approach is the most flexible, but a pain for developers that simply want to add new listeners that do not alter the underlying behavior of Hibernate.)

Finally, note that current version of ConfigurableListenerBeansHibernatePersistence only provides configurability of the PostLoadEvent for now.