EAV vs AM

One important area of concern in terms of database flexibility is the data model structures used for integrating new content.    This post will look at two completely different strategies used for building model structures that can easily accommodate change (at least compared to the typical 3rd normal form but not so for the typical noSQL form).     These are :

1. Use an Entity Attribute Value structure (EAV)
2. Use an Anchor Modeling structure (AM)  

In some way, these two structures are opposed to each other in their way to provide greater flexibility: EAV ingests new data content through its generic model, while AM uses highly normalised model for ingesting new content to adapt and grow incrementally without .   Note the term content here implies the modification of data structure and adding new entity and not simply the addition of new record, although this distinction is blurred for the case of EAV.

I'll discuss the pros/cons of these two here.

1) EAV structure

The Entity Attribute Value structure uses an open schema to store Parameter/Value pair.  This provides extreme flexibility in dealing with evolving scope of entities and parameters (here Parameter=Attribute).

This structure is good for : 1) high Sparsity Attribute values; 2) Attributes highly variable and unknown. 

Pros:

1. Fully evolutive: Data model and API layer can accommodate addition of any new Parameter type with some magic use of metadata (only metadata must be changed) 
2. Efficient storage for highly sparse data
3. Seem common in neuroinformatics world (ref to metadata standard: odML)

Cons:

1. EAV is more of an exception or anti-pattern in data modelling and is applicable for a few limited use cases
2. Require a rich metadata infrastructure: data-related business rules no longer enforced by relational engine but by its metadata layer
3. Higher complexity in data request, ad-hoc query (especially those cross-referencing multiple Attributes), data bulk extraction etc..

Implementation Impact:

1. often need a dedicated layer for managing data query complexity (e.g. ElasticSearch)
2. a no-SQL backend maybe more adapted (except for the metadata) unless there are clear requirements for concurrent transactional integrity and some form of isolation level 
3. When Historisation is a requirement, need to enrich EAV to support temporal attributes, and thus increasing querying complexity further.  There does not seem to be any temporal field inside the EAV instance table of its typical reference implementation (see ycmi as an example)  

2)  AM structure

AM favors strong (and fine-grained) typing of database schema while reducing a few symptoms common to rigid relational data model.  This is achieved by using higher normalisation structure (6NF).  

The AM structure is good for 1) integration of unforeseen and evolutive data model piece-wise, 2) offering clear and semantic-rich schema.

Pros:

1. Allows for data model evolution through organic extension without revisiting existing model structure
2. Historisation of all attributes is built-in without any compromise  
3. Not penalised by highly sparse data (NULL never needed)  
4. Produce models that are expressive and clear while carrying insights to the particular domain

Cons:

1. Require extension of both data model and API layer with each new data model type 
2. May suffer from table explosion (mitigated with a virtual access-layer) 
3. Need special Query execute plan optimisation (table elimination) 

Implementation Impact:

1. need DB engine with necessary optimisation (seem PostgreSQL meets some requirement, but should be further validated)
2. assume entities have strong identity with available candidate for natural keys
3. could require applying some form of generalisation to be less sensitive to addition of parameter (equivalent of ETL pattern when working in PULL mode)
4. Data modeller needs to know enough insight and have good Data Domain knowledge (data model end-result actually looks similar to domain ontologies) 

-----------

Data Model example

Here I try to model the scope that was presented to me at work.  However, I have skipped a few details and generalise the concept furthermore,  

In AM we should correctly separate Entity having clear identity (the Hub , shown in blue) from their attributes/descriptive/measures (the Attribute, shown in red), and identify relationship/association /transaction (the Tie shown in yellow).   The important thing to retain is that AM offer end-result model that are expressive and provide insights to the particular domain.  This is more easily done with one ore more SME.
   

An example of a Anchor Data model

Note: this does not follow a strict AM form :  Tie have attributes, naming convention not follow, more than one Attribute per table, audit metadata inside primitive, etc...

For the EAV, I used one reference implementation:
http://ycmi.med.yale.edu/nadkarni/EAV_CR_frame.htm.  I've added historisation capability at the attribute/value pair (EAV tables) as well as adding explicit common attributes to all Entities type.  Here, I've added natural-key for lookup and update existing entities (oddly enough there was none in the EAV ref, I guess these are buried inside one EAV attribute?).

This is inspired from the concepts of odML and its  metadata relationship: Section →0..n Property →1..m Value.   Section is used for grouping entities of same type and associated with a set of Property relevant to these entities.  These Section/Property/Value will grow as we add new data type into the PARAM_SPACE:  

An example of a EAV data model

Martin

Anchor modeling (part-2)

In previous post, I describe the Anchor modeling approach.  Here I will attempt to comment on its merit and providing a very small example of a data model .

My take on it:

Although, I never implemented AM on a large scale project myself, I can certainly see a few points that are very attractive at first glance.  Here are some that quickly come to mind :

• The model prescription really goes up to the 6NF (except when we decide to include multiple non-identifying Anchor FK or knotted attribute into a single Tie).  This 6NF implies that no other attributes are stored along with the Anchor surrogate-key, not even the natural-key(s).  Potential benefits are :  
  • Improved maintenance when dealing with evolutive or unstable natural-key.  This follows the principle of model extension where we can accommodate new or change of natural keys in the future.  

• The modelling prescription details specific and strict rules .
• Advantage: This reduces the likelihood of our model to go “wrong”.   As a data modeller you are offered much smaller degrees of freedom, at least after you have defined Anchors and Ties.  Modelling Tie may be more flexible, as it seems, you are free to add more or less non-identifying keys in the Relationship. 

• The model guideline recognises the importance of State often describing relationship (these sort of state are always mutually exclusive and exhaustive).  
• Advantage: Tie can be Knotted for storing this data in a straightforward way with no need to come up with external structure besides the Knot.   In DV, you’d need external Satellites for storing the attributes and their time-segment.  This feature is convenient as in real-world project we often see this pattern where relationship goes through some form of lifecycle changing state.  
• Disadvantage:  The flip side of this is that once your relationship is built like that, then it becomes less flexible and can hardly accommodate new stuff...In this respect, DV Links modelled as immutable intersection of Keys offer better longterm flexibility.

• The model guideline also recognise the importance of having at most one key, called Role, in the relationship outside the Tie key identifiers.  
• Advantage: For Historical Tie, this makes explicit what changes have triggered an update in the relationship.

• The model guideline recommends keeping Tie’s width small (#of Roles).   In practice, we should try breaking down larger Tie into smaller constituents.  This helps in regard to : 
  • handling of late arrival facts (or asynchronous) which would delay the capture of the relationship as recording can only happen once all roles are known;
  • decreased data redundancy generated from historised relationships with frequent changing state/property.  In this scenario, lots of nearly identical rows arise since all roles, except the non-identifying state/property role, stay identical.
  • increased stability:  larger relationship are more likely to become deprecated rendering your AM model less evolutive with more frequent replacement of existing Tie with new ones.

• The Knot is a primitive modeling construct holding important referential data:
• Advantage: static set text values are easily recognised.  We can avoid disseminating these text values, typical of lesser normalised structures, and thus reducing the data redundancy throughout our data model.

• The model guideline proposes a clear separation between metadata and data.  Auditing info is stored and maintained externally from data into their own metadata structures.  Metadata is referred to by FK pointing to a global Audit table in a standard and global way.   
• Maintaining data separate from metadata leads to cleaner data structures and less chance of misinterpreting timestamp data fields.

However an AM implementation has its drawback, the most important one being that we probably need a complete toolset and a separate access layer to help us manage the explosion of data structures (table, views..).  No one would want to interact directly at the physical level any real size AM implementation.

Other important issues could come from DB vendor's limitation which may not support some functions required (ex. Table Elimination is almost mandatory for query optimisation).   Bottom line, you certainly need a lot of experiment before leveraging this technique, and see if the additional complexity can be managed and mitigated in a large scale implementation.

A small Example:

I conclude this post by presenting a small example.   Using the available modeling tool, you can quickly start creating your own model.  So let’s imagine a DB used to collect any info, statistics, indicators/ metrics that are produced about countries and cities in the world.   As these data would need to be refreshed periodically it is important to capture the history as well as sourcing, format, and other meta-info.

The diagram model (each entity type has dedicated appearance) presented next, gives the overview of the data model.  Red squares are used for representing Anchor, circle for Attribute, rounded square for Knot while grey diamond-shaped represents Tie.   Optionally, we may choose to keep Historisation for Tie or Attribute, in which case we have outline represented as double-line.

General view of a simple AM model

The naming and mnemonic convention is used to automatically provide physical tables names (as well as views and functions).    We can toggle between mnemonic or real name in the diagram.

We see below the two main Anchor: City and Country naturally linked through a Tie, as well as with a few other Anchors tied together.

Model zoom-in around Hubs Country and City

Let’s suppose that during the lifetime of our Data repository, we had decided to keep as natural-key the ISO ALPHA-2 (2-letter code).  This was used as the country look-up with the help of Natural-key View.  However later on, we realised that a  lot of data sources actually used the ISO ALPHA-3 (3-letter code) instead as country identifier.   No problem here, our AM model supports transparently the addition of any sort of Attributes.  The only impact is that we now have two Natural-key View to choose from for the look-up.

We later discovered with disbelief, that our natural-keys are NOT immutable… quite annoying for a database key.   As always, this seemed easy to know in hindsight, but at the time who could have guessed that our ISO standard key will betray us.   Our data model must then allow for natural-keys evolution in time!  Again no problem, our AM model has “Historisation by Design” built-in, and the only impact is the addition of a new field for temporal validity inside the natural Attribute tables.  We would also need a slight change to the "Natural-key View"  to accommodate for the time point entry necessary to manage the new look-up logic.

Important note: had we kept our natural-key(s) along the surrogate, these evolutions would have involved breaking existing elements of our model!  This illustrates the benefit of higher normalisation with AM.

Conversely, from our small business case, we clearly see where the 6th NF could cause us real harm… If we start adding new each indicator as one Attribute around the Country and City (and historized as each can be updated in future), there will quickly be an explosion of tables making the entire data model unsustainable!!

To avoid this pitfall we make use of data model generalization principles and create more abstract entities.  So in place of adding new tables to store values for each new indicator, we turned these tables into rows.   In our simple scenario, this becomes possible by defining new abstract entities :  Indicator and the Indicator-instance.      Each Indicator has a clear identity (Anchor: Indicator) and is described by its name, its definition, its source (i.e. its surrounding Attributes).   The Indicator-value is represented as an anchor (IndInstance), whose identity correspond to the Indicator-value given for a particular country (or city) and at a given year.   One  Attribute holds the exact value while a Tie is needed to tie everything together (the indicator, the country and the year-period).

Model entities for Indicator and their instance values

More meta-info could be used in order to locate the information sourcing on the web, and other info related to format, document type.


Martin

ETL plugins for Data Vault

I recently finished a project whose goal was to provide an easy way to load data into a Database modelled using the Data Vault approaches.    At the time, I had chosen an open source ETL tools coming called PDI from Pentaho's suites.

Unfortunately, due to some issues there was a shift of strategy and this tool would no longer be needed.  In the sake of sharing code, I decided to upload this work to Git-Hub, so anyone willing to use/share/improve could do it freely.  Please note, that the code works as specified however a lot more realistic performance test and improvement is expected.  I know for a fact that some level of caching is necessary for the plugin to work in a realistic data warehousing environment.  Anyone feeling up to it, feel free to fork it!

Update note:
The whole project called "DV-loaders" with code source is now moved to github ( https://github.com/mart2010/pdi-datavault-plugin ).


The rest of this post gives more detail on the tools.

DV-loaders provide custom Transformation steps for loading Data Vault compliant objects: Hub, Link and Satellite. These are developed using plugin extension capability offered by Pentaho Data Integration tool (PDI, aka Kettle).

With these custom Steps, you can:

1. Load quickly and easily Hub, Link and Satellite
2. Define as many Steps as needed inside the same Transformation
3. Load mandatory attributes compliant with logical Data Vault rules
4. Load non-mandatory fields as-is (ex. batch-Id for auditing)
5. Adjust buffer size to fine-tune performance

As an example, consider this simple DV data model:

Then one can load it with this simple Transformation:

INTRODUCTION

---

- Data Vault is a methodology suited for building an integration data layer of the Enterprise Data Warehouse that relies on a simple and repetitive pattern-based approach.

- ETL tool gives you access to any data input format and any backend output out of the box, without re-inventing the wheel.

- PDI is an open source ETL tool that can be extended by custom plugin

- Data Vault + ETL + PDI = DV-Loader plugin

- DV-Loader plugin makes it even easier and more performant to load any Data Vault model from data stream processed by PDI.

Details

---

Features

The PDI DV-Loader plugin offers :

1. Simplicity
• easy configuration setting using a consistent ETL design-model
• same process usable for full AND incremental loading
• Load Hub and Link "expose" technical key (PK) to downstream steps (usable for dependent Satellites)
• ETL data flow structure closely follows the DV data model
2. Robustness
• fully re-entrant and re-startable ETL process
• support duplicated record in input stream
• support unsorted satellite historical records
• support "a posteriori" satellite historical catch-up
3. Perfomance
• leverage JDBC batch processing
• query lookup done not on individual key but rather using a set of keys to minimize the number of sluggish DB round-trip
• batch and key lookup size is defined by the parameter Buffer size
4. Compliancy
• load mandatory DV fields using DV logic:
• technical keys
• business keys
• satellite fields
• temporal fields setting satellite record validity
• audit mandatory fields
• load other none-mandatory field as-is (pass-through attribute like batch-Id for meta-audit)

Assumptions

A very small number of assumptions is assumed:

• Hub and Link must have primary key defined as unique sequence integer (support any DB data type used for storing integer)
• Temporal validity of the Satellite rows are determined through a single temporal field (Date type) available in the incoming record stream (entrant hop)
• Business keys must all be Not Nullable

Rules

The ETL flow design must respect these logical rules:

• Hub can be loaded once its business key(s) are available in input record stream
• Link can be loaded once all referred Hub primary keys are available in input record stream
• Satellite can be loaded once referred hub (or Link) primary keys are available in input record stream (normally appended upstream by Load-Hub step)

USER GUIDE

ETL data flow

Data model is loaded by attaching Load Steps through hops :

1. Load Hub
• Connect Step through a hop containing all business/natural key(s) in input stream
• Step will look-up business/natural key(s) and append the associated tech-key (Hub's PK)
• When found: append the tech-key returned by DB
• When not found: generate new key (using sequence method defined) and append it
• Step will "expose" the tech-key in the output stream as <Table-Name.Techkey-Name>
2. Load Link
• Connect Step through hop containing all tech keys of the relationship
• Step will look-up Hub tech keys and append the associated Link tech-key (its PK)
• When found: append the tech-key returned by DB
• When not found: generate new key (using sequence method defined) and append it
• Step will "expose" the tech-key in the output stream as <Table-Name.Techkey-Name>
3. Load Satellite
• Connect Step through a hop containing the Hub or Link's PK tech-key, the attributes and the "From-Date" temporal attribute controlling satellite record's lifecycle
• Step will load new satellite record based on different use-case:
• Temporal Satellite with Idempotent=true: sat record is loaded unless there is an identical consecutive records (default)
• Temporal Satellite with Idempotent=false: sat record is loaded unless irrespective of consecutive records
• Static Satellite (no "From-Date" temporal attribute defined): sat record is loaded unless one already exist for the Hub

---

Setting common to all Step

General Properties:

Property	Definition
Step name	Name of the Step must be unique within the Transformation
Connection	Database connection to use
Hub/Link/Sat table	Target table name to load
Buffer size	This determines the number of input rows processed at once. Too large value may generate Query lookup or batch insert causing JDBC driver error (typical values are of range 500 or more)

Audit-related Properties:

Property	Definition
Sys-creation Date/Time	The column holding the Timestamp indicating when record was loaded into DB (leave empty when not used
Load Record Source	Column holding the Audit record source information (leave empty if not used)
Record source value	Value to store in the column "Load Record source" (may be set by a variable substitution)

---

Setting specific to Step "Load Hub"

Property	Definition
Attribute Mapping	Mapping between input stream field and database column
Attribute of type Business/Natural Key	field(s) corresponding to the business key(s)
Other type	pass-through field simply loaded as-is (useful for attribute like batch-id, etc.
PK Sequence-Id setting	Define the technical PK column and which sequence generation method to use

---

Setting specific to Step "Load Link"

Property	Definition
Attribute Mapping	Mapping between input stream field and database column
Attribute of type "Relationship Key"	field corresponding to the Relationship keys (i.e. each Hub's Primary key)
Other type	pass-through field simply loaded as-is (useful for attribute like batch-id, etc.)
PK Sequence-Id setting	Define the technical PK column of the Link and which sequence generation method to use

---

Setting specific to Step "Load Sat"

Property	Definition
Idempotent transformation	Idempotent ignores records having identical state at two consecutive time (all attributes are equal). Data Vault standard is Idempotent, but you may have different requirements
Attribute Mapping	Mapping between input stream field and database column
Attribute of type "Foreign-Key to Hub"	field representing the Hub's PK (typically provided by an upstream "Load Hub" step)
Attribute of type "From-Date Temporal"	field representing the temporal attribute controlling the timeline of each Satellite record. Using an input field instead of a fixed adds more flexibility: if is appropriate, then we simply append it upstream, but other attribute could also be used (ex. when using file input, when using table input, etc..). You use the temporal attribute most adapted for your use-case
Attribute of type Normal	fields recorded inside the Satellite. All these control sat record timeline (or lifecycle). Pass-through fields may only be added if their changing values do not impact Satellite lifecycle with regard to the Hub record (the ETL batch-id attribute is a valid example of this)

INSTALLATION

Pre-requisite

 PDI version 5.x

Download and Install

* Download latest archive/package

* Unzip it inside folder: ${PDI_HOME}/plugins/steps

Check installation

* new folder 'DV-loader' should now exist in: ${PDI_HOME}/plugins/steps

* Re-start PDI Spoon UI (${PDI_HOME}/spoon.sh or ${PDI_HOME}/spoon.bat)

* Create a PDI Transformation and add DV-loader steps found under category Experimental:

• Load Hub
• Load Link
• Load Sat

Martin

What is Anchor Modeling? (part-1)

Anchor Modeling (AM) is another approach well suited for modeling the integrated/consolidated data layer within an enterprise data warehouse.  AM was originally created by Olle Regardt with formalisation done by Lars Rönnbäck.  It offers an open source database modeling technique accompanied by an online modeling tool available through an MIT license.  Most info is maintained around AM home.

The approach has some theoretical background with ideas borrowed from the information modeling world.  More specifically it is inspired by a method called Object Role Modeling (ORM) developed by Dr. Terry Halpin.   ORM, not to be confused with the Object Relation Mapping, is a modeling notation designed to help non-expert doing conceptual modeling of database schema.

The theoretical formalisation has also helped provide a very strict and prescriptive methodology compared to some more ad-hoc approaches used in the world of data warehouse (often heuristic-based developed with time and experience).  

It also offers some implementation ecamples, but at this time, the only complete implementation is ported onto the Microsoft SQL-Server engine.

Comparison to Data Vault

I will not go into details as to what is Anchor Modeling here.   Instead I will describe the similarities it shares with Data Vault (DV),  since both aims to decompose any data model into smaller and more basic constituents.    For those interested in detail description, please refer to this excellent article Anchor modeling - agile information modeling in evolving data environments written by its creators.  

1. Anchor ≈ Hub

• Both Anchor and Hub store Entities having strong identity.   There is an important difference:  in AM, an Anchor stands completely on its own, i.e. it only stores the surrogate-key (aka technical key) so keep natural-key (aka business-key) in separate Attribute table.   

IMO, this offers additional flexibility especially when natural-key(s) are chosen based on operational systems. These tend to have shorter lifecycle than the EDW whose lifetime should be, at least what EDW team hope for,  infinite!    By keeping natural-key(s) outside the Hub, we avoid making any assumption as to what are the natural-key(s) and their data type(s) in future.    AM propose to use a Natural-key View for handling the loading and look-up logic, and any change in natural keys would involve updating this View and adding the new Attribute table only.

2. Attribute ≈ Satellite

• Attribute is similar to the Satellite notion in DV.    Both adds contextual and descriptive data which are inherently time-dependent.   However to preserve the 6NF temporal integrity, AM enforces that each Attribute lands in dedicated table.   DV is more relaxed here and accepts any number of attributes in the same Satellite.  

AM supports either temporal Attribute (referred to as historised in AM modeling front-end tool), or static and immutable Attribute where no temporal validity period is required.   DV modeling guideline assume historisation by default, and requires to store the full time-segment with two distinct time points: Valid-from and Valid-to.  Again the higher-normalised nature of AM restricts us from storing the redundant Valid-to time point (= subsequent Valid-from time point).

The multi-attributes in DV imply that we must proceed with row comparison to know which one has changed between active record and previous one.  However, I'd argue that AM rule 1-Attribute = 1-table is too strict, and could be relaxed for some exceptional cases.   One example are attributes having strict functional dependency on a master attribute; think of user-friendly text attribute providing descriptive info for technical short code.  Other examples are for immutable attributes guaranteed to
remain so in the lifetime of the entity they describe.

3. Tie ≈ Link

• Tie is similar to the Link notion in DV whose role is to store the relationship between Anchor/Hubs entities .   However Tie does not support over-hanging Attributes the same way Link have their own Satellites.  This limitation is due to the fact that historisation is built-in for Tie as with Attribute.   Switching-on historisation adds Valid-from time point to the Tie's Primary-key, and consequently any Attributes referring to the Tie would be left dangling (or else requires duplicating the Attribute row in violation of 6NF rule).   

The drawback of AM follows that for any natural many-to-many associations having inherent attributes will involve the creation of a “tied anchor”.   DV is more flexible here, as it allows attaching Satellites to Link.   In DV time-period is not built-in inside the Link structure, so we are left with constructing the history of the relationship ourselves by adding time segment in external Satellite.  This is required when we want to keep track of time-period validity of any relationship.

Tie may also have some form of attributes like when we deal with relationship having state/role information. Under this circumstance, one can add a Knot attribute to the Tie, becoming a "Knotted-tie" that can now support historical change of its state/role (the knot carries this state information).   With this scheme, you should have at most one Knotted attribute outside the relationship identifier (or another un-identifying relation in extra inside the Tie) for keeping track of the state or role information of the relationship (or the extra role-key tie). 

4. Knot ≈ Reference 

• Knot is similar to the Reference notion in DV.    Both of them hold immutable data of lower cardinality.  They normally represent a set of fixed categories, codes or static attributes and are normalised into separate table to avoid update anomalies and duplicating string values throughout the database.   Main difference is that Reference in DV can store multiple strings whereas again in AM only one string code per table is allowed.    It seems though that Knots are first class citizen, whereas in DV, Reference seems more like an ad-hoc optional structure.   Knots also enhance the semantic of its reference, for ex, an Attribute referring a knot becomes a knotted Attribute, a Tie becomes a knotted Tie, both being either static or historised.  

A small digression on Time

Temporal aspect of AM: The notion of Time in AM is well framed semantically with clear definition.   It provides support of 3 notions of Time: Happening time.   Corresponds to when some events/transactions occurred: as represented by specific Attribute(s) attached on Anchor Changing Time.   Corresponds to when some Attribute/Tie (when historised) are valid: as represented by the built-in “Valid-From” Time point in Attribute/Tie. Note that no redundant “Valid-To” is added to close the time-interval in order to comply with the 6NF requirement.  This avoids the nasty “update” during data loading process, and the potential risk of recording invalid time period.  But this comes at the price of additional complexity for reporting valid temporal attribute at specific time.  Recording time.  Corresponds to when the data was recorded into our platform (i.e. loading metadata):  as represented by FK’s  referencing a particular log entry in metadata Audit tables. Externalising the “Recording time” has the advantage of separating data from metadata, where all timestamp found in the data structures correspond to functional dates whereas metadata is located outside these data structures.  This contrasts with DV where metadata-info sits alongside the data.  Although the metadata timestamp “Load_Start” is not supposed to carry any functional meaning, it is confusing that most Query/Report examples given in DV documentation use this metadata timestamp for returning the valid data at given point in time!   Time when data is loaded are rarely correlated with Time of data validity.  This can happen in some occasion, like we have no other alternatives (see case 3 below).   The way I see things, when we need to determine the right functional “Valid-from”, we can be faced with 3 different situations : Dates do exist explicitly at source, so we load them as-is (typical of data having specified lifespan with effective/expiration date , etc..) Dates do not exist explicitly at source but can be deducted from some technical dates t source, so we use these technical dates (typical example is source having LastStatusUpdate meta-info allowing us to know when the data has changed) Dates do not exist explicitly at source neither can they be deducted from other technical dates, so that one is left with the only option of using the Loading time Ideally, situation 3 is more the exception than rules in most EDW implementation.

 

Main advantages

Most advantages of the AM approach results from its highly normalised structure.  This offers benefits such as:

• Simpler Data loading involves cheap “insert” and no expensive “update/delete”
  • Fault recovery is more easily done since all rows data have a unique batch load traceability and can be deleted on exceptional situation (when erroneous)
  • Strict "No Null" policy :   null values never appear, thus eliminating difficulties like interpretability, indexing, and other issues.
  • Non-persistent elements never removed but flagged with Knot indicating their state of persistence
  • Maintenance is overall improved and simplified

• Data model evolution only involves extension of previous version
  • New content always involve incremental addition of either Anchor / Tie / Attribute / Knot structures 
  • These New data structures may, sometimes, render some existing structure obsolete, then we can simply leave them as-is with no onward refresh.

• Potential Higher Performance (here, it’s more tricky as the higher normalisation can also penalise performance, but the authors have highlighted a few valid points worth mentioning)
  • Narrow Query (hitting small number of attributes) can have very good performance with the aid of Table Elimination
    
  • Storage is highly efficient since practically no data redundancy exist, only a small number of indexes is actually needed due to the use of clustered table (aka index-organised table in different DB vendor) and also narrowness of table.   On this last point, it'd be nice to see how column-store architecture could be leveraged to exploit the AM unique schema.
  • Data loading practically free from locking and other concurrency issues as only “insert” are processed.

• Reduce translation logic between different representation:  the unique AM graphical representation is used for both conceptual and logical modeling
  
  • Furthermore, the physical representation maps directly model entities onto tables, simplifying modeling abstraction levels or even eliminates the need for translation logic. 

Martin 

Working in Jordan

At work we have won a contract in Jordan recently.  This gives me the opportunity to make longer stay in this country and enjoy more than just the few days typically spent at the hotel.

This is my first work experience in middle east, and there is a lot to learn being surrounded by a very different culture and habits than what you are used to… which is nice, I’m always keen in discovering other way of life.

I am working in Amman, capital of Jordan, with roughly 3 million people or about half the population of the whole country.   What strikes any visitor about this city is its high density!   You can appreciate this in various places as it is located in a hilly surrounding:




Depending on your location and elevation, you see buildings with very similar look and color as far as your line of sight can reach:

Shot taken from my hotel top floor 

Picture taken from the archeological site Citadel (Jabal al-Qal'a).

The streets are very animated ... and noisy as lots of car are honking for no apparent reason.   I later realised that a lot were directed toward me!!  Yes I look pretty much foreigner and I certainly do not act like a local (I haven't seen many walking long distance on the street of the city, so all taxis saw me as a potential client).

You can really get a sense of the popular crowd scene during friday bazaar (or Souk to be more accurate):



The whole region's history traces back quite a long time in the past, and many historical sites can be visited within the city.   For ex, the site Citadel is located on a “belveder” at the heart of the city.   This site has witnessed a number of different civilisation dating back from era as far back as neolithic.  It was also an important place during the Ottoman period and the Nabataeans, a very ancient arab civilisation that constructed Petra.

Various civilisation that were established at the Citadel

Apparently the lack of water in this region is not new as proved by the presence of these vast reservoirs.  The old civilisation quickly had to find ways to capture the precious rain water so scarce in this region of the world (there is less than 20mm of precipitation during 7 months in the summer with some month with literally no rain!).


There is also the presence of what is believed to be Hercule's hand... one of the remains of probably the largest statue of the roman empire:



Although there are not many Hammams found in the city as you'd expect from a middle-east city, there are ruins of this centuries-old tradition in the Citadel.



The artefacts found on sites is astonishingly old…. some cave dates back from Bronze era!




There is also just below the Citadel a roman theatre which I found very well preserved… and they actually still host event there:







The Mosque Abdallah is one of the very few that non-muslim can visit.  It was recently constructed in the memory of the very first king Abdallah of Jordan kingdom.




Although quite “meridional” I found the city's climate rarely unbearable at least until late spring.  I doubt that during mid-summer we can enjoy cool night that I did.   But still the air is quite dry and the fact that it is located on a high plateau helps mitigate the heat.

Further toward the east, we can go to the lowest point on earth:  the Dead sea located below the -400m (the exact elevation is constantly decreasing due to the accelerated evaporation!).   Going to this sea from Amman is quite an adventure, first the temperature will typically soar well over 10 degrees and the air pressure will also increase quite a bit!  The night we went there, it was chilly in Amman about 18C but at the sea it was still around in the range 0f 28-30C degree!  You quickly feel this place is special even before going to the sea.  It must be related to its high pressure atmosphere, low allergen air content and low UV radiation!

The sea water is also highly mineralised that it seems we swin in a visqueous soup with smell that was closer to metalic than water ...very strange!  Don't even mention the floating aspect which makes swimming a bit dangerous as your body has the tendency to pivot along its center of gravity, and point your head under the water line!  And you certainly don't want your eyes to even contact this highly corrosive water.

My first "Selfie" taken by my front camera... the only functioning lens I had.

Unfortunately I just had a half-broken camera over there.   I could not have my picture taken while comfortably reading a newspaper and lying in a gravity-free position (this is a must for any well respected tourist at Dead Sea).  So instead I ended up taking my very first "selfie" (my daughter later made me realise this) as a tentative to keep a trace of the incredible -400meter mark!

In a different weekend, I also went to visit Petra!  This was amazing and would deserve a full dedicated post.

Martin

Beautiful Creta

Il m'arrive parfois d'écrire sur un voyage quelques temps après l'avoir terminé.   Ici, je pousse l'expérience encore plus loin, car presqu'un an s'est écoulé depuis notre retour de la Crète.  Ceci s'avère un excellent exercice de mémoire et en même temps me fait réfléchir sur les raisons qui m'incitent à écrire sur ces expériences familiales.

Avant toute chose, pourquoi ces écrits?  Il y a bien-sûr l'aspect d'exposition ou d'exhibition au monde extérieur, si recherché par la génération facebook.  Mais je ne crois que ça soit ma motivation, première, d'ailleurs je n'ai ni compte facebook, ni instagram, ni twitter, ... et je n'ai jamais fait d'effort pour médiatiser ou faire connaître ce site.   Donc, plus vraisemblable, je crois que c'est un besoin fondamental de mémoire, mémoire pour moi et ma famille rapprochée.   Je me réjouis à l'idée que mes enfants et leurs futurs enfants pourront consulter ces souvenirs de famille dans cinquante ans!

C'est vrai que je pourrais procéder de manière complètement privé, mais au rythme des changement de logiciels, d'ordinateurs ou encore de support de mémoire, le risque de tout perdre m'apparaît assez élevé.   Et admettons que j'y mette toute l'effort pour conserver ces souvenirs, qui assurera cette conservation parmi mes générations futures... ok, ok pas sûr que j'aurais plus de chance avec ce "host".

J'ai donc choisi la facilité en évitant ces efforts et en pariant sur la survie d'un géant du web comme Google ... au prix d'une partie de ma vie privée.   Quoiqu'ici, je choisi seul, sans contrainte et de façon délibéré, tout le contenu exposé, ce qui est loin d'être le cas du modèle facebook qui est souvent incite les gens à exposer inconsciemment et de façon insidieuse une bonne partie de leur vie privée (ex. lors d'échange et de commentaires extérieurs).

Après ce long aparté, revenons au voyage.   Nous avons donc été en Crète à l'automne 2012 et ce du côté nord-ouest de l'ile tout près de Chania (La Canée).   Mes intérêts premiers, pour les voyages, sont plus tournés vers l'observations de phénomènes naturels que celui des traces laissées par l'homme.  Les transformations géologiques aux échelles de temps non accessibles aux humains restent pour moi, sans commune mesure aux constructions humaines!

J'avais pour objectif de voir deux points d'intérêts: les gorges de Samaria et les oliviers de Vouves, les premières étants les plus longues de toute l'Europe et les seconds  étant apparemment, les plus vieux (et peut-être un des 10 plus vieux du monde)!  Mes enfants avaient plutôt émis le souhait de voir le temple supposé du roi légendaire Minos (et le fameux monstre Minotaure), situé à Knossos (près d'Heraklion),  et ce suite à leur cours sur la mythologie grecque.   Sinon, je ne connaissais rien d'autres de cette île, sauf sa réputation d'offrir une des cuisines le plus santé de monde: le régime crétois!

Donc, voici un résumé de ce voyage, en photo montages, devenus en quelque sorte l'extension de notre mémoire d'humain ;-) !

Voici le fameux olivier à l'âge vénérable de 3000ans
... qui n'est pas touché devant un être vivant de cet âge? 

Et oui il se visite même de l'intérieur comme ont pu constater mes enfants!

ceci est le dernier escarpement rocheux toute à l'ouest de l'île,
de l'autre côté se trouve un trésor naturel nommé le Lagon de Balos 

Les chèvres de montagnes (souvent nommé Kri-Kri) arrivent miraculeusement à survivre sur ces rochers arides... 

La particularité de l'ile est sans aucun doute son eau crystaline...

..ce qui a permis aux enfants de découvrir les joies du snorkeling

et de profiter de la température encore clémente de l'eau
malgré un mois d'octobre bien avancé

Dans le vieux port de La Canée se trouve des vestiges de la conquête islamique turc (église convertie en mosquée) 

..le climat aride donne des teintes particulières aux nombreuses falaises plongeants dans la mer

une vue du port de La Cana situé au sud de l'ile, partie difficile d'accès!  

... comme on peut bien le voir ici, côté sud de l'ile est plus déchiqueté et beaucoup plus
sauvage que la partie nord

D'ailleurs c'est là qu'on y retrouve les gorges Samaria... 

Je me suis rendu seul à cette gorge, mais la montée en valait le coup...
au point le plus serré, seule quelques mètres séparent les deux parois 

les traces d'érosion sur le roc laissées par l'eau se retrouvent tout au long de la gorge 

Nous nous sommes rendus à ce rocher à la demande de la mamie ...
qui, est en fait, le lieu de tournage d'un film marquant de l'époque des années 60!

.. qui a reconnu le film ... et oui "Zorba Le Grec"  

Curieux arbre avec des fruits énormes aux formes étranges de courges?

Pour finir, voici LA photo du voyage prise ma mère!
le coucher du soleil à rendre jaloux les meilleurs photographes
  

Pour la petite histoire, nous avons malheureusement raté notre visite du temple de Minos.., avec les difficultés financières du pays, les heures d'ouvertures des sites historiques ont été réduites.  Nous avons raté de quelques minutes l'ouverture du site, désolé les enfants!!  Par contre, ce n'est que parti remise, car on devra revisiter cette île, surtout suite au visionnement d'un documentaire qui met en lumière la disparition de la civilisation Minoenne!   Selon de récentes découvertes, cette disparition serait due à un tsunami...  à regarder!

Martin