Martin Ouellet Notes

Personal Knowledge Management

2011-04-11T09:15:00.007+02:00

I've always been looking for a tool that could simplify all notes I take to store and gather business knowledge I accumulate along my consulting activities. This could even be leveraged for other aspect of life for that matter, but for now my need is to optimize the knowledge acquisition and retention done when taking on new professional projects.

The level of cognitive load is directly linked to all the new information (principles, fact, rules, definition, etc.. ) that is to be assimilated early on any new project inception. Any method or tool helping me to optimize my knowledge acquisition and management could certainly be a big benefit for the project.

The key spec I'm looking for in such a tool are:

must be flexible and easy to add new content without having to first organize and structure the content, the relationship among concept and taxonomy
must be light (must not "get in the way" ) and ideally available from the net and stored the info centrally
must easily allow for structuring, re-structuring the content (either through tag, hierarchy, taxonomy, etc...)
and obviously must be intuitive...

There are tools more oriented toward online collaboration which have been adapted for personal use: personal wiki ( e.g. twiki for personal, moinmoin personal). But I find these too html-presentation oriented... my concern is to gather information not how should it be represented.

Although I did not yet find the optimal solution, I find the tools that falls into the area of "mind-mapping" (e.g. theBrain, freeMind, thinkgraph) to offer good characteristics. You are not too worried with how info is presented (since it pretty much follow a fix canvas) so you can fully dedicate yourself with the information content. However, it does suffer from a few limitation: volume content (I don't see collpasing/exploding thousands or more nodes), versioning and/or time-dependent information, ...

So until I find a better match, I will keep on relying on mind mapping software, and it always help when you can have a good open source and free version such as Freemind. And for an online solution, there is also mind42.com.

As an example, I just used it to gather some background knowledge on rules/best practices applicable when one wants to define data layer of aggregation in a typical data warehouse implementation relying solely on relational technology (inspired from Mastering data warehouse aggregates). Here is a partial view of this guideline (done with Freemind):

Here is the same guideline shown as a dynamic mind map :

Martin

Thesis

2010-08-05T17:38:00.018+02:00

One of the reason why I'm holding this blog is to archive things I consider important enough to be kept in a longterm platform as I hope this one to be? A while back, I wanted to put online the thesis I wrote in the mid-90's... wow that sounds old! Being optimistic I though this would be a simple matter... it is not as if I wrote this with a typewriter requiring me to scan the entire document! Word processor existed back then and I even used what was considered at the time the top of word processor: WordPerfect.

Well this wishful thinking was ignoring limitation of technology and its rapid obsolescence. When I consider what I ended up using to archive this important piece of work> not USB Flash drive, not CD , not DVD, not even Zip drive.. yes on those 3.5inch floppy disk: there was simply no chance of getting it back. I did manage to restore one diskette, but was less successful in reading and converting its format (of course I kept the files on their original WP 5.2 format ! ).

I bet nowadays most modern Universities preserve all post-grad work in their original digital content.

Anyway, enough allusion to my age, and here is the main content of this post> A full scan in pdf of my thesis dated back from 1994:

Martin

p.s. Note that the paper copy I could get a hold on to still contains some minor editing to be done, so for the real final copy I'll need to get to McGill University.

PALO MOLAP SERVER and Excel Add-in

2010-04-19T17:50:00.001+02:00

What is Palo MOLAP Server:

Palo is cell-related
Palo is an in-memory database
Palo is multi-dimensional
Support Hierarchies in Dimension
Support Attributes for Dimension member (e.g. to allow for localization, provide longer name … ) in addition to Element names. Note that these Attributes are stored in separate tables.

A local server is installed on your computer when you install the Palo client. You can log into the local servers of other computers in a network. In practice, this makes each local server public. You can create users and groups and also assign passwords to manage necessary access rights to your server. But you can't add another server on the same computer.

A Server can host many different Databases which contain one or more Data Cubes. Dimensions can be shared among different Cubes. A server can be installed and run privately (Localhost, by default: http://127.0.0.1:7777) or on a shared server for concurrent multi-users access.

Palo OLAP Server with Excel Add-In is open source designed to help working with large data quantities represented multidimensionally. Palo server can host multiple end-users on a network and thus eliminate the need to distribute all Excel tables with all well known problems associated to that.

Although its main focus is to work on top of Excel, it seemed to be supporting the OpenOffice Calc equivalent as well, although I did not try it.

My observations done on the Palo BI Server 3.0 & Excel Add-in:

Very robust and well designed with a full integration with Excel. So if you are a big Excel fan and don’t like the mess that this lead too in big organization this can help mitigate the often disastrous Excel over-used in today’s large enterprise.

Who has not suffered from Excel hell where merely all managers start their own mini Reporting app initiative within Excel…another illustration of the Peter Principle applied to software application ;-)!!!

So if you are very good with Excel, then you can basically create Cube right-off the bath without merely leaving your comfort zone. Even all Cube management and Administration is done through a dedicated Menu Palo added in Excel.

Ok, at one point you will need to import data from external source, but even then you can easily use Excel built-in function to load a cube from a simple CSV file and leverage Palo provided Excel function to generate your hierarchies levels, your calculated metrics, your ragged dimensions (for example you could easily create a time dimension from a simple date format by deriving the quarter, week, months and year data using normal Excel functions).
…It is quite cool you actually see these import in real-time with cells being refreshed in front of you (ok yes not so good for performance but you have the option to stop the refresh).
Concerning the dimension, I’ve encountered a few limitations concerning its dimensions structure and Elements. Besides the documented naming restrictions concerning Elements (see below), consolidated Elements cannot have the same names (case insensitive, in contradiction with user-doc) as one of its sub-elements (i.e. parent can’t have the same name as children). It seems to be applicable throughout all the dimension hierarchy, no duplicates can exist at any level! (I remember Essbase having similar constraints?)
Also, complex dimension structure having multiple independent hierarchies in parallel tends to degrade performance considerably.

Documented limitation and rules about Characters allowed in element-names:
Characters allowed in element name:
ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_-.
Rules to follow for name of an element:
- The length cannot be 0.
- It cannot begin with a blank.
- It cannot end with a blank.
- Names in Palo are case sensitive!

The most commons functions you would use within Excel are:

--- PALO.DATA()

This is used to represent data of a cube inside Excel (work in both read/write mode ) based on the arguments listed. The arguments are used to identify the host, database and the cube. Obviously we must include the dimension/measures elements in order to address the cube cell precisely.
Other similar function exist to represent data (PALO.DATAC and .DATAV) but these differ only by how the data is fetched: each sheet cell individually (PALO.DATA being the slowest), all sheet cells having PALO.DATAC formulas will be calculated in one-pass, whereas the PALO.DATAV is even faster because it leverages Excel array formula (not always possible).

---PALO.EADD()

This is used to Add Elements within existing Dimension. We have also arguments to provide server/database as well one to provide the Dimension name. Other argument is to give the Type (‘n’ : base element, ‘c’: consolidated) specify the type of elements. The use of type "n” can always be used since it will anyway be converted to ‘c’ later on during import when sub elements are assigned to parent element.
We need obviously to provide the Element name, the Parent name (optional since top parent may not have any) and Weight (number used to describe the consolidation factor). A final important argument is Clear : this decides if the dimension should be cleared for all elements before importing, so when this parameter is set to TRUE all elements are deleted before importing and when it is set to FALSE it just updates old and inserts new elements.
So you should change this parameter in all your PALO.EADD functions from TRUE to FALSE to avoid clearing the dimensions.
Possible parameters values are 0 (or FALSE), 1 (or TRUE) and 2
0: Removes before importing no elements of the dimension.
1: Removes before importing all elements of the existing dimension.
2: Removes before importing all existing elements of the C-dimension, basic
elements are not deleted.

Attention with Problem of removing Dimension element with PALO.EADD:
I had this problem until I realised that PALO imports work in the entire workbook (not only the active worksheet)! So when you are importing using the PALO.SETDATA function in one worksheet any PALO.EADD function in other open workbook will also be run. But since new rows are only imported in the active sheet, only the currently displayed (last) row in inactive sheets are imported again using PALO.EADD.

---PALO.EADD()

And obviously you also have a function to load fact data into the Cube: PALO.SETDATA. This function works similarly as PALO.EADD (must specify server/database/cube), as well as all the coordinates elements (for which dimension member the value is loaded) and obviously the cell value (the value of the measure specify in the coordinate). It has also a Splash (a funny terminology ?!!) argument which indicates whether we can write to a consolidated element (and distribute data evenly among children afterwards..).

Important Design PALO features:

1. Attributes

Attributes are used to provide additional element properties (e.g. other languages, elements description like Product description, code and other attributes, and also special number like Product Unit Prices, etc..).
A very Palo-specific way to manage these Attributes is done through a dedicated special Cube (Attribute-type) stored independently of the Cubes having the fact data. These independencies between Attributes and Elements data is flexible, but force you to switch back and forth between different Cubes during normal Reporting….

2. Hierarchies

Palo support Hierarchy inside a Dimension, and these hierarchies can be Balance or Unbalanced and even Ragged,. You can even have multiple hierarchies within the same dimension (same elements can belong to different consolidations element, i.e. different Parent).

3. Comments

You can add Comment elements (defined as normal elements in Dimension but important to set in String format and not Numeric format typically used for normal Elements).

4. Calculated Elements

To create calculated Elements (typically Calculated measures), we have two options:
1) set up so-called server Enterprise rules in the database where computations are done within the database… these Enterprise rules may involve complex models and large data volumes impacting potentially the speed as a great deal of computing may be required to calculate the entire cube).
2) a simpler approach and sometimes good alternative is the use of Push Rules, i.e. operations, which are performed in Excel and the results of which are then written back into the cube, so no need for Server-processing. This is done through the use of PALO.SETDATA function to upload into the cube any elements cells and then leverage all Excel functions used to calculate the Elements.

5. Consolidation Factors

With hierarchical structures you may factor elements differently among all children of a Parent. Typically you normally have an equal factor (=1), but you may need to weight the elements differently: negative weightings for Measure-type element for which you would derive a consolidated Parent (e.g., when consolidating expenditures and revenue to have a Profit-type consolidated member).

How-tos:
- Creating new cubes

The following sequences are normally necessary to Create a complete new Cube:
I. Create a new database
II. Create dimensions
III. Create a cube with the dimensions
IV. Importing elements into the dimensions
V. Import data cells
Important: Dimensions are created independently of Cube (they are stored at the Database-level), so that we can delete complete Cube without loosing any Dimension data. On the other hand, updating a Dimension structure/data will impact all associated Cubes.

- Importing External Data

Although it is quite simple to populate Dimension and Cube directly doing data entry Excel, it is certainly not practical with large volume of data. So other options exist.
Import options (through special Excel built-in functions) from different source:
- text files
- other Palo cubes
- database through ODBC driver
- from direct excel Worksheet
- some external ETL standard tools (including PALO ETL Server application)
Final option is through Database back-up: although Palo is memory-based, it obviously keeps a copy of each database into dedicated directory. Each Database-directory has three different files types (1-, 2- and 3- ) directly stored within the server file system.
It is convenient to do backup copies of database by simply copying the full directory content (after having stopped the PaloServerService)…that you can also use to transfer full Database (with all their cubes/dimensions) into a different host.

- Working with Snapshot

Also convenient is to work on a normal Excel Worksheet without being connected to the Palo server. This is possible on any worksheet tied to a PaloServer with the options Save as Snapshot… this will change all Palo-specific formulas in cells by normal data values… as any standard Excel sheet.
Similarly but not quite the same, you can also work on an offline mode (so the cells keep their Palo-specific formulas) to be able to see data without any connection to the Palo MOLAP Server (this requires to activate the Use client Cache in the Database settings).

Martin

La région du bassin du Souss-Massa

2010-04-13T19:10:00.044+02:00

Les dernières vacances m’ont permis de voir pour la première fois un pays du Maghreb, le Maroc.

Premières impressions depuis l’avion: c’est beaucoup plus vert que ce que j’aurais cru étant donné la proximité du désert… mais peut-être que la conjonction de l’océan et des chaînes de montagne Atlas fait en sorte que la saison des pluies arrosent tout de même un peu une partie de ce grand pays ?

Après vérification, la région du bassin du Souss-Massa où l’on se trouvait est plutôt caractérisée par un climat subdésertique handicapant sérieusement les cultures céréalières. Par contre, grâce à l’exploitation des nappes phréatiques, la culture arboricole est très présente ce qui explique la présence de grandes étendus de champs de verdure correspondant à exploitation d’arbres agrumes (oranger, clémentine, citronnier,..). On comprend mieux l’omniprésence de tous les fruits marocains dans nos marchés. L’agriculture est très importante pour l’économie du pays et est apparemment très conditionnée aux quantités de pluie reçu…

Paysage pris des collines tout près d'Agadir.

Aussi typique de ce pays sont les épices que l’on retrouvent partout exposés et frais dans les souks … entre la coriandre, le safran (épice les plus chers au monde extrait à partir d’une fleur au pétale mauves), le cumin, le gingembre, la cannelle, le clou de girofle, la noix de muscade, ou encore les mélanges préparés pour les tajines et couscous comme le Ras el hanout (composé traditionnellement de plus d'une vingtaine d'ingrédients!). Le choix est vraiment impressionnant comme en témoigne la photo pris du marché berbère de Taroudant. La cuisine marocaine en est bien imprégnée surtout dans ses couscous et autres tajines apprêtés avec plusieurs épices raffinées.

L'épice la plus mythique est sans aucun doute le safran cultivé et exporté principalement du creux de l’anti-Atlas entre Taroudant et Ouarzazate. Y'a aussi le piment doux, la coriandre en grains et bien-sûr la menthe douce qui demeure un symbole fort grâce au thé à la menthe sucrés tradition importante de la vie sociale dans tout le Maghreb. La menthe du Maroc serait considérée comme une des meilleures du monde !

Tous ces épices se retrouvent en abondance dans le souk et autres marchés. Un exemple de marché authentique est celui de la petite ville de Taroudant très recommandée avec ses 2 souk berbères et arabes. On a un meilleur ratio qualité/prix qu’à Agadir, donc conseil aux touristes il faut se déplacer des villes les plus touristiques pour trouver de bonnes affaires.

Dans le souk on trouve presque de tout... des céréales, des meubles artisanaux jusqu'aux animaux de la ferme!

C'est assez impressionnant sur cette photo du souk, comment les femmes s'empressent à se cacher le visage: dans la culture musulmane les gens refusent souvent de se faire photographier et ce plus particulièrement les femmes ... ce serait pour éviter toute idôlatrie de reproduction d'être vivant! J'ai pris réellement connaissance de ce comportement une fois retourné à la maison ...et oui j'ai été digne du parfait touriste.

Autre particularité du monde végétale du coin est l'Arganier, petit arbre endémique du Maroc. Son amande a des belles qualités et se transforme en huile soit de cuisine ou de produit de soins pour le corps. D'ailleurs on voit de plus en plus cette huile dans nos magasins...

Me voici avec Mathias explorant les champs de l'Arganier dans un bolide de son choix ! Plus tard, il m'avouera qu'il s'était caché les yeux par peur... peur qu'il combattait en se répétant inlassablement que son papa était un bon conducteur !!

Une autre photo de l'arganeraie où les plus fins observateurs apercevront les fameuses chèvres grimpeuses d'Arganier:

From Maroc_2010

une fois le fruit recueilli de cet arbre, on peut en extraire l'huile à partir de son amande:

Autre attrait à voir de Taroudant est la synagogue qui a été convertie en lieu de trésor aux plaisirs pour les yeux depuis le départ de toute la population juive de cet endroit (la communauté juive du Maroc est aujourd'hui seulement présente dans les grandes villes).

Vue de l'intérieur de la synagogue avec toutes ces décorations ostentatoires (par ailleurs tous à vendre, à condition d'y mettre le prix )

Y'a une autre facette du pays qui nous moins plu: cette culture de marchandage et cette pression commerciale très présente dans les rues... surtout dans les villes plus touristiques où tout touriste est assurément racolé à peu près à chaque coin de rue.

Ok il faut savoir composer avec cela et c’est pas trop agressif non plus. Bon il est vrai qu’on va trouvé de beaucoup meilleur prix sur la rue (la voiture louée à 5fois moins cher que celle trouvé via les agences touristiques normales ), mais certains voudront éviter d’avoir à gérer ces négociations et le pourboire une fois le service fourni !

Parlant de marchandage, on ne pourra passer sous silence le fameux marchand de tapis. On a eu une agréable surprise d'ailleurs avec cette rencontre d'un marchand des plus typique qui a fait toute une mise en scène pour expliquer les différences entre Berbère, nomades et bédouins, la signification des dessins et style des tapis et leur origine. Ok, y’a beaucoup de mise en scène pour augmenter les chances de vente, mais les enfants ont été conquis... et les parents ont finalement plié suite aux milliers de tapis déroulés!

Ce fut aussi accompagné de cette pratique de convivialité et d’hospitalité avec thé à la menthe. On a eu plusieurs occasions d'y goûter et même les enfants se sont mis au thé.

On a aussi parcouru le littoral entre Agadir et Essaouira via une route très agréable longeant plus ou moins la rive. Attention aux flics cachés derrière les panneaux de limite 60 souvent précédés d'une distance d'à peine 200m par d'autres panneaux indiquant plutôt 80. Je ne sais pas si c'est une attrape touriste, mais je me suis bien fait avoir.

Ce littoral est parfois bien découpé, et on a pu constater la force impressionnante de l’océan de ce côté de l'Atlantique!!

Le long de la route, on voit aussi les avancés du désert qui semble se jeter littérallement dans l'océan!

Aussi plusieurs dromadaires nous apparurent dans ces parcs tout près de la route... sont-ils à l'état sauvages, je ne crois pas?

On a fini l'excursion de la journée à la ville fortifiée d’Essaouira. Cette ville apparemment très venteuse (confirmée lors de notre séjour) fut fondée au 18e siècle par MohammedIII et construit par des architectes européens. Elle comporta un port marchand de grande importance mais qui est maintenant qu'un port dédié à la pêche.

From Maroc_2010

Voici quelques autres photos prises aux alentours et à l'intérieur de la ville fortifiée :

Les enfants toujours aussi sérieux:

Finalement mes enfants m'en voudraient vraiment si ja passait sous silence le charmeur de serpent (c'est ce qu'ils ont le plus apprécié du voyage... dit avec toute l'honnêteté et la candeur propres aux enfants)

Mathias n'en revient toujours pas de sa rencontre avec le cobra argenté!

Le dernière anecdote du voyage est la chute de Mathias ...qui a atterri sur la tête pendant une escapade avec Laurie. Laurie, soeur attentive et toujours aux petits soins avec son frère benjamin, donna les premiers soins et ensuite se précipita vers sa mère avec cette phrase assassine: ... pendant que toi tu étais en train de te prélasser au bar Mathias se fendait le crâne!

Martin

Pentaho platform introduction

2009-11-28T11:09:00.000+01:00

A few weeks ago, I've done a presentation to introduce the Java/J2EE world to some BI consultant at Sword. During this presentation, I also covered a little bit the Pentaho platform as it is a good illustration of the challenges and issues one face when trying to integrate various open source heterogeneous components.

Here is a slideshow on this second part which introduces the Pentaho platform ....

I'm hoping to investigate the ROLAP engine (Mondrian) soon ...

Martin

Kas, Turkey

2009-11-13T16:42:00.019+01:00

Last October we headed to Turkey for vacations. Having a couple of Turkish friends, I had been warned by its natural beauty and cultural richness, but it did surprise me anyway! It is a bit surprising that we do not hear much coming from Turkey compared to similar Mediterranean places (do you know anyone not having been to either Greece, Italy, Spain or France?).

Landing on Istanbul, we are quickly reminded that it is a Muslim world by seeing the number of Minaret breaking through the skyline of the city. One of the first sound we heard the next morning outside our hotel (at the airport), was the Adhan (call to prayers in Islamic religion) coming out from the speakers installed onto the Minaret. It was our first pleasant change of culture but a bit of stress as we were expected to catch another flight transiting to Antalya but had to wait in the small bus for, what I suspect, the driver to complete his Salah (muslim prayers). Traditionally, the prayers are the obligatory rites of the religion, and are to be performed five times a day. However, we found the religious fever in Turkey to be quite moderate ... and apparently the Salah is now spoken in Arabic language which most Turkish do not even understand!

So that will be it for the religious signs since the religion is mostly left outside the public affair in this officially "laïque" society (secular). For the more curious, you can refer to Atatürk (founder of the Republic of Turkey) revolution and modernization of the society just after the Ottoman empire fall down... BTW, Turkish women received rights to vote before most westerner countries: 1930 (ref. http://www.ipu.org/wmn-e/suffrage.htm).

The place we chose in Turkey was the Mediterranean region just south west of Antalia as this area offers quite a lot of variety in terms of natural and cultural attractions.

Our kids enjoying the view from the house rented from the french tenant. BTW, quite a lot of retired Europeans are staying over here and enjoy the nice climate at low price¨.. rent can average around 300€!

Ok a little bit of history ( Turkey's history is anything but little ;-)) . I remember not being very good at it during my young age , because I've never been motivated in learning stuff simply for what they were without involving some kind of logical thinking and reasoning..... (it could be related to how history was taught as it seemed to me that the goal was to cram and memorize as many events as possible without much hindsight on the relations and intricacies between these).

Having said that, my country history was limited to about 400 years! I can help but have .. a small thought for turkish kids having to learn their country history at school!

Referring to my Guide Bleu de Turquie (Blue guides are probably the oldest and best of tourism guide.. ), the long history of Turkey has been characterized by many different change of reigns and regime overruled which evolved in time almost from the Bronze age up to now!! A very brief summary would need to at least give these over-lapping periods :

pre-historic Bronze (XI to 1200 BC) with the first establishment, the Hittites empire taking control of Babylon, until the fall of Troy and of the empire.
Greek (XII to I) civilization from the conquest of Alexander the Great to his death and the later control by Roman.
Roman empire (300 to 565 AD) with the change of Byzance to Constantinople (new Roma) done by Constantin.
Byzantine (726 to X AD, the Roman empire continuation into the middle age) from the iconoclasm (ban of religious icons) to the schism seperating the Orthodox and the Roman church.
Seljuq dynasty (1055 to 1303) originating from the Turcoman tribal around Bagdad and going into a long westerner migration to the fall following the invasion of Mongol
Ottoman empire (1326 to 1923) emerging from the Turco-Persian Seljuq in northern Antolia, the Ottomans will remain in control of the region for a few centuries and developping a brillant civilization reaching passed the Balkans, the middle-east and north Africa. Unable to modernize they will succomb to the europeans and russians and finally collapse after the end of WW1.
Modern Turkish (1923-) from the recognition of a reduced territory sovereignty (compared to Ottoman period) and the establishment of a laic and parlementary-based republic. It is during this period that a massive exchange of people occured (1.3 million) between Greece and Turkey.

This evolution has left many traces that can still be witnessed in various part of this region. The evolution of the city of Kas is a perfect illustration of this long history....see next.

Kas and vicinity:

The archaeological search done in and around Kas, revealed the city of Antiphellus. Originally this ancient town of Antiphellus, was the harbour of the town of Phellus which stood in front of it. Alexander the Great (Alexander III from Macedon) annexed this region into his kingdom to make it part of Greek empire. During Roman empire Antiphellus gained importance and became an episcopal centre during the Byzantine period. After, it was conquered by the Arab who rename it Andifli (there are turkish companies with this name nowadays), and later ruled by the Tekeogullan Principality (I'm confused here...it is probably the first time it happens: I like to grab info from touristic pamphplet/guide while travelling and later get back home and google these ... I obviously get other info, nice photos, people comments, etc.. ! Now nothing came out with this principality name !!! I guess the Tekeogullan had had very limited historical & geographical impact ... I should ask for the Turkish equivalence and look for it in Turkish).

Obviously, the region passed under the control of the Ottoman, where the peninsula became important as sea trade increases. Now Kas county has become a predominant touristic region admired for its natural & cultural richness settled in between the handsome Taurus mountains and the crystal clear water of the Mediterranean.

The area around Kas is called Teke peninsula located just beneath the Taurus mountains. The larger region between the bays Antalya and Fethiye corresponds to, what was known in the ancient time, as Lycia which regroup different cities considered the first federation in the world with democratic principles (according to wikipedia, the US constitution would have been influenced by the Lycian). It later became part of the roman empire.

Typical views seen around Kas with the sea, mountain and vegetation..

That is Laurie standing on the uphill just above Kas with its bay and peninsula (the island further away into the sea is actually the most estearn's Greek island Mesogeios... basically located a few kilometers from Kas)

Again some nice view of the bay at sunset.

Small creek commonly seen on the coast from Kas to Fethiye.

The familiy just outside the small port of Kas.

Also in Kas, you can find a very impressive sarcophagus (Kral Mezan or King's tomb) carved from a single block with scripting in the Lycian language (dating from the 4^th century BC), see photos:

We finally visited during the week a few other sites: ...Patara (with the longest sandy beach in the rergion..), Kekova (the underwater anciant city), Xanthos (a protected Unesco site), the gorge of Saklikent, Myra and Olympos.

Patara:

Patara integrated Lucian union in the 2^nd century AD as one of its six cities (we have visited 3 of these 6: Patara, Xanthos and Myra). It was also the birthplace of St. Nicholas (born c.260-280 AD) which later lead to the legend of Santa Claus (see Myra below).

The site is quite impressive and very well located. Near the Victory Monument (see photos below of the three arches), excavation 100 meters ahead revealed the world's oldest detailed road map which shows distances between Lycian cities. Here I presume the map has been moved to a museum since after walking around and checking all the place out (not much help available on-site ), no sign of the map could be seen. This turned out a bit of disappointment for a former GIS analyst!... Actually, checking later on the web, there is neither trace nor any image of the map ...just a description! So this is difficult to verify.

The famous Victory monument

The amphitheatre of Patara.

Also exceptionally in Patara is that they also have found what is possibly the world's oldest Lighthouse ...that could have been destroyed following a Tsunami (http://www.lycianturkey.com/lycian_sites/patara-lighthouse.htm).

The ongoing effort of restauration, excavation and clearing the sand have allowed to show the main street with its re-erected columns ...

Some monument with Lycian-language writing

Kekova:

This region characterized by the eponym island is quite incredible as a lot of its ruins are completely under the sea surface level. An earthquake is believed to have sunken partly the region during the 2^nd century. Being quite touristic, we were not too wary on our way to the site and actually hitchhiked one guy who apparently needed a ride to the village (yes that was quite naive). Ok we finally got free parking but had to deal with an overpriced boat trip with a local fisherman... anyway we made a good trip and had our own personal boat for the visit...and that the kids could steer for free!

Various ruins in and above sea level

The famous well preserved sarcophagus

Laurie steering the boat with our friendly fisherman

Xanthos:

This place is somewhat more valued and better serviced with touristic information (there are gov. people working at the site), probably due to its Unesco inscription as a world heritage site. Based on findings on the acropolis, the city dates back to the 8^th century BC but had mostly grew during the Roman empire who practically re-build the entire city after its destruction caused by Brutus!

The place offered a large variety of ruins for the .. and for kids nice collection of turtles spread out here and there ... our kids were obviously more motivated by searching turtles than anything else.

Here are some pics taken there:

Unfortunatly, the initial excavation carried out in mid-1800 by the english have moved many valuable piece of art to the British museum. This is one example where we only see the relief re-created by clay:

and... yes Mathias did find one turtle

Also not too far from Xanthos are the gorge of Saklikent the second largest in Europe with its 20km long (after Samera in Crete, so it is the longest and deepest in Turkey). When most of the snow has felt (after april), we can walk about 4km into the gorge.

Myra:

Another beautiful place belonging to Lycian league is Myra located at the current town called Kale (or Demre, apparently its new name). It features a necropolis with the tombs scultured directly onto this mountain.

Also, Myra is where St. Nicolas was bishop, so we can find his church and tomb which had been partly renovated by the russian when the Tsar Alexander II bought the church in 1863. The place has been obviously turned into a touristic shopping stop...what else to expect from Santa Claus ancestor!! The place is full of small shops of Byzantine art.

Olympos

Not much to say here, as my intention was to stop by on our way back to Antalya. I was mostly curious about the eternal flames to be found there and known as the Chimaera (see mythical link http://en.wikipedia.org/wiki/Chimera_(mythology) ) and which is fueled naturally by methane liberated from the earth cracks. However, again we underestimated the time to reach the place and finally only reached a very strange place filled with funny looking tree houses and mud everywhere (I guess the place was under major restauration? ..and the family was not so inspired by the place! ):

But this gave us the opportinuty to check out the Olympos mountain, which gave its name to the city. In turkish, it is called the Tahtalı Dağı (or Timber Mountain) and alsmots reaches 2400m.

We finsihed the trip to Antalia where we have stayed for a night in the old city district (Kaleici) strictly reserved for pedestrians. It has a lot of Ottoman-type architectural buildinds...

the famous, Kesik Minare (litterally the broken Minaret) going through all sort of transformation from various periods and finally partly destroyed during a fire in 1896!

The view of the Taurus mountain from Antalya

Finally some generic comments about our trip, that could be useful to anyone planning a first-time trip to this place:

+the food: great and healthy. Very much appreciated the Mezeler (starters) which serve very similar purpose as Tapas in Spain and typically very heathly. A must is the Hummus probably the best I ever had!

+Roads : travelling around the region is not so simple as it is quite remote and isolated by mountain and sea ....the mountain litterally reaches the sea making any road construction quite expansive and a more direct roads with tunnel makes it proably much more expansive than a longer roads avoiding the most predominant obstacles. For example, Kas – Kaleucagiz à vol d'oiseau is 20km ... but when we got there by roads to see Kekova, it took us about 1 hour 30min from our rented house!? It seems as if the place is more accessible by boats, as confirmed by te number europenan boats at the port, and some having sailed as far away as Norway (actually one couple we met reached Turkey from the east after a complete trip around the world!!).

+ the accessibility of the ruins & vestige. We can litterally use these sites as a playground...this caused some problem as people can easily bring rocks or other artefacts with them which is obviously illegal. It is just very different from Italy where they are understandably obsessed by security and preservation issues.

+ trees blossom amid the very dryness soil (however it is surely high in nutrient.. and we often witness what was probably Terra Nossa (http://en.wikipedia.org/wiki/Terra_Rossa) with its reddish color! (reminded me of a trip to Prince Edward Island with its famous red beach ...). A lot of Grenada Pomegranate apple, Orange and Lime trees grow about anywhere ... sometimes amazingly out of concrete and dry soil in dense cities!

- On the negative side: the continuous buying pressure when walking around touristic and commercial district and neighborhood (ok that is also very culture-related, and for them that is also a way of communicating but it is not something I'm comfortable with). Also there is often confusion with price, as most of the time price not shown and sometimes after negociating we just realized that they were talking in euros, or yet that there is a sur-tax with some plate since they were apparently over-sized... this infamous touristic price versus local .. . and some nice picture at a friend price : (this photo was worth 2.50 euros !)

Ok that is about it for what I can remember... overall it is a place higly recommeded and we will surely go back to visit other regions in the future ... Cappadocia (http://www.cappadociaturkey.net/) looks like a very good candidate!.

Martin

Summer break

2009-08-13T15:45:00.005+02:00

During our last summer vacation we first headed to France to enjoy its renown culinary art...pretty much guaranteed to eat very well no matter where we end up to!

We spent a week around the region of Drôme, the department located in southeast of France.
Although its geography and climate is different from where we live, we only had to drive about 3.5hours to get there (ok this can vary depending on traffic condition mostly an issue in and around the bottleneck A7 autoroute du Soleil). This region is less touristic than Provence, so it does not have its disadvantages like the overly pricy restaurants and over-crowded places.. This department along with the ones nearby (e.g. Ardèche, Vaucluse) seems to have been better preserved from the wars destruction effect.

Just next to our house we rented, there was the village of Savasse, located north of Montélimar and on the east bank of the Rhone:

The house located on the south side of the hill offered the natural advantage of protecting us against the sometimes brutal northerly Mistral wind. Looking at all the vegetation found at the house or nearby, we would say that the climate is very Mediterranean-like.

Here are some of the plants we found at the house "Laurier-rose", Artichoke, "Figuier", "monnaie-du-pape"..

Also present throughout the region are the small perched and fortified villages typical of the middle-age period (built explicitly as a defensive position). During a ride with my bike, I actually stumbled into one of these villages: Rochemaure (an officially classified site), where we returned at night to visit and eat out :

We encountered quite a few other similar village during a getaway to the Gorges d'Ardèche (the french version of the Grand Canyon), there is one that attracted our interest just by its funny name “Bidon” (http://www.ardeche.guideweb.com/ville/bidon/), literally meaning fake or phony in french!

The Gorges itself are less spectacular than the Grand Canyon, but they are worth the trip. This is apparently shared by a great number of people when we consider the number of tourists seen camping and canoeing at the gorge. I now better understand the familiar complains of Europeans about the tourists invasions in France, and why someone could appreciate a country like Canada for its quietness or emptiness !

One destination that I've reserved only to myself was the Mont Ventoux located in the Vaucluse. Yes, I took the long ride up this mountain often called Géant de Provence. The riding difficulties are mostly caused by temperature (the first 10km are quite hot although protected by shade of the pinède -pine forest), the length of the ascent (1617 m over 21,8km from the small and beautiful village of Bédouin), and the constantly grade offering no break or pause getting even worse near the summit with the last 2-3km having the steepest gradient (over 10%).

Just too bad my mobile phone did not have enough battery to take picture of the beautiful 360degree view up there. In today's Internet era, you are never too far from being exposed to the web. There were people taking pictures to publish them on the Internet and offering the possibility to buy a full resolution image. These guys make a living off vanity of most cyclers...they'll surely get rich!! ...yes as this site is all about self-exposure (;-)), here you go with some snapshots:

as seen by the poor resolution of these advertised pics, I did not actually purchase these pictures..

Speaking of cycling, during my vacation I also cycled to Verbier with a friend that visited over. Verbier has a well-known classical festival.... however this time it conflicted with the 15^th stage of tour de France. It turned out to be the decisive stage where Cantador completely dominated during the 15km ascent (yes this is again a non-human performance, but nothing new for this sport always bringing its share of controversy).

I've never thought that the tour could be such a popular event... there were near 100,000 people all gathered in the Verbier ascent! Probably a good 25% were cyclers, so it was quite exciting after the race to ride among all pelotons scattered on the way down to Martigny. We also rode with the pros, as most of them quickly returned to their team camper located below!

Later on in our vacation, we visited Yvoir which is also very near our place. It is a small medieval village located just on the other side of the Leman, and that celebrated its 700^th year of history in 2006. coming from Switzerland, the easiest is to go by boat leaving Nyon and only lasting about half an hour.

Although the village is very nice-looking with all its flowers and old stones, the shopping industry completely took over. And this has become as expensive or even more than Switzerland.

Ok that's about it, a special thanks to all my family who shared these visits with us and also provided some of the pictures shown here.

Martin

My open source toolset

2009-07-09T08:59:00.003+02:00

I'm always trying to look for open source alternative when it comes to software I'm using. Here's my software application toolsets originating almost exclusively from the open source community (this list is to be maintained).

Editor/Player/Viewer:

Open Office: I used to only exploit Writer as a replacement for Word, now I even see myself doing presentation stuff using Impress (although I rarely used powerpoint at all), Math, and next target will be Calc as an Excel replacement!

jEdit: although originally meant for developer with syntax color highlighting... this is a great and lean txt editor written in Java and a very good notepad replacement.

freeMind: a great mind-mapping tool. Warning, its multi-purposes usage can become addictive!

yEd: I'm often using this Graph Editor at work to create some hierarchical-based or tree-based graphing. It has a few glitches but well compensated by its simplicity of use.

Gimp: I don't have the patience and talent to do any photo/picture edition, however this one came in handy a few times...

VLC: simply the best media player around. With its independence of systems codecs, it supports merely all video & audio encoding format. It now finally reached release 1.0, so to be downloaded before it gets acquired!

Misc tools:

KeePass: a password manager, with this you will only need to remember a unique password: the Master Password!

Avg: a free anti-virus...although this one does not come from open source.

WinSCP: a ftp/sftp client.

jZip: alternative to winzip or winrar.

Internet:

Firefox and all its add-ins: been a while since this needed introduction!

Rssowl: supports RSS, RDF and Atom newsfeed

Twiki: a nice and easy to install stand-alone Wiki.

Plucker: a great tool for my palm to read Web content and e-book offline on the train (not open source)!

Miro: I hooked up my old PC into my TV with HDMI at home. My kids can now enjoy exciting, interesting and fun content from Internet (yes obviously this needs some form of supervision)

Thunderbird: another good Mozilla application !

Development and database:

Eclipse: don't use it more directly since I don't code much anymore, however its pervasiveness is such that you end-up using it indirectly elsewhere (e.g. Talend, Pentaho, Websphere studio, BIRT, SAS...)

StarUML: Without going overboard with reverse-engineering and code generation, I like this one to create some UML-type diagram ... can even be used to get quick ERD!

SQuirreL SQL: a good alternative to the too famous TOAD... which is limited only to Oracle. Also has a nice Tabbing capacity for your query result (a parallel with what distinguished for a long time Firefox from IE). You just need to avoid trying to cache all DB schema objects from a big Oracle datawarehouse!
Apache Derby or any of its reincarnation... very useful when you need a small footprint DB for some SQL prototyping.

BI and Data mining tools:

Pentaho: these guys have looked around and bundle the best open source projects to offer a full BI platform. Their original concept is that the platform is more geared toward Process and Workflow concept than other traditional BI solution.

Weka: the Java-based data mining originating from the University of Waikato and now integrated into the Pentaho platform.

Yale (now known as RapidMiner): A more commercial approach has now been adopted as suggested by its new Rapid-I branding, however a community offer is still offered.

Talend: The ETL/ELT open source offer now associated with JasperSoft.

Martin

Star Schema Limitation

2009-06-12T15:29:00.008+02:00

The star schema is probably the most over-used data modeling paradigm used today in data warehousing. It is very attractive due to its simplicity (especially important for business) and its pervasiveness, and as such people has the tendency to overuse it at every layer of the BI data platform.

There is one use case where its use quickly become problematic and causes more harm than benefice. This is when we want to track-down all slowly changes occurring around the Customer of the enterprise (this term is used loosely here, as it can mean different thing such as subscriber, client, account, etc..). The slowly changes referred here must not be confused with simple attributes (e.g. customer age, name, ...) usually handled within the dimension itself using specific slowly changing dimension rules (SCD type 1,2,3,.., 6 and still counting). Here, we refer to important Indicator information usually spread out in separate Dimensions or other data structure.

More specifically, these slowly changes can be comprised of very general Indicator characterizing the Customer either directly obtained from operational system (such as account type, price plan, subscriber status, address, dealer and channel, etc) or features derived/calculated from data within the integration layer of the BI platform (such as customer value, churn likelihood, behavior, customer segmentation, etc).

The star schema can be used for this goal: we simply design the model around all Dimension representing the features being tracked down, along with the dimension Customer and two links to Date/time Dimensions in order to define the validity period of the fact row (the start and end date). In other words, each fact row represents a new state for the Customer in time and is created whenever one or more slowly changes (characterized by dimension) have occurred.

This approach offers a more or less standardized approach in BI (star schema model) allowing a very simple Query logic with multiple dimensions (indicators) to be reported simultaneously. However, this advantage is trade-off by a number of important negative issues :

Flexibility: Physical data model is not amenable to easily permit changes in terms of new dimension/indicator as this requires a new version of the model every time (people working in large organization will surely appreciate the implication cost of changing data model inside database in Production!!).
Extensibility: There is a limit on the number of dimension/indicator that can be added into the structure as this requires a significant cost in terms of disk storage for every new dimension/indicator. The star schema is simply not designed to track down a variable number of indicators dynamically.
Volumetry: Data structure leads to a great deal of disk storage waste as dimension/indicator are duplicated in all rows even when their values remain unchanged (ex. Dealer or Geography indicator are very unlikely to change throughout the Customer history but their values will be repeated for each and every new Customer fact-row state).
Clarity: This particular model mixes two concepts into a single fact table (1. event-based fact and 2. snapshot-based fact representing the state at fixed point in time) blurring its role and increasing query complexity for all EOP, BOP type of reports.
Loading: The ETL feed is likely to be more complex with this type of structure, which results that only minor changes are done over time and typically no new information element is rarely added.

There are better solution that we can model in order to provide such typical historized view of the Customer evolution in time. One such solution is presented next with a simple logical model:

This solution is sometimes dubbed 1-dimensional model (because everything evolves around the generic Indicator table), or I've seen alternative terminology like this one ...
The more relational inclined will note that this model is flawed as the effective and expiration date are duplicated info (the expiration date is directly linked to effective date of the subsequent row for a specific customer.. ) and as such should be constraint to ensure integrity by the DBMS engine. This is a good point and can simply be avoided by removing one of the field (see one good example), but to me it is a physical implementation detail: should the data model enforce this rule or could we trust the SQL feeding codes to handle it (obviously more risky). To keep things simple, I've decided to only show the logical data model, leaving this implementation choice as a side-note.

The table 'Customer Indicator Association' is central to this data structure and is similar to the Fact table whose role is to capture all association customer-indicator in time. Within this context, the Indicator table corresponds to dimension records found in various dimension table, whereas the Indicator Type corresponds to the actual Dimension.

The pre-requisite of this data model is that all Customer must be only associated to only one Indicator value by Indicator Type at any point in time. Some other important aspects of this data structure is summarized here:

Each time state of Customer would change for one Indicator value, we would generate a new fact row with the corresponding Effective start date (and Expiration from date set to some infinite key date) and the previous state would be expired (update done on the expired to).
Only necessary changes are tracked down, .... meaning a Customer whose state never changed from Activation (for a specific Indicator) would only have a single fact row for this particular Indicator.
Date (start and end) correspond to real date when the change of indicator occurred: usually based on operational system date (for Indicator obtained from these systems) or business rule for Indicator derived from integration layer (e.g. Customer Value re-calculation period).
No elapse time period is allowed between the previous Expired date and the current Effective Date. This ensures data integrity state to state, so that

Key implementation decisions that need to be chosen with this data structure :

How to align all Indicators once a new customer is activated. Two options are possible: concurrently aligned all Indicators with the first Start date even if some Indicator still has no value, or yet allow for Indicator in time.
How to close Indicators once a Customer cancels all activity (no more customer). Again, we can align all Indicators with a transition state such as 'no longer customer', or leave with the latest Indicator value before cancellation occurred.
How to treat technical changes of Indicator (i.e. it is not rare in operational system that Customer change status within a short period of time to return to original status... technical changes without real consequence). A grace period can be defined to account and ignore for technical changes.

The indicator association table will easily supports the full historization of all slowly changing Customer related indicators (existing ones, new ones, modified one and future ones) without any model change. This has the potential to become the central and only place in your BI data platform where to hold atomic indicators for Customer.Publish Post

Obviously, this solution will present the flip-side disadvantage of the more traditional Star schema solution. The most important one aspect concerns the Reporting extraction logic which is more complex and less optimal. In essence, this structure is not aimed toward end-user Reporting since more complex and far less performant Query logic must be employed for data extraction (this is somewhat mitigated with the use of latest SQL analytical functions). This is especially true since most query involve multiple predicate conditions imposed on a number of dimension. So usually, a downstream data structure would exist more geared toward end-user query (a sort a flatten version where all Indicators value would be transposed back to columns...).

Martin

Half-blog: the container is blog-style but not its content

2009-04-27T20:53:00.004+02:00

I haven't had much time nor energy to post anything here... usually this happens when I'm overloaded with technical documentation at work. The past months have been very busy writing business functional specification, modeling and design and other architecture document. Not that I despise this or anything, but it seems that whenever I exceed a number of written words threshold (for a given time period) then writing becomes less enjoyable. I guess this is applicable for a lot of things we do in life!

I'm now taking the opportunity to change the title of this blog to something that reflects more what it's all about: Everything but a blog.

Ok don't have to point out the contradiction... however I could have also defined it more judiciously: Half-blog : the container is blog-style but not its content. This is particularly true with the everything-must-be-instantaneous trend reaching new level with Twitter fad! That is simply not my style, as I'd much rather take the time to reflect on things then simply comment them...which is a probably not a bad thing considering the amount of time I spent doing analysis and architecture type of work.

So for the upcoming time, I will continue using this space more as an archival document repository, a reference tool for technology, a handy place to keep my notes and .... probably more and more as a tool for family experience sharing.

Martin

Ski à Tignes

2009-02-22T10:57:00.007+01:00

Parmi les avantages d'avoir plusieurs semaines de vacances en Europe, est l'organisation du calendrier scolaire nous incitant (lire contraignant) à tout arrêter lors des congés scolaires!! Mais ceci s'avère des plus appréciés, par exemple pendant une semaine durant le mois de février, toutes les écoles ferment pour les vacances d'hiver et y'a pas trop de service de garde, donc ça donne un congé bien mérité pour toute la famille.

Nous avons voulu en profiter pour initier les enfants au ski et sachant que tout le canton de Vaud prenait congé, nous avons eu l'idée d'aller en France pour éviter la cohue sur les pistes... en ignorant bien-sûr que toute la région parisienne se trouvait aussi en vacances!!

Bref, malgré ce tourisme de masse ce fût quand même de chouette vacances passées dans la station de Tignes située dans le parc national de la Vanoise en Savoie. La station est d'ailleurs voisine de Val d'Isère où se tenait la coupe du monde de ski 2009. Nous avons assisté au cortège du trafic de retour lors de notre monté vers la station..

Voici quelques chiffres de Tignes + Val d'Isère:

-10,000 ha de domaine skiable avec 300km de piste
-altitude maximum: 3450m, altitude à la station : 2100m (Tignes), 1550m (Val d'Isère)
-pistes: plus de 150, remontées mécaniques: 89

Quelques dates:
-visite des premiers alpinistes: 1860
-les premiers skieurs: années 1920
-premier remonte pente: 1936
-construction du barrage de Tignes mettant un terme aux projets de village: 1946
-mise à l'eau du barrage et dispersion des villageois: 1952
-première saison officielle: 1957

Les +:

- les "Naturides" qui sont des pistes noires balisées, sécurisées mais non damnées qu'on retrouve un peu partout!
- la bouffe témoignant du savoir faire culinaire français (sauf les mets plus typique savoyard qui sont un peu lourd à mon goût)
- l'école du ski français... les enfants ont bien progressé tout en profitant de leur semaine.

Les -:

- pistes beaucoup trop occupées
- la quantité de neige moyenne considérant l'hiver exceptionnel que connaît l'Europe (nous avons pas vu la même quantité de poudreuse qu'on trouve en ce moment en Suisse)
- soleil pas mal puissant car la station se trouve à plus de 2100m! (apparemment la force des rayons UV s'accroît de 4% par tranche de 300m d'altitude) Ok, j'avoue que la plupart des gens recherchent justement ce soleil d'été en plein hiver.

Ces vacances nous ont aussi permis de (re)confirmer les caractères assez distincts de nos deux enfants:

Laurie: plus posée, méthodique et prudente ... elle progresse tout au long de son séjour pour finir dans les pistes rouges avec ses propres bâtons! Plus sociable, elle en profite aussi pour ce faire une copine ... Je crois qu'elle peut nous accompagné pratiquement partout et ceci après une semaine de cours.

Mathias: voulant excellé dès le départ avec son orgueil et sa fierté bien à lui, il n'aime pas trop l'apprentissage et les leçons du débutant ...par contre une fois ce cap franchi il donne des conseils à sa soeurs et insiste grandement pour me montrer ses prouesses en descente downhill sur toute une pente et ce, en se mettant en boule (ce qui donna une syncope à sa mère).

Voici quelques clichés! (malheureusement pris à partir du téléphone portable)

From Tignes_2009

Laurie avec sa monitrice lors de la remise de médaille.

From Tignes_2009

Kim profitant de son nouveau genou!

From Tignes_2009

La vue du village prise à partir de notre chalet.

From Tignes_2009

Laurie avec ses copains "flocons"

From Tignes_2009

Mathias montrant aussi fièrement sa médaille.

From Tignes_2009

Mathias se reposant des nombreuses descentes.

From Tignes_2009

Ma tentative de ski hors-piste avec mes peaux de phoque et mes fixations adaptées pour mixer ski de rando + descente (je vais essayer de faire un blog sur cette passion qui m'a redonné goût au ski).

From Tignes_2009

Les enfants avant leur dernière journée de cours.

From Tignes_2009

Martin

vacations

2008-09-03T18:13:00.004+02:00

The last summer vacation was a bit short, at least relative to the European standard, taking two non-consecutive weeks. However, it is impressive how much variety you have access to in a very short travel time here. It all comes down to distance which seems to be compressed in Europe, and with Switzerland right in the middle of it (at least geographically speaking, since the political integration has not progressed much!), we can enjoy a lot of small weekend getaways.

I can't help but compare this with Canada... when I was about 18, I took a train ride of about 42 hours which took me from Montreal to Regina (roughly 2400 km). During all this time one could only witness two types of landscape (forest and prairie) and a single culture, english canada!

Being totally unfair and bias, this is what I get when looking at all the places I've visited in Europe during my first stay in 2000-2002:

- Algarve in Portugal
- France (Paris, Lyon, the Alpes region, Provence and Côte d'Azur)
- Spain (Madrid, Barcelona, Alicante, Seville, Valence)
- Baviere region in Germany (Munich)
- Italy (Aoste, Piedmont, Lombardie regions)
- England (London, Wales)
- and of course a lot of regions in Switzerland (Romandie, Suisse Alémanique, Tessin)

All these trips offering such a diversity of culture within less than 1000km (except probably south of Spain and Portugal which are a bit further)!

However, now with the kids we are less ambitious ... actually we can reach places within less than 2 hours drive away from home... the exact time before our kids stuck in the car start their rampage! So that is what we did this summer.

Here are some pictures of these trips.

Enjoying the small boat in the lake Du Bourget at Aix-les-Bains.

Here is my mom and sister with the kids while visiting Gstaad.

Eating out at Thoune with my daughter!

From Juillet - Aout 2008 - partie 2

I did also quite a bit of biking .. with my longest ride from Lausanne to Bern or about 100km. My longest ascenscion, I did near Thoune (actually between Thoune and Interlaken) where I climb about 1400m.

Next pics were taken about half way toward Beetenberg and Nierderhorn.

You feel lucky to go through this tunnel knowing the quality of road in Switzerland, it was so dark the only thing visible is the small spot of light at the end of tunnel.

Actually the highest altitude road usually winds upward (lacet in french) quite smoothly... they are not nearly as challenging as city roads I regularly ride in and around Lausanne (I guess the logical explanation to this is related to accessibility during winter). For example, a small road I often take has average slope at 20% that rides up to reach "Les trois Chasseurs" from Pully-la_Rosiaz.

From From-mobile-phone

BTW, the pictures taken while riding are from my 2 Megapixel Nokia6300, not too bad for a mobile phone!

Martin

Gruyères

2008-08-03T15:37:00.005+02:00

Ce blog a pris une tournure plus personnelle depuis notre arrivée en Suisse, ceci en grande partie pour tenir au courant ma famille de nos escapades. J'ai donc décidé que ma mère (Normande Lapointe, la mamie assez bien branchée et à la page en ce qui concerne l'internet) qui profite normalement bien de ce blog, pourrait maintentant y contribuer en écrivant un petit blog-photo le temps d'une balade! Je lui cède donc le clavier. (Martin)

Quel plaisir pour moi de relater brièvement notre excursion au château de Gruyère en cette magnifique journée du 2 août 2008, lendemain de la fête nationale suisse ! Un ciel bleu agrémenté de quelques nuages, une douce brise, de magnifiques montagnes en toile de fond, voilà comment s'amorce notre visite. Des conditions gagnantes, quoi !

Voici une vue du château de Gruyères prise par ma petite-fille Laurie qui, à six ans, manifeste déjà des talents de photographe.

Le château de Gruyères est un des plus prestigieux de Suisse, il est perché sur une colline dans un paysage verdoyant au coeur des Préalpes fribourgeoises (à 830 m d'altitude). Il domine la cité médiévale de Gruyères (nom qui dérive de la grue héraldique). L'histoire dit que 19 comtes y ont habité du XIe au XVIe siècle. Michel, le dernier comte de Gruyères, connut de grandes difficultés financières et fit banqueroute en 1554. Le territoire fut alors partagé entre les créanciers : les villes de Berne et Fribourg.

En compagnie de ma belle-fille Kim et de mes deux petits-enfants, Laurie et Mathias, nous montons vers le château.

On est presque arrivés... Voici le passage et les fortifications du Belluard, accès principal de la cité de Gruyères
Nous franchissons les portes.

En famille, devant une partie du Calvaire qu'on aperçoit à moitié derrière.

Au coeur de la petite place en bas du château... devant Le Chalet, maison qui a été construite ailleurs et récupérée ici.

Mathias devant une ouverture qui nous donne un aperçu des montagnes environnantes.

Quel visage radieux ! Le clocher de l'église de Gruyères nous apparaît derrière Mathias.

Vue de la montagne sous un autre angle.

Laurie fait une petite moue, le paysage est toujours époustoufflant.

Laurie, heureuse dans les bras de son père.

A l'intérieur du château, elle se prend pour un chevalier.

Nous voici dans la salle des baillis. Le château fut le siège des baillis fribourgeois de 1555 à 1798, après la faillite du dernier comte de Gruyères. Ceux-ci y ont laissé leur marque en aménageant des intérieurs baroques.

Mathias et Laurie sur le poêle en catelles (tuiles en céramique dans le langage populaire suisse-romand).

Laurie prend des airs de comtesse !

Une galerie avec vue... Le cours d'eau derrière moi est la rivière Sarine, bassin du lac de la Gruyère artificiellement créé suite à la construction du barrage de Rossens, donnant naissance au plus jeune lac de toute la Suisse!

Nous nous promenons dans le jardin français aménagé derrière le château.

La belle Kim, en plein centre ville.

Notre visite tire à sa fin. Nous flânons encore un peu dans la rue bordée de magasins de souvenirs et de restaurants. On est ravis de notre visite mais on a le sentiment d'avoir à peine effleuré des siècles d'histoire.
Peut-être reviendrons-nous un jour...

Normande

Thermalp Ovronnaz

2008-07-07T21:07:00.005+02:00

Ce weekend j'avais réservé une journée surprise pour ma petite vieille! Et oui Kim a eu 38 ans la semaine passée et avec tout ce stress accumulé avec les années, rien de tel qu'une fin de semaine passée aux bains thermales accompagnée de soins divers!

Le site d'Ovronnaz se trouve à une heure de la maison en prenant l'autoroute qui longe le Rhône dans le Valais.

Petite parenthèse écolo:

Dans le Valais, on croise les plus puissantes éoliennes de la Suisse, fait assez rare en Suisse car le pays commence à peine à développer ce secteur. Il faut dire que les grands vents se trouvent plus en altitude et que j'imagine assez bien l'opposition des gens à gâcher ces magnifiques paysages alpins avec de gros ventilateur! La zone située à Collonges est particulièrement bien placée car il y a un phénomène de vent constant dû à l'encaissement de la vallée. La vallée du Valais située entre les plus hautes montagnes de la Suisse a d'ailleurs des caractéristiques météorologiques très différentes du plateau de la Suisse, comme le foehn et son très faible taux de pluviométrie (même quantité de pluie que Barcelone). D'ailleurs, j'ai bien remarqué ce minuscule plan d'eau qui semble idéal pour la planche à voile.... je vais peut-être considéré une reprise de ce sport qui n'est pas évident à pratiquer autour de l'arc lémanique. Le Léman est plutôt une cuvette avec peu de vents thermiques contrairement à d'autres lacs dans les alpes comme le lac de Côme.

Les enfants nous ont forcément rejoint plus tard dans la journée avec comme espoir de se baigner dans le "gros bain en forme de piscine".

Quant à moi, je me suis aussi réservé un peu de "détente" en grimpant les petites routes qui montent en laçais vers les haut sommets autour du village d'Ovronnaz! La monté fût assez pénible mais l'arrivée en valait le détour.

Je me suis finalement arrêté à la buvette Loutze à plus de 1700 mètres d'altitude. Je me sens maintenant prêt pour un vrai col!!

Martin

Le barrage d'Emosson

2008-06-22T21:28:00.002+02:00

Aujourd'hui nous nous sommes payés une petite visite en hauteur pour se rafraîchir par ce temps caniculaire (plus de 33C en plaine...). Nous avons opté pour Le Châtelard situé dans le Valais à deux pas de la vallée française de Chamonix.

A partir de ce petit village, nous avons pris le plus abrupte funiculaire à deux cabines de monde avec une pente maximale de 87% (ce qui donne à peu près 41 degrée! ... présumant que mes quelques souvenirs de la trigonométrique soient justes ).

Laurie au moment du départ, en plein confiance?

Nous parcourons par la suite un petit trajet à flanc de côteau en train à la vapeur fonctionnant au charbon.

Ce parcours nous offre une vue imprenable sur le Mont Blanc et ses sommets avoisinnants (on voit très bien les pics des Aiguilles du midi).

Nous arrivons à un 2e funiculaire qui lui nous mène au barrage du lac d'Emosson (à poids voûte).
Ce barrage avec sa hauteur de près de 200 mètre impressionne et donne le vertige! Il est le 2e plus gros de la Suisse en terme de volume d'eau retenu avec plus de 227 million de mètres cubes d'eau (la Suisse compte pas moins de 160 barrages ... ce qui me semble pas mal compte tenu de sa taille).

Voici quelques autres photos...

Et non nous n'avons pas adopté 2 nouveaux enfants, mais plutôt fait le voyage avec un couple d'amis avec enfants!

Coup de chance en me rendant dans une zone plus isolée pour prendre quelques photos, j'aperçois un chamoix qui ne semble pas trop appeuré de notre présence! Mon fils l'est un peu plus et nous le laisserons tranquille dans son jolie coin de pays: "papa, on ne devrait pas trop l'approcher... il a quand même des cornes!!"

Petite déception, nous n'avons pas pu atteindre les empreintes de dinosaures malgré toute la bonne volonté et motivation de Mathias.... la fatigue a tout simplement eu le dessus!!! À une prochaine fois.

Les enfants se sont montrés bien brâves et tenaces avec ces longs parcours, ces dénivellés et ascensions ... rien de mieux que d'avoir des amis de leurs âges pour stimuler leur orgeuil!!

Martin

Claire Ouellet

2008-06-08T15:24:00.002+02:00

This is a shameless blog-plug!

My sister Claire Ouellet, is a professional pianist. Although, I was sort of annoyed by it while I was younger (these continuous and endless practices caused me a few nervous breakdown!), I must admit that all these long years of discipline did pay off. She is now a teacher at the Conservatoire of Montreal, and her last record out is a hot seller (ok, ok... selling statistics for classical albums are certainly not on the same scale as pop music, so I'm not expecting her to move to Montreux soon ;-).

Here's her most recent album: Orford Six Pianos with details here.

Previous albums can also be found at

- Stravinski & Bartok: two pianos
- Invitation à la danse
- Nostalgia

Martin

Technology adoption rate in corporate world

2008-05-30T15:50:00.003+02:00

Corporations are usually slow in adopting new technology internally... there is usually strong resistance usually among employees to change old habit (the famous comfort zone) or upgrade application implementation barely meeting requirement and specification. A good illustration of this is the too often seen requirement with spec similar as "files X containing all needed info will be sent to Mr. Y mailbox at some specified time schedule".

The email messaging paradigm is still the most natural and the least controversial way for disseminating information to specific recipients among the organization. However, many drawbacks are related to this approach:

The publisher application must manage all recipient emails (additions, removals, modifications, etc)

The publisher application is also responsible in managing temporary substitution among recipients email (e.g. with vacations, etc..)

Sending a large number of messages to various different recipients increase complexity of the publishing application whose scope should focus on the generation of the information

Email messaging is less practical for larger data volume content

Lack of security: the data sent over mailing application is rarely protected

Recipient mailbox is not resilient to failure (e.g. limit space exceeded) resulting in the end-user missing valuable new content delivery

Email used as the communication channel of type Application-to-EndUser, is very frequent everywhere, although a much better and flexible solution exist. RSS feed: RSS stands for Really Simple Syndication. Without going into too much details RSS feed (aka RSS channels) is a mechanism to allow any end-user to receive asynchronously some form of message (e.g. headlines, updates) or even data content published periodically by a provider. The end-user utilizes a RSS Reader (aka RSS aggregator) to stay up to date with new content published without having to know and worry about the publishing schedule. Simple RSS aggregators are directly available inside common web browser and more advanced ones are also available for free.

Some commonly uses of RSS feed include:

Notification of new data content within a web site (either just a summary or the full content), typically used by weblogs and news websites.
Notification of the arrival of new products in a store.
Listing and notifying you of newsletter issues, including email newsletters.
Notification of additions of new items to a database, or new members to a group.
Weather and other alerts of changing conditions.

Advantages of RSS:
-sophisticated security model can be implemented with RSS by leveraging the web security rules and mechanisms available in the company intranet/portal ;
-the management of all different messages content to all different recipients are off loaded from the publisher responsibility into the individual end-user recipients (except for security which can be managed at the web page level).... allowing the publisher to focus on the data content instead of delivery concern;
- Publisher application has a more standard and accepted way to publish and disseminate ad-hoc type message content (report, data, news, etc,,) among the end-user community within the organization.

Also interesting is the possibility that RSS can offer to corporate intranet and portal: RSS in the intranet. Some people even suggest RSS as a replacement of the email, as a way to get it over with the spamming issue or other shortcomings such as not being able to embed video within email directly.

Martin

Sel des Alpes

2008-05-18T21:04:00.003+02:00

Et oui, le sous-sol de la Suisse ne contient pas trop de pétrole mais on y trouve du sel! En fait, dans une autre ère géologique (200 million d'années.. ces échelles de temps m'apparaîtront toujours inconcevables??), la mer se trouvant sur ce territoire s'évapora et le sel laissé par cette évaporation se trouva pris sous le plissement de la formation alpine.

En jugeant de l'effort qu'ils ont mis pour l'extraire, on comprend bien que le sel était le pétrole de l'époque (début de l'extraction à partir des galeries souterraines fut réalisée en 1684). Les galeries étaient tout simplement creusées au marteau et à la cisette... avançant quelques cm par jour, un seul puit pouvant prendre plus de 26 ans à construire!

Aujourd'hui, le sel de la mine de Bex est extrait en injectant de l'eau sous pression dans les trous forés. Ce forage se fait avec des couronnes (têtes des foreuse couvertes de diamant industriel) qui carottent le roc à plus de 10m par jour! La saumure arrivant par le retour d'eau est saturée à 30% de sel (10x l'eau de mer) et est par la suite canalisée vers une usine afin d'y extraire le sel. La mine peut se visiter car, quoiqu'encore fonctionnelle, tout est automatisé et aucun mineur n'est normalement présent (ils recoivent des sms en cas de pannes ou anomalies!).

La particularité de cette mine qui se trouve par endroit à plus de 450m sous-terre (en fait sous la montagne) est qu'il fait une température stable de 17c. Ceci est dû au gypse déshydraté (roche anhydrite) qui absorbe une grande quantité de l'humidité ambiante... dégageant ainsi de la chaleur. Donc, température fixe, humidité fixe... endroit idéal pour la conservation du vin. Et oui ceci est bien une cave à vin en plein mine sous terre:

On y voit aussi d'autres curiosités comme des fistules (sorte de stalactites creuses... l'eau y coulent de l'intérieure) composées de sel formé par la condensation de l'humidité ambiante... elles ont une croissance rapide du mètre par an (lignes fines sur le fond de la photo):

Mathias ici absorbé par toutes ces explications...

Laurie et Mathias avant le retour au monde extérieur grâce au train franchissant plus de 1.5 kim de couloir étroit:

Du sel, on en trouve un peu partout...en voici formé le long des parois d'une galerie.

En conclusion, ce fut une journée très agrèable et enrichissante passée en famille! Maintenant on sait l'origine du sel des Alpes (tout le sel acheté dans le canton de Vaud provient de cette mine), et sa grande pureté conservée par des millions d'années passé au coeur du roc alpin.

Martin

EAI

2008-05-07T13:23:00.016+02:00

Most BI architecture found within enterprise today does not exploit the integration capacity offered by more recent technology. These are more typically leveraged by other application systems in the operational area inside the enterprise IT infrastructure. But before giving some thoughts on the reason why (more on this in a coming blog), first I’d like to give more detail on the concept of application integration. Traditionally, this subject has been referred to as the TLA EAI.

Instead of relying on a more prose-like form, I thought to make use of the mind map diagram for describing the concept. I really like Mind map as a way to illustrate, navigate, and represent knowledge on specific subject.

So here is my mind map of EIA (using FreeMind ) .

1. static view:

2. Dynamic View at Mind42

T~~he mind map is hosted by mindshare.com which is still in beta, so may not be available at all times!~~

Here is the same dynamic view embedded as iframe :

From my personal observation when it comes to EAI, BI applications typically limit its integration to:

For the upstream spectrum (i.e. fetching data in) : Bacth-file or database loading in otherwords ETL processing.
For the downsteam spectrum (i.e. disseminating the info out) : things usually are limited to sending flat files to external system at scheduled time. Here I only refer to integration between different system applications without including desktop user-type application, where things are different.

Martin

Visite au château

2008-05-01T19:10:00.004+02:00

Aujourd'hui nous sommes allés visiter le Château de Chillon! Nous étions déjà allés avant que les enfants soient nés, mais avec les enfants ça comportait un peu plus d'émerveillement et de magie! Ce château est vraiment typique avec ses tourelles, ses toits en pente, ses douves naturels (le château se situe en fait sur une île rocheuse), son donjon.... c'est son emplacement qui le rend unique. Il est protégé d'un côté par les escarpements des alpes vaudoises et de l'autre par le lac Léman. Les plus anciennes mention du château date de 1150 et sa période la plus marquante est liée aux Ducs de Savoie qui contrôlaient ce territoire stratégique chevauchant la fontière franco-italienne.

Voici quelques photos de cette après-midi passé en famille...

Laurie et Mathias !

Kim et les enfants se racontant des peurs..

une vue sur l'autoroute traversant longeant le lac.

une vue sur le lac

moi et les enfants avec le château en arrière plan.

Martin

Decorating RPC server call

2008-04-27T14:07:00.003+02:00

In RPC service layer, I often need special processing to occur before/after some of the call methods. For example, I may want to gather various access statistics related to some of the methods, compress the response when it exceeds some maximum threshold, or yet simply providing a cache response to speed things up!

So in essence, I need to intercept specific RPC calls to provide extra service to these calls and optionally even taking responsibility of handling the call without reaching the real service layer (e.g. as in the case of caching). The logical answer to this need would be to use a AOP solution (Aspect-Oriented Programming), however my service layer are already intercepted by some aspects to manage orthogonal concern such as transactional processing and security handling. And defining additional aspects around my service methods seem counter-intuitive especially for aspects that are relatively sparse (i.e. only apply to a few methods/service).

My solution to this problem is simply leveraging the Decorator design pattern. Conceptually, this pattern provides a way to attach additional responsibilities to object dynamically, i.e. here I could add auditing ability or caching ability to the service object. The next UML figure presents the class diagram involved in my solution. It is a slightly modified version of the classic pattern because both my concrete service object (to be wrapped by decorator) and my decorators inherit the same super class. This is more a convenience to centralize the various steps originating from GWT RPC handling mechanism with the template method pattern. The drawback is that all decorators have an annoying dependence on the GWT library. This is fine for my scenario, but I may have to refactor this in order to improving testability.

I’m providing the skeleton source code for the main classes involved, feel free to adapt them to your needs.

Martin

BI and technology adoption

2008-04-24T09:21:00.009+02:00

To follow-up on a question raised by someone at my consulting firm... why is BI group usually late adopter (compare to other group inside the enterprise) in terms of technology?

In fact, I've been asked this question by a manager, and at first, I did not exactly what to respond, actually my first reaction was to think that the big organizations were reluctant and hesitant to add more technology to a platform already loaded (or bloated?) with tools, difficult enough to manage and for which ROI is certainly not obvious to calculate.

After putting additional thoughts, I now feel that reasons are found inside and not outside BI. Traditionally the BI designer and architect come from a database background and culture, and these have a tendency to be more skeptical about new technology (they may well be right on that one!), whereas the general software development crowd usually come from computer science background very much inclined in looking into latest innovations ...

In general I have observed that it is among people running more operational systems, who are already used in adopting technologies, that will turn down proposal to connect to the BI platform using less traditional way. It is rather BI group itself which keeps proposing to send flat files to feed other external systems or give read-only access to their database (because these are familiar communication channel coming from the familiar ETL process). And when you propose to expose these information as web services instead: you get a funny look!

The outcome of this fact is that we often end up with BI platform that is very isolated (silo-type application) from the rest of the corporate integrated architecture. This penalizes the potential usefulness of BI... which could be leveraged better and exploit in a more transparent and integrated way into the existing corporate Business Process.

Hence there is no doubt that BI would gain by adopting technologies like web services, and architecture such as SOA, ESB? And there are a lot of initiatives along these lines... this is inevitable.

Martin

Are we settled now?

2008-04-12T11:08:00.018+02:00

All the places I once called home!

I was counting the number of places I lived in recently and this number seemed completely unreasonable... so I decided to blog about it mostly for my wife so she can see that it may well be the time to settle. Although things were quite stable at first (I actually lived for the first 18 years in a single place), it all changed once I moved out of my mom's place!

In case you are wondering, no I'm not 65 but only 36 years old...ok almost 37 now.

Here's the map:

View Larger Map

Here's the list:

Chicoutimi: My first Home!
Well almost the first, but I did move there at the age of 2 months-old...and left at the age of 19 years-old! Year: 1971-1990
Regina : My first summer trip
Spent about two months there to learn our national second official language! Year: 1989
Vancouver
Supposed to spend a full summer there with a friend, but we actually returned home prematurely ... I had to return right after we found our first job. Year: 1991
Montreal: My first apartment
2510 Bldv Édouard-Montpetit, Montréal. Did my first two years of University there..with two other good friends. Year:1990-91
Worcester: WPI in the dorms
I've completed my 3rd year of B.Eng. Year: 1992-93
Montreal: Last apartment where I completed my B. at Ecole Polytechnique
3210 Bldv Édouard-Montpetit Year:1993
Montreal: First apartment with Kim
2910 Av Linton Year: 1994
Montreal: Second apartment with Kim
2910 Av Linton Year: 1994
Montreal: Third apartment with Kim
Av Papineau Nice view of the park! Year: 1995
Montreal: Fourth apartment with Kim
Av Papineau Yes believe it or not, we moved next door to have an extra room, sigh!! Year: 1996
Montreal: Fifth apartment with Kim
6654 de Lanaudiere Well things are getting out of control..I have a solution! Year:1997
Iasi: for Work
I've been in and out of this place for work. It literally opened my eyes to the world, and I also met quite interesting and fun people. Year: About 4 months during 1998-1999
Montreal: The first place we owned
6554 rue de la Roche. Here's the solution: we bought a nice duplex...things should stabilize now, well that was my plan! Year: 1998-1997
Tonbridge Wells: my sister place
Stayed at my sister place for a couple of months waiting for my Swiss work permit Year: 2000
Lausanne: Seventh place with Kim
Saint-Roch. First apartment we rented in Lausanne. Small and..expensive (ok this is Switzerland) Year: 2000
Lausanne: Eighth place with Kim
Cite-Devant. Our second apartment in Lausanne, much nicer and right in the old part of Lausanne. This is where my daughter was born. Year: 2001-02
Vaudreuil-sur-le-lac: Nineth place with Kim
26 des Caryers. The house we bought. Year: 2003-2007
Laval: Tenth place with Kim
Du Bouvreuil. The place we rent for 6 months before going back to Europe. Year: 2007
Epalinges: Eleventh place with Kim
De la Jaquiere. This is our current location! Year: 2008-?

Martin

SQL Analytic

2008-04-01T17:55:00.007+02:00

In my current consulting activity, I’m involved in a project relying on Oracle 10g for most of its data warehouse infrastructure. I thought it would be a nice opportunity for me to brush-up on my SQL skills.

The OLAP SQL functions (also referred to as Analytic Functions) come in very handy when dealing with data reporting or analytical needs. So this post will summarize the SQL analytic functions available in Oracle (some of them since 8i and other since 9i).

Although reports produced using these functions could very well be done using SQL or PL/SQL workarounds, these would be more clumsy and potentially less than optimal solution (depending on how well Oracle Optimize process these functions).

These functions can be roughly classified into these categories:

1. Ranking (show top 5 or bottom 5?),

2. Reporting Aggregation (compare values at different level of aggregation, e.g. region sales versus national, market share?),

3. Window Aggregation (moving average, cumulative sum?)

4. Row comparison (studying change or variation over sequential row, LAG/LEAD, or get first and last value of sorted groups with FIRST/LAST?)

It is important to note that these functions operate just before the ORDER BY clause of a query (which occurs last on the SQL process sequence order), or at the very last step when no such clause is present. Also important, these functions are not allowed within the WHERE, FROM and HAVING clauses, and are really meaningful within the result set of a query (i.e. in the SELECT clause) and more rarely in the ORDER BY clause.

The follow the following generic form:

function(<arguments>) OVER(<analytic clause>)

where

+function is the analytical function (e.g. AVG, CORR, COVAR_POP, COVAR_SAMP, COUNT, CUME_DIST, DENSE_RANK, FIRST, FIRST_VALUE, LAG, LAST, LAST_VALUE, LEAD, MAX, MIN, NTILE, PERCENT_RANK, PERCENTILE_CONT, PERCENTILE_DISC, RANK, RATIO_TO_REPORT, STDDEV, STDDEV_POP, STDDEV_SAMP, SUM, VAR_POP, VAR_SAMP, VARIANCE)

+arguments is optional and could include more than one (<argument1, argument2, argument3>)

+analytic clause can be of one the following: Query Partition clause (to break a result set into multiple partitions according to a criteria set expression); Order-By clause (for sorting data into each partition); and Windowing clause (to define a window of data).

1. Ranking functions

These functions role is to provide ordering info to records based on some given attribute. The basic three functions available (i.e. ROW_NUMBER, RANK and DENSE_RANK) only differ in the way they manage ordering ties. Here’s a very simple sample to illustrate the differences:

RANK and DENSE_RANK rules allow for the top-n query, which is not possible with ROW_NUMBER as it does not treat tie.

Note that we can also control the orderings of nulls value by specifying the self-explanatory clauses NULLS LAST or NULLS FIRST (default) after the “ORDER BY attr” clause.

Other functions can be classified within the Ranking category : PERCENT_RANK, CUME_DIST. They operate very similarly as the previous ones (e.x. PERCENT_RANK() OVER(ORDER BY attr), but compute the cumulative fraction typical of histograms.

In conclusion, the ranking functions offer a better alternative than relying on the well-known hack (creating a subselect virtual table with ORDER BY used in combination with the ROWNUM function) that can handle tie correctly.

2. Reporting Aggregation functions

Most common aggregation SQL functions (like SUM, COUNT, AVG, MAX, MIN, VARIANCE, etc) can be used as analytic functions as well, eliminating the constraint of having the aggregate value to be at the same level as the row set (mandatory when using GROUP BY). We can then have a global aggregated value presented with all data rows, as well as an aggregated value at a different partitioned window level all in a single query statement. Here’s a self-explanatory example with AVG:

Also belonging in this category is the RATIO_TO_REPORT which gives ratio of a value versus an aggregated comparator. Actually, this function is simply a shortcut version of : value / SUM(value) OVER() == RATIO_TO_REPORT (value) OVER().

We also include in this category the function ROLLUP and CUBE which both provide aggregated values at higher level (the difference being that CUBE will provide aggregate values throughout all combination of attributes) as illustrated by this simple example:

Notice the use of GROUPING function (which return 1 when aggregation was done over a specified attribute and 0 otherwise) to help differentiate between NULL values in the table and artificial NULL generated for reporting purposes.

Window Partitioning Operator

We often used in combination with the Ranking and Reporting aggregation functions, an operator known as the window partitioning operator. This operator role is to provide a way to separate data into grouping and perform the functions on these groupings. It relies on the use of PARTITION BY placed in the analytic clause. Here’s a simple example illustration the window partitioning used with the RANK() function:

We can see that the second RANK function now operates on a defined grouping (here defined by Grouping column) and not the overall set of rows.

3. Window Aggregation functions

These functions permit the calculation of moving average on a set of ordered data records. For example, to get a moving average using three data points window (one in the past, one current and one in future), we could use :

AVG(value) OVER(ORDER BY t ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING)

As found in the following sample:

Note that for missing values on the first and last record, the preceding and following records are considered NULL, and thus not used in the AVG function (see the first moving AVG value: 150 which is calculated from only 100 and 200).

This is a general rule applicable for treating NULL values when doing aggregation function, i.e. always ignored in the calculation.

The window aggregate function can also be used to calculate running or cumulative sum. This is possible using the ROWS UNBOUNDED PRECEDING and CURRENT ROW clauses for the window boundary definition. This gives:

As a final comment about this category of functions, let’s note that as in the case of ranking and report aggregation, it is possible to partition their calculation using the PARTITION BY clause.

4. Row Comparison

The row comparison functions are very useful for showing previous and subsequent records data on the same row as the current one within an ordered sequence of records. These functions are LAG and LEAD. These again are very simple to use as demonstrated in this small example:

The more generic format allow us to define offset and default values for NULL, i.e.: LAG(attr, offset, default_null).

Again, these functions can also be partitioned similarly as the previous ones. Anyone who has hacked a solution for this type of report by relying on self-join query can certainly appreciate the clearness and compactness of these functions.

Martin

Functional or Technical Focus

2008-03-22T19:31:00.004+01:00

In my professional life, I’m constantly confronted and hesitating between two career tracks:

Focus more on functional area and increase my business expertise and knowledge on specific industries or business domains;
Focus more on technical area and increase my IT technical knowledge on specific set of technologies, language, framework without any specific business focus.

Note that I've highlighted the verb focus here, since most IT work done in enterprise has a very important business aspect. So unless you are working for a highly technical business sector, you can't get away in IT without any business knowledge and culture.

Recently I’ve leaned more toward the technical side, mainly because in my freelance practice, I had a lot of software development activity. Surely the best way to learn and acquire real knowledge about technology is to actually deep into it by doing real development yourself (expressed clearly in this famous B. Franklin quote: "Tell me and I forget, teach me and I remember, involve me and I learn"). However, this requires a lot of learning and adaptation because of the fast pace of technology change and evolution.

Since my arrival to Sword in Switzerland, it seems as if I may have to reconsider this focus as the bulk of software development work is done offshore and I'm more involved in functional specification. I’m not making a big deal out of this, since I’m currently involved in the telco industry which has a lot of technical focus and is also very challenging and constantly evolving!

And besides, there will always be the open source world where a lot of possibility exist to practice and maintain my software development skills. But for the time being, I can't even consider that…, even my own personal projects had to be put on the backburner.

Martin

Vacation over!

2008-02-14T22:45:00.002+01:00

In our hectic occidental life, we are not used to stop doing what we usually do, i.e. work (and shop by judging on the popularity of shopping mall)! My break lasted for more than 4 months, in which I did enjoyed playing with the kids, reading more books than the last 4 year, biking around 1000km (although it was late in the year) and did various other things we usually don't have time doing during our short 2 days weekend break (BTW, yes I must confess I also did work during this break!). I guess it looked fun, since my wife got jealous and decided to take a break with us as well!

But it seems we are well programmed to live our active life which is not difficult to understand considering that we are brought up to it at very early stages (when I see those 4-5months old at aycare, sigh!). So after a while I was quite motivated to go back to work, but first I needed to find one.

Although I have my own consulting company, contracts don’t come so easily when you are a foreigner in a foreign country. I’ve worked in Lausanne before, but mostly involved in job at the backend, i.e. the RDBMS. My experience acquired within my consulting firm helped me broaden my knowledge in the J2EE environment and improve my ability as a consultant. This is the sort of skills I want to exploit in my new job.

So that means starting all over again and look for jobs the normal way, i.e. sending resumes, meeting new people and company and go through interview. Not the most fun, however I decided to take it seriously. The thing is, in my previous two jobs, I did not bother much about sending resumes and going through interviews. Actually I targeted only a single company, went through the normal hiring process and got lucky enough to get the job I was interested in.

So before leaving Canada, I actually checked the job market, identified a few potential firms/jobs and send them a link with my online resume. I started with about 5 or 6 of them, from which there were middle man ("intermédiaire"). I though that this would be enough as a first step and I could find additional target once arrived here. I must say the job market for my skills is well active in Switzerland, because this initial sending lead to an impressive list of follow-up:

-6 interviews
-1 psychometry test
-1 technical test
-3 phone interviews, etc… and 4 job proposal!!

p.s. Don’t underestimate the effect of the middle-man which can have a multiplicative effect.

Comparing this with my first job interview at Lausanne, for which I felt I was being sold a job rather than the opposite, I must say that things are more professional and most firms take recruitment very seriously. I've been through very high-level technical interview, challenging test, and very precise questions requiring thought-out answers and leaving not much place to subjectivity.

This took me quite some time and energy but I’m happy to have dedicated this effort because at the end this pays off and offered me the luxury to choose a job that meets most of my criteria (in order of importance):
1- type of work involved
2- type of technology involved
3- advantages like salary, vacation (of course!)
4- location of work
5- type of work environment

I’m happy to announce that I have accepted a job as a consultant for SWORD. This job will give me the opportunity to see former colleagues (because the first contract is at Orange), work not too far from home (Geneva looked cool but the commuting would be a bit too long, at least until they finally open the new metro M2 line at Lausanne), and I'm given the opportunity to exploit two of my expertise BI and J2EE!

Martin

Arrivée en Suisse

2008-01-12T14:25:00.000+01:00

Les deux dernières semaines ont été très occupées pour toute ma petite famille… nous avons organisé notre départ de Montréal pour aller s’établir à Epalinges juste au nord de Lausanne à quelques 830 mètres d’altitude (donnée importante dans cette région du monde puisqu’elle détermine la présence de neige en hiver).

En plus de toute l’organisation reliée au départ d’un pays, on a dû affronter les tâches reliées à l’arrivée dans un autre pays. Et l’arrivée fût un peu perturbante puisque le container avec tout notre mobilier est en fait arrivé en retard et ma femme n’a pas encore officiellement son permis de travail (une simple formalité … mais qui nous empêche pour l’instant d’être des citoyens avec les mêmes droits que les autres…. dans nos sociétés “modernes” ceci veut dire consommer). Donc nous avons campé dans notre nouvelle maison avec matelas d’appoint et le stricte minimum pour plus d’une semaine. Ceci veut dire pas de tv, pas de radio, pas de téléphone (on a finalement pu acheté un téléphone avec carte prépayée seulement), et ….le plus difficile pour moi, par d’Internet!

C’est inquiétant comment je suis rendu dépendant à l’Internet, en fait j’ai pas eu de souci pour la tv, le téléphone et la radio mais impossible de vivre sans ce monstre. Je m’aperçois que j’ai comme plusieurs pratiquement tout réorienté mes sphères d’activités vers le TCP/IP… scary !!! Ex: communication avec la famille sur Skype, email, et blog, transfert et opération bancaire via site bancaire, projets de développement storés sur site en-ligne, mis au courant des nouvelles, guide d’achat et localisation des commerces, etc... (Parenthèse professionnelle: pour quelqu’un qui fait de l’architecture de système, je devrais modifier cette dépendance des plus fragile afin que mes moyens de communication re-deviennent plus résiliente et robuste aux échecs « tolerance to failure »).

En ce qui concerne notre villa, tout est vraiment bien et nous avons en bonus une belle vue sur les alpes à partir des fenêtres des combles (Bernoises/Valaisannes et de la Haute-Savoie française)-- voir photo en debut de texte.

Martin

p.s. étant éloigné des miens restés au Québec, ce blog va prendre une tangente plus orientée personnelle, avec par contre toujours des sujets techniques qui devraient êtres nourris par mes expériences dans le domaine de la consultation.

Quick solution for Spring & GWT integration

2007-12-10T23:05:00.001+01:00

As promised in my last post, I'll describe here how I proceeded to integrate my service bean layer managed by the Spring container with the RPC mechanism offered by GWT.

GWT toolkit offers a custom servlet (RemoteServiceServlet) from which we can inherit in order to let client code calls server-side code asynchronously. The solution I implemented was to create a base Controller which simply extends the Google RemoteServiceServlet. See code below.

By subclassing this new base Controller, all my service-bean classes become now GWT-RPC compatible. This simple solution has some drawback such as adding a compile-time dependency between my service-bean and the Spring framework/GWT, however that is just fine for my specific use-case. More clever solution could also make use of reflection in order to leave your service leaving the service-layer classes free of such dependency, however this will come at a price of more intense processing needs. For those interested see the GWT Server Library initiative.

However for me, this simple solution offered a very convenient caching mechanism to improve performance all client accessing these service-beans. Since I have some RPC service objects on the server layer for which result are cached to avoid the expensive database access hit. Because these result objects contain long object graphs (a few number of association at various depth level..), I also wished to avoid the GWT serialization process involved at every hit of these cached object graph. To achieve this, I wanted to store the GWT-serialized version
of my result objects instead of the normal de-serialized graph. The cache is planned to refresh itself at regular interval...

All my service-bean achieve this by optionally intercepting the RPC request (i.e. override the method 'processCall') and use the GWT-serialized value when available in cache (see example below).

Martin

Web development

2007-11-23T04:15:00.000+01:00

Up until my current project which uses GWT to develop a rich web client, I always followed the best practices in terms of Web development. I used to develop thin and dumb client view (e.g. jsp) which were fed and render based on data model objects and eventually sent back to the web client.

All web requests are taken care by some dedicated controller (or action in Struts parlance) that are responsible in analyzing the user request, delegating the request to the service layer (the core of the application where business rules usually sits), obtaining the return domain object from the service and preparing the response by selecting the proper view to be rendered and sent to client. Ok, that is a very brief and incomplete description of the MVC pattern adapted to the Web server and HTTP protocol. Although this design pattern may be overkill for smaller applications, the advantages (e.g. makes web applications that are easier to maintain and are logicially divided into layers that are inherently decoupled and whose focus are on very different software concern) usually outweight the additional learning curve and relative complexity.

A great deal of coding must be done on the controller aspect of the MVC. And a lot of this includes details originating from handling the request/response of the stateless HTTP protocol...which has very little to do with the main application business logic. If I'd have the choice, I'd rather spend this time developing and improving the business objects domain, the related business rules logic and the database model, i.e. my preferred subjects and expertise!

With the advent of AJAX Web 2.0 fat client, a great deal of the weight that used to sit on the web server shoulder implementing this MVC pattern has now moved toward the client sitting on the other side of the communication channel. Web clients now make asynchronous RPC (remote procedure call) to the server without having to deal with the of HTTP concern details (for example the RPC mechanism of the GWT API isolates client code these). They can also now handle their own session state by keeping and handling the data model locally.

The server can focus less on the "C" of the MVC, and more on the service core layer designed to enforce and apply all business rules logic. The client code can choose to implement the traditional MVC pattern, when its complexity justifies it.

In my current project, all web controller codes disappeared and my web application does not move anymore from one page to another (following subsequent user request), but rather change its unique page appearance (by replacing/changing section of page through DOM manipulation) followed by user action and RPC response from the server.

My next post will present the way I've implemented my RPC client code to talk directly to my service layer using the Spring Framework. Now, I need to get back to work!

Martin

my Web2.0 experience with GWT

2007-10-14T22:42:00.002+02:00

My last month as a freelancer allowed me to have made quite important progress on my current project. I find it quite motivating since I'm the main analyst/designer/developer/tester and even business analyst (actually, I'm all alone on that one, so no one to blame or complain about;-). Technology wise, it's a perfect situation since I can freely chose all components based on design merit and no decision is based on political constraint of some sort.

Currently the technologies involved are:

On the server:

- Spring framework, iBatis, ehcache, Quartz, Velocity, and eventually the servlet container (not yet decided.. but the development is obviously done using Tomcat).

On the client:

- GWT and a few velocity pages.

The novelty for me is the use of GWT (Google Web Toolkit). I've already mentioned my frustration in developing client side code especially using http-based web client. The time spent on the page design messing around with jsp tag, html and css styling seems to me as a big wasted time and effort. That's why in my professional life I'm usually involved muck closer to the back end where my expertise is more profitable for the client.

Now with the advent of GWT, client side component is no longer restricted to thin and dumb page presenting the view/model already prepared on the server. A lot can happen on the client with the capability to hold on great deal of state information and business logic. And this magic is done without adopting Javascript and sacrificing the strongly-typed property of Java.... the toolkit does the Java-->Javascript conversion for you!!!

So far I'm quite pleased with the toolkit as it practically eliminate the need to write any html markup... however all styling still depends on css. I'm probably half-way there for the client part and only have a single html page with just a few lines of markup html code.

The server integration is done through RPC call (done asynchronously à la AJAX style) with servlet on the server end. A great deal of design effort is done by keeping the server scalability as a top priority. Because the application will target the web public at large, most design decisions are affected by this, e.g.

Keeping minimal state on the server. As a matter of fact, with the help of GWT most state is kept locally while the server only keeping a session token (cache in memory) to identify valid client connection. This will avoid issues involved when relying on HttpSession object (to hold client state information within servlet container) such as server affinity and more importantly larger memory footprint per connecting session.

Data is cached both on the client side and on the server side. Because some data is not mission critical, the client keeps some data content (limited size) and the server keeps all data content in cache eliminating the need to hitting the database back-end during most client request.

The cache server data content is refreshed at fixed interval of time which is configurable depending on the data volume and the database query extraction time

The cache server data content is kept in a de-serialized GWT form, avoiding the need to serialize the payload of server response at each client request.

I'm confident to be able to put a beta version of the application before the end of year... stay tuned!

Martin

La gang de Chicout!

2007-10-12T21:43:00.008+02:00

Bon ce qui semblait anodin à l'époque relève maintenant de l'exploit: avec un petit groupe d'amis datant de l'époque du Séminaire de Chicoutimi, nous avons réussi à rester chums et garder contact à travers tout ce temps passé (ouch, près de 30 ans ou plus pour certains!!! ).

Les bases de cette amitié sont assez simples: on ne se doit rien et aucun reproche pour les manques de nouvelles ou absences plus ou moins prolongées... et le tout reposant sur un passé commun durant une période de vie assez agitée. Cette période demeure encore la source principale de nos discussions et anecdotes interminables.

Ca sera mon post le plus court car malheureusement on ne peut en dire beaucoup sans risquer d'entâcher et compromettre des vies d'adultes responsables ... quoique le risque d'en voir un tombé sur ce site reste faible!

Donc pour conclure voici deux clichés pris avec pas mal d'années, de kilos et d'enfants en plus:

Party d'une date quelconque mais assez lointaine :

Dernier party en date :

From photos_amis

Martin

Scoring function

2007-09-27T11:53:00.005+02:00

This part discusses scoring issues. Please refer to this for info.

Definition:

instances = data objects observed and analysed (sometimes referred to as objects, data points...)
variables= characteristics measured (for continuous) or observed (for categorical) for each instance

Notation:

n data objects (sample size)
X generic input variables. When it is a vector, its component variable j is expressed with subscript: X_j
x denotes some observed instance, and when we have p-variables, we denote x₁ .. x_p as the the real-valued for the 1.. p variables measured on the particular object or instance.
x_k(i) correspond to the measure for variable X_k of the i-th data objects, where i has 1 .. N.
x (in bold) correspond to the vector of n observation of a single variable x.
X (capital in bold) correspond to the matrix N x p, containing N input p-vector x_p(1..N).

Classification model

2007-09-27T09:46:00.002+02:00

This part is on classification model. Please refer to this post for more info.

Definition:

instances = data objects observed and analysed (sometimes referred to as objects, data points...)
variables= characteristics measured (for continuous) or observed (for categorical) for each instance

Notation:

n data objects (sample size)
X generic input variables. When it is a vector, its component variable j is expressed with subscript: X_j
x denotes some observed instance, and when we have p-variables, we denote x₁ .. x_p as the the real-valued for the 1.. p variables measured on the particular object or instance.
x_k(i) correspond to the measure for variable X_k of the i-th data objects, where i has 1 .. N.
x (in bold) correspond to the vector of n observation of a single variable x.
X (capital in bold) correspond to the matrix N x p, containing N input p-vector x_p(1..N).

Laurie and Mathias

2007-08-16T05:41:00.001+02:00

I never spent too much time writing about my family here... naturally I tend to value more the time that I can spend with them instead of the time that I could write about them!

I just wanted to post a few pictures looking at my kids growing up. The thing that come into my mind looking at these pictures is: "Warning: Kids are less calm than they appear in photograph!"

But nevertheless we've got really two great kids, Laurie my daughter is now 5 and Mathias 3 years old and I'm confident that they'll continue making me a very proud dad!

Martin

That's it I'm changing career path!

2007-08-06T03:23:00.000+02:00

I've eluded in my previous blog all my frustration concerning closed source software... and today I'm very happy that I finally decided to move on to a different career path: I'm quitting my job at IMS Health where for the last 4 years I've held various positions as BI developer, team leader, etc.

My goal will be to focus on doing more consultating type of work that will rely primarily on open source libraries/framework to deliver high-quality application. During the last few years, I've been lucky enough to get some external contract which made use of these libraries. And I really appreciated the possibilities that OSS can offer:

in terms of quality of source code which helps to increase the quality of the application to build as well as your productivity
but also in terms of the education opportunities it gives to increase your expertise and knowledge as a software developer

This career move almost coincides with our plan to return back to Europe... my wife and I have finally decided to go back in Switzerland by the end of this year or early next year. More on this on a future post...

What will I do for the these 4 months left to 2007? Well, I'm very anxious to continue a personal project that I started a few months ago like a fun 'passe-temps', but now is starting to occupy too much time in my mind and I need to get rid of it ... meaning putting it online asap.

Enough said for now, I need to get back to work.

Martin

OSS, the future of software development?

2007-05-17T19:15:00.000+02:00

Came across this interview the other day from Lars Trieloff... "I believe that in ten years OS will be the predominant business model in the software industry". I usually tend to be quite wary and skeptical about these sort of long-term prognostics, especially in such a dynamic and rapid growth industry as the IT industry. However, Open Source to me is really the most effective way to produce complex piece of software. Without re-iterating the most cited advantages of this de-centralized approach, I'd just like to comment on a more personal experience.

At the office we are stuck with proprietary closed source software (no for long though.. more on this in a future post). As expected with software we hit a few limitations (read bugs) that were quite difficult to find a workaround or a graceful solution... all we were left with was a potential resolution in some future release of the framework (Reporting Services from Microsoft for this specific example)!

I can't help but compare this situation with a recent limitation I came across with Spring Framework a few weeks ago.

When trying to use EhCache for a project, I actually want to use the caching not only for what it is ... well i.e. caching, but also as a way to gather usage stats on the access of some of my service object methods. So I've defined a EhCacheFactoryBean while configuring it with the standard ehcache property file and decorated with a SelfPopulatedCache by injecting an implementation of the CacheEntryFactory. However, reviewing the code I realized that there was an issue, so in less than 5 minutes I raised a jira log and about 2 hours later my issue was assigned to one of the author of Spring, and corrected 2 days later!!

Well the fact that it took such as small amount of time to resovle the issue is very impressive and honor the commiters at Spring, although this could have very well happened in any other companies... but do you know a lot software vendors with such a quick reactive turnaround on bugs correction? However to me, the real advantage lies in the possibilities of having all source code at your disposal, it becomes trivial to simply write a corrected version and move on without having to wait for some future release. I can't help but compare this with similar situations we're faced with when issues are found at work with some Microsoft components.... well just too bad!

Martin

Consultation informatique

2007-04-09T23:45:00.000+02:00

Dans ma vie professionnelle j'ai la chance de combiner deux types de travail assez distinct. Le premier est relativement commun car il implique un emploi comme "Lead technique" d'une équipe de développeur d'application de type BI.

Par contre, ma deuxième vie professionnelle est plus stimulante et comporte son lot de défis. Dans cette seconde vie, je fais de la consultation informatique (freelance) dans mon domaine d'expertise: i.e. tout ce qui gravite autour des tâches d'analyse, de design et développement d'application développée avec Java et exigeant des interactions avec des systèmes de base de données.

Mon cheminement professionnel m'a permis de maîtriser ces deux spécialités comportants certaines incompatibilités, soit l'implantation de systèmes de base de donnée basés sur un design de type relationnelles des données, et l'implantation d'applications basées sur un design de type Orienté Objet (dans mon cas c'est Java). Il faut noter que certaines incompatibilités sont quelques fois une conséquence de la mauvaise fois des gens de chacun des deux clans, mais ceci est une autre histoire....

La connaissance de ces deux spécialités s'avère d'autant plus utile avec l'explosion d'application sur l'internet dont les caractéristiques justifient l'utilisation de ces deux spécialités.

Tout cela pour dire, que je suis bientôt mûr pour un changement de carrière ... mon emploi chez IMS étant moins stimulant car nous servons une industrie beaucoup plus conservatrice avec de long cycle de vie des applications logiciels (je travail en ce momnent sur une application vieille de plus de 30a ns fonctionnant sur le Mainframe). Il faut dire que j'arrivais du monde un peu fou de la télécommunication qui est assez intense et garnie en terme d'innovation.

Martin

JDBC convenient programming with Spring

2007-02-06T22:40:00.000+01:00

There is no need to re-iterate what has already been said and spread about the usefulness of the Spring framework... but it seems that every time I make use of a feature I come to the same conclusion: it simply makes you more satisfactory of your code, that is how I see it anyway !

Most of my database-related code usually made use of Hibernate because the sizing and the extend of the domain model simply justified leveraging the complexity of this framework.

However, in my current project I decided to give a shot at Spring jdbc support packages mostly because my domain model is simple enough (less than 10 entities with basic relationship) and because... well exploring more of the Spring library!

Although the library only really offers some thin wrapper around jdbc api (as opposed to a complete ORM solution), it does it in a way that you can create your data access code in a more object oriented fashion while keeping a close access to the underlying jdbc low-level api.

To illustrate this, I've created some helper classes (one per each business object entity) wrapping Spring jdbc data access object (e.g. SQLUpdate, JdbcTemplate, MappingSqlQuery) and offering a convenient way to centralize all sql-related string (sql command, table name and field, etc..).

On top of these classes, you can actually implement all generic sql access code (e.g. delete by id) and generic sql commands. Here's how look like the superclass, refer here as BaseSqlHelper:

And Here's one example of how one could implement a particular subclass (in this specific example the subclass handles sql for the User business entity object):

What does this offer you:
1- get all your sql strings contralized in one convenient place
2- benefit from code re-use by moving-up all your common fields (e.g. id, createDate,..) and common sql operations (e.g. delete from ... where id=, select ... from ... where id=) for all business entities into the superclass
3- exploit the MapRow capability to treat query response as real business entities and not merely as data field.
4- Ease for accomodating new fields and remove existing ones

Martin

Regression model

2007-01-27T11:42:00.006+01:00

This is yet another post on theoretical background and explanation related to data mining...

This part is on regression model.

Definition:

instances = data objects observed and analysed (sometimes referred to as objects, data points...)
variables= characteristics measured (for continuous) or observed (for categorical) for each instance

Notation:

n data objects (sample size)
X generic input variables. When it is a vector, its component variable j is expressed with subscript: X_j
x denotes some observed instance, and when we have p-variables, we denote x₁ .. x_p as the the real-valued for the 1.. p variables measured on the particular object or instance.
x_k(i) correspond to the measure for variable X_k of the i-th data objects, where i has 1 .. N.
x (in bold) correspond to the vector of n observation of a single variable x.
X (capital in bold) correspond to the matrix N x p, containing N input p-vector x_p(1..N).

Utilisation d'Hibernate

2007-01-04T15:44:00.000+01:00

Pour faire suite au post précédent, j'ai observé que certaines applications sont construites sans tenir compte des limites et contraintes d'Hibernate... ils utilisent cette librairie comme le "silver bullet" qui va se charger de tout ce qui touche la question de la persistance et ce sans effort ou configuration spéciale!

Par exemple concernant l'utilisation du Paramètre Lazy dans le mapping, plusieurs applications sont configurées avec lazy=false. Ceci facilite évidemment la question du fetching des graphes d'objet, mais ceci à un prix et ce prix s'appelle performance!

Recommandation:

Pratiquement toutes les entités et les associations de compositions devraient être configurées de façon Lazy (i.e. lazy=”true”). Il ne faut pas confondre le paramètre lazy et le paramètre fetch, le premier indique seulement à Hibernate s’il pourra créer un proxy (qui exigera une initialisation en session ouverte) pour la classe en question ou la collection, tandis que dernier indique comment les relations doivent être fetchées (eager ou non).

En utilisant lazy=”false” ceci a comme « side-effect » de pre-fetché toutes les dépendances de l’entité et ultimement la base de donnée complète sera chargée en objet ! Cela peut être pratique puisqu’on n'a pas à se poser la question de devoir initialiser ou pas les dépendances, mais ça devient vite catastrophique lorsque le volume de donnée est important.

La décision de fetcher ou ne pas fetcher (eager fetch) est une stratégie qui doit être défini en runtime puisque différente utilisation exige différent accès aux données. La couche DAO peut être utilisée en ce sens en fournissant des options de loader ou ne pas loader les dépendances (avec Hibernate.initialize()) avant de fermer la session Hibernate. Configurer avec lazy=”false” élimine tout simplement cette flexibilité.

Si pour une raison quelconque, une dépendance d’une entité doit toujours être pre-fetché, alors utiliser l’option fetch=join dans le mapping mais en gardant le lazy à false pour cette entité dépendante.

L’auteur d’Hibernate a lui-même reconnu ce fait et recommande depuis Hibernate 3.0 que toutes les mapping des class et des collections soient lazy= ‘true’ (ceci est maintenant le défaut utilisé).

Martin

Model & Pattern

2006-11-27T09:30:00.005+01:00

This is yet another post on theoretical background and explanation related to data mining...

This part is on descriptive modelling and pattern structure recognition.

Definition:

instances = data objects observed and analysed (sometimes referred to as objects, data points...)
variables= characteristics measured (for continuous) or observed (for categorical) for each instance

Notation:

n data objects (sample size)
X generic input variables. When it is a vector, its component variable j is expressed with subscript: X_j
x denotes some observed instance, and when we have p-variables, we denote x₁ .. x_p as the the real-valued for the 1.. p variables measured on the particular object or instance.
x_k(i) correspond to the measure for variable X_k of the i-th data objects, where i has 1 .. N.
x (in bold) correspond to the vector of n observation of a single variable x.
X (capital in bold) correspond to the matrix N x p, containing N input p-vector x_p(1..N).

Performance des applications aves des outils ORM

2006-11-10T23:15:00.000+01:00

Evidemment certains aspects sont plus importants que d'autres lorsqu'il est temps de considérer les performances des application se connectant avec les bases de données de type relationnel. Si on utilise de surcroît des outils de mapping ORM, alors la mauvaise utilisation de l'outil peut venir brouiller encore plus les cartes.

De façon général, voici par ordre d’importance les éléments qui doivent être considérés lors d’optimisation des applications interagissant avec une base de donné avec un outil ORM:

Le model ER (Entité Relationel) da la base de données Peu importe comment l’application et les modules ORM sont optimisés si le modèle ER de la base de donnée est pourri il sera difficile de produire des applications performantes. Normalement en utilisant des outils ORM, ce problème ne se pose pas puisque la création du modèle suit plus ou moins le modèle UML. Ce problème se pose normalement lorsqu’on dépend du base de données « legacy » qui ne peut pas être modifié.

La configuration du ORM (« fetching strategies », « caching », « report queries »). Cet aspect est critique car une mauvaise configuration résultera forcément en de très mauvaises performances peu importe les autres aspects et avec une dégradation linéaire en fonction du volume de données.

Le model physique de la base de donnée (tables, contraintes, indexes) Présumant que l’aspect précédent est respecté, ceci va garantir de bonnes performances avec l’accroissement du volume de données.

Le tuning général de la base de données (vendor-specific) Cet aspect est intimement lié au type de base de donnée, chaque vendeur fournit différent mécanisme afin d’améliorer la performance des applications de type OLTP ou bien OLAP. Cet aspect est difficilement généralisable et est la responsabilité du DBA.

Il reste toujours des cas d'exceptions mais en général ces principes s'appliquent assez bien pour les application de type OLTP avec base de donnée.

Martin

Muret de pierre sèche

2006-10-22T00:49:00.000+02:00

Ouf!

Cet été j'ai finalement complété mon petit aménagement "style naturel" autour de notre piscine. Le plus dur fût sans contredit le mur de pierre sèche qui a nécessité plusieurs centaines de grosses pierres traînant ici et là. Après la construction de notre maison il restait à faire tout l'aménagement paysager. Nous avons donner quelques contrats mais les coûts exhaustifs de ce type de main d'oeuvre saisonier m'a convaincu d'en faire une partie moi-même.

Le sol sur lequel notre maison est construite est littéralement envahi par les pierres, donc j'ai décidé de faire 1 pierre 2 coups (mauvais jeux de mots): ne pas seulement les transporter pour m'en débarasser (à grosses sueurs), mais de les réutiliser du même coup (à plus grosses sueurs) pour en faire un muret, une terrase et des marches pédestres. Le seul principe important pour le muret est qu'étant donné qu'il n'y a aucun ciment pour assurer la cohésion (il tient que par gravité), il faut s'assurer qu'une pierre repose sur au moins trois autres pierres.

Et après un grand merci à ma copine et à ma mère pour l'esthétique jardin de fleurs (ok, ok aussi à mes enfants qui se sont enfargés plus d'une fois), voilà le résulat de cet aménagement...!

Martin

Statistical General Concept

2006-10-16T14:58:00.009+02:00

This post is part of the notes I'm gathering from various references providing theoretical background and explanation related to data mining, analytics and computer learning (see this post for the book reference). I'm gathering these notes in hope to be a little smarter in applying and interpreting data mining algorithms taken out-of-the-box from mining tools (I must admit I'm also doing this to serve my endless quest for understanding).

This is actually linked (through an "iframe" tag) to a Google Doc that I keep updating as I face projects making use of new mining algorithms... so this is work-in-progress. I realize that blog is probably not the best way to publish live text, but is the easiest one for me.

This first part gathers basic topics from statistics difficult to classify into a very precise subject... should pretty much serve as a refresher for most people in this domain.

Definition:

instances = data objects observed and analysed (sometimes referred to as objects, data points...)
variables= characteristics measured (for continuous) or observed (for categorical) for each instance

Notation:

n data objects (sample size)
X generic input variables. When it is a vector, its component variable j is expressed with subscript: X_j
x denotes some observed instance, and when we have p-variables, we denote x₁ .. x_p as the the real-valued for the 1.. p variables measured on the particular object or instance.
x_k(i) correspond to the measure for variable X_k of the i-th data objects, where i has 1 .. N.
x (in bold) correspond to the vector of n observation of a single variable x.
X (capital in bold) correspond to the matrix N x p, containing N input p-vector x_p(1..N).

I hate UI-type development

2006-09-29T22:39:00.000+02:00

Why do I hate having to deal with UI-type development... I don't know but I have some hints:

I simply suck at it!

Although I enjoy using intuitive UI and appreciate the design value of it, I neither have the patience nor the talent to do it! In my view so much time and effort spent simply in designing a nice HTML/JSP page or rich client equivalent (with SWT for ex.) is too frustrating for the end result. I sometimes have to do it when delivering an end-to-end product for clients, and typically most of my time will be wasted on these UI stuff! I guess I could outsource all these, actually I even tried it once... but finding a good designer willing to develop inside JSP page is another challenge on its own!!!

When I first did some RCP stuff in Eclipse, I appreciated all the advanced design patterns available in library such as JFace but I soon got bored and tired again in dealing with all these widgets details consideration, I'm hopeless.

I guess I'll stick to creating domain business layer, service business layer, data access layer, and other more non-visual feature!

Martin

Parapente

2006-09-19T21:48:00.002+02:00

J'ai eu la chance d'essayer le parapente pendant mes dernières vacances et j'ai adoré les sensations !!! Ca confirme mon attirance pour les sports exploitants la dynamique du vent: j'adore la magie qui opère lorsqu'on est porté par la force du vent...et la prise de vitesse en dépit de l'absence de bruit de moteur.

J'aime la planche à voile mais trouve sa plage d'utilisation beaucoup trop restreinte (du moins ou j'habite en ce moment)... c'est pourquoi j'insiste toujours à y avoir accès lors de voyages familiales vers le sud. La dernière fois (à Punta Cana) par contre fut assez décevante: les journées avec un bon vent furent accompagnés d'interdiction, et pour les journées de levée d'interdiction alors les petites voiles suffisaient à peine à sortir des vagues donc impossible à éxécuter, genre de catch22 de la planche!

Le parapente semble mieux adapté au variation des conditions car on peut y exploiter deux types de vent: dynamique et thermique, ce dernier permet une ascension importante en altitude tandis que le premier est exploité le long de falaise ou parois de montagne! Le site ou j'ai pratiqué était assez bien, quoique loin des altitudes alpiennes, mais offre l'avantage de départ des quatres points cardinaux, assurant un départ garanti (sauf journée orageuse ou trop venteuse). Pour les intéressés:

Voici une petite séquence de la fin de ma descente en tandem.... juste après avoir fait quelques bonnes vrilles à 360 degré, accompagné de je ne sais combien de G, assez pour sentir mon cerveau descendre tout juste au niveau des pieds! Et oui, la fameuse force centrifuge (ou serait-ce centripète... mes cours de physique du secondaire sont quelque peu enfouis sous la masse d'expérience qui ne cesse d'expansionner avec le temps!) agit beaucoup dans ce genre de manoeuvre...

Martin

Unit Testing but...

2006-09-12T22:35:00.000+02:00

I really appreciate developing using the unit testing approach and as such I always have a JUnit library somewhere in my classpath while building an application. This really brings up my confidence into my code and allow me to refactor at ease without worry about breaking all existing functionality.
However, there are a few stricter recommendations commonly found among the unit tester fanatic or the extreme programmer advocate that I find, to say the least, debatable:

your unit test should focus on a single method of a single class, otherwise it is not really unit test.
always build your unit test first and then your application class afterward.

Point 1 actually emphasizes the term unit, and violating this makes your unit test more like integrated test which I agree. But in my view these tend to be more meaningful and practical.

First of all, methods should be small, precise, have a single clear responsibility and have descriptive name that conveys their purposes. As such I tend to agree with recommendation that limit the size of a single method (R. Johnson gave as a ballpark figure between 30-40 lines of code including all comments, while J. Kerievsky goes as far as recommending ten lines of code or fewer, with the majority of them using one to five lines of code). Keeping methods small and giving them intuitive name produce much easier and self-documented code: I like this idea since it helps reduce the need for documenting your code!

This is why I feel that the principle 1. above is opposed to the "writing short method" approach, since small method do not contain enough complex logic that requires a dedicated unit test on its own.

A junit class that test and validate the effect of each and every single method produces on the state of current object or some other dependents (through Mock-up objects) is often straightforward and thus overkilled! Also, a large number of method may not deserve a full dedicated test on them, since not only their logic is simple but also the impact on state is minimal.

That's why I twist my unit test a bit to make them more integrated test, i.e. test only important methods in the class in relation with their impact on itself and on its dependencies (external library, other piece of my code..). Ok, this is not always possible especially when the dependency library is costly and resource intensive component (then I'll use Mock-up for such case), but in very frequent usage, this allows me to validate and better understand the external library during my test as well as testing my code against its dependency. I find myslef even doing such integrated test with code at the service layer level (above the DAO layer) and validating its effect at the database tier. Using a small memory-based database engine such as HSQLDB helps negating the perfomance penalty of doing this.

As for the point 2, I usually adopt more of a concurrent approach, i.e. draft the application class and once it stabilizes create the test class and making it evolve simultaneously. The first few version of my class/interface are a bit too dynamic and sketchy to really have an accompanying test class. So to limit the need to duplicate my changes in both, I'd rather wait till I'm more comfortable with the class/ interface and then proceed with writing test case.
The only advantage I see in creating the test case first, is when I really don't know how my object's going to be used in the client code. However, in that case, I'd rather use a pencil and sketch some use case scenario beforehand...

Martin

Handling persistence with ORM

2006-09-07T22:21:00.000+02:00

In Java application development, using an Object Relational Mapper to connect to database typically will offer many advantages :

avoiding the necessity to code against lower level JDBC API
dealing with data persistence concern in a more transparently way more aligned with object oriented code paradigm
providing isolation to database vendor specifics allowing easy porting to a number of different DB backend
providing additional services support built-in such as connection pooling, caching, etc .
reducing the need to be highly skilled in SQL, although ignoring about relational concept and SQL is definitely not realistic.
writing less code

On the flip side, I've realized that there are drawbacks as well, such as:

providing least common denominator functionality to achieve DB neutrality
losing control of the SQL statetement automatically generated for us
some performance degradation, no matter what the tool vendor will pretend (ORM will always be one layer on top of JDBC...), however smart caching strategy can mitigate this
requiring additional knowledge of the ORM API (so less code to write but more library code to understand and make use of)
failing when the application use case is more focused on data reporting and aggregation of large data volume rather than on data entry transaction-based use case.

Typically on the last project I built using Hibernate, I've enjoyed spending more time on the design of a good domain model layer since I've spent less on the persistent logic concern. However, I discovered later through a more realistic usage and data volume test, that it suffered some nasty performance degradation in some specific use case that were not discovered through unit testing (unit testing is only concerned with functionality testing and not performance scaling issues).

Without going into details, the problem had to do with the number of round-trip Hibernate was triggering to fetch object data graph. I had designed some relation (1:1 or N:1) to be eager fetch (always fetch the related object) instead of using Lazy fetching strategy (fetch in database only when necessary). This was good in some scenario, since some data dependencies were always needed and this avoided a second database call to get this dependent object data. However when confronted with getting collection of data, the effect was actually a separate DB call for every single data elements within the collection. So getting a long list of N items resulted in N+1 DB call! Alternative solutions exist for this, but the recommendation is to model most (if not all) object relation using a Lazy strategy and adjust this default by specifying different fetch mode during run-time.

Bottom line, there is no magic bullet especially when it comes with database interaction. We need a good grasp in relation database concept in order to build application interacting with database, no matter what tools or framework you'll be working on.

Martin

Data mining background

2006-09-03T18:10:00.003+02:00

Data mining has its root, among other thing, in statistics and computer learning. To generalize things greatly, it is observed that depending on your background, these two will tend to view data mining performance very much differently..... Statistical background will rate performance on the statistical significance and inference power and score, whereas the computer scientist tend to measure performance on both the algorithm efficiency and scalability. However I realize that the two approaches really are two sides of the same coin, and this is reflected in the most recent scientific literature.

Various definition of data mining can be found in the literature, but I personally appreciate the more academic point of view than the marketing one commonly marketed by vendors. Here's some explanation excerpt taken from the excellent book « Principles of Data Mining » from David Hand, Heikki Mannila and Padhraic Smyth (MIT Press) (seems to be one of the few data mining books respected inside the statistical community).

It (Data mining) is recognized as well defined procedures that take data as input and produces output in the form of models (a summarized descriptive form of the data globally) or patterns (a descriptive form of some local phenomenon happening on a fraction of the data). The well defined procedure contrast with computational method which does not guarantee to terminate after a finite number of steps as opposed to a data mining procedure.

Data mining is concerned with building empirical models that are not based on some underlying theory about the mechanism through which the data arose but rather models consisting of a description of the observed data. In this sense, a good model is qualified as « generative »in the sense that data generated according to the model will share the same characteristics as the real data from which the model was generated.

They also offer an interesting decomposition of data mining algorithms into orthogonal components which contrasts with the magical and reductionist view marketed by tool vendors (always around the idea of simply applying specific algorithms to magically accomplish your task at hand). In essence, Data mining algorithms intends are to perform specific task on a sample or a complete multivariate datasets. But the task (1^st component) is one of many other component that a mining algorithm usually address:

Obviously, the Data mining task in question: whether it be visualization, prediction (classification or regression), clustering, rule pattern discovery, summarization through descriptive model, pattern recognition, etc.
The functional form of the model or the structure of the pattern. Example include linear regression forms, non-linear functions such as the one resulting from Neural network, a decision tree form, a hierarchical clustering model, an association rule, etc. These forms delimit the boundary of what we can expect to approximate or learn.
The score function used to judge on the quality of the fitted model used to summarize the observed data or of the pattern used to characterize a local structure of the data. This score function is what we try to maximize (or minimize) when we fit parameters to our model. It can be based on goodness-of-fit (how well the model describes the observed data), or also on the generalization performance, i.e. how well it describes on the data not yet observed (for prediction purposes).
The search or optimisation method used: the computation procedures and algorithm used for maximizing the score function for a particular model. The search can limit itself to select the best parameters value within the k-parameters space (as in the case of k-th order polynomial function form) when the structure is fixed. And we may have to select first from a set or families of different structures.
The data management technique to be used for storing indexing and retrieving the data. This aspect becomes primordial when it is time to process massive data sets excluding the use of the main memory alone.

The ideal tool would allow you to use different components independently from each other during some data mining activity. This level of agility and flexibility is however not possible in tools today... which may be justified and reasonable when it comes to the components optimization and data management but much less for the functional form and the score function components.

In practice, tools usually offer pre-packaged algorithms from which you can easily fall into the algorithm trap where you are only expected to apply some well established mining algorithm to accomplish magically the specific task at hand. This is the typical black-box paradigm that I've learned to despise in data mining (note that black box abstraction is overall beneficial especially in software OO programming model).

My curiosity simply forces me to step back and discover what's all the different mining algorithms applied to my data. After reviewing some of the literature, I've realized that I came across a lot of theory/practice in statistics during my curriculum, however I cannot say the same for machine learning (although I did some cluster analysis during my master thesis). So in the spirit of increasing your theoretical grasp of the underlying principles, let me give you a list of the books I highly recommend (in ascending order of statistical theory prerequisite):

Intelligent Data Analysis. Michael Berthold (Editor), David J. Hand (Editor). Springer (note that this one leans a bit toward AI with the inclusion of material such as fuzzy set logic and genetic algorithms)
Principles of Data Mining. David Hand, Heikki Mannila and Padhraic Smyth. MIT Press
Pattern Recognition and Machine Learning. Christopher Bishop. Springer
Elements of statistical learning Data mining, inference, and prediction. Tibshirani & Friedman. Springer.

Martin

p.s. I will try to summarize some the principles found in these books that I consider more than useful for any data mining practitioners to have. Although, blogs are not the ideal platform for such knowledge sharing, it is the most convenient one I have at hand (at least currently).

Préférences

2006-08-15T12:06:00.000+02:00

Avec le temps et surtout l'expérience, j'arrive à mieux savoir mes préférences en terme de travail... voici une petite liste de truc dans le format "me convient versus me convient moins":

exécuter versus faire exécuter
travail sur un plus petit nombre de tâches concurrentes plus exigeantes versus un plus grand nombre de tâches plus routinières et monotones
travail dans un cadre précis et concret versus cadre plus flou et plus théorique
travail d'analyse et de raisonnement dépendant de la logique versus travail de recherche dépendant du bagage de connaissance
comprendre versus apprendre
apprendre grâce à la compréhension versus apprendre grâce à la mémorisation
travail de gestion technique des projets/ressources versus travail de gestion administrative
travail varié et exploratoire versus travail fixe et récurrent donc redondant.

Cette liste est bien évidemment dynamique mais disons que, depuis quelques années déjà, elle se stabilise assez bien. Les fluctuations semblent être en terme d’addition et non de modification!

Martin

Note: From time to time and usually when my post will be more personal, I'll blog in french. As you may have already noticed, english is not my first language, but rather the language I often use in my professional life.

Java and Oracle

2006-08-03T20:21:00.000+02:00

Oracle has been committed since Oracle8i in integrating Java within its database/application architecture. Being confronted to the development of a particular application highly tied to Oracle, I'm taking this opportunity to review the current state of affair as of Oracle 10g. Here's what I found:

Originally the strategy was to follow a database-centric strategy where merely all software layer would be offered and hosted directly inside the database engine. This controversy strategy (to say the least) has since been reversed from 9i and 10g where some J2EE technologies already integrated inside the database (e.g. EJB container, JSP and servlet) have been desupported.

The focus is now on providing a complete Application Server suite (J2EE compliant) outside the database offering a vast number of services and support, pretty much like IBM WebSphere, BEA WebLogic or JBoss Application Server.

However, this strategy leads to the development (from beginning of Oracle 8i) of a fully functional and compatible Java Virtual Machine inside the database: OracleJVM.

Each of these two components are commented next.

1- OracleJVM

As of 10g release the OracleJVM offers these characteristics:

support the J2SE 1.4.2 as specified by Sun Microsystems
supports only the headless mode of the Java AWT (i.e. no GUI will be materializable on the server or remotely)
java classes (bytecode), resources files and java source code (optional), all reside at the database and stored at the schema level (knows as the Java schema object)
each session (user connecting to the database and calling Java code) will see its own private JVM (although for performance reason the implementation does share some part of Java library between session)
core Java class libraries are run natively through the use of Ahead-of-time compilation to platform-specific C code before runtime
core Java libraries are stored and loaded within the PUBLIC schema and thus available to all other schema
application specific Java classes are stored and loaded within the user schema (the owner)
besides writing the Java class, compiling it and running it, OracleJVM requires two extra steps in its development/deployment cycle: 1- class needs to be loaded into the database (done through a utility called loadjava, 2- class needs to be published when callable from SQL or PL/SQL (done by creating and compiling a call specification or a.k.a. PL/SQL wrapper) to map the Java's method parameter and return type to Oracle SQL type.
granting execution rights is also needed when running a Java classes located in other user's schema
class loading is done dynamically as in conventional JVM, however it is done into shared memory, so only one-time loading speed hit is encountered among all users code requiring the class
instead of a global classpath defined at runtime to resolve and load all application classes, OracleJVM uses a resolver per each class during class installation specifying in which schema their depending classes reside
multi-threading is usually achieved using the embedded scalability of the database server, making Java language-threads needless since they won't help improve the concurrency of the application (this helps avoid complex multi-threading issue inside Java code)
OracleJVM offers adapted version of JDBC (called server-side internal driver) which is specially tuned to provide fast access to Oracle data from Java stored procedure, as well as a optimized SQLJ server-side translator.

Execution control:

How do we exactly start off a Java application located inside the Oracle database or in other words what is the equivalent entry point of the static main method in a "normal" application launched by a conventional JVM? This process is referred to in Oracle terminology as a Call and can be done by calling any static method within available loaded and published classes. These published classes must then contain a static method entry point, and are qualified as the Java counterpart of a PL/SQL procedure (referred to by the term Java Stored Procedures).

Some possible scenario of a Java called includes:

a SQL client program running a Java stored procedure
a trigger (i.e. event fired off by defined SQL DML statement) running a Java stored procedure
a PL/SQL program calls a Java code

These Java Stored Procedures are callable from PL/SQL code but can also call PL/SQL procedure.

Some thoughts: Even though I've never played with OracleJVM, I'm yet to be convinced about its advantage: stored Java procedures seems a bit like writing Java code with a procedural mindset? It seems that the only advantage is the possibility to write and centralize business rules that are more portable and powerful than PL/SLQ code and that are available to application written to bypass the Application Server tier?

2- Oracle OC4J J2EE Application Server (or a.k.a. OracleAS):

This server referred to as OC4J, now includes an ever growing number of components (Web server, J2EE technology, ORM with TopLink, Portlet, wireless, Business Intelligence, etc). Its J2EE support includes: JSP, servlet, JSF and ADF framework (using event-based model for web http processing), EJB, JNDI, XML support (schemas, namespace, DOM, SAX, XPath...), Web Services (WSDL, UDDI, SOAP).

The type of applications supported by this infrastructure are usually large and complex, i.e.

involve multiple application tier: the central AS tier where the business logic is maintained, a web tier (maybe part of the AS tier) interacting with Web clients, a backend database tier where persistent data is preciously stored, client tier from fat to thin.
involve multiple user with different role and rights accessing concurrently common data
involve different remote user sites (i.e. implies Web access) and heterogeneous environment
involve sophisticated business rule;
involve interaction with other EIS enterprise information system through the J2EE connector Architecture (ERP such as SAP, legacy information system)
involve web services support

Of course not all application will need all these, but to pull its weight and leverage this considerable software infrastructure weight, the application specification should meet a fair level of complexity before committing to this framework. This technology overweight is probably responsible of the creation of lighter and simpler initiative coming from opens source community (lightweight Framework only requiring a web jsp/servlet container, such as the one I described here).

Martin

J2EE development

2006-05-29T02:31:00.000+02:00

Before starting to do any J2EE Web development I did my own research on tools and libraries that would best meet my web transaction-based application requirement (e.g. things like flexibiltiy, simplicity, availability, cost, adoption..). I finally decided to go with the Spring Framework for all integration code and parameterization settings, Hibernate on the data tier to handle the ORM aspect, and Struts for the Web tier. I discovered since then that these exact set of tools are promoted by Source Labs (http://www.sourcelabs.com/?page=software&sub=sash) as the SASH stack. Although I appreciated developing using these libraries, I enjoyed even more the best practices that these frameworks encourage through the adoption of sound principle: loose coupling between component, seperation of concerns, design pattern uage like MVC or dependency inversion, etc.

You have a feeling when you're building applications along these principles that it is well architect and clean, however you enjoy it even more when 5-6 months later the client calls you to update its requirements!

Without going into details, the web application developped with these frameworks usually follows architecture along these lines:

A seperate and "dumb" View layer (JSP page);
A seperate Control layer (action and action setting files in Struts)
A separate Model/Business layer (business layer is using simple POJOs following java bean rules which allow dependency injection with Spring)
A seperate Data layer (thourgh DAO and Hibernate ORM)
An integration and configuration to glue all layers through Spring bean application context file
And finally a simple servlet/JSP container server ( e.g. Tomcat) to service and host the application deployment.

As a good advocate of open source, I put my principle into practice by putting such an application available to anyone intrested, just contact me by email and I'll send you a copy of the project.

Martin

JDMAPI: some info

2006-04-15T16:45:00.000+02:00

Here's some information I've gathered by looking at the API (JDMAPI spec version 1.0) and at the material found here and there:

1- Mining Function

First the features that are probably central to the API are called Mining Function. These functions relate to the objective one wish to achieve and operate on the individual record or case (collection of related attributes belonging to one entity or transaction used as input for building the data mining model, e.g. customer, purchase, etc..).

These can also be categorized by whether they are supervised (the model is trained and tested using a known target attribute value) or unsupervised (no target value variable is of used).

Looking at the Enum type called MiningFunction, we find these supervised functions :

Classification (predict a target attribute of type categorical)
Regression (predict a target attribute is of type numerical, i.e. continuous)
*Supervised Multi-Target (for model predicting multiple target attribute at once)

For unsupervised functions, we have

Clustering (associate record to natural cluster that "closed" )
Association (discover hidden interrelationship or correlation among the variables)
*Anomaly Detection (identify rare and unexpected case)
*Time Series (understand the pattern and forecast the time-ordered series of case)
*Feature Extraction (project the set of all attributes into a much smaller set of features useful for visualization capturing important characteristics of the data)

There is also a function applicable to both supervised and unsupervised called Attribute Importance which is helpful to reduce the number of attributes and complexity of the model to build. This function helps identify the most relevant attributes and reduce nose when building mining model.

2- Mining Algorithm

To build mining model for each of these functions, we need to apply specific algorithms. By checking the Enum class called MiningAlgorithm, we find:

Decision Tree
Feed Forward Neural Net
kMeans (a k-means clustering algo)
Naive Bayes
SVM Classification (for classification)
SVM Regression (for regression)
*Arima (for time series)
*Arma (for time series)
*Auto Regression (for time series)
*NMF (Non-negative Matrix Factorization algorithm for feature extraction)

3- Mining Task

The API includes a definite set of task used to construct the various mining object (see Mining Named object) : These task defines by the Enum class 'MiningTask' are given next:

buildTask (to construct a mining model, see mining function)
testTask (for validating the mining model on an independent test dataset, only for supervised)
applyTask (for applying the mining model to a new data source)
computeStatisticsTask (to get basic statistics of attributes from source physical data)
importTask/exportTask (for interacting with external application/framework)

4- Mining Objects

Typically when one submit mining task to the DME (data mining engine), this will generate some persistent objects called NamedObjects. These objects are normally stored in the mining repository and can be saved and restored with the API:

buildSetting (used to specify the model to be built, i.e. the mining function, the source data/attribute, the algorithm to be used with its settings, etc.)
testMetrics (used to produce a test result)
applySetting (used to define the data on which a model needs to be applied)
physicalDataSet (a pointer to the original data used for build, test and apply e.g. a database table a file )
logicalData (optionally describe the physical data set to change attribute name and their types)
model (used to store the resulting mining model)
task (used to refer to existing task and their status )
taxonomy (used to define hierarchical grouping definition of categorical attribute value)
costMatrix (matrix used with classification to associate cost with the actual versus prediction value )

Note: * refers to release 2.0 functions.

Martin

Data Mining has now its open interface: JDMAPI

2006-03-27T23:23:00.001+02:00

I've always been interested in data mining because it mixes some advanced statistical or mathematical methods with complex data computation algorithm (typically developped for computer learning). On the negative side, its application may have a bad press (sometimes well deserved) because of potential abuse it can lead to... I will not dwell time on identity and privacy sensitive issues, but when the goal is respectful of people privacy's right, one can leverage data mining to bring normal analysis to a much higher level. This is achieved through data induction (let the data speak for itself...) as opposed to data deduction (deducts conclusion based on specific report produced) typically encountered in more classical BI application.

Data mining functionality is now built-in inside Oracle 10g through a standardized JSR (http://www.jcp.org/en/jsr/detail?id=73). Although this first effort is limited in scope, it will ensure application could be developed independent on vendor proprietary API. It is currently being continued through a more complete initiative (JSR-247, http://www.jcp.org/en/jsr/detail?id=247) which will bring more mining functionality and advanced algorithm. Most big players contribute to this standardization effort: Oracle, SAS, Hyperion, SPSS, SAP, IBM, etc.., but yes of course except Microsoft.

The standard API also offers extensibility (each vendor can provide additional functionality not explicitly defined within the standard) and also covers the use of Web Services which will ensure complete independence of platform and language implementation.

More info can be found by googling JDMAPI.

I'll try to analyse this API in more detail and let you know my discoveries...

From ER to OO

2006-03-19T23:54:00.000+01:00

In my previous posts I've been sharing knowledge valuable for people dealing with the technology underlying relational database management system (RDBMS for short). This technology is used to store literally any information held by organization. I've been dealing with this technology since about 1996, and still continue to do so mainly because of its ubiquity in IT world. Relation databases store information using set theory and implement transaction and concurrency control to handle a large number of simultaneous connection, and as such their scope are fairly limited (although most big players are trying to include more functionality and processing flexibility into their engine, e.g. Oracle experiencing with the inclusion of JRE within their database...).

After doing modeling and designing architecture for database for some time, I started designing and developing stuff in Object Oriented language environment (around 2002). At first this can be quite daunting with all the flexibility OO programming can offer... Compared to database modeling where you have a rigid framework and theory guiding your work, OO modeling seems more to stimulate your artistic and creativity ability than your analytical expertise.

To overcome this new paradigm, here's some pragmatic steps I did and applied in learning Java, free of charge (or almost):

Getting and reading good reference documentation, such as the free resource from Bruce Eckel, Thinking in Java. This first step will only help you gaining some knowledge, but to be able to do it yourself in an elegant and flexible way you'll definitely need more experience. After some practice you'll seem to face recurrent problem over and over... this is where step 2 kicks in.

Getting a good reference on Design Pattern, this will teach in developing code with better quality (from aspects such as flexibility, robustness, adaptive, less error prone, etc.) following pattern developed by experienced developer. A good introduction book would be Head First Design Pattern, but to get the real reference document you should go to Gang of Four.

If you're still shy and afraid of downloading a free copy of Eclipse to experiment and code yourself (at this point maybe you should simply reconsider coding ;-), then what is still available to you are millions of lines of quality code (mostly in Java) available in the best open source project, more on that later. However, most likely you'll actually be programming your own stuff relying one or many open source components, at least that's how I did it.

Martin

Design vs Coding, and DDD

2005-12-16T00:07:00.002+01:00

I have the chance (some would probably not call it that way) to deal with architecture/design stuff in my work, in addition to doing real coding and hands-on development work during my consulting activities.

For me, it's sort of an experimental way aiming to find which of one of these two would fulfill my day the most. After doing the two for the past 2 years: I came up to the conclusion: the mix of both is what fits me since coding allow to stay more in touch with the technical aspect and the technology while the design and architecture helps me keep some perspective as to which technology can make some difference and which adds up to ... well mostly hype!

An interesting approach that I find in OO-type of development that actually stresses the importance of having both competence in order to build complex system having all usual quality indicators. This approach originated from Eric Evans and is now well accepted and referred to as Domain-Driven Design (or DDD). I will not describe it here, please refer to his book.

However let me just comment of one aspect that I find particularly useful during OO design of a typical business application. It really helps in producing well organized, well designed, well documented and logical Domain Model construction. Very briefly the way I see this particular concept, is similar to a form of taxonomy that helps you organize and classified each object that you design in your Domain Model based on their roles:

Entity Objects are those with a very precise identity and transactional lifecycle, and for which we usually worry when multiple instances of the same identity could be found at runtime. They would typically be mapped to a unique row in the persistence database.

Value Objects are those without any identity where its field attribute are only compared by value (i.e. two instances with same value are considered equal).

Factory Objects are those concerned with the creation of object(s)... more often justified when creating group of objects along with their associated graph. This is required when this creation involve enough complexity to warrant encapsulation through factory.

Service Objects are those used to handle and implement business service along with associated business rules. These service objects usually operate on multiple collaborators entity objects as well as their repository.

Repository Objects are those used to handle entity lookup and persistence through an abstract layer hiding any details and idiosynchracies of the related persistence engine. They are typically used by Service objects, but sometimes by Entity Object, in which case Entity object would have a reference to the Repository passed inside its methods requiring Repository functionality.

I find this taxonomy very insighful and convenient, and this is certainly useful to adopt..still worth it when only used to help standardizing design terminology among developers and architect!

Martin

Intelligent Agent part 4: Implementation with OO programming

2005-01-11T20:19:00.002+01:00

Intelligent agent implementation using Object Oriented programming paradigm

Wooldridge in Multiagents Systems provides a comparison between object oriented system and agent system paradigm, which seems awkward in the sense that the two paradigms serve different purposes. According to his reasoning, OO does share agent system characteristics by exhibiting autonomy over its state (and object control its state by making it private), however fails in controlling its behaviour (because once object methods are made public, control is switched to external object) contradicting the autonomy notion of agent. Other observations include the fact the agent have their own thread of control whereas in standard object model, a single thread of control is allocated to the entire system.

Actually, OO paradigm is sophisticated enough to provide mechanism required by the agent specification. For example, the access control granularity of object could allow to make only state information visible through methods and no operation done on the objects themselves (or condition could be applicable to the operation in order to respect the autonomy notion of the object), furthermore most OO implementation offers robust multi-threading control capacity, etc.). Surprisingly, the article refers to active object, which shares the property just mentioned, and also mentions that OO system can be used to build agent based system. We could add that it is actually being used, see Bigus for specific application example. Furthermore, from an ad-hoc search on the current software architecture currently available in building agent-based system, we see that the majority of development frameworks are actually originated from OO language implementation (reference: http://www.multiagent.com/) and most of them are actually from a well accepted general purposes language, i.e. Java:

Academic research based:

Cougaar Agent Architecture (http://www.cougaar.org/)
Jadex BDI Agent System (http://vsys1.informatik.uni-hamburg.de//projects/jadex/)
JATLite (http://java.stanford.edu/)
JATLiteBean (http://kmi.open.ac.uk/people/emanuela/JATLiteBean/)
Java Communicating Agents (http://jacomma.sourceforge.net/)
JESS, the Java Expert System Shell (http://herzberg.ca.sandia.gov/jess/)
MAP, a Mobile Agent Platform (http://sun195.iit.unict.it/MAP/)
Excalibur (http://www.ai-center.com/projects/excalibur/)
Bee-gent Multi Agent Framework (http://www2.toshiba.co.jp/beegent/index.htm)
Zeus Agent Building Toolkit (http://more.btexact.com/projects/agents/zeus/index.htm)

Commercial based product:

Jade (http://jade.tilab.com/)
CybelePro (http://www.cougaar.org/)
JACK Intelligent Agent (http://www.agent-software.com.au/jack.html)
Living Systems Technology Suite (http://www.whitestein.com/pages/solutions/ls_ts.html)
TSpaces (http://www.almaden.ibm.com/cs/TSpaces/)

Taken from

“Constructing Intelligent Agents with Java” (1998) from J.P. Bigus and Internet ad-hoc researches

Intelligent Agent part 3: Multiagent and Societies

2004-12-12T19:58:00.003+01:00

Multiagent systems and Societies of Agents

Analyzing and considering only individual agent architecture is neither convenient nor realistic as most environments contain multiple agents interacting together in various ways. An important aspect of these multiagent systems or societies of agents consists of the infrastructure facilitating the interaction between agents during their course of action. These interactions should not compromise autonomy of agent whether they are cooperative, competitive or simply self-interested.

During message exchanging, some form of protocol or mechanisms dictating the communication are necessary to ensure that agents “understand” each other. These protocols are defined at various communication levels. We have, from lowest to highest respectively, level specifying interconnection, level responsible of format and syntax and level for meaning or semantic. At a more abstract level, another protocol is also necessary to organize interaction between agents as to mimic “real conversation” taking place not just single message exchanged.

Communication Level Protocol

To simplify software agent design, “speech act theory” serves as a model for handling the various aspect of communication. Speech act views human language as actions (e.g. request, suggestion, commitment, replies, etc). These actions have three dimensions; locution (the words), illocution (the intended meaning) and the perlocution (the actions resulted of the locution).

One good example of a protocol for exchanging information and knowledge that rely on “speech act theory” performative model is called KQML (or Knowledge Query and Manipulation Language). This protocol is specifically useful for decoupling both the message and the communication semantic. This is achieved by separating the message into multiple fields; language (e.g. Knowledge Interchange Format -KIF or Lisp, Prolog, etc), content (i.e. representing the message itself and expressed using the language specified) and ontology (representing the objects and their inter-relationship in a domain of interest) are all part of separate fields.

Interaction Level Protocol

As for the higher level of communication, concepts such as coordination, cooperation, contract net, blackboard system, negotiation and market mechanisms have all been devised in order to specify agent interaction protocol.

Coordination protocol ensures that global goals are achieved while respecting global constraint and reducing redundant problem solving. Coordinated systems have often been designed in DAI field (distributed artificial intelligence) through techniques that distribute both control and data. However, this leads to some difficulty such as agents having only partial and imprecise information knowledge on the global goals and also having less coherent global behaviour. Coordination uses two keys structures to formalize itself in distributed agent system. The first one called commitment is used by the agent as a pledge to undertake a specified course of action, while the second one called convention is a means of managing commitments in changing circumstances. Commitment ties agent into future activities and convention dictates the rules and constrains under which these commitments are re-evaluated in case of changes and new reality.

Coordination is not the only abstract level protocol existing, the following gives a brief definition of alternative mechanisms existing to support agent interaction:

Task decomposition and distribution. Often utilized in a more cooperative type environment in which the system designer takes care of the task decomposition of agents. This can be done by spatial decomposition where agents are responsible of task based on the information source and decision point location, or by functional decomposition where agents are defined and allocated by expertise.

Contract Net. This mechanism is modelled following the goods/services exchange economic rule. The roles of agent are not specified before hand but rather through a bidding system in which managers (agent announcing task for bidding) award contractors (agent winning a contract after submitting a bid) specific problem resolution task.

BlackBoard System. Central interaction point shared and used by all agents to reach common solution.

Negotiation. Utilized in situation involving agents having conflicting goal. This needs some form of concession and alternatives search from the agents to reach and move toward agreements.

Market mechanism. Often a direct interaction between agents is just impossible due to the significant number of agents involved. In this scheme, a voting system can be put in place allowing agents to vote between a set of alternatives solution proposed by a smaller group of agent.

Taken from

“Multiagent Systems, a Modern Approach to Distributed Modern Approach to Artifical Intelligence” Chapter 2 Mutliagents Systems and Societies of Agents, Huhns and Stephens (1999)

Intelligent Agent part 2: Architecture

2004-05-04T18:02:00.002+02:00

Here the second part of my research study on software intelligent agent.

Intelligent Agent Architecture

Current agent architecture can be classified as either 1) logical based agent, 2) reactive agent, 3) BDI agent (belief-desire-intention) and 4) layered architectures. The following summarizes very briefly these approaches and provides some insight in relation to their pros/cons.

The logic based approach is when the decision making strategy reduces to a problem of proof (predicate logic), i.e. the agent uses its symbolic representation of the world (usually through logical formulae) to accomplish logical deduction in order to select an appropriate action. In this scheme, the deduction rules that determine the agent behaviour are the “program” (execution), whereas the symbolic information/ representation of the environment is the current database (data).

This traditional approach originating from early AI work, suffers from disadvantages such as the following:

Because of Calculative rationality (the optimal action is deduced at time j based on the environment condition as seen at time i, where j>i), we must assume that the environment did not change during the time agent chooses its action, reducing the practicality of such agents. Although alternative compilation of the logical specification into a more amenable form can provide better efficiency, this issue still persist in time-constraint and highly dynamic environment.
Perception of the agent is realised through a non-obvious mapping from real environment condition to a set of symbolic formulae. It has been acknowledged that as seductive and useful logic is to conceptualise and specify agent, it is quite difficult when one tries to represent and reason about complex, dynamic of physical or non-physical environment using a set of explicit symbolic reasoning.

The reactive agent architecture (alternatively referred to as behavioural or situated) proposed by R. Brooks intends to simplify and avoid the difficulty of symbolic representation by providing a more economical, elegant and robust agent whose behaviour is simply a function of its local environment. One of the most famous technique called Subsumption architecture is characterized by an action-perceptual input mapping (situation  action) and a layer controlling hierarchy in charge of resolving possible simultaneous action conflicts (lower layer action takes precedence over higher or more abstract layer). Agent performance and complexity which depends on the number of behaviour and number of percept (raw sensor input is used as opposed to complex symbolic representation) can result in a constant decision time when hard-wiring decision making process, making such agent computational simple and attractive. However, this simplicity leads to some disadvantages:

Only a local environment information is gathered making non-local information not possibly taken into account
How could purely reactive paradigm lead to some form of experience learning and performance improvement over time?
Implementation engineering relying heavily on experimentation, trail and error methodology
The hierarchical form of action control can lead to confusion when a large number of different behaviours are modelled.

The BDI or Belief-Desire-Intention architecture is influenced by the study of practical reasoning (human reasoning taking place during continuous decision-action process). This philosophical concept explains any action taken by a combination of intention (what goals are to be achieved) and a chosen option (how to proceed for achieving these goals). It is assumed in BDI system that course of action taken to satisfy goals should not stop simply because of failure but in the mean time it should not persist without any reasonable chance of success (or without the original intention remaining relevant). This represents a dilemma between the right balance of reactiveness and proactiveness behaviour contained in a BDI agent. Pure goal directed agents will never stop to reconsider their intention even after important environment changes that could invalidate the original goal. Meanwhile pure event driven agents never spend enough time trying to achieve their goals and waste all computational time in reacting to their environment. In general the more static an environment is the more pro-active agents should be, whereas in more dynamic environment agents need to have the ability to react to changes.

The layered architecture aims to accomplish a better balance between agent pro-activeness and re-activeness by introducing the concept of specialized dedicated layer. Layering architecture into subsystems allows one to design specialized subsystems adapted for specific task, for example one subsystem would deal with reactive behaviour while another one with pro-active behaviour. Most of these architectures use either horizontal layering or vertical layering for their information and control flow mechanisms. Horizontal layering requires some form of mediation between all layers since they all have access to input and can all produce some action output. Vertical layering involves a flow sequence between all layers, with lowest level layer receiving the input and forwarding it to highest or more abstract layer if necessary (i.e. when current layer expertise and competence cannot process or deal with input). Input information flowing from lowest to highest is referred to by bottom-up activation while command flowing from highest to lowest is referred to by top-down execution. This architecture is very popular due to its pragmatism and intuitive mapping from pro-active, reactive, social behaviours into specific and dedicated subsystems.

Taken from

“Multiagent Systems, a Modern Approach to Distributed Modern Approach to Artifical Intelligence” Chapter 2 Mutliagents Systems and Societies of Agents, Huhns and Stephens (1999)

Intelligent Agent part 1: Definition and Theory

2004-05-01T17:46:00.007+02:00

I did some research study for a friend of mine (no more details are possible ;-) on intelligent agents (software). This is a quite a broad area and very interesting. While doing this academic-oriented research reminded me on the period of my master degree, I must admit that working around a theoretical framework greatly contrasted with my daily work at the job... which is all about practical and not much about theoretical.

I've decided to publish some of the theoretical material I've surveyed... here is the first part.

What are intelligent agents

Agents are loosely defined as being a computer based system characterized by their autonomy (no direct human intervention required), their social ability allowing them to communicate with other agents or human, their reactivity allowing them to react to the changing environment and finally their pro-activeness which also permit them to take initiative to meet their goal. In fact the key characteristics of reactivity, pro-activeness and social ability can be considered as main differentiator between “normal” agent and “intelligent” agent.

Other more formal definition would also include some human alike concepts such as knowledge, belief, intention and obligation, as well as additional properties like mobility, veracity, benevolence and rationality.

An important aspect of agent has to do with their surrounding environment which they operate in. To see the impact of environmental properties on agent, one can classify them as being:

Accessible/inaccessible: accessible environment would be one for which all states are known up-to-date and accurately. In all practical cases, the environment is inaccessible, e.g. surrounding physical world, Internet, etc.
Deterministic/non-deterministic: except for the most trivial case, agents has only partial and non-deterministic control over their environment are, i.e. same action performed twice on seemingly identical circumstances may appear to have different consequences. Although non-deterministic lead to more complex system, to all intents and purposes it is recognized that a highly complex deterministic environment is just as difficult as a non-deterministic.
Episodic/non-episodic: episodic system has performance dependant only on the current discrete episode. This simplifies design since all possible scenarios and interaction in future episodes can be ignored in relation to deciding which actions are to be taken by the agents.
Static/dynamic: static environment only observe changes in reaction to action performed by the agent, in most situation other unknown/known processes operate on the environment making it dynamic.
Discrete/continuous: Discrete environment implies only a finite number of actions and percept to be possible.

It is recognised that with respect to building agent, most complex class of environment are those that are inaccessible, non-deterministic, non-episodic, dynamic and continuous.

Agent theory

A substantial part of agent theories are derived around the concept of intentional system, i.e. system with behaviour that are predicted based on similar human type attitudes: belief and knowledge (referred to as information category), and desire, intention, obligation, commitment and choice (referred to as pro-attitudes). Following this concept, merely all objects from the most simples automata-like objects to the most complex ones, can be described by having some form of intentional stance. This fact helps to provide a level of abstraction useful when dealing with the complexity of the latter.

As “compelling” as the intentional notions are for an intuitive representation, description, understanding and explanation of agent based system, it can certainly represent a difficult challenge when one wishes to reason about this inside a logical theoretical framework. Theory for representing intentional notion cannot be based using classical relying on truth functional operator, because this notion is said to be referential opaque. Substitution rule applicable to first order logic are not possible with intentional sentence such as “Janine believes p”, since replacing p with an equivalent will not preserve meaning.

Because of this limitation, other formalisms have been proposed for handling the syntactic rules and the semantic model associated with intentional system. One of the best known semantic model is characterized by the so-called “possible worlds” which is supported by an attractive mathematical tools called the “correspondence theory”. However as compelling as this model is for its logic representation, it suffers from limitations related to the famous “logical omniscience” (see problem due to infinite knowledge caused by statement logical tautology). In this 1995 study, the authors recognized that alternatives theories developed for either belief/knowledge representation (information attitudes) or goals/desire (pro-attitudes) all suffer from various deficiencies and/or limitations representing challenges to the agent theorist.

A more useful and complete theory for agent system should not only try representing the intentional stance but should also try embracing a larger logical framework interested in other components such as the dynamic aspects of agency, the relationship between the agent’s information and its pro-attitudes, the cognitive states change over time and reaction to the environment stimuli, etc. Some tentative work aims to achieve this very challenging task.

Of particular interest is communication between agents, which are modelled using the “speech act theory” stating that all communications are considered as actions performed to bring change to the world (usually to the listener internal state). This theory attempts to classify between the various forms of speech acts, with the two prominent ones being representative speech act for informing objectives and the directives one for requesting objective. See Multiagents system discussion below.

Taken from

“Intelligent Agents: Theory and Practice”, Wooldridge and Jennings (1995)
“Multiagent Systems, a Modern Approach to Distributed Modern Approach to Artifical Intelligence” Chapter 1 Intelligent Agents, Wooldridge (1999)

Data Warehouse Modeling guideline Part 3: Aggregate Data Modeling

2003-03-11T20:55:00.000+01:00

Here's the third and final part of my guideline, ....this time I'm covering very briefly a complex subject : the aggregation strategy.

General

1. A first level of aggregation should be built on top of the atomic database level, mainly for 1) boosting query performance done on atomic data, 2) for storing pre-calculate KPI which are usually aggregated data, and optionally to archiving historical data not kept at atomic level.

2. The first level aggregation is best when in the same infrastructure as the atomic level (i.e. relational model). It can be accompanied with a second-level which uses OLAP independent Cube technologies.

Dimension Entity

1. Two roles dimension

1.1. In the context of a migration count aggregation, only the dimension of interest must have the dual roles (Origin and Destination), all other dimension functionally dependent on the migration i.e. are modified after the migration, must be defined and fixed as either Origin or Destination.

Fact Entity

1. Definition of the type of fact

1.1. In aggregation the rule of not mixing two types of facts into the table (instantaneous event-based vs. interval or end of period snapshot-based) can be violated. This is because aggregation usually needs to provide a wealth of information and because there is a need to limit the number of aggregate to build/maintain

1.2. (1.1 continued) The mix can only be possible under the condition that it does not affect the granularity. Granularity must always be fixed, i.e. a state-based EOP aggregation cannot accept event-based counts if these affect the subscriber-EOP by axis grain.

1.3. (1.1 continued) If any event measure affect the grain (may potentially require to have BOP by axis to be represented because number of subs that have cancelled some conditions may not exist in the original grain) then it can't be integrated. In this case, it is much more suited to use OLAP Cube which deal easily with the BOP-EOP comparison.

1.4. (1.1 continued) Furthermore, great care must be applied in doing this, because the occurrence of all axis is inconsistent, event should track occurrence of axis at the time of the event while EOP state number track them at the EOP snapshot time. This implies that computation between measures is not possible by axis.

2. Definition of Measures

2.1. In the aggregation, it is common practice to store explicitly independent measures (e.g # of active subs, # of ghost subs, or Nb of sms, Nb of normal calls) that correspond to various dimension condition. This helps decreasing the granularity of the aggregation table and ignores the dimensions only needed to identify the independent measures.

Martin