Designing an airline passenger reservation system

15 November 2008

Data Model - Basics - Part 11 - The HuagatiRes data model converted to “M”

Filed under: Data Model, Tools — Tags: , , — Kristofer @ 18:35

Happy over Microsoft’s new commitment to data modelling, I have been playing around a bit with the “Oslo” CTP. It is time they come up with a replacement for Visio 2003 EA. The latest thing I tried was a conversion of the HuagatiRes datamodel into an M script. There’s not much to say about it - “M” and “Oslo” are still CTPs so some things can be defined in M models (e.g. table defs, foreign keys, PKs) while others can not be described in M (e.g. indexes, alias types, precision/scale on numeric/decimal types etc). I also haven’t figured out how to declare both nullability and length on members of type “Text”.   “:Text where value.Count <=100” works fine while “:Text? where value.Count <=100” will throw an error, but that might just be me not fully understanding the M syntax.

The M model for HuagatiRes can be downloaded from here:

http://blog.huagati.com/downloads/HuagatiResMmodel.zip

Huagati MGen / “Aker Brygge”

Now, I wasn’t stupid enough to actually type all of that into Intellipad. Instead, I did something even more stupid: I wrote a code snippet that extracts schema, type, table, column, foreign key, and index defs and generates M files.

If you’re interested in taking the schema->M tool (”Huagati MGen” a.k.a. “Aker Brygge”) for a test-spin, you can download it from here:

http://blog.huagati.com/downloads/HuagatiMGen.zip

It requires .net 3.5 SP1 and works against SQL Server 2005 and SQL Server 2008 databases only. Give it a connect string and an output folder and it will generate one M file per schema in the database you point it at, complete with code comments (based on object names and extended property descriptions) and all. Even [commented out] index definitions - prepared for the day when M gets support for index definitions…

Huagati MGen

 

MGen Output

In the guts of MGen

The first version of MGen was a single exe, but I have since divided it up into a DLL containing a simple object model (MCodeProvider) for generating M code, along with a class that will connect to a SQL Server to extract schema definition and generate M using the MCodeProvider.

The exe file adds the simple user interface shown above, but if you prefer you can use the MCodeProvider or the DBSchemaMGenerator from your code instead. Note that this is just an experimental piece of sample code - it does not support everything that can be done in M, and likewise the MCodeProvider and related classes also support [db-side] features that don’t exist in the CTP version of M. The generated M code will reflect unsupported things such as indexes, extended properties etc as code comments.

Usage example 1 - generate M from an existing database:

string connectString = "Data Source=(local);Initial Catalog=AdventureWorks;Integrated Security=True";
string targetFolder = "c:\temp";
//generate M from an existing db
Huagati.MGenCore.DBSchemaMGenerator mGenerator
    = new Huagati.MGenCore.DBSchemaMGenerator();
mGenerator.SchemaToM(connectString, targetFolder);
 

Usage example 2 - generate M code using the MCodeProvider:

//create a new module Â
MCodeProvider code = new MCodeProvider();
MModule module
    = new MModule
    {
        Name = "TestModule",
        DescriptionComment = "Test Module"
    };
code.Modules.Add(module);
//define a simple type for identifying codes, string less than or equal to 10 characters in length Â
MSimpleType idCodeType
    = new MSimpleType
    {
        Name = "CodeType",
        Inherits = new MTextType(),
        DescriptionComment = "Simple type used for storing identifying codes."
    };
idCodeType.Constraints.Add(
    new MMemberConstraint
    {
        MemberMember = ConstraintMemberEnum.Length,
        Operator = ConstraintMemberOperator.LessThanOrEqual,
        ConstraintValue
            = new MIntLiteral
            {
                Value = 10
            }
    });
module.Members.Add(idCodeType);
//define a simple type for names, string less than or equal to 255 characters in length Â
MSimpleType nameType
    = new MSimpleType
    {
        Name = "NameType",
        Inherits = new MTextType(),
        DescriptionComment = "Simple type used for storing names."
    };
nameType.Constraints.Add(
    new MMemberConstraint
    {
        MemberMember = ConstraintMemberEnum.Length,
        Operator = ConstraintMemberOperator.LessThanOrEqual,
        ConstraintValue
            = new MIntLiteral
            {
                Value = 255
            }
    });
module.Members.Add(nameType);
//define a base type for all ref tables Â
MComplexType refTableType
    = new MComplexType
    {
        Name = "RefTableType",
        DescriptionComment = "Reference table base type."
    };
MMember idCode
    = new MMember
    {
        Name = "Code",
        Inherits = idCodeType
    };
refTableType.Members.Add(idCode);
refTableType.Members.Add(
    new MMember
    {
        Name = "Name",
        Inherits = nameType
    });
refTableType.Members.Add(
    new MMember
    {
        Name = "SortSeq",
        Inherits = new MInt32Type()
    });
refTableType.Members.Add(
    new MMember
    {
        Name = "Active",
        Inherits = new MLogicalType()
    });
refTableType.IdentityMembers.Add(idCode);
module.Members.Add(refTableType);
//define a test table, inheriting from the ref table type Â
MExtent testTable
    = new MExtent
    {
        Name = "Test",
        DescriptionComment = "Test reference table.",
        Inherits = refTableType
    };
module.Members.Add(testTable);
//add an index to the test table Â
MIndex testTableIndex1 =
    new MIndex
    {
        Name = "ix_TestTable_SomeOtherField"
    };
testTableIndex1.Members.Add(refTableType.GetMember("Name"));
testTable.Indexes.Add(testTableIndex1);
//save all modules   
code.SaveCode("c:\\temp");  

The output from the code above will be an M file containing the following:

//Test Module
module TestModule
{
    //Simple type used for storing identifying codes.
    type CodeType : Text where value.Count<=10;
    //Simple type used for storing names.
    type NameType : Text where value.Count<=255;
    //Reference table base type.
    type RefTableType
    {
        Code : CodeType;
        Name : NameType;
        SortSeq : Integer32;
        Active : Logical;
    } where identity (Code);
    //Test reference table.
    //[ExtendedProperty("MS_Description", "Test reference table.")]
    Test : RefTableType*;
    // index ix_TestTable_SomeOtherField(Name);
}

3 November 2008

Quiet blog, gone fishing, is Linq-to-SQL dead?, etc

Filed under: off-topic — Tags: , , , , , , — Kristofer @ 1:48

It has been a while since I posted anything on the blog. Did I stop blogging? Nope, just been having fun with other stuff. Like optimizing indexes for an oracle database, adding new features to my L2S/EF toolkit etc.

Gone Fishing

Right now I am “busy” fishing. It is Sunday and we’re at our vacation house on the countryside 130km to the west of Bangkok, where we have a small lake full of fish. I have been trying to catch a fish since yesterday but so far it is 2-0 to the fish. Over night, two tackles and hooks just disappeared. One large triple pointed hook attached to a steel wire and baited with a chicken skin is gone, the steel wire had snapped or been cut off. So I either got a very large fish with razor sharp teeth hanging out in the lake, a fish with a wire cutter, or maybe our resident monitor lizard got it.

The second missing tackle is also a triple pointed hook attached to half litre water bottle. I use water bottles as floats as they allow the fish to swim around while making it easy for me to find them. Until now. I have checked everywhere, and it is just gone. Not in the lake, and not anywhere in the irrigation canals out in the back of the garden. If the monitor lizard got it I would have expected to find it outside of her nest but no trace there either. (I know it’s a ’she’ because she laid eggs a few months ago. Unfortunately someone poached the nest a couple of weeks later so we won’t get any cute baby monitors. Monitor lizard eggs are turned into “performance enhancing” omelet by the locals.)

Some time whatever animal got it will have to surface, no fish can stay under water forever with an extra half kilo bouyancy. One remote possibility is that I got a fish on it and the fish got eaten by something larger. Like a python. They’re not resident (they’re more like consultants :) ) so if a python got it, it has probably already left our garden. If anyone comes across a large python dragging around a Minere bottle attached to a string, please return it to me. :)

 

EF performance

Enough off-topic stuff. I promised a while ago that I would write something about Microsoft’s new ADO.NET Entity Framework and performance. I was planning to do that, but I feel it is no longer necessary. There has been a lot written about it already and it isn’t pretty. Somehow they forgot to test EF and L2E beyond “Hello World”. Complex LINQ-to-entities queries are translated into really poor SQL. Less complex queries like a simple “from person p in oc.Persons where p.LastName.StartsWith(lastName) select p” are translated into equally bad SQL that will always force a table scan. Basic Math library operations such as “Round” in L2E queries are not supported at all by EF. The list goes on and on, but in short it EF v1 was simply released way too early, the current version should have been labelled “CTP” instead of “RTM”.

In contrast, L2S generates remarkably good and efficient SQL queries from just about any Linq expression thrown at it.

The future of Linq-to-SQL and Entity Framework/Linq-to-Entities

Microsoft made an announcement regarding Linq-to-SQL and Linq-to-Entities last week. Actually they made a lot of announcements regarding Linq-to-Entities last week during PDC. During one session the EF program manager, Tim Mallileau showed what they’re working on for future versions of Entity Framework. A lot of the “new” stuff in EF vNext are features that already exist and work in Linq-to-SQL such as deferred loading, POCO support with object-relational mappings described as attributes instead of in a bloated embedded EDMX file. That’s all nice, and shows that they have learnt at least a few things from the EFv1 fiasco, so now they’re trying to merge what is already there in L2S into EF. Wouldn’t it be smarter to cut the losses on EF and instead continue forward with Linq-to-SQL? At a minimum, opening up the provider model in L2S would be a nice first step…

So instead, they came out with an announcement that a large part of the L2S user base interpreted as a death certificate for Linq-to-SQL. Read it here: http://blogs.msdn.com/adonet/archive/2008/10/29/update-on-linq-to-sql-and-linq-to-entities-roadmap.aspx

A lot of heated discussions and upset blog entries ensued, people asked “Has Microsoft really killed Linq-to-SQL?” on stackoverflow etc. As always, Roger Jennings does a great job at summarizing what is being written around the blogosphere so head over to http://oakleafblog.blogspot.com/ for a summary…

Is Linq-to-SQL dead?

So is Linq-to-SQL dead? No, Linq-to-SQL is not dead. Heck no. (to paraphrase some ’softies). Microsoft can’t kill it. Linq-to-SQL is built into the .net framework as System.Data.Linq etc and it will stay there. To my knowledge, so far they haven’t ever removed/killed any features from the .net framework, but some namespaces don’t get a lot of new stuff added to them. That doesn’t mean they’re dead.

What I think the statement from Microsoft can be interpreted as is: “We’re not going to commit resources to improving Linq-to-SQL because EF is our big strategic object-relational mapping tool.“. And that is not really a big problem. Linq-to-SQL works really good out of the box, from a developer’s perspective it is easy to get started with, easy to use and really speeds up development of database-driven apps. Sure there are areas that can be improved, but the majority of those things can be solved without Microsoft having to make changes in the framework. More importantly, Linq-to-SQL is ready to use in app development today while EF is still really just a CTP release although Microsoft keeps touting it as RTM.

New features for the designer is one good example where L2S can be improved without the help of Microsoft, and I think I have already demonstrated that it can be done as VS addins. Code generation extensibility has been covered by Damien Guard in his T4 templates for L2S. There are plenty of runtime enhancements and code samples out there as well. Linq-to-SQL fans just need to follow Roger Jenning’s blog and they’ll find everything there is to find about Linq-to-SQL. So as long as Microsoft takes care of the occassional bug that might surface in L2S that’s fine and I’m sure they will. Although the documentation don’t mention it, they seem to have included a bunch of improvements in .net framework 3.5 - for example queries with multiple aggregates now come out clean and much more optimized than they did in the RTM version.

Oslo - the city with the most expensive beer in the world

Another interesting thing that was announced and demonstrated at MS PDC last week is something called “Oslo”. Besides being a city with the most expensive beer in the world, “Oslo” is also the code name for a new set of modelling tools from Microsoft, slated for release in 2010. I first heard about it a couple of months ago when I got an email from Don Box asking me if I ever considered working for Microsoft.

I have considered that a couple of times, the first time in 1997 or 1998 when I was interviewed for a job at Microsoft in Stockholm. That time I went for five separate in-person interviews (I lived 500km from Stockholm at the time), one which included filling out a “personality test” that bore a resemblance to the OCA test. They (MSFT) still couldn’t make up their mind after the fifth interview so I cut it short.

But since I got an email from a famous guy I did consider working for MSFT once again and so I attended a couple of phone interviews. This time there was no personality test involved so either they got rid of that test or maybe it was just something used by Microsoft in Stockholm. This time I got to the third interview, the tech interview. I botched it on a couple of questions on swapping items in linked lists, and on QSort. Don’t know what I can blame that on but I guess it being 7am and me not being a morning person is one semi-valid excuse. Anyway, they were cool about it and said it had to do with budget constraints and not me messing up the linked list… :) Budget constraints at Microsoft? That’s like snow in Bangkok…

So no, I’m not going to become a ’softie. Anyway, the whole experience got me interested in knowing more about “Oslo” (mainly because Don Box is leading the Oslo team, making it highly likely that something cool has to come out of it), so I downloaded the Oslo SDK CTP when it was released last week.

Oslo includes a graphical modelling tool, Quadrant (which is not included in the CTP download), a text-based modelling language called M, some lower level languages for defining domain specific languages, a repository design pattern that appear to be aimed at ensuring the generated databases will play nice in their new “Cloud” services platform, called “Windows Azure” (with a worse nickname). The “Oslo” tools are still in their infancy so there’s a lot of stuff missing, but it is still interesting to play around with it.

If you are interested/active in model driven development, drift over to the msdn Oslo Developer Center,  download the SDK, take it for a test spin, browse the Oslo msdn forum and let them know what features you would like to see in Oslo.

Personally, I am hoping that this time around MSFT will release a modelling tool that [on the database modelling side]:

  • is at least as easy to use as Visio, and as flexible as ERStudio and ERWin together
  • supports all basic/common data modelling scenarios
  • will successfully be able to reverse-and-forward engineer at a minimum Northwind and AdventureWorks without losing any database objects
  • will handle versioning of data models and schemas; schemas evolve over time in any system so the ability to update an existing database with model changes is a fairly basic requirement in my opinion
  • will be flexible enough to support the different naming conventions commonly in use

Now I’m off to check if I got a fish or if the missing bottle/tackle has surfaced anywhere. If not, I’m going to switch to net-fishing next week.

Powered by WordPress