mfp: The solution to the element ID problem

[Blog bot]

Источник: http://blogs.msdn.com/b/mfp/archive/...d-problem.aspx
==============
About 3 years ago heading into the design phase of AX 2012 I wrote the Solving the element ID problem post. As the post describes element IDs in AX have constituted an inherent problem in all past AX releases. The problem is so fundamental that when I met with the original Axapta architects (at the Damgaard Data 25 years reunion party) they considered it insolvable without a major rewrite. In AX 2012 we did solve the problem. This post outlines the solution.

Design pillars

Before I dive into the solution these are our design pillars:

1. Backward compatible. We cannot introduce any inconsistencies in business data or meta data.
2. Minimal impact on customizations. The uptake of required changes must be minimal and easily absorbed when upgrading to AX 2012.
3. Completed by AX 2012 RTM. We do not desire a staged multi-release solution to this problem.

The solution part A - 32 bit IDs

First we needed to break the upper limit of 60.000 IDs. We decided to make IDs 32 bit (used to be 16 bit). The primary deciding factor for int32 (and not int64 or guid or str) was that there would be no X++ uptake cost of using an int32, as the int data type in X++ already is a 32 bit integer - in other words all X++ code (and table fields) handling/containing IDs would simply just continue to work untouched. In the SYS layer we have >500 fields containing IDs. Not having to convert these (e.g. to GUIDs), and provide upgrade scripts is highly desirable.

As the ID for table fields already were 32 bit (16 bit for the element ID and 16 bit for the array index) we decided to keep the field IDs as is. In other words the element ID for fields are still just 16 bit. Given these are scoped by the table, there are no upper limit concerns involved here.

Besides a lot of kernel refactoring there was one significant X++ impact. The intrinsic function TypeID had to be deprecated. It returned a 32 bit integer where the upper 16 bits contained the Type (either Types::Enum or Types:UserType), and the lower 16 bits the element ID. This function was primarily used in relation to the Dialog framework. The solutionwas to change the dialog framework to use the name of the type instead of the bit mangled integer.

The solution part B - Installation specific IDs

To solve the uniqueness problem, we need to ensure that two (or more) solutions can be implemented independently, and installed on the same system without any ID conflicts. In AX 2009 the assignment of IDs happens at creation time (i.e. when the element is created in the AOT). In AX 2012 we are deferring the assignment of IDs to installation time. This means that the model file containing the elements does not contain any element IDs. The IDs are being assigned when the model is imported by AxUtil. The consequence of this is that the same class will have different IDs on different installations. We call it "Installation specific IDs".

Caution: Deleting a model and reimporting it (or a newer version of it) will randomize IDs and thus cause data integrity issues. The right procedure when upgrading a model is to not delete the model, but just import on top of the existing model.

Given the requirement to be backwards compatible we must ensure that elements released in AX 2009 (and AX 4) retained their IDs. To satisfy this requirement we introduced a new int property on all ID based elements: LegacyID, and assigned it the ID value from past releases. Now we have captured the ID the element used to have in a simple (and editable) property, and we can use it during import.

As it is vital that an element retain its ID across releases, we need to find a way to support rename scenarios. Consider the scenario where a new AX 2012 class is renamed in AX7. As the class is new it will have an installation specific ID; once the AX7 model is being imported we need to retain the ID - but if the only match criteria is the class name, and the class has been renamed, then we cannot fulfill the requirement. To solve this we introduced a new guid property on all ID based elements and all root-elements: Origin. This property is set when an element is created, and remains static for the lifetime of the elements. It is the element's fingerprint. Besides enabling invariant IDs across releases for renamed elements the Origin property has proven valuable in data export scenarios, and rename scenarios of newly fine grained meta data (like forms). This will be covered by another blog post soon.

AxUtil will during import assign IDs based on these rules:

1. If an element already exists with the same Origin, then replace the element and reuse its ID - else
2. If an element already exists with the same Type, Name and ParentID, then replace the element and reuse its ID - else
3. If the import element has a Legacy ID, and the LegacyId is available on the target system, then add the element setting ID = LegacyID - else
4. Assign a new installation specific ID from a guaranteed free range that will not collide with any LegacyIDs (>60000 for fields, >1000000 for all other elements)

Notice: IDs are no longer tied to layers, e.g. Layers no longer have an ID range. Nor do you need the Team Server (aka. ID Server) anymore when using a version control system.

Data can be exported from one installation to another. It is vital that any ID on the source system is mapped to the corresponding ID on the target system. The data import feature has been improved so all columns containing element IDs will be converted automatically (given they are using the proper extended data type). However; we also needed to address eventual IDs in blobs (containers). Data import and model import has no insights into the structure of the blobs. The only solution to this is to replace all ID-based element references in blobs with name-based references. We have done this for:

• All kernel classes returning a container - e.g. Query.pack(), QueryRun.pack(), Map.pack(), Set.pack() etc.
• All X++ classes packing element IDs (typically Runbase derived classes). A best practice rule is in place to detect these.
• All references in meta data - e.g. when a form data source refers to a table.

Wrapping up

It was a significant investment to solve this inherent problem. We did it in one release, we know this addresses the two major pain points around IDs: Uniqueness and Upper limit. The solution we implemented has a minimal impact on existing X++ implementations, yet it has an impact. My next blog post will describe the impact.




==============
Источник: http://blogs.msdn.com/b/mfp/archive/...d-problem.aspx